Compare commits
14
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3742b3e453 | ||
|
|
98bc52b9f4 | ||
|
|
30fea9394f | ||
|
|
e966e913b6 | ||
|
|
ba6248b90b | ||
|
|
6cad02bc82 | ||
|
|
fdc57822b5 | ||
|
|
4e1748f65a | ||
|
|
b20331aa39 | ||
|
|
104fbcce09 | ||
|
|
302fd30af1 | ||
|
|
8d0a069950 | ||
|
|
cdb44658cf | ||
|
|
3772a69c6b |
No files matched your search
@@ -40,6 +40,7 @@ MEMORY_EMBEDDING_DIMS=1024
|
||||
# MEMORY_EMBEDDER_TIMEOUT=15
|
||||
# MEMORY_EMBEDDER_MAX_RETRIES=2
|
||||
# MEMORY_EMBEDDER_RETRY_DELAY=0.3
|
||||
# MEMORY_EMBEDDER_BATCH_SIZE=64
|
||||
# MEMORY_EMBEDDER_EXTRA_HEADERS={}
|
||||
# MEMORY_EMBEDDER_EXTRA_BODY={}
|
||||
|
||||
|
||||
@@ -4,6 +4,73 @@ Hy Memory and MemHub are versioned and released independently.
|
||||
|
||||
## Unreleased
|
||||
|
||||
## Hy Memory 2.0.10 — 2026-11-01
|
||||
|
||||
- New `hy_memory.reembed` module for switching the embedding
|
||||
model. Writers must be paused for the whole run. Nothing is derived from
|
||||
the provider or endpoint: source and target are named explicitly, and the
|
||||
identity of a space is the model name plus the width (`space_identity`).
|
||||
Only Chroma and Kuzu are supported.
|
||||
- `vector_space_names(config, base_name=None)` returns every physical
|
||||
collection a client opens for the six vector roles: memory, entities,
|
||||
coding keys and working facts, each in the working and coding variants.
|
||||
- `reembed_vector_space(source, target, embedding=...)` pages every raw
|
||||
source collection, covering all users and all layers including L1_RAW. It
|
||||
embeds the text the write path embedded (`reembed_text`: the canonical
|
||||
state's `state_retrieval_text`, a schema's `retrieval_text`, otherwise the
|
||||
document). It then writes each record into the target with the same id,
|
||||
document and metadata. Each batch's widths are checked before anything is
|
||||
written. A cancel stops at a batch boundary (`ReembedCancelled`). A rerun
|
||||
skips the ids the target already holds, so it resumes, and a merge never
|
||||
overwrites the target. The source is never written.
|
||||
- `plan_reembed`, `verify_reembed` and `drop_vector_space` complete the
|
||||
vector side. `verify_reembed` checks ids, metadata, per-user counts and
|
||||
the width, and runs a self-recall sample. `drop_vector_space` refuses
|
||||
collections that a live client in this process holds open.
|
||||
- `scan_vector_spaces(persist_directory)` is read-only. It lists every
|
||||
space on disk with its roles, counts, stored width, per-user counts and
|
||||
the models recorded in metadata. `diff_spaces(source, target)` returns the
|
||||
ids that the source holds and the target lacks, whatever their status.
|
||||
- `reembed_graph(source_path, target_path, embedding=..., target_dims=...)`
|
||||
builds a new Kuzu graph at the new width. It copies every node and edge
|
||||
and recomputes `Memory.embedding`, `Memory.beh_embedding`, and the Topic
|
||||
and HierarchyTopic embeddings, then rebuilds the HNSW index. The source
|
||||
graph is opened read-only and kept for rollback. A run that fails or is
|
||||
cancelled leaves no target behind, and a rerun starts over.
|
||||
`verify_graph_reembed` and `drop_graph` complete the graph side.
|
||||
- `EmbedService.embed_batch` now splits large batches into
|
||||
requests of `embedder.batch_size` texts each (`MEMORY_EMBEDDER_BATCH_SIZE`,
|
||||
default 64); before, it sent them all in one request. On a 429 it waits for
|
||||
the `Retry-After` time plus jitter, capped at 60 s. It takes a per-call
|
||||
`max_retries`. A returned vector whose width differs from the configured
|
||||
`embedding_dims` now raises `EmbeddingDimensionMismatchError` and is not
|
||||
retried.
|
||||
- Raw conversations are stored already trimmed. MemHub now keeps every raw
|
||||
forever and lite mode stores nothing else, so tool output and long pasted
|
||||
logs no longer stay in full. One rule (`hy_memory/utils/raw_trim.py`) is
|
||||
applied to each message when the raw is written:
|
||||
- `user` and `system` turns are kept in full;
|
||||
- `assistant` turns and other speakers (`[John]:`) longer than 500
|
||||
characters, after stripping, are cut to the first 500 plus `...`. This is
|
||||
the limit history injection used before;
|
||||
- `tool` turns keep their head and tail, and the middle becomes
|
||||
`[...N bytes omitted by truncate...]`, with the coding limit
|
||||
`coding.tool_result_max_bytes` (`MEMORY_CODING_TOOL_RESULT_MAX_BYTES`,
|
||||
default 2048).
|
||||
|
||||
The rule is idempotent. It covers chat `L1_RAW` in lite, pro and ultra (lite
|
||||
keeps tool turns, because it has no coding route) and the `memory_history`
|
||||
ADD record. The coding/working `coding_trace` was already trimmed when it
|
||||
was written. Its tool and assistant truncations are now idempotent, so
|
||||
reading the history back no longer rewrites the omitted-count markers. The
|
||||
text embedded for `L1_RAW` is the trimmed text. New raws carry
|
||||
`custom.raw_trimmed=true`.
|
||||
- Extractor history ("last k messages") now joins the stored raws directly
|
||||
and does not trim them again. Raws written before this change have no
|
||||
`raw_trimmed` flag, so they are still trimmed with the same rule when they
|
||||
are read. The extractor still receives the current batch in full (tool
|
||||
turns are still removed on the chat route).
|
||||
|
||||
## MemHub 1.0.5 — 2026-10-28
|
||||
|
||||
Build 36, on `hy-memory==2.0.8`. Linux x64/arm64 Runtime and macOS arm64
|
||||
|
||||
@@ -48,6 +48,7 @@
|
||||
| `MEMORY_EMBEDDER_TIMEOUT` | `15` | 单次超时,秒 |
|
||||
| `MEMORY_EMBEDDER_MAX_RETRIES` | `2` | 最大重试次数 |
|
||||
| `MEMORY_EMBEDDER_RETRY_DELAY` | `0.3` | 重试间隔,秒 |
|
||||
| `MEMORY_EMBEDDER_BATCH_SIZE` | `64` | `embed_batch` 单次请求最多发送的文本条数,超出按此切块 |
|
||||
| `MEMORY_EMBEDDER_EXTRA_HEADERS` | — | JSON object,附加 HTTP headers |
|
||||
| `MEMORY_EMBEDDER_EXTRA_BODY` | — | JSON object,附加请求 body |
|
||||
|
||||
|
||||
@@ -55,6 +55,33 @@ except Exception:
|
||||
from .client import HyMemoryClient
|
||||
from .inspector import MemoryInspector
|
||||
from .runtime import SharedRuntime
|
||||
from .reembed import (
|
||||
EmbeddingSpaceIdentity,
|
||||
GraphReembedPlan,
|
||||
GraphReembedProgress,
|
||||
GraphReembedReport,
|
||||
GraphVerification,
|
||||
ReembedCancelled,
|
||||
ReembedPlan,
|
||||
ReembedProgress,
|
||||
ReembedReport,
|
||||
ReembedVerification,
|
||||
ScannedSpace,
|
||||
SpaceDiff,
|
||||
VectorSpace,
|
||||
diff_spaces,
|
||||
drop_graph,
|
||||
drop_vector_space,
|
||||
plan_reembed,
|
||||
reembed_graph,
|
||||
reembed_text,
|
||||
reembed_vector_space,
|
||||
scan_vector_spaces,
|
||||
space_identity,
|
||||
vector_space_names,
|
||||
verify_graph_reembed,
|
||||
verify_reembed,
|
||||
)
|
||||
from .snapshot import (
|
||||
DataPlaneRestore,
|
||||
DataPlaneSnapshot,
|
||||
@@ -129,6 +156,32 @@ __all__ = [
|
||||
"DataPlaneSnapshotRejected",
|
||||
"restore_local_data_plane",
|
||||
"snapshot_local_data_plane",
|
||||
# 换 embedding 模型:把向量库和图重新 embed 进新的 embedding 空间
|
||||
"EmbeddingSpaceIdentity",
|
||||
"GraphReembedPlan",
|
||||
"GraphReembedProgress",
|
||||
"GraphReembedReport",
|
||||
"GraphVerification",
|
||||
"ReembedCancelled",
|
||||
"ReembedPlan",
|
||||
"ReembedProgress",
|
||||
"ReembedReport",
|
||||
"ReembedVerification",
|
||||
"ScannedSpace",
|
||||
"SpaceDiff",
|
||||
"VectorSpace",
|
||||
"diff_spaces",
|
||||
"drop_graph",
|
||||
"drop_vector_space",
|
||||
"plan_reembed",
|
||||
"reembed_graph",
|
||||
"reembed_text",
|
||||
"reembed_vector_space",
|
||||
"scan_vector_spaces",
|
||||
"space_identity",
|
||||
"vector_space_names",
|
||||
"verify_graph_reembed",
|
||||
"verify_reembed",
|
||||
# 配置
|
||||
"MemoryConfig",
|
||||
# 公共扩展协议 / 精确读取类型
|
||||
|
||||
+11
-2
@@ -3089,9 +3089,18 @@ class HyMemoryClient:
|
||||
if self._history_store is not None:
|
||||
try:
|
||||
isolation_key = self._build_isolation_key(uid, agent_id)
|
||||
# 从 content 或 messages 中提取用于记录的文本
|
||||
# 从 content 或 messages 中提取用于记录的文本。messages 与 L1_RAW
|
||||
# 同一规则先裁剪再落审计库(工具输出 / 长回复不原样长期保存)。
|
||||
from .utils.raw_trim import trim_raw_messages
|
||||
history_content = content or "; ".join(
|
||||
f"{m.role}: {m.content}" for m in chat_messages
|
||||
f"{m.role}: {m.content}"
|
||||
for m in trim_raw_messages(
|
||||
chat_messages,
|
||||
tool_max_bytes=getattr(
|
||||
getattr(self._config, "coding", None),
|
||||
"tool_result_max_bytes", None,
|
||||
),
|
||||
)
|
||||
)
|
||||
await self._history_store.record_add(
|
||||
memory_id=result.memory_id,
|
||||
|
||||
@@ -11,6 +11,7 @@ Coding Memory 预处理工具(规则,零 LLM)
|
||||
使用方式与 scene 路由见 docs/scene_coding_usage.md。
|
||||
"""
|
||||
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from ..pipelines.base import ChatMessage
|
||||
@@ -63,6 +64,13 @@ DEFAULT_MAX_TOOL_RESULT_BYTES = 2048
|
||||
DEFAULT_HEAD_BYTES = 1024
|
||||
DEFAULT_TAIL_BYTES = 512
|
||||
|
||||
# 本模块自己产出的省略标记。截断结果会再次经过截断(落库前一次、读历史时又一次),
|
||||
# 认出自己的标记才能保证幂等:已截断的文本再截一次原样返回,而不是把省略计数改写。
|
||||
_TOOL_OMIT_MARKER_RE = re.compile(r"\n\n\[\.\.\.\d+ bytes omitted by truncate\.\.\.\]\n\n")
|
||||
# 标记本身的字节数上限(数字位数有限),用于判断「已截断且只超出一个标记的长度」。
|
||||
_TOOL_OMIT_MARKER_MAX_BYTES = 128
|
||||
_ASSISTANT_OMIT_MARKER_RE = re.compile(r" \[\.\.\.\d+ chars omitted\.\.\.\] ")
|
||||
|
||||
|
||||
def truncate_tool_result_text(
|
||||
text: str,
|
||||
@@ -81,6 +89,12 @@ def truncate_tool_result_text(
|
||||
raw = text.encode("utf-8")
|
||||
if len(raw) <= max_bytes:
|
||||
return text
|
||||
# 已截断过(max_bytes 很小时标记本身会让结果略超上限):原样返回,保证幂等。
|
||||
if (
|
||||
len(raw) <= max_bytes + _TOOL_OMIT_MARKER_MAX_BYTES
|
||||
and _TOOL_OMIT_MARKER_RE.search(text)
|
||||
):
|
||||
return text
|
||||
# 自定义 max_bytes 可能小于默认 head+tail(1024+512)。按原比例缩放内容窗口,
|
||||
# 给省略标记预留空间,不能因为窗口和过大而把超限原文直接返回。
|
||||
marker_budget = min(96, max(24, max_bytes // 4))
|
||||
@@ -161,6 +175,10 @@ def truncate_assistant_text(
|
||||
text = "" if text is None else str(text)
|
||||
if len(text) <= head_chars + tail_chars:
|
||||
return text
|
||||
# 已截断过:中段恰好是本函数的省略标记 → 原样返回,保证幂等(否则会把
|
||||
# 省略计数改写成标记自身的长度)。
|
||||
if _ASSISTANT_OMIT_MARKER_RE.fullmatch(text[head_chars:len(text) - tail_chars]):
|
||||
return text
|
||||
omitted = len(text) - head_chars - tail_chars
|
||||
return (
|
||||
text[:head_chars]
|
||||
|
||||
@@ -260,6 +260,7 @@ class EmbedderConfig:
|
||||
max_retries: int = None # 最大重试次数
|
||||
retry_delay: float = None # 重试间隔(秒)
|
||||
timeout: int = None # 请求超时(秒)
|
||||
batch_size: int = None # 单次 HTTP 请求最多带多少条 input(embed_batch 按此切块)
|
||||
|
||||
# 内部平台扩展(extra_headers / extra_body 会透传给 OpenAI client)
|
||||
extra_headers: Optional[Dict[str, str]] = None
|
||||
@@ -294,6 +295,10 @@ class EmbedderConfig:
|
||||
# 请求切断并制造重试风暴。SDK 内层重试已关闭,15s 是每个显式 attempt
|
||||
# 的真实上限;部署可通过 MEMORY_EMBEDDER_TIMEOUT 覆盖。
|
||||
self.timeout = _get_env_int("MEMORY_EMBEDDER_TIMEOUT", 15)
|
||||
if self.batch_size is None:
|
||||
# 大批量(如重新 embed 整个库)不能一次 HTTP 发几千条:多数服务对
|
||||
# input 数组有上限,超了直接 400。按 batch_size 切块逐块发送。
|
||||
self.batch_size = _get_env_int("MEMORY_EMBEDDER_BATCH_SIZE", 64)
|
||||
if self.extra_headers is None:
|
||||
val = os.getenv("MEMORY_EMBEDDER_EXTRA_HEADERS")
|
||||
if val:
|
||||
|
||||
+105
-19
@@ -38,10 +38,12 @@ import importlib.util
|
||||
import json
|
||||
from json import JSONDecodeError
|
||||
import logging
|
||||
import random
|
||||
import threading
|
||||
import time
|
||||
|
||||
from ..config import MemoryConfig
|
||||
from ..embedding import EmbeddingDimensionMismatchError
|
||||
from .workload import embedding_slot
|
||||
from ..utils.log_setup import get_request_id
|
||||
|
||||
@@ -187,6 +189,41 @@ def _redact_headers(headers: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any
|
||||
return out
|
||||
|
||||
|
||||
# Retry-After 的上限:服务端给出超长等待(或恶意值)时不让单次调用挂几十分钟。
|
||||
_RETRY_AFTER_MAX_S = 60.0
|
||||
|
||||
|
||||
def _retry_after_seconds(exc: Exception) -> Optional[float]:
|
||||
"""从 429 异常的响应头里取 Retry-After(秒数或 HTTP 日期),取不到返回 None。"""
|
||||
response = getattr(exc, "response", None)
|
||||
headers = getattr(response, "headers", None)
|
||||
if not headers:
|
||||
return None
|
||||
try:
|
||||
raw = headers.get("retry-after") or headers.get("Retry-After")
|
||||
except Exception:
|
||||
return None
|
||||
if raw is None:
|
||||
return None
|
||||
raw = str(raw).strip()
|
||||
try:
|
||||
seconds = float(raw)
|
||||
except ValueError:
|
||||
try:
|
||||
from email.utils import parsedate_to_datetime
|
||||
from datetime import datetime, timezone
|
||||
|
||||
when = parsedate_to_datetime(raw)
|
||||
if when.tzinfo is None:
|
||||
when = when.replace(tzinfo=timezone.utc)
|
||||
seconds = (when - datetime.now(timezone.utc)).total_seconds()
|
||||
except Exception:
|
||||
return None
|
||||
if seconds != seconds: # NaN
|
||||
return None
|
||||
return min(max(0.0, seconds), _RETRY_AFTER_MAX_S)
|
||||
|
||||
|
||||
def _embeddings_api_url(base_url: str) -> str:
|
||||
"""OpenAI SDK posts to ``{base_url}/embeddings`` (base_url usually ends with /v1)."""
|
||||
base = (base_url or "").rstrip("/")
|
||||
@@ -205,7 +242,8 @@ class EmbedServiceConfig:
|
||||
api_key: str = ""
|
||||
base_url: str = ""
|
||||
embedding_dims: int = 1536
|
||||
batch_size: int = 100
|
||||
# embed_batch 单次 HTTP 最多带多少条 input;超出按块顺序发送
|
||||
batch_size: int = 64
|
||||
enable_cache: bool = True
|
||||
cache_size: int = 10000
|
||||
max_retries: int = 5
|
||||
@@ -287,6 +325,7 @@ class EmbedService:
|
||||
max_retries=embedder.max_retries if embedder.max_retries is not None else 5,
|
||||
retry_delay=embedder.retry_delay if embedder.retry_delay is not None else 3.0,
|
||||
timeout=embedder.timeout if embedder.timeout is not None else 60,
|
||||
batch_size=max(1, int(getattr(embedder, "batch_size", None) or 64)),
|
||||
queue_batch_size=int(os.getenv("MEMORY_EMBED_BATCH_SIZE", "8")),
|
||||
queue_batch_window_ms=float(os.getenv("MEMORY_EMBED_BATCH_WINDOW_MS", "100")),
|
||||
extra_headers=embedder.extra_headers,
|
||||
@@ -640,6 +679,8 @@ class EmbedService:
|
||||
self,
|
||||
texts: List[str],
|
||||
use_cache: bool = True,
|
||||
*,
|
||||
max_retries: Optional[int] = None,
|
||||
) -> List[List[float]]:
|
||||
"""
|
||||
批量向量化
|
||||
@@ -647,17 +688,23 @@ class EmbedService:
|
||||
Args:
|
||||
texts: 文本列表
|
||||
use_cache: 是否使用缓存
|
||||
max_retries: 本次调用的重试次数;None 用配置值。长批量任务(重新
|
||||
embed 整个库)可以给更宽的重试预算,不影响在线路径。
|
||||
|
||||
Returns:
|
||||
向量列表
|
||||
"""
|
||||
vectors, _metas = await self.embed_batch_with_meta(texts, use_cache=use_cache)
|
||||
vectors, _metas = await self.embed_batch_with_meta(
|
||||
texts, use_cache=use_cache, max_retries=max_retries,
|
||||
)
|
||||
return vectors
|
||||
|
||||
async def embed_batch_with_meta(
|
||||
self,
|
||||
texts: List[str],
|
||||
use_cache: bool = True,
|
||||
*,
|
||||
max_retries: Optional[int] = None,
|
||||
) -> Tuple[List[List[float]], List[Dict[str, Any]]]:
|
||||
"""批量向量化,同时返回每条的 provider metadata(非 vector)。"""
|
||||
if not texts:
|
||||
@@ -682,7 +729,12 @@ class EmbedService:
|
||||
indices_to_embed.append(i)
|
||||
|
||||
if texts_to_embed:
|
||||
vectors, batch_metas = await self._embed_texts_with_meta(texts_to_embed)
|
||||
# max_retries 只在显式给出时透传:保持 `_embed_texts_with_meta(texts)`
|
||||
# 这个既有调用形状不变(测试与下游替身按它打桩)。
|
||||
retry_kwargs = {} if max_retries is None else {"max_retries": max_retries}
|
||||
vectors, batch_metas = await self._embed_texts_with_meta(
|
||||
texts_to_embed, **retry_kwargs,
|
||||
)
|
||||
|
||||
for j, vector in enumerate(vectors):
|
||||
idx = indices_to_embed[j]
|
||||
@@ -700,9 +752,9 @@ class EmbedService:
|
||||
return results, metas
|
||||
|
||||
async def _embed_texts_with_meta(
|
||||
self, texts: List[str]
|
||||
self, texts: List[str], *, max_retries: Optional[int] = None,
|
||||
) -> Tuple[List[List[float]], List[Dict[str, Any]]]:
|
||||
"""实际的向量化调用:**一次 HTTP 带走整批**;返回 (vectors, metas)。
|
||||
"""实际的向量化调用:**一次 HTTP 带走一块**;返回 (vectors, metas)。
|
||||
|
||||
早期实现是 `for text in texts: await _embed_openai([text])` 逐条串行,
|
||||
原意是"避免 batch size 限制",但代价极大:线上一批 8 条 × ~5.6s 串行
|
||||
@@ -719,21 +771,34 @@ class EmbedService:
|
||||
截断在这里做(发送前逐条过 `_truncate_for_embedding`):所有 embedding
|
||||
路径(embed / embed_batch / embed_queued / *_with_meta)最终都落到这个
|
||||
方法 —— 放在各入口会漏。理由与阈值见模块头部 `_EMBED_MAX_TOKENS`。
|
||||
|
||||
切块:一次带走整批只适用于小批。几千条的 input 数组会被多数服务直接
|
||||
400(input 上限),所以按 `batch_size`(默认 64)切块,块之间顺序发送。
|
||||
不超过一块时行为与切块前逐字相同(`batch_mode="single_request"`)。
|
||||
"""
|
||||
if not texts:
|
||||
return [], []
|
||||
|
||||
safe_texts = [_truncate_for_embedding(t) for t in texts]
|
||||
vectors, meta = await self._embed_openai(safe_texts)
|
||||
base_meta = dict(meta or {})
|
||||
base_meta["batch_mode"] = "single_request"
|
||||
base_meta["batch_input_count"] = len(safe_texts)
|
||||
|
||||
chunk = max(1, int(self._embed_config.batch_size or 64))
|
||||
vectors: List[List[float]] = []
|
||||
all_metas: List[Dict[str, Any]] = []
|
||||
for idx in range(len(safe_texts)):
|
||||
m = dict(base_meta)
|
||||
m["batch_index"] = idx
|
||||
all_metas.append(m)
|
||||
chunk_count = (len(safe_texts) + chunk - 1) // chunk
|
||||
for start in range(0, len(safe_texts), chunk):
|
||||
part = safe_texts[start:start + chunk]
|
||||
retry_kwargs = {} if max_retries is None else {"max_retries": max_retries}
|
||||
part_vectors, meta = await self._embed_openai(part, **retry_kwargs)
|
||||
base_meta = dict(meta or {})
|
||||
base_meta["batch_mode"] = "single_request" if chunk_count == 1 else "chunked"
|
||||
base_meta["batch_input_count"] = len(part)
|
||||
if chunk_count > 1:
|
||||
base_meta["batch_chunk_index"] = start // chunk
|
||||
base_meta["batch_chunk_count"] = chunk_count
|
||||
for idx in range(len(part)):
|
||||
m = dict(base_meta)
|
||||
m["batch_index"] = start + idx
|
||||
all_metas.append(m)
|
||||
vectors.extend(part_vectors)
|
||||
|
||||
self._last_embed_meta = all_metas[-1] if all_metas else {}
|
||||
return vectors, all_metas
|
||||
@@ -783,7 +848,7 @@ class EmbedService:
|
||||
)
|
||||
|
||||
async def _embed_openai(
|
||||
self, texts: List[str]
|
||||
self, texts: List[str], *, max_retries: Optional[int] = None,
|
||||
) -> Tuple[List[List[float]], Dict[str, Any]]:
|
||||
"""
|
||||
使用 OpenAI 兼容接口向量化
|
||||
@@ -873,7 +938,9 @@ class EmbedService:
|
||||
"endpoint_overridden": bool(_ov),
|
||||
}
|
||||
|
||||
_max_retries = max(0, int(self._embed_config.max_retries or 0))
|
||||
_max_retries = max(0, int(
|
||||
(self._embed_config.max_retries or 0) if max_retries is None else max_retries
|
||||
))
|
||||
_retry_delay = max(0.0, float(self._embed_config.retry_delay or 0.0))
|
||||
last_exc: Optional[Exception] = None
|
||||
|
||||
@@ -919,6 +986,17 @@ class EmbedService:
|
||||
f"got={len(vectors)}"
|
||||
)
|
||||
dims = [len(v) for v in vectors]
|
||||
expected_dims = self._embed_config.embedding_dims
|
||||
if expected_dims:
|
||||
bad = [d for d in dims if d != int(expected_dims)]
|
||||
if bad:
|
||||
# 宽度不对的向量写进集合会把集合的维度钉死在错误值上(Chroma
|
||||
# 首次 add 定维),或在已有集合上整批失败。确定性错误,不重试。
|
||||
raise EmbeddingDimensionMismatchError(
|
||||
"embedding dimension mismatch: "
|
||||
f"expected={int(expected_dims)} got={bad[0]} "
|
||||
f"model={self._embed_config.model}"
|
||||
)
|
||||
meta = {
|
||||
"http_status": raw.status_code,
|
||||
"response_headers": resp_headers,
|
||||
@@ -954,6 +1032,8 @@ class EmbedService:
|
||||
},
|
||||
)
|
||||
return vectors, meta
|
||||
except EmbeddingDimensionMismatchError:
|
||||
raise
|
||||
except Exception as e:
|
||||
last_exc = e
|
||||
_err_name = type(e).__name__
|
||||
@@ -987,12 +1067,18 @@ class EmbedService:
|
||||
if _retryable and attempt < _max_retries:
|
||||
# 逐出可能卡死的 client,下个 attempt 重建干净连接(恢复原防御语义)
|
||||
self._evict_client(_eff_base_url, _eff_api_key, _eff_timeout)
|
||||
_delay = _retry_delay
|
||||
_retry_after = _retry_after_seconds(e) if _status == 429 else None
|
||||
if _retry_after is not None:
|
||||
# 限流时服务端明确告诉了要等多久:按它等,再加一点抖动,
|
||||
# 避免多个 worker 在同一时刻一起醒来再次撞限流。
|
||||
_delay = max(_delay, _retry_after) + random.uniform(0, 0.25 * _retry_after)
|
||||
logger.warning(
|
||||
f"[embed] attempt {attempt + 1}/{_max_retries + 1} failed "
|
||||
f"({_err_name}): {_err_msg[:200]}; retry in {_retry_delay}s"
|
||||
f"({_err_name}): {_err_msg[:200]}; retry in {_delay:.2f}s"
|
||||
)
|
||||
if _retry_delay > 0:
|
||||
await asyncio.sleep(_retry_delay)
|
||||
if _delay > 0:
|
||||
await asyncio.sleep(_delay)
|
||||
continue
|
||||
self._log_embed_call(
|
||||
status="error",
|
||||
|
||||
@@ -22,6 +22,7 @@ import logging
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
import weakref
|
||||
|
||||
from ..models.memory import MemoryNode, MemoryLayer, MemoryStatus
|
||||
from ..config import MemoryConfig
|
||||
@@ -77,6 +78,21 @@ async def _run_in_kuzu_pool(func, *args, **kwargs):
|
||||
return await run_in_executor_guarded(_kuzu_executor, _measured)
|
||||
|
||||
|
||||
# 本进程里已打开、尚未 close 的 store。drop_graph 据此拒绝删掉一个正被
|
||||
# 活着的 client 使用的图库。WeakSet:store 被回收后自动消失。
|
||||
_LIVE_STORES: "weakref.WeakSet[KuzuGraphStore]" = weakref.WeakSet()
|
||||
|
||||
|
||||
def open_graph_paths() -> List[str]:
|
||||
"""Real paths of the graph databases live stores in this process hold open."""
|
||||
import os
|
||||
return [
|
||||
os.path.realpath(str(store._db_path))
|
||||
for store in list(_LIVE_STORES)
|
||||
if store._db is not None
|
||||
]
|
||||
|
||||
|
||||
def kuzu_executor_snapshot() -> Dict[str, Any]:
|
||||
with _kuzu_stats_lock:
|
||||
return {
|
||||
@@ -350,6 +366,7 @@ class KuzuGraphStore(GraphStoreBase):
|
||||
)
|
||||
self._db = db
|
||||
self._conn = conn
|
||||
_LIVE_STORES.add(self)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning(
|
||||
f"GraphStore init timed out after 5s (Kuzu file lock contention). "
|
||||
@@ -2441,6 +2458,7 @@ class KuzuGraphStore(GraphStoreBase):
|
||||
async def close(self) -> None:
|
||||
"""关闭连接"""
|
||||
conn, db = self._conn, self._db
|
||||
_LIVE_STORES.discard(self)
|
||||
self._conn = None
|
||||
self._db = None
|
||||
self._available = False
|
||||
|
||||
@@ -89,6 +89,17 @@ def passes_created_after(
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def physical_collection_name(base_name: Optional[str], dims: Optional[int]) -> str:
|
||||
"""The collection name a store opens for ``base_name`` at ``dims``.
|
||||
|
||||
The dims suffix keeps two embedding widths from sharing one collection.
|
||||
`hy_memory.reembed.vector_space_names` uses this too, so callers never
|
||||
re-derive the rule.
|
||||
"""
|
||||
base = base_name or "agent_memories"
|
||||
return f"{base}_{dims}" if dims else base
|
||||
|
||||
|
||||
class VectorStoreBase(ABC):
|
||||
"""
|
||||
向量存储抽象基类
|
||||
@@ -98,13 +109,10 @@ class VectorStoreBase(ABC):
|
||||
|
||||
def __init__(self, config: MemoryConfig):
|
||||
self.config = config
|
||||
base_name = config.vector_store.collection_name or "agent_memories"
|
||||
dims = config.vector_store.embedding_dims
|
||||
# 自动拼接维度后缀,避免不同 embedding 模型写入同一 collection 导致维度冲突
|
||||
if dims:
|
||||
self._collection_name = f"{base_name}_{dims}"
|
||||
else:
|
||||
self._collection_name = base_name
|
||||
self._collection_name = physical_collection_name(
|
||||
config.vector_store.collection_name, config.vector_store.embedding_dims,
|
||||
)
|
||||
|
||||
_supports_retrieval_clauses: bool = False
|
||||
_supports_narrow_episode_queries: bool = False
|
||||
|
||||
@@ -9,10 +9,12 @@ Agent Memory V2 - VectorStore (Chroma)
|
||||
- 同步 API → _run_in_vdb_pool() 包装为异步
|
||||
"""
|
||||
|
||||
from typing import Optional, List, Dict, Any
|
||||
from typing import Optional, List, Dict, Any, Set, Tuple
|
||||
import concurrent.futures
|
||||
import functools
|
||||
import logging
|
||||
import os
|
||||
import weakref
|
||||
|
||||
from ..models.memory import (
|
||||
MemoryNode, MemoryLayer, MemoryStatus,
|
||||
@@ -33,6 +35,30 @@ _vdb_executor = concurrent.futures.ThreadPoolExecutor(
|
||||
)
|
||||
|
||||
|
||||
# 本进程里已 initialize、尚未 close 的 store。drop_vector_space 据此拒绝删掉
|
||||
# 一个正被活着的 client 使用的集合。WeakSet:store 被回收后自动消失。
|
||||
_LIVE_STORES: "weakref.WeakSet[ChromaVectorStore]" = weakref.WeakSet()
|
||||
|
||||
|
||||
def _location_key(vs_config) -> Tuple[str, str]:
|
||||
if getattr(vs_config, "host", None):
|
||||
return ("http", f"{vs_config.host}:{vs_config.port or 8000}")
|
||||
return ("local", os.path.realpath(str(vs_config.persist_directory or "")))
|
||||
|
||||
|
||||
def open_collection_names(location: Tuple[str, str]) -> Set[str]:
|
||||
"""Collections that live stores in this process hold open at ``location``."""
|
||||
names: Set[str] = set()
|
||||
for store in list(_LIVE_STORES):
|
||||
if store._client is None or _location_key(store.config.vector_store) != location:
|
||||
continue
|
||||
names.add(store._collection_name)
|
||||
if store._entity_collection is not None:
|
||||
from ..pipelines._retrieval.config import entity_collection_name
|
||||
names.add(entity_collection_name(store._collection_name))
|
||||
return names
|
||||
|
||||
|
||||
async def _run_in_vdb_pool(func, *args, **kwargs):
|
||||
"""Run a synchronous Chroma call in the dedicated VDB executor."""
|
||||
call = functools.partial(func, *args, **kwargs) if args or kwargs else func
|
||||
@@ -88,6 +114,7 @@ class ChromaVectorStore(VectorStoreBase):
|
||||
return client, collection
|
||||
|
||||
self._client, self._collection = await _run_in_vdb_pool(_init)
|
||||
_LIVE_STORES.add(self)
|
||||
logger.debug(f"VectorStore initialized (Chroma), collection={self._collection_name}")
|
||||
|
||||
# ================================================================
|
||||
@@ -561,6 +588,7 @@ class ChromaVectorStore(VectorStoreBase):
|
||||
async def close(self) -> None:
|
||||
"""关闭客户端"""
|
||||
# Chroma PersistentClient 无需显式关闭
|
||||
_LIVE_STORES.discard(self)
|
||||
self._client = None
|
||||
self._collection = None
|
||||
self._entity_collection = None
|
||||
|
||||
@@ -280,6 +280,22 @@ class MemoryWriter(WritePipeline):
|
||||
custom["source_raw_memory_ids"] = source_ids
|
||||
return custom
|
||||
|
||||
def _flat_trimmed_raw(self, request: WriteRequest) -> str:
|
||||
"""把 request.messages 按 raw 统一规则裁剪后拼成 L1_RAW 文本。
|
||||
|
||||
只裁落库副本;request.messages 不动(本次 extractor 输入仍用完整 messages)。
|
||||
"""
|
||||
import copy as _copy
|
||||
from ..utils.raw_trim import trim_raw_messages
|
||||
|
||||
_coding_cfg = getattr(getattr(self, "config", None), "coding", None)
|
||||
_trimmed_req = _copy.copy(request)
|
||||
_trimmed_req.messages = trim_raw_messages(
|
||||
request.messages,
|
||||
tool_max_bytes=getattr(_coding_cfg, "tool_result_max_bytes", None),
|
||||
)
|
||||
return _trimmed_req.get_flat_content()
|
||||
|
||||
@staticmethod
|
||||
def _attach_client_meta(custom: Dict[str, Any], request: Optional[WriteRequest]) -> None:
|
||||
"""给新建节点挂上调用方 meta(单份 list)。
|
||||
@@ -2554,9 +2570,14 @@ class MemoryWriter(WritePipeline):
|
||||
try:
|
||||
# 参数校验
|
||||
content = request.content
|
||||
# raw 在写入时就按统一规则裁好(见 utils/raw_trim.py):工具输出、长
|
||||
# assistant 回复不再原样长期存储;历史拼接直接用存下来的文本,不再二次裁剪。
|
||||
# request.messages 保持原样——本次 extractor 输入仍是完整 messages。
|
||||
_raw_trimmed = False
|
||||
if not content and request.has_messages():
|
||||
content = request.get_flat_content()
|
||||
content = self._flat_trimmed_raw(request)
|
||||
request.content = content
|
||||
_raw_trimmed = True
|
||||
|
||||
if not content:
|
||||
response.error_code = 400
|
||||
@@ -2627,6 +2648,9 @@ class MemoryWriter(WritePipeline):
|
||||
vector_store = await self._get_vector_store()
|
||||
layer_enum = MemoryLayer.from_string(suggested_layer)
|
||||
_raw_custom: Dict[str, Any] = {}
|
||||
if _raw_trimmed:
|
||||
from ..utils.raw_trim import RAW_TRIMMED_KEY
|
||||
_raw_custom[RAW_TRIMMED_KEY] = True
|
||||
self._attach_client_meta(_raw_custom, request)
|
||||
mem_node = MemoryNode(
|
||||
user_id=request.user_id,
|
||||
@@ -2929,13 +2953,16 @@ class MemoryWriter(WritePipeline):
|
||||
agent_id: Optional[str] = None,
|
||||
exclude_memory_id: str = "",
|
||||
max_turns: int = 20,
|
||||
max_chars_assistant: int = 500,
|
||||
max_chars_assistant: Optional[int] = None,
|
||||
) -> str:
|
||||
"""
|
||||
获取该用户最近的 L1_RAW 对话记录,按轮次拆分后拼成 history context。
|
||||
|
||||
每条 L1_RAW 的 content 是多轮对话([user]: ...\n[assistant]: ...)。
|
||||
拆成单条 message 后,user 消息完整保留,assistant 消息截取前 max_chars_assistant 字符。
|
||||
raw 在写入时已按 utils/raw_trim.py 的规则裁好(custom[RAW_TRIMMED_KEY]),
|
||||
这里拆成单条 message 后直接拼接,不再二次裁剪。该规则之前写入的旧 raw
|
||||
没有这个标记,仍按同一(幂等)规则裁剪后再拼,避免旧的长回复 / 工具输出
|
||||
撑大 prompt。max_chars_assistant 只作用于这些旧 raw,默认同写入规则。
|
||||
最终取最近 max_turns 条 message(注意单位是 message 条数,非对话轮数;
|
||||
调用方按 k 轮 × 2 传入)。
|
||||
|
||||
@@ -2977,23 +3004,33 @@ class MemoryWriter(WritePipeline):
|
||||
# 避免把正文里形如 "[注]: xxx" 的普通文本误判成消息边界。
|
||||
import re as _re
|
||||
from .base import parse_user_name_line
|
||||
_KNOWN_ROLES = {"user", "assistant", "system", "tool"}
|
||||
# 带方括号的任意 label(含对方真人名字)
|
||||
_labeled_prefix = _re.compile(r"^\[([^\]\n]{1,32})\]:\s?")
|
||||
# 无方括号的已知 role(历史数据容错,沿用旧正则的 `\[?...\]?` 宽容度)
|
||||
_bare_role_prefix = _re.compile(r"^(user|assistant|system|tool):\s?")
|
||||
|
||||
# (label, content) 元组列表;label 为 role 或对方真人名字
|
||||
parsed: List[Tuple[str, str]] = []
|
||||
from ..utils.raw_trim import (
|
||||
DEFAULT_RAW_ASSISTANT_MAX_CHARS,
|
||||
RAW_TRIMMED_KEY,
|
||||
trim_raw_body,
|
||||
)
|
||||
if max_chars_assistant is None:
|
||||
max_chars_assistant = DEFAULT_RAW_ASSISTANT_MAX_CHARS
|
||||
_coding_cfg = getattr(getattr(self, "config", None), "coding", None)
|
||||
_tool_max_bytes = getattr(_coding_cfg, "tool_result_max_bytes", None)
|
||||
|
||||
# (label, content, 是否旧 raw 需兼容裁剪) 元组列表;label 为 role 或对方真人名字
|
||||
parsed: List[Tuple[str, str, bool]] = []
|
||||
user_name: str = ""
|
||||
for node in nodes:
|
||||
content = node.content or ""
|
||||
_legacy = not bool((node.custom or {}).get(RAW_TRIMMED_KEY))
|
||||
for _line_idx, line in enumerate(content.split("\n")):
|
||||
m = _labeled_prefix.match(line) or _bare_role_prefix.match(line)
|
||||
if m:
|
||||
label = m.group(1)
|
||||
body = line[m.end():]
|
||||
parsed.append((label, body))
|
||||
parsed.append((label, body, _legacy))
|
||||
continue
|
||||
# 名字行只可能是每个 node content 的第一行(get_flat_content 插在文首)。
|
||||
# 逐 node 判断而非用 `not parsed`:parsed 跨 node 累积,第二个 node
|
||||
@@ -3006,18 +3043,18 @@ class MemoryWriter(WritePipeline):
|
||||
continue
|
||||
if parsed:
|
||||
# 续行:拼回上一条 message(保留换行)
|
||||
prev_label, prev_body = parsed[-1]
|
||||
parsed[-1] = (prev_label, prev_body + "\n" + line)
|
||||
prev_label, prev_body, prev_legacy = parsed[-1]
|
||||
parsed[-1] = (prev_label, prev_body + "\n" + line, prev_legacy)
|
||||
else:
|
||||
# 开头就是无前缀行(异常/老数据):当成 user 消息兜底
|
||||
parsed.append(("user", line))
|
||||
parsed.append(("user", line, _legacy))
|
||||
|
||||
# 取最近 max_turns 条 message(按 message 截断,绝不从消息中间切开)
|
||||
recent = parsed[-max_turns:] if len(parsed) > max_turns else parsed
|
||||
if not recent:
|
||||
return ""
|
||||
|
||||
# 格式化:非 user 侧整条截断(含对方真人),user 完整保留
|
||||
# 格式化:新 raw 直接拼接;旧 raw 按写入同一规则兼容裁剪
|
||||
all_messages: List[str] = []
|
||||
if user_name:
|
||||
from .base import render_user_name_line
|
||||
@@ -3025,17 +3062,18 @@ class MemoryWriter(WritePipeline):
|
||||
|
||||
_name_line = render_user_name_line(
|
||||
user_name,
|
||||
is_chinese("\n".join(b for _, b in recent)),
|
||||
is_chinese("\n".join(b for _, b, _ in recent)),
|
||||
)
|
||||
if _name_line:
|
||||
all_messages.append(_name_line)
|
||||
for label, body in recent:
|
||||
for label, body, legacy in recent:
|
||||
if legacy:
|
||||
body = trim_raw_body(
|
||||
label, body,
|
||||
assistant_max_chars=max_chars_assistant,
|
||||
tool_max_bytes=_tool_max_bytes,
|
||||
)
|
||||
body = body.strip()
|
||||
# 截断口径:assistant + 对方真人(非已知 role 的 label)。
|
||||
# user 完整保留;system / tool 沿用历史行为(不截断)。
|
||||
_truncate = label == "assistant" or label not in _KNOWN_ROLES
|
||||
if _truncate and len(body) > max_chars_assistant:
|
||||
body = body[:max_chars_assistant] + "..."
|
||||
all_messages.append(f"[{label}]: {body}")
|
||||
|
||||
return "\n".join(all_messages)
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,91 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
原始对话(raw)落库前的统一裁剪规则(规则,零 LLM)。
|
||||
|
||||
raw 会被长期保存(MemHub 已取消 raw 的保留期限;lite 模式下 raw 是唯一存储),
|
||||
不裁剪会把工具输出、大段粘贴日志原样存下来。因此 raw 在**写入时**就裁好,
|
||||
读取方(extractor 的「最近 k 条」历史注入)直接拼接,不再二次裁剪。
|
||||
|
||||
规则按说话人标签(ChatMessage.speaker_label() / L1_RAW 里的 ``[label]``):
|
||||
|
||||
- ``user``:完整保留;
|
||||
- ``system``:完整保留(与历史注入原口径一致);
|
||||
- ``tool``:头尾保留、中段替换为 ``[...N bytes omitted by truncate...]``,
|
||||
上限为 ``coding.tool_result_max_bytes``(env ``MEMORY_CODING_TOOL_RESULT_MAX_BYTES``,
|
||||
默认 2048),与 coding 链路同一个函数;
|
||||
- 其余(``assistant``、对方真人名字等非已知 role 的 label):strip 后超过
|
||||
``assistant_max_chars``(默认 500,即历史注入原来的 ``max_chars_assistant``)
|
||||
时截成前 N 个字符 + ``...``。
|
||||
|
||||
所有规则确定且幂等:对已裁剪的文本再裁一次结果不变。
|
||||
"""
|
||||
|
||||
from typing import List, Optional
|
||||
|
||||
from ..pipelines.base import ChatMessage
|
||||
|
||||
# 历史注入原 `_get_recent_history(max_chars_assistant=500)` 的默认值,现为唯一来源。
|
||||
DEFAULT_RAW_ASSISTANT_MAX_CHARS = 500
|
||||
|
||||
# L1_RAW 节点 custom 上的标记:内容已按本模块规则在写入时裁剪。
|
||||
# 历史拼接据此区分「新 raw 直接拼接」与「本规则之前写入的旧 raw 需兼容裁剪」。
|
||||
RAW_TRIMMED_KEY = "raw_trimmed"
|
||||
|
||||
# 不裁剪的标签(tool 单独处理)
|
||||
_KEEP_LABELS = frozenset({"user", "system"})
|
||||
|
||||
|
||||
def trim_raw_body(
|
||||
label: str,
|
||||
body: str,
|
||||
*,
|
||||
assistant_max_chars: int = DEFAULT_RAW_ASSISTANT_MAX_CHARS,
|
||||
tool_max_bytes: Optional[int] = None,
|
||||
) -> str:
|
||||
"""按说话人标签裁剪单条消息正文。幂等。"""
|
||||
if not isinstance(body, str):
|
||||
body = "" if body is None else str(body)
|
||||
if label in _KEEP_LABELS:
|
||||
return body
|
||||
if label == "tool":
|
||||
from ..coding.preproc import (
|
||||
DEFAULT_MAX_TOOL_RESULT_BYTES,
|
||||
truncate_tool_result_text,
|
||||
)
|
||||
|
||||
return truncate_tool_result_text(
|
||||
body, max_bytes=int(tool_max_bytes or DEFAULT_MAX_TOOL_RESULT_BYTES),
|
||||
)
|
||||
stripped = body.strip()
|
||||
if len(stripped) > assistant_max_chars:
|
||||
return stripped[:assistant_max_chars] + "..."
|
||||
return body
|
||||
|
||||
|
||||
def trim_raw_messages(
|
||||
messages: List[ChatMessage],
|
||||
*,
|
||||
assistant_max_chars: int = DEFAULT_RAW_ASSISTANT_MAX_CHARS,
|
||||
tool_max_bytes: Optional[int] = None,
|
||||
) -> List[ChatMessage]:
|
||||
"""返回裁剪后的新列表;不修改入参(messages 仍要原样给 extractor)。"""
|
||||
out: List[ChatMessage] = []
|
||||
for m in messages:
|
||||
new_content = trim_raw_body(
|
||||
m.speaker_label(),
|
||||
m.content,
|
||||
assistant_max_chars=assistant_max_chars,
|
||||
tool_max_bytes=tool_max_bytes,
|
||||
)
|
||||
if new_content == m.content:
|
||||
out.append(m)
|
||||
continue
|
||||
out.append(ChatMessage(
|
||||
role=m.role,
|
||||
content=new_content,
|
||||
name=m.name,
|
||||
tool_calls=list(m.tool_calls),
|
||||
tool_call_id=m.tool_call_id,
|
||||
tool_name=m.tool_name,
|
||||
))
|
||||
return out
|
||||
+2
-2
@@ -13,8 +13,8 @@
|
||||
接口先补 SDK。Runtime writers 停止后的 App 启动迁移可直接操作底层存储,
|
||||
但不得把迁移专用方法加入公共 Client。
|
||||
3. MemHub **不存 memory 实体**;**可以存 `memory_id`** 做组织/回执。原文进入 **SDK raw**。
|
||||
4. SDK raw 只保留滚动三天;一条 raw 对应一个完整 turn,只含真人 user 与
|
||||
assistant,tool 不持久化。提取后的高层 memory 不受该窗口影响。
|
||||
4. SDK raw 全部保留,任何模式(lite / pro / ultra)都不删除;一条 raw 对应一个
|
||||
完整 turn,只含真人 user 与 assistant,tool 不持久化。
|
||||
5. `source_raw_ids` 是 SDK 内部溯源关系,不是产品 Evidence 合同;记忆详情不提供
|
||||
“原始依据”或按 raw ID 展开原文的入口。
|
||||
|
||||
|
||||
@@ -315,6 +315,8 @@ describe("catalog exit codes", () => {
|
||||
"models",
|
||||
"memories",
|
||||
"profile",
|
||||
"settings",
|
||||
"evolve",
|
||||
"graph-width",
|
||||
"host",
|
||||
"recall",
|
||||
|
||||
+220
-3
@@ -188,6 +188,25 @@ import {
|
||||
removeManagedCliLink,
|
||||
rollbackManagedCliLink,
|
||||
} from "./managed-cli-link.js";
|
||||
import {
|
||||
type FeaturesDocument,
|
||||
featuresExitCode,
|
||||
parseFeaturesCommand,
|
||||
renderFeaturesText,
|
||||
runFeaturesCommand,
|
||||
} from "./memory-features-flow.js";
|
||||
import {
|
||||
type EvolveDocument,
|
||||
evolveExitCode,
|
||||
type ModeDocument,
|
||||
modeExitCode,
|
||||
parseEvolveCommand,
|
||||
parseSettingsCommand,
|
||||
renderEvolveText,
|
||||
renderModeText,
|
||||
runEvolveCommand,
|
||||
runModeCommand,
|
||||
} from "./memory-mode-flow.js";
|
||||
import {
|
||||
addMemory,
|
||||
DEFAULT_USER_ID,
|
||||
@@ -325,6 +344,8 @@ export interface RunControls {
|
||||
watchSignal?: AbortSignal;
|
||||
watchWait?: (milliseconds: number, signal: AbortSignal) => Promise<void>;
|
||||
watchNow?: () => string;
|
||||
/** `memhub evolve --wait` polls on this; tests inject an instant one. */
|
||||
sleep?: (milliseconds: number) => Promise<void>;
|
||||
readStdin?: () => Promise<Buffer | null>;
|
||||
stdinIsTTY?: boolean;
|
||||
/** Injectable onboarding prompter so the wizard can be driven in tests
|
||||
@@ -2018,12 +2039,17 @@ async function runCommand(
|
||||
return sourceProgressExitCode(document);
|
||||
}
|
||||
if (named === "profile") {
|
||||
const profileFlags = arguments_.slice(1);
|
||||
if (
|
||||
arguments_.filter((item) => item === "--json").length > 1 ||
|
||||
arguments_.slice(1).some((value) => value !== "--json")
|
||||
profileFlags.filter((item) => item === "--json").length > 1 ||
|
||||
profileFlags.filter((item) => item === "--versions").length > 1 ||
|
||||
profileFlags.some((value) => value !== "--json" && value !== "--versions")
|
||||
) {
|
||||
return usage();
|
||||
}
|
||||
if (profileFlags.includes("--versions")) {
|
||||
return await profileVersionsCommand(environment, { asJson, userId });
|
||||
}
|
||||
const connected = await connectRuntime(environment, {
|
||||
readerTimeoutMs: 15_000,
|
||||
});
|
||||
@@ -2223,6 +2249,89 @@ async function runCommand(
|
||||
);
|
||||
return outcome.state === "committed" ? EXIT_OK : EXIT_FAILED;
|
||||
}
|
||||
if (
|
||||
named === "settings" &&
|
||||
arguments_.slice(1).find((value) => value !== "--json") === "features"
|
||||
) {
|
||||
const parsed = parseFeaturesCommand(
|
||||
arguments_
|
||||
.slice(1)
|
||||
.filter((value, index, all) => index !== all.indexOf("features")),
|
||||
);
|
||||
if (!parsed) return usage();
|
||||
const connected = await connectRuntime(environment);
|
||||
if (!connected.ok) {
|
||||
const document: FeaturesDocument = {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
command: "settings features",
|
||||
state: "unavailable",
|
||||
code: connected.code,
|
||||
};
|
||||
emit(
|
||||
asJson ? `${JSON.stringify(document)}\n` : renderFeaturesText(document),
|
||||
);
|
||||
return connected.exitCode;
|
||||
}
|
||||
const document = await runFeaturesCommand(
|
||||
parsed,
|
||||
connected.reader,
|
||||
connected.operations,
|
||||
);
|
||||
emit(
|
||||
asJson ? `${JSON.stringify(document)}\n` : renderFeaturesText(document),
|
||||
);
|
||||
return featuresExitCode(document);
|
||||
}
|
||||
if (named === "settings") {
|
||||
const parsed = parseSettingsCommand(arguments_.slice(1));
|
||||
if (!parsed) return usage();
|
||||
const connected = await connectRuntime(environment);
|
||||
if (!connected.ok) {
|
||||
const document: ModeDocument = {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
command: "settings mode",
|
||||
state: "unavailable",
|
||||
code: connected.code,
|
||||
};
|
||||
emit(asJson ? `${JSON.stringify(document)}\n` : renderModeText(document));
|
||||
return connected.exitCode;
|
||||
}
|
||||
const document = await runModeCommand(
|
||||
parsed,
|
||||
connected.reader,
|
||||
connected.operations,
|
||||
);
|
||||
emit(asJson ? `${JSON.stringify(document)}\n` : renderModeText(document));
|
||||
return modeExitCode(document);
|
||||
}
|
||||
if (named === "evolve") {
|
||||
const parsed = parseEvolveCommand(arguments_.slice(1));
|
||||
if (!parsed) return usage();
|
||||
const connected = await connectRuntime(environment);
|
||||
if (!connected.ok) {
|
||||
const document: EvolveDocument = {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
command: "evolve",
|
||||
state: "unavailable",
|
||||
code: connected.code,
|
||||
};
|
||||
emit(
|
||||
asJson ? `${JSON.stringify(document)}\n` : renderEvolveText(document),
|
||||
);
|
||||
return connected.exitCode;
|
||||
}
|
||||
const document = await runEvolveCommand(parsed, {
|
||||
userId: userIdParameters(userId).user_id,
|
||||
reader: connected.reader,
|
||||
operations: connected.operations,
|
||||
sleep: controls.sleep,
|
||||
});
|
||||
emit(asJson ? `${JSON.stringify(document)}\n` : renderEvolveText(document));
|
||||
return evolveExitCode(document);
|
||||
}
|
||||
if (named === "graph-width") {
|
||||
const parsed = parseGraphWidthCommand(arguments_.slice(1));
|
||||
if (!parsed) return usage();
|
||||
@@ -3888,6 +3997,108 @@ async function emitModelReceipt(
|
||||
|
||||
/** Attach to a verified Runtime, telling a stopped one (exit 3) from one
|
||||
* that is up but unusable (exit 1). */
|
||||
/**
|
||||
* `memhub profile --versions`: the long-term profile's L0 version chains,
|
||||
* latest first, so a value that replaced a better one can be found by id.
|
||||
*/
|
||||
async function profileVersionsCommand(
|
||||
environment: NodeJS.ProcessEnv,
|
||||
options: { asJson: boolean; userId?: string },
|
||||
): Promise<number> {
|
||||
const fail = (state: string, code: string, exitCode: number): number => {
|
||||
emit(
|
||||
options.asJson
|
||||
? `${JSON.stringify({ schema_version: 2, product: PRODUCT, state, code })}\n`
|
||||
: `画像版本读取失败:${code}\n`,
|
||||
);
|
||||
return exitCode;
|
||||
};
|
||||
const connected = await connectRuntime(environment, {
|
||||
readerTimeoutMs: 15_000,
|
||||
});
|
||||
if (!connected.ok) {
|
||||
return fail("unavailable", connected.code, connected.exitCode);
|
||||
}
|
||||
const outcome = await connected.reader<unknown>(
|
||||
"/api/v2/user-profile/versions",
|
||||
userIdParameters(options.userId),
|
||||
);
|
||||
const chains = outcome.state === "ok" ? profileChains(outcome.data) : null;
|
||||
if (outcome.state !== "ok" || chains === null) {
|
||||
return fail(
|
||||
outcome.state === "ok" ? "invalid" : outcome.state,
|
||||
outcome.state === "ok" ? "response_invalid" : outcome.code,
|
||||
EXIT_FAILED,
|
||||
);
|
||||
}
|
||||
if (options.asJson) {
|
||||
emit(
|
||||
`${JSON.stringify({
|
||||
schema_version: 2,
|
||||
product: PRODUCT,
|
||||
state: "ok",
|
||||
...(outcome.data as object),
|
||||
})}\n`,
|
||||
);
|
||||
return EXIT_OK;
|
||||
}
|
||||
if (chains.length === 0) {
|
||||
emit("尚未形成用户画像版本。\n");
|
||||
return EXIT_OK;
|
||||
}
|
||||
const lines: string[] = [];
|
||||
for (const chain of chains) {
|
||||
lines.push(`画像 ${chain.memory_id}(${chain.status})`);
|
||||
for (const version of chain.versions) {
|
||||
const when =
|
||||
typeof version.gmt_created === "number"
|
||||
? new Date(version.gmt_created * 1000).toISOString()
|
||||
: "时间未知";
|
||||
lines.push(
|
||||
` ${version.current ? "*" : " "} ${version.memory_id} ${when} ${version.content}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
lines.push(
|
||||
"* 表示当前生效的版本;删除错误版本:memhub memories delete <memory-id> --yes",
|
||||
);
|
||||
emit(`${lines.join("\n")}\n`);
|
||||
return EXIT_OK;
|
||||
}
|
||||
|
||||
type ProfileChainView = {
|
||||
memory_id: string;
|
||||
status: string;
|
||||
versions: {
|
||||
memory_id: string;
|
||||
content: string;
|
||||
gmt_created?: number | null;
|
||||
current: boolean;
|
||||
}[];
|
||||
};
|
||||
|
||||
function profileChains(value: unknown): ProfileChainView[] | null {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
const chains = (value as { chains?: unknown }).chains;
|
||||
if (!Array.isArray(chains)) return null;
|
||||
const valid = chains.every(
|
||||
(chain) =>
|
||||
chain &&
|
||||
typeof chain === "object" &&
|
||||
typeof (chain as ProfileChainView).memory_id === "string" &&
|
||||
typeof (chain as ProfileChainView).status === "string" &&
|
||||
Array.isArray((chain as ProfileChainView).versions) &&
|
||||
(chain as ProfileChainView).versions.every(
|
||||
(version) =>
|
||||
version &&
|
||||
typeof version.memory_id === "string" &&
|
||||
typeof version.content === "string" &&
|
||||
typeof version.current === "boolean",
|
||||
),
|
||||
);
|
||||
return valid ? (chains as ProfileChainView[]) : null;
|
||||
}
|
||||
|
||||
async function connectRuntime(
|
||||
environment: NodeJS.ProcessEnv,
|
||||
options?: { readerTimeoutMs?: number },
|
||||
@@ -6707,7 +6918,13 @@ const OWN_DATA_DIR_COMMANDS = new Set([
|
||||
/** Remove one `--data-dir <absolute path>` from argv; null when it is
|
||||
* repeated, missing its value, or not an absolute path. */
|
||||
/** Memory commands that honour the global `--user-id`. */
|
||||
const USER_ID_COMMANDS = new Set(["memories", "recall", "profile", "write"]);
|
||||
const USER_ID_COMMANDS = new Set([
|
||||
"memories",
|
||||
"recall",
|
||||
"profile",
|
||||
"write",
|
||||
"evolve",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Take `--user-id <id>` out of the arguments. The default user (`local`) is
|
||||
|
||||
@@ -434,9 +434,68 @@ export const COMMANDS: readonly CommandDescriptor[] = [
|
||||
},
|
||||
{
|
||||
name: "profile",
|
||||
summary: "读取 SDK 组装后的用户长期画像(L0 + 已启用的长期画像补充)。",
|
||||
usage: "memhub profile [--user-id <id>] [--json]",
|
||||
options: [USER_ID_OPTION, JSON_OPTION],
|
||||
summary:
|
||||
"读取 SDK 组装后的用户长期画像(L0 + 已启用的长期画像补充);--versions 列出 L0 画像的版本链(最新在前,* 为当前版本)。",
|
||||
usage: "memhub profile [--versions] [--user-id <id>] [--json]",
|
||||
options: [
|
||||
{
|
||||
flag: "--versions",
|
||||
summary:
|
||||
"列出 L0 基本信息的版本链(get_user_basic_info),用于找出覆盖了正确值的错误版本。",
|
||||
},
|
||||
USER_ID_OPTION,
|
||||
JSON_OPTION,
|
||||
],
|
||||
exit_codes: RUNTIME_EXITS,
|
||||
},
|
||||
{
|
||||
name: "settings",
|
||||
summary:
|
||||
"查看或切换记忆模式(lite / pro / ultra)。lite 不调用模型,只在原始对话里检索;pro 由模型提取记忆(System 1);ultra 在 pro 之上每晚自动进化(System 2,使用知识图谱)。任何模式都保留全部原始对话。切换不迁移数据:已处理的不会重做,排队中的和新的按新模式处理;需 --yes,不加时只显示计划。环境变量 ONE_MEMORY_GRAPH_STORE 会强制 ultra。settings features 查看或开关 SDK 的可选检索与提取功能(全局设置,默认全部保持 SDK 默认值):graph_rerank 图谱重排(仅 ultra)、same_series_timeline 同系列时间线(需 graph_rerank)、episode_recall 片段召回(pro / ultra,需先开 session_episodes 写入片段)、session_episodes 片段写入、episode_summary 片段摘要(每个片段一次模型调用)、intention 意图提取(默认开)、few_shot 提取示例;set 需 --yes,不加时只显示计划;reset 恢复默认值。",
|
||||
usage:
|
||||
"memhub settings mode [lite|pro|ultra] [--yes] [--json]\n" +
|
||||
" memhub settings features [--json]\n" +
|
||||
" memhub settings features set <name>=<true|false> [...] [--yes] [--json]\n" +
|
||||
" memhub settings features reset [<name> ...] [--json]",
|
||||
options: [
|
||||
{
|
||||
flag: "mode [lite|pro|ultra]",
|
||||
summary: "不带模式只显示当前模式;带模式时先显示切换计划。",
|
||||
},
|
||||
{
|
||||
flag: "features set <name>=<true|false>",
|
||||
summary:
|
||||
"开关一个或多个 SDK 功能;功能在当前模式下不生效时会注明原因(例如 graph_rerank 需要 ultra)。",
|
||||
},
|
||||
{
|
||||
flag: "features reset [<name> ...]",
|
||||
summary: "把指定功能(不写则全部)恢复为 SDK 默认值。",
|
||||
},
|
||||
CONFIRM_OPTION,
|
||||
JSON_OPTION,
|
||||
],
|
||||
exit_codes: RUNTIME_EXITS,
|
||||
},
|
||||
{
|
||||
name: "evolve",
|
||||
summary:
|
||||
"立即为一个用户运行记忆进化(System 2,仅 ultra 模式)。在后台运行并返回运行 ID;--wait 等待结果。同一用户同时只运行一次;每晚的自动进化由 Runtime 按设置安排。",
|
||||
usage:
|
||||
"memhub evolve [--user-id <id>] [--agent-id <id>] [--max-batches <n>] [--wait] [--json]",
|
||||
options: [
|
||||
USER_ID_OPTION,
|
||||
{
|
||||
flag: "--agent-id <id>",
|
||||
summary:
|
||||
"进化哪个 agent 的记忆,默认 all-apps(MemHub 写入记忆用的 agent)。",
|
||||
},
|
||||
{
|
||||
flag: "--max-batches <n>",
|
||||
summary: "每轮最多处理的批次数(1–1000),默认由 SDK 决定。",
|
||||
},
|
||||
{ flag: "--wait", summary: "等待这次进化结束再返回。" },
|
||||
JSON_OPTION,
|
||||
],
|
||||
exit_codes: RUNTIME_EXITS,
|
||||
},
|
||||
{
|
||||
|
||||
@@ -76,6 +76,8 @@ const COMMANDS = [
|
||||
"models",
|
||||
"memories",
|
||||
"profile",
|
||||
"settings",
|
||||
"evolve",
|
||||
"graph-width",
|
||||
"host",
|
||||
"recall",
|
||||
|
||||
@@ -17,7 +17,14 @@ const READ_ONLY_PATHS = [
|
||||
"/api/v2/home-summary",
|
||||
"/api/v2/memory-library",
|
||||
"/api/v2/user-profile",
|
||||
"/api/v2/user-profile/versions",
|
||||
"/api/v1/settings/memory-mode",
|
||||
"/api/v1/settings/memory-features",
|
||||
"/api/v2/evolve",
|
||||
] as const;
|
||||
/** One evolve run's receipt (`memhub evolve --wait`). */
|
||||
const EVOLVE_RUN =
|
||||
/^\/api\/v2\/evolve\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
|
||||
const SDK_MEMORY_DETAIL = /^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}$/;
|
||||
/** Status of one `POST /api/v2/memories` addition, by its request id. */
|
||||
const MEMORY_ADDITION =
|
||||
@@ -220,6 +227,7 @@ export function isReadOnlyPath(path: string): boolean {
|
||||
if (IMPORT_JOB.test(path)) return true;
|
||||
if (OPERATION_RECEIPT.test(path)) return true;
|
||||
if (MEMORY_ADDITION.test(path)) return true;
|
||||
if (EVOLVE_RUN.test(path)) return true;
|
||||
if (!SDK_MEMORY_DETAIL.test(path)) return false;
|
||||
const segment = path.slice("/api/v2/memories/".length);
|
||||
return !(RESERVED_MEMORY_SEGMENTS as readonly string[]).includes(segment);
|
||||
|
||||
@@ -26,6 +26,11 @@ const SOURCE_IMPORT_JOB_PATH = "/api/v1/import-jobs";
|
||||
const IMPORT_JOB_ACTION_PATH =
|
||||
/^\/api\/v1\/import-jobs\/[A-Za-z0-9][\w.-]{0,127}\/(?:pause|resume|cancel)$/;
|
||||
const GRAPH_WIDTH_PATH = "/api/v1/graph-width";
|
||||
/** `memhub evolve` starts a run; `memhub settings mode <m> --yes` sets the mode. */
|
||||
const EVOLVE_PATH = "/api/v2/evolve";
|
||||
const MEMORY_MODE_PATH = "/api/v1/settings/memory-mode";
|
||||
/** `memhub settings features set|reset` sets the SDK features. */
|
||||
const MEMORY_FEATURES_PATH = "/api/v1/settings/memory-features";
|
||||
const ONBOARDING_COMPLETE_PATH = "/api/v1/onboarding/complete";
|
||||
const MAX_BYTES = 4 * 1024 * 1024;
|
||||
|
||||
@@ -97,6 +102,7 @@ export function createCoreOperations(attachment: {
|
||||
!SOURCE_AUTHORIZE_PATH.test(path) &&
|
||||
!SOURCE_REVOKE_PATH.test(path) &&
|
||||
path !== GRAPH_WIDTH_PATH &&
|
||||
path !== EVOLVE_PATH &&
|
||||
path !== ONBOARDING_COMPLETE_PATH &&
|
||||
!MODEL_PROFILE_ACTIVATE.test(path)
|
||||
) {
|
||||
@@ -132,7 +138,11 @@ export function createCoreOperations(attachment: {
|
||||
body: Record<string, unknown>,
|
||||
signal?: AbortSignal,
|
||||
) {
|
||||
if (!SDK_MEMORY_PATH.test(path) || path.includes("?")) {
|
||||
if (
|
||||
path !== MEMORY_MODE_PATH &&
|
||||
path !== MEMORY_FEATURES_PATH &&
|
||||
(!SDK_MEMORY_PATH.test(path) || path.includes("?"))
|
||||
) {
|
||||
throw new Error("core_operation_path_not_allowed");
|
||||
}
|
||||
let response: { status: number; data: unknown };
|
||||
|
||||
@@ -0,0 +1,318 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { run } from "./cli.js";
|
||||
import { parseFeaturesCommand } from "./memory-features-flow.js";
|
||||
import { composeStatusReport, renderStatusText } from "./status-report.js";
|
||||
|
||||
const transport = vi.hoisted(() => ({
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
put: vi.fn(),
|
||||
patch: vi.fn(),
|
||||
delete: vi.fn(),
|
||||
}));
|
||||
vi.mock("axios", () => ({
|
||||
default: {
|
||||
create: vi.fn(() => transport),
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
isAxiosError: () => false,
|
||||
},
|
||||
}));
|
||||
|
||||
const runtime = vi.hoisted(() => ({
|
||||
inspectRuntimeStatus: vi.fn(),
|
||||
discoverRuntimeAttachment: vi.fn(),
|
||||
}));
|
||||
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
|
||||
...runtime,
|
||||
}));
|
||||
|
||||
const DEFAULTS: Record<string, boolean> = {
|
||||
graph_rerank: false,
|
||||
same_series_timeline: false,
|
||||
episode_recall: false,
|
||||
session_episodes: false,
|
||||
episode_summary: false,
|
||||
intention: true,
|
||||
few_shot: false,
|
||||
};
|
||||
|
||||
function featuresState(
|
||||
configured: Record<string, boolean> = {},
|
||||
mode = "pro",
|
||||
extra: Record<string, unknown> = {},
|
||||
) {
|
||||
const nonDefault: Record<string, boolean> = {};
|
||||
const features = Object.entries(DEFAULTS).map(([name, fallback]) => {
|
||||
const enabled = configured[name] ?? fallback;
|
||||
if (enabled !== fallback) nonDefault[name] = enabled;
|
||||
const needsUltra = name === "graph_rerank";
|
||||
const reason = !enabled
|
||||
? "disabled"
|
||||
: needsUltra && mode !== "ultra"
|
||||
? "requires_ultra"
|
||||
: null;
|
||||
return {
|
||||
name,
|
||||
enabled,
|
||||
configured: name in configured ? configured[name] : null,
|
||||
default: fallback,
|
||||
active: reason === null,
|
||||
reason,
|
||||
side: "read",
|
||||
requires_mode: needsUltra ? "ultra" : "pro",
|
||||
requires_feature: null,
|
||||
cost: "none",
|
||||
sdk_field: `recall.${name}`,
|
||||
env: `MEMORY_${name.toUpperCase()}`,
|
||||
description: name,
|
||||
};
|
||||
});
|
||||
return { mode, features, non_default: nonDefault, ...extra };
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
for (const method of Object.values(transport)) method.mockReset();
|
||||
runtime.inspectRuntimeStatus.mockResolvedValue({
|
||||
state: "running",
|
||||
code: "runtime_verified",
|
||||
instanceId: "instance-1",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
version: "1.2.3",
|
||||
});
|
||||
runtime.discoverRuntimeAttachment.mockResolvedValue({
|
||||
instanceId: "instance-1",
|
||||
apiToken: "synthetic-runtime-token",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
servicePid: 4242,
|
||||
health: {
|
||||
status: "ok",
|
||||
service: "one-memory-core",
|
||||
version: "1.2.3",
|
||||
instance_id: "instance-1",
|
||||
},
|
||||
});
|
||||
transport.get.mockImplementation(async (url: string) =>
|
||||
url === "/api/v1/settings/memory-features"
|
||||
? { status: 200, data: featuresState() }
|
||||
: { status: 404, data: {} },
|
||||
);
|
||||
});
|
||||
|
||||
describe("memhub settings features", () => {
|
||||
it("lists every feature with its value and whether it acts", async () => {
|
||||
const result = await capturedRun(["settings", "features", "--json"]);
|
||||
expect(result.code).toBe(0);
|
||||
const json = JSON.parse(result.stdout);
|
||||
expect(json.state).toBe("ok");
|
||||
expect(json.features.features).toHaveLength(7);
|
||||
expect(transport.put).not.toHaveBeenCalled();
|
||||
const text = (await capturedRun(["settings", "features"])).stdout;
|
||||
expect(text).toContain("graph_rerank:关(默认)");
|
||||
expect(text).toContain("intention:开(默认)");
|
||||
expect(text).toContain("先开启 session_episodes");
|
||||
});
|
||||
|
||||
it("prints the plan and changes nothing without --yes", async () => {
|
||||
const result = await capturedRun([
|
||||
"settings",
|
||||
"features",
|
||||
"set",
|
||||
"graph_rerank=true",
|
||||
"episode_recall=true",
|
||||
]);
|
||||
expect(result.code).toBe(0);
|
||||
expect(transport.put).not.toHaveBeenCalled();
|
||||
expect(result.stdout).toContain(
|
||||
"memhub settings features set graph_rerank=true episode_recall=true --yes",
|
||||
);
|
||||
});
|
||||
|
||||
it("sets features with --yes and reports the ones inactive in this mode", async () => {
|
||||
transport.put.mockResolvedValue({
|
||||
status: 200,
|
||||
data: featuresState({ graph_rerank: true, episode_recall: true }, "pro", {
|
||||
changed: ["graph_rerank", "episode_recall"],
|
||||
}),
|
||||
});
|
||||
const result = await capturedRun([
|
||||
"settings",
|
||||
"features",
|
||||
"set",
|
||||
"graph_rerank=true",
|
||||
"episode_recall=true",
|
||||
"--yes",
|
||||
]);
|
||||
expect(result.code).toBe(0);
|
||||
expect(transport.put).toHaveBeenCalledWith(
|
||||
"/api/v1/settings/memory-features",
|
||||
{ graph_rerank: true, episode_recall: true },
|
||||
undefined,
|
||||
);
|
||||
expect(result.stdout).toContain("已更新:graph_rerank、episode_recall");
|
||||
expect(result.stdout).toContain(
|
||||
"graph_rerank:开(已设置,当前模式下不生效(需要 ultra))",
|
||||
);
|
||||
});
|
||||
|
||||
it("does nothing when the values are already set", async () => {
|
||||
transport.get.mockResolvedValue({
|
||||
status: 200,
|
||||
data: featuresState({ few_shot: true }),
|
||||
});
|
||||
const result = await capturedRun([
|
||||
"settings",
|
||||
"features",
|
||||
"set",
|
||||
"few_shot=true",
|
||||
"--yes",
|
||||
]);
|
||||
expect(result.code).toBe(0);
|
||||
expect(result.stdout).toContain("没有需要改动的设置");
|
||||
expect(transport.put).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("resets the named features, or every set one, to the default", async () => {
|
||||
transport.get.mockResolvedValue({
|
||||
status: 200,
|
||||
data: featuresState({ few_shot: true, intention: false }),
|
||||
});
|
||||
transport.put.mockResolvedValue({
|
||||
status: 200,
|
||||
data: featuresState({}, "pro", { changed: ["intention", "few_shot"] }),
|
||||
});
|
||||
expect(
|
||||
(await capturedRun(["settings", "features", "reset", "few_shot"])).code,
|
||||
).toBe(0);
|
||||
expect(transport.put).toHaveBeenLastCalledWith(
|
||||
"/api/v1/settings/memory-features",
|
||||
{ few_shot: null },
|
||||
undefined,
|
||||
);
|
||||
const all = await capturedRun(["settings", "features", "reset", "--json"]);
|
||||
expect(JSON.parse(all.stdout).state).toBe("committed");
|
||||
expect(transport.put).toHaveBeenLastCalledWith(
|
||||
"/api/v1/settings/memory-features",
|
||||
{ intention: null, few_shot: null },
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
|
||||
it("refuses unknown names and malformed values before the Runtime", async () => {
|
||||
for (const bad of [
|
||||
["settings", "features", "set"],
|
||||
["settings", "features", "set", "local_rerank=true"],
|
||||
["settings", "features", "set", "graph_rerank=yes"],
|
||||
["settings", "features", "set", "graph_rerank"],
|
||||
["settings", "features", "set", "few_shot=true", "few_shot=false"],
|
||||
["settings", "features", "reset", "nope"],
|
||||
["settings", "features", "toggle"],
|
||||
]) {
|
||||
const result = await capturedRun(bad);
|
||||
expect(result.code, bad.join(" ")).toBe(2);
|
||||
}
|
||||
expect(transport.put).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("surfaces a 422 from the Runtime as rejected", async () => {
|
||||
transport.put.mockResolvedValue({
|
||||
status: 422,
|
||||
data: { detail: "invalid_request" },
|
||||
});
|
||||
const result = await capturedRun([
|
||||
"settings",
|
||||
"features",
|
||||
"set",
|
||||
"few_shot=true",
|
||||
"--yes",
|
||||
"--json",
|
||||
]);
|
||||
expect(result.code).toBe(1);
|
||||
expect(JSON.parse(result.stdout).state).toBe("rejected");
|
||||
});
|
||||
|
||||
it("parses the forms the help shows", () => {
|
||||
expect(parseFeaturesCommand([])).toEqual({ kind: "show" });
|
||||
expect(parseFeaturesCommand(["set", "intention=false", "--yes"])).toEqual({
|
||||
kind: "set",
|
||||
values: { intention: false },
|
||||
confirmed: true,
|
||||
});
|
||||
expect(parseFeaturesCommand(["reset"])).toEqual({
|
||||
kind: "reset",
|
||||
names: [],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("status pipeline features", () => {
|
||||
it("shows the features that differ from the defaults", async () => {
|
||||
const processing = {
|
||||
pending: 0,
|
||||
fresh_pending: 0,
|
||||
backlog_pending: 0,
|
||||
outside_window_pending: 0,
|
||||
running: 0,
|
||||
waiting_for_model: 0,
|
||||
completed: 3,
|
||||
failed: 0,
|
||||
blocked_stages: [],
|
||||
pipeline: {
|
||||
mode: "pro",
|
||||
configured_mode: "pro",
|
||||
forced_by_env: false,
|
||||
last_evolve_at: null,
|
||||
graph_build: { state: "idle", users: {} },
|
||||
evolve: { mode: "pro", schedule: null, running: [], users: {} },
|
||||
features: { episode_recall: true, intention: false },
|
||||
},
|
||||
};
|
||||
const report = await composeStatusReport({
|
||||
dataDir: "/tmp/isolated",
|
||||
runtime: { state: "running", code: "ok", host: "127.0.0.1", port: 1 },
|
||||
background: null,
|
||||
reader: (async (path: string) =>
|
||||
path === "/api/v1/memory-processing"
|
||||
? { state: "ok", data: processing }
|
||||
: { state: "unavailable", code: "x" }) as never,
|
||||
});
|
||||
expect(report.pipeline?.features).toEqual({
|
||||
episode_recall: true,
|
||||
intention: false,
|
||||
});
|
||||
const text = renderStatusText(report, "MemHub");
|
||||
expect(text).toContain("SDK 功能:episode_recall=开、intention=关");
|
||||
});
|
||||
});
|
||||
|
||||
async function capturedRun(arguments_: string[]) {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const stdoutSpy = vi
|
||||
.spyOn(process.stdout, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stdout += String(chunk);
|
||||
return true;
|
||||
});
|
||||
const stderrSpy = vi
|
||||
.spyOn(process.stderr, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stderr += String(chunk);
|
||||
return true;
|
||||
});
|
||||
try {
|
||||
return {
|
||||
code: await run(arguments_, { ONE_MEMORY_DATA_DIR: "/synthetic/data" }),
|
||||
stdout,
|
||||
stderr,
|
||||
};
|
||||
} finally {
|
||||
stdoutSpy.mockRestore();
|
||||
stderrSpy.mockRestore();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,273 @@
|
||||
/**
|
||||
* `memhub settings features [set <name>=<true|false> ... | reset [<name> ...]]`.
|
||||
*
|
||||
* The SDK's optional retrieval and extraction features (graph rerank, episode
|
||||
* recall, the episode write side, intention, few-shot). One global setting in
|
||||
* the Runtime; an unset feature keeps the SDK default, so an install nobody
|
||||
* touched behaves as before. `set` prints its plan first and needs `--yes`,
|
||||
* like `settings mode`, because some features spend model calls; `reset`
|
||||
* only returns features to their defaults and applies at once.
|
||||
*/
|
||||
|
||||
import type { CoreReader } from "./core-client.js";
|
||||
import type { CorePutOperations } from "./core-operations.js";
|
||||
import { PRODUCT } from "./version.js";
|
||||
|
||||
export const MEMORY_FEATURES = [
|
||||
"graph_rerank",
|
||||
"same_series_timeline",
|
||||
"episode_recall",
|
||||
"session_episodes",
|
||||
"episode_summary",
|
||||
"intention",
|
||||
"few_shot",
|
||||
] as const;
|
||||
export type MemoryFeature = (typeof MEMORY_FEATURES)[number];
|
||||
|
||||
const FEATURE_LABEL: Record<MemoryFeature, string> = {
|
||||
graph_rerank: "图谱重排(沿知识图谱的边传播支持度;仅 ultra)",
|
||||
same_series_timeline:
|
||||
"同系列时间线(同一系列的记忆按时间放在一起;需图谱重排)",
|
||||
episode_recall:
|
||||
"片段召回(补入同一片段的记忆并多通道打分;pro / ultra,需片段写入)",
|
||||
session_episodes: "片段写入(同一会话提取的记忆归入同一片段)",
|
||||
episode_summary: "片段摘要(片段结束时总结一次;每个片段一次模型调用)",
|
||||
intention: "意图提取(随事实一起提取用户的计划与意图)",
|
||||
few_shot: "提取示例(在提取提示里附上示例;不加调用,输入更长)",
|
||||
};
|
||||
|
||||
const REASON_TEXT: Record<string, string> = {
|
||||
disabled: "未开启",
|
||||
requires_pro: "lite 模式下不生效(需要 pro 或 ultra)",
|
||||
requires_ultra: "当前模式下不生效(需要 ultra)",
|
||||
requires_graph_rerank: "需要先开启并生效 graph_rerank",
|
||||
};
|
||||
|
||||
export type FeaturesCommand =
|
||||
| { kind: "show" }
|
||||
| {
|
||||
kind: "set";
|
||||
values: Partial<Record<MemoryFeature, boolean>>;
|
||||
confirmed: boolean;
|
||||
}
|
||||
| { kind: "reset"; names: MemoryFeature[] };
|
||||
|
||||
function isFeature(value: string): value is MemoryFeature {
|
||||
return (MEMORY_FEATURES as readonly string[]).includes(value);
|
||||
}
|
||||
|
||||
/** Arguments after `settings features`; null = usage. */
|
||||
export function parseFeaturesCommand(
|
||||
values: readonly string[],
|
||||
): FeaturesCommand | null {
|
||||
const rest = values.filter((value) => value !== "--json");
|
||||
if (rest.length === 0) return { kind: "show" };
|
||||
const [verb, ...tail] = rest;
|
||||
const yes = tail.filter((value) => value === "--yes").length;
|
||||
if (yes > 1) return null;
|
||||
const positional = tail.filter((value) => value !== "--yes");
|
||||
if (verb === "set") {
|
||||
if (positional.length === 0) return null;
|
||||
const parsed: Partial<Record<MemoryFeature, boolean>> = {};
|
||||
for (const item of positional) {
|
||||
const match = /^([a-z_]+)=(true|false)$/.exec(item);
|
||||
if (!match || !isFeature(match[1]) || match[1] in parsed) return null;
|
||||
parsed[match[1]] = match[2] === "true";
|
||||
}
|
||||
return { kind: "set", values: parsed, confirmed: yes === 1 };
|
||||
}
|
||||
if (verb === "reset") {
|
||||
if (new Set(positional).size !== positional.length) return null;
|
||||
if (!positional.every(isFeature)) return null;
|
||||
return { kind: "reset", names: positional as MemoryFeature[] };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export interface FeatureState {
|
||||
name: MemoryFeature;
|
||||
enabled: boolean;
|
||||
configured: boolean | null;
|
||||
default: boolean;
|
||||
active: boolean;
|
||||
reason: string | null;
|
||||
cost: string;
|
||||
requires_mode: string;
|
||||
description: string;
|
||||
}
|
||||
|
||||
export interface FeaturesState {
|
||||
mode: string;
|
||||
features: FeatureState[];
|
||||
non_default: Partial<Record<MemoryFeature, boolean>>;
|
||||
changed?: MemoryFeature[];
|
||||
}
|
||||
|
||||
function featuresState(value: unknown): FeaturesState | null {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
const item = value as Record<string, unknown>;
|
||||
if (
|
||||
typeof item.mode !== "string" ||
|
||||
!Array.isArray(item.features) ||
|
||||
!item.non_default ||
|
||||
typeof item.non_default !== "object"
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
for (const feature of item.features) {
|
||||
if (
|
||||
!feature ||
|
||||
typeof feature !== "object" ||
|
||||
typeof (feature as FeatureState).name !== "string" ||
|
||||
typeof (feature as FeatureState).enabled !== "boolean" ||
|
||||
typeof (feature as FeatureState).active !== "boolean"
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
return item as unknown as FeaturesState;
|
||||
}
|
||||
|
||||
export type FeaturesDocument = {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
command: "settings features";
|
||||
} & (
|
||||
| { state: "ok"; features: FeaturesState }
|
||||
| {
|
||||
state: "preview";
|
||||
features: FeaturesState;
|
||||
target: Partial<Record<MemoryFeature, boolean>>;
|
||||
applied: false;
|
||||
next_step: string;
|
||||
}
|
||||
| { state: "unchanged"; features: FeaturesState }
|
||||
| { state: "committed"; features: FeaturesState; changed: MemoryFeature[] }
|
||||
| { state: "rejected" | "unknown" | "unavailable"; code: string }
|
||||
);
|
||||
|
||||
const BASE = {
|
||||
schema_version: 1 as const,
|
||||
product: PRODUCT,
|
||||
command: "settings features" as const,
|
||||
};
|
||||
|
||||
const PATH = "/api/v1/settings/memory-features";
|
||||
|
||||
export async function runFeaturesCommand(
|
||||
command: FeaturesCommand,
|
||||
reader: CoreReader,
|
||||
operations: CorePutOperations,
|
||||
): Promise<FeaturesDocument> {
|
||||
const read = await reader<unknown>(PATH);
|
||||
if (read.state !== "ok") {
|
||||
return { ...BASE, state: "unavailable", code: read.code };
|
||||
}
|
||||
const current = featuresState(read.data);
|
||||
if (current === null) {
|
||||
return { ...BASE, state: "unavailable", code: "response_invalid" };
|
||||
}
|
||||
if (command.kind === "show") {
|
||||
return { ...BASE, state: "ok", features: current };
|
||||
}
|
||||
let body: Record<string, unknown>;
|
||||
if (command.kind === "set") {
|
||||
const configured = new Map(
|
||||
current.features.map((feature) => [feature.name, feature.configured]),
|
||||
);
|
||||
const differs = Object.entries(command.values).some(
|
||||
([name, value]) => configured.get(name as MemoryFeature) !== value,
|
||||
);
|
||||
if (!differs) return { ...BASE, state: "unchanged", features: current };
|
||||
if (!command.confirmed) {
|
||||
const pairs = Object.entries(command.values)
|
||||
.map(([name, value]) => `${name}=${value}`)
|
||||
.join(" ");
|
||||
return {
|
||||
...BASE,
|
||||
state: "preview",
|
||||
features: current,
|
||||
target: command.values,
|
||||
applied: false,
|
||||
next_step: `${PRODUCT} settings features set ${pairs} --yes`,
|
||||
};
|
||||
}
|
||||
body = { ...command.values };
|
||||
} else {
|
||||
const names =
|
||||
command.names.length > 0
|
||||
? command.names
|
||||
: current.features
|
||||
.filter((feature) => feature.configured !== null)
|
||||
.map((feature) => feature.name);
|
||||
if (
|
||||
names.every(
|
||||
(name) =>
|
||||
current.features.find((feature) => feature.name === name)
|
||||
?.configured === null,
|
||||
)
|
||||
) {
|
||||
return { ...BASE, state: "unchanged", features: current };
|
||||
}
|
||||
body = Object.fromEntries(names.map((name) => [name, null]));
|
||||
}
|
||||
const result = await operations.put<unknown>(PATH, body);
|
||||
if (result.state !== "committed") {
|
||||
return { ...BASE, state: result.state, code: result.code };
|
||||
}
|
||||
const changed = featuresState(result.data);
|
||||
if (changed === null) {
|
||||
return { ...BASE, state: "unknown", code: "response_invalid" };
|
||||
}
|
||||
return {
|
||||
...BASE,
|
||||
state: "committed",
|
||||
features: changed,
|
||||
changed: changed.changed ?? [],
|
||||
};
|
||||
}
|
||||
|
||||
export function featuresExitCode(document: FeaturesDocument): number {
|
||||
return ["ok", "preview", "unchanged", "committed"].includes(document.state)
|
||||
? 0
|
||||
: 1;
|
||||
}
|
||||
|
||||
function featureLine(feature: FeatureState): string {
|
||||
const label = FEATURE_LABEL[feature.name] ?? feature.name;
|
||||
const value = feature.enabled ? "开" : "关";
|
||||
const origin = feature.configured === null ? "默认" : "已设置";
|
||||
const activity =
|
||||
feature.enabled && !feature.active && feature.reason
|
||||
? `,${REASON_TEXT[feature.reason] ?? feature.reason}`
|
||||
: "";
|
||||
return `${feature.name}:${value}(${origin}${activity})— ${label}`;
|
||||
}
|
||||
|
||||
export function renderFeaturesText(document: FeaturesDocument): string {
|
||||
if ("code" in document) {
|
||||
return `SDK 功能未能${document.state === "unavailable" ? "读取" : "修改"}:${document.code}\n`;
|
||||
}
|
||||
const state = document.features;
|
||||
const lines = [`当前模式:${state.mode}`];
|
||||
for (const feature of state.features) lines.push(featureLine(feature));
|
||||
lines.push(
|
||||
"片段召回(episode_recall)只对带片段的记忆有用:先开启 session_episodes,之后提取的记忆才会带片段。",
|
||||
);
|
||||
if (document.state === "unchanged") lines.push("没有需要改动的设置。");
|
||||
if (document.state === "preview") {
|
||||
lines.push("将设置:");
|
||||
for (const [name, value] of Object.entries(document.target)) {
|
||||
lines.push(`- ${name} = ${value ? "开" : "关"}`);
|
||||
}
|
||||
lines.push(`未执行任何改动。确认后运行:${document.next_step}`);
|
||||
}
|
||||
if (document.state === "committed") {
|
||||
lines.push(
|
||||
document.changed.length > 0
|
||||
? `已更新:${document.changed.join("、")}。正在进行的写入按原设置完成,之后的写入和检索使用新设置。`
|
||||
: "已保存,生效的值没有变化。",
|
||||
);
|
||||
}
|
||||
return `${lines.join("\n")}\n`;
|
||||
}
|
||||
@@ -0,0 +1,309 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { run } from "./cli.js";
|
||||
import { modeEffects, parseSettingsCommand } from "./memory-mode-flow.js";
|
||||
import { composeStatusReport, renderStatusText } from "./status-report.js";
|
||||
|
||||
const transport = vi.hoisted(() => ({
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
put: vi.fn(),
|
||||
patch: vi.fn(),
|
||||
delete: vi.fn(),
|
||||
}));
|
||||
vi.mock("axios", () => ({
|
||||
default: {
|
||||
create: vi.fn(() => transport),
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
isAxiosError: () => false,
|
||||
},
|
||||
}));
|
||||
|
||||
const runtime = vi.hoisted(() => ({
|
||||
inspectRuntimeStatus: vi.fn(),
|
||||
discoverRuntimeAttachment: vi.fn(),
|
||||
}));
|
||||
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
|
||||
...runtime,
|
||||
}));
|
||||
|
||||
const RUN_ID = "11111111-2222-4333-8444-555555555555";
|
||||
|
||||
function modeState(mode: string, configured = mode) {
|
||||
return {
|
||||
mode,
|
||||
configured_mode: configured,
|
||||
forced_by_env: false,
|
||||
modes: ["lite", "pro", "ultra"],
|
||||
graph_build: { state: "idle", users: {} },
|
||||
};
|
||||
}
|
||||
|
||||
function receipt(state: string, extra: Record<string, unknown> = {}) {
|
||||
return {
|
||||
run_id: RUN_ID,
|
||||
user_id: "wife",
|
||||
agent_id: "all-apps",
|
||||
state,
|
||||
trigger: "manual",
|
||||
started_at: "2026-11-01T03:00:00Z",
|
||||
rounds: state === "running" ? 0 : 2,
|
||||
tasks_processed: state === "running" ? 0 : 5,
|
||||
batch_failures: 0,
|
||||
capped: false,
|
||||
elapsed_ms: 10,
|
||||
results: { schemas: ["never printed"] },
|
||||
...extra,
|
||||
};
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
for (const method of Object.values(transport)) method.mockReset();
|
||||
runtime.inspectRuntimeStatus.mockResolvedValue({
|
||||
state: "running",
|
||||
code: "runtime_verified",
|
||||
instanceId: "instance-1",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
version: "1.2.3",
|
||||
});
|
||||
runtime.discoverRuntimeAttachment.mockResolvedValue({
|
||||
instanceId: "instance-1",
|
||||
apiToken: "synthetic-runtime-token",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
servicePid: 4242,
|
||||
health: {
|
||||
status: "ok",
|
||||
service: "one-memory-core",
|
||||
version: "1.2.3",
|
||||
instance_id: "instance-1",
|
||||
},
|
||||
});
|
||||
transport.get.mockImplementation(async (url: string) =>
|
||||
url === "/api/v1/settings/memory-mode"
|
||||
? { status: 200, data: modeState("pro") }
|
||||
: { status: 404, data: {} },
|
||||
);
|
||||
});
|
||||
|
||||
describe("memhub settings mode", () => {
|
||||
it("shows the current mode without changing it", async () => {
|
||||
const result = await capturedRun(["settings", "mode", "--json"]);
|
||||
expect(result.code).toBe(0);
|
||||
expect(JSON.parse(result.stdout).mode.mode).toBe("pro");
|
||||
expect(transport.put).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("prints the plan and changes nothing without --yes", async () => {
|
||||
const result = await capturedRun(["settings", "mode", "lite"]);
|
||||
expect(result.code).toBe(0);
|
||||
expect(transport.put).not.toHaveBeenCalled();
|
||||
expect(result.stdout).toContain("更高模式生成的记忆仍保存着");
|
||||
expect(result.stdout).toContain("memhub settings mode lite --yes");
|
||||
const json = JSON.parse(
|
||||
(await capturedRun(["settings", "mode", "ultra", "--json"])).stdout,
|
||||
);
|
||||
expect(json.state).toBe("preview");
|
||||
expect(json.applied).toBe(false);
|
||||
expect(json.effects).toEqual(["existing_memories_added_to_graph"]);
|
||||
});
|
||||
|
||||
it("changes the mode with --yes and reports what changed", async () => {
|
||||
transport.put.mockResolvedValue({
|
||||
status: 200,
|
||||
data: {
|
||||
...modeState("lite"),
|
||||
previous_mode: "pro",
|
||||
effects: ["higher_mode_memories_hidden"],
|
||||
},
|
||||
});
|
||||
const result = await capturedRun(["settings", "mode", "lite", "--yes"]);
|
||||
expect(result.code).toBe(0);
|
||||
expect(transport.put).toHaveBeenCalledWith(
|
||||
"/api/v1/settings/memory-mode",
|
||||
{ mode: "lite" },
|
||||
undefined,
|
||||
);
|
||||
expect(result.stdout).toContain("已切换");
|
||||
expect(result.stdout).toContain("原始对话:全部保留");
|
||||
});
|
||||
|
||||
it("does nothing for the mode already set, and refuses bad input", async () => {
|
||||
const same = await capturedRun(["settings", "mode", "pro", "--yes"]);
|
||||
expect(same.code).toBe(0);
|
||||
expect(same.stdout).toContain("已经是这个模式");
|
||||
expect(transport.put).not.toHaveBeenCalled();
|
||||
for (const bad of [
|
||||
["settings"],
|
||||
["settings", "mode", "max"],
|
||||
["settings", "mode", "--yes"],
|
||||
["settings", "mode", "lite", "pro"],
|
||||
["settings", "language", "en"],
|
||||
]) {
|
||||
expect((await capturedRun(bad)).code, bad.join(" ")).toBe(2);
|
||||
}
|
||||
});
|
||||
|
||||
it("plans the same effects Core reports", () => {
|
||||
expect(modeEffects("lite", "pro")).toEqual([
|
||||
"earlier_raw_not_extracted",
|
||||
"lite_search_stops",
|
||||
]);
|
||||
expect(modeEffects("ultra", "lite")).toEqual([
|
||||
"higher_mode_memories_hidden",
|
||||
"evolve_stopped",
|
||||
]);
|
||||
expect(modeEffects("pro", "pro")).toEqual([]);
|
||||
expect(parseSettingsCommand(["mode", "ultra", "--yes"])).toEqual({
|
||||
kind: "set",
|
||||
mode: "ultra",
|
||||
confirmed: true,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("memhub evolve", () => {
|
||||
it("starts a run for the named user and returns at once", async () => {
|
||||
transport.post.mockResolvedValue({ status: 202, data: receipt("running") });
|
||||
const result = await capturedRun([
|
||||
"evolve",
|
||||
"--user-id",
|
||||
"wife",
|
||||
"--agent-id",
|
||||
"all-apps",
|
||||
"--max-batches",
|
||||
"3",
|
||||
"--json",
|
||||
]);
|
||||
expect(result.code).toBe(0);
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v2/evolve",
|
||||
{ user_id: "wife", agent_id: "all-apps", max_batches: 3 },
|
||||
undefined,
|
||||
);
|
||||
const document = JSON.parse(result.stdout);
|
||||
expect(document.state).toBe("running");
|
||||
expect(document.run.results).toBeUndefined();
|
||||
});
|
||||
|
||||
it("follows the run with --wait until it ends", async () => {
|
||||
transport.post.mockResolvedValue({ status: 202, data: receipt("running") });
|
||||
const reads = [receipt("running"), receipt("succeeded")];
|
||||
transport.get.mockImplementation(async (url: string) =>
|
||||
url === `/api/v2/evolve/${RUN_ID}`
|
||||
? { status: 200, data: reads.shift() }
|
||||
: { status: 404, data: {} },
|
||||
);
|
||||
const result = await capturedRun(["evolve", "--wait"], {
|
||||
sleep: async () => undefined,
|
||||
});
|
||||
expect(result.code).toBe(0);
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v2/evolve",
|
||||
{},
|
||||
undefined,
|
||||
);
|
||||
expect(result.stdout).toContain("进化完成");
|
||||
expect(result.stdout).toContain("2 轮");
|
||||
});
|
||||
|
||||
it("names the mode when Core refuses outside ultra", async () => {
|
||||
transport.post.mockResolvedValue({
|
||||
status: 409,
|
||||
data: { detail: "memory_mode_not_ultra" },
|
||||
});
|
||||
const result = await capturedRun(["evolve"]);
|
||||
expect(result.code).toBe(1);
|
||||
expect(result.stdout).toContain("memhub settings mode ultra --yes");
|
||||
expect((await capturedRun(["evolve", "--max-batches", "0"])).code).toBe(2);
|
||||
expect((await capturedRun(["evolve", "--now"])).code).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("status pipeline", () => {
|
||||
it("reports pipeline.mode and the last evolve from memory-processing", async () => {
|
||||
const processing = {
|
||||
pending: 0,
|
||||
fresh_pending: 0,
|
||||
backlog_pending: 0,
|
||||
outside_window_pending: 0,
|
||||
running: 0,
|
||||
waiting_for_model: 0,
|
||||
completed: 3,
|
||||
failed: 0,
|
||||
blocked_stages: [],
|
||||
pipeline: {
|
||||
mode: "ultra",
|
||||
configured_mode: "ultra",
|
||||
forced_by_env: false,
|
||||
last_evolve_at: "2026-11-01T03:00:05+08:00",
|
||||
graph_build: { state: "succeeded", users: {} },
|
||||
evolve: {
|
||||
mode: "ultra",
|
||||
schedule: { enabled: true, time: "03:00" },
|
||||
running: [],
|
||||
users: {
|
||||
local: {
|
||||
last_run_at: "2026-11-01T03:00:05+08:00",
|
||||
last_result: { state: "succeeded", tasks_processed: 4 },
|
||||
last_error: null,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
const report = await composeStatusReport({
|
||||
dataDir: "/tmp/isolated",
|
||||
runtime: { state: "running", code: "ok", host: "127.0.0.1", port: 1 },
|
||||
background: null,
|
||||
reader: (async (path: string) =>
|
||||
path === "/api/v1/memory-processing"
|
||||
? { state: "ok", data: processing }
|
||||
: { state: "unavailable", code: "x" }) as never,
|
||||
});
|
||||
expect(report.pipeline?.mode).toBe("ultra");
|
||||
expect(report.pipeline?.last_evolve_at).toBe("2026-11-01T03:00:05+08:00");
|
||||
expect(report.pipeline?.evolve_users.local?.state).toBe("succeeded");
|
||||
const text = renderStatusText(report, "MemHub");
|
||||
expect(text).toContain("记忆模式:ultra");
|
||||
expect(text).toContain("每日 03:00 自动进化");
|
||||
});
|
||||
});
|
||||
|
||||
async function capturedRun(
|
||||
arguments_: string[],
|
||||
controls: Parameters<typeof run>[2] = {},
|
||||
) {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const stdoutSpy = vi
|
||||
.spyOn(process.stdout, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stdout += String(chunk);
|
||||
return true;
|
||||
});
|
||||
const stderrSpy = vi
|
||||
.spyOn(process.stderr, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stderr += String(chunk);
|
||||
return true;
|
||||
});
|
||||
try {
|
||||
return {
|
||||
code: await run(
|
||||
arguments_,
|
||||
{ ONE_MEMORY_DATA_DIR: "/synthetic/data" },
|
||||
controls,
|
||||
),
|
||||
stdout,
|
||||
stderr,
|
||||
};
|
||||
} finally {
|
||||
stdoutSpy.mockRestore();
|
||||
stderrSpy.mockRestore();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,372 @@
|
||||
/**
|
||||
* `memhub settings mode` and `memhub evolve`.
|
||||
*
|
||||
* The mode (lite / pro / ultra) decides what the SDK does with every
|
||||
* conversation, and switching it migrates nothing: what one mode made is not
|
||||
* shown by a lower one, and raw kept by lite is not extracted until it is
|
||||
* imported again. So a change prints its plan first and needs `--yes`, like
|
||||
* every other command whose effect a person cannot simply take back.
|
||||
*
|
||||
* `memhub evolve` starts System 2 (ultra only) and, with `--wait`, follows
|
||||
* the run until it ends.
|
||||
*/
|
||||
|
||||
import type { CoreReader } from "./core-client.js";
|
||||
import type { CoreOperations, CorePutOperations } from "./core-operations.js";
|
||||
import { PRODUCT } from "./version.js";
|
||||
|
||||
export const MEMORY_MODES = ["lite", "pro", "ultra"] as const;
|
||||
export type MemoryMode = (typeof MEMORY_MODES)[number];
|
||||
|
||||
export type ModeEffect =
|
||||
| "earlier_raw_not_extracted"
|
||||
| "higher_mode_memories_hidden"
|
||||
| "existing_memories_added_to_graph"
|
||||
| "evolve_stopped"
|
||||
| "lite_search_stops"
|
||||
| "graph_backfill_already_running";
|
||||
|
||||
const MODE_SUMMARY: Record<MemoryMode, string> = {
|
||||
lite: "lite:不调用模型,保留全部原始对话,只在原始对话里检索",
|
||||
pro: "pro:模型从对话中提取记忆(System 1)",
|
||||
ultra: "ultra:pro + 每晚自动进化(System 2,使用知识图谱)",
|
||||
};
|
||||
|
||||
const EFFECT_TEXT: Record<ModeEffect, string> = {
|
||||
earlier_raw_not_extracted:
|
||||
"lite 下已处理完的对话不会再被提取;只有尚未处理的和之后的新对话会由模型整理成记忆,检索立即改为记忆检索",
|
||||
higher_mode_memories_hidden:
|
||||
"更高模式生成的记忆仍保存着,但在这个模式下不会显示",
|
||||
existing_memories_added_to_graph:
|
||||
"已有记忆会在后台加入知识图谱(每个用户一次)",
|
||||
evolve_stopped: "每晚的自动进化(System 2)停止",
|
||||
lite_search_stops: "lite 只在原始对话里的检索立即停止,改为检索记忆",
|
||||
graph_backfill_already_running: "知识图谱的补建已在进行中",
|
||||
};
|
||||
|
||||
/** The same rule Core applies (`memory_mode_effects`), for the plan. */
|
||||
export function modeEffects(
|
||||
previous: MemoryMode,
|
||||
next: MemoryMode,
|
||||
): ModeEffect[] {
|
||||
if (previous === next) return [];
|
||||
const rank: Record<MemoryMode, number> = { lite: 0, pro: 1, ultra: 2 };
|
||||
const effects: ModeEffect[] = [];
|
||||
if (previous === "lite") {
|
||||
effects.push("earlier_raw_not_extracted", "lite_search_stops");
|
||||
}
|
||||
if (rank[next] < rank[previous]) effects.push("higher_mode_memories_hidden");
|
||||
if (previous === "ultra") effects.push("evolve_stopped");
|
||||
if (next === "ultra") effects.push("existing_memories_added_to_graph");
|
||||
return effects;
|
||||
}
|
||||
|
||||
export type ModeCommand =
|
||||
| { kind: "show" }
|
||||
| { kind: "set"; mode: MemoryMode; confirmed: boolean };
|
||||
|
||||
/** `settings mode [lite|pro|ultra] [--yes]` after `settings`; null = usage. */
|
||||
export function parseSettingsCommand(
|
||||
values: readonly string[],
|
||||
): ModeCommand | null {
|
||||
const rest = values.filter((value) => value !== "--json");
|
||||
if (rest[0] !== "mode") return null;
|
||||
const tail = rest.slice(1);
|
||||
const confirmed = tail.includes("--yes");
|
||||
const positional = tail.filter((value) => value !== "--yes");
|
||||
if (tail.filter((value) => value === "--yes").length > 1) return null;
|
||||
if (positional.length === 0) return confirmed ? null : { kind: "show" };
|
||||
if (positional.length !== 1) return null;
|
||||
const mode = positional[0] as MemoryMode;
|
||||
if (!(MEMORY_MODES as readonly string[]).includes(mode)) return null;
|
||||
return { kind: "set", mode, confirmed };
|
||||
}
|
||||
|
||||
export interface ModeState {
|
||||
mode: MemoryMode;
|
||||
configured_mode: MemoryMode;
|
||||
forced_by_env: boolean;
|
||||
graph_build?: { state?: string } | null;
|
||||
previous_mode?: MemoryMode;
|
||||
effects?: ModeEffect[];
|
||||
}
|
||||
|
||||
function modeState(value: unknown): ModeState | null {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
const item = value as Record<string, unknown>;
|
||||
if (
|
||||
!(MEMORY_MODES as readonly unknown[]).includes(item.mode) ||
|
||||
!(MEMORY_MODES as readonly unknown[]).includes(item.configured_mode) ||
|
||||
typeof item.forced_by_env !== "boolean"
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return item as unknown as ModeState;
|
||||
}
|
||||
|
||||
export type ModeDocument = {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
command: "settings mode";
|
||||
} & (
|
||||
| { state: "ok"; mode: ModeState }
|
||||
| {
|
||||
state: "preview";
|
||||
mode: ModeState;
|
||||
target: MemoryMode;
|
||||
effects: ModeEffect[];
|
||||
applied: false;
|
||||
next_step: string;
|
||||
}
|
||||
| { state: "unchanged"; mode: ModeState }
|
||||
| { state: "committed"; mode: ModeState; effects: ModeEffect[] }
|
||||
| { state: "rejected" | "unknown" | "unavailable"; code: string }
|
||||
);
|
||||
|
||||
const BASE = {
|
||||
schema_version: 1 as const,
|
||||
product: PRODUCT,
|
||||
command: "settings mode" as const,
|
||||
};
|
||||
|
||||
export async function runModeCommand(
|
||||
command: ModeCommand,
|
||||
reader: CoreReader,
|
||||
operations: CorePutOperations,
|
||||
): Promise<ModeDocument> {
|
||||
const read = await reader<unknown>("/api/v1/settings/memory-mode");
|
||||
if (read.state !== "ok") {
|
||||
return { ...BASE, state: "unavailable", code: read.code };
|
||||
}
|
||||
const current = modeState(read.data);
|
||||
if (current === null) {
|
||||
return { ...BASE, state: "unavailable", code: "response_invalid" };
|
||||
}
|
||||
if (command.kind === "show") return { ...BASE, state: "ok", mode: current };
|
||||
if (current.configured_mode === command.mode) {
|
||||
return { ...BASE, state: "unchanged", mode: current };
|
||||
}
|
||||
if (!command.confirmed) {
|
||||
return {
|
||||
...BASE,
|
||||
state: "preview",
|
||||
mode: current,
|
||||
target: command.mode,
|
||||
effects: modeEffects(current.mode, command.mode),
|
||||
applied: false,
|
||||
next_step: `${PRODUCT} settings mode ${command.mode} --yes`,
|
||||
};
|
||||
}
|
||||
const result = await operations.put<unknown>("/api/v1/settings/memory-mode", {
|
||||
mode: command.mode,
|
||||
});
|
||||
if (result.state !== "committed") {
|
||||
return { ...BASE, state: result.state, code: result.code };
|
||||
}
|
||||
const changed = modeState(result.data);
|
||||
if (changed === null) {
|
||||
return { ...BASE, state: "unknown", code: "response_invalid" };
|
||||
}
|
||||
return {
|
||||
...BASE,
|
||||
state: "committed",
|
||||
mode: changed,
|
||||
effects: changed.effects ?? [],
|
||||
};
|
||||
}
|
||||
|
||||
export function modeExitCode(document: ModeDocument): number {
|
||||
return ["ok", "preview", "unchanged", "committed"].includes(document.state)
|
||||
? 0
|
||||
: 1;
|
||||
}
|
||||
|
||||
export function renderModeText(document: ModeDocument): string {
|
||||
if ("code" in document) {
|
||||
return `记忆模式未能${document.state === "unavailable" ? "读取" : "修改"}:${document.code}\n`;
|
||||
}
|
||||
const mode = document.mode;
|
||||
const lines = [`当前模式:${MODE_SUMMARY[mode.mode]}`];
|
||||
if (mode.forced_by_env) {
|
||||
lines.push(
|
||||
`设置里是 ${mode.configured_mode};环境变量 ONE_MEMORY_GRAPH_STORE 强制使用 ultra。`,
|
||||
);
|
||||
}
|
||||
lines.push("原始对话:全部保留(任何模式都不删除)");
|
||||
if (document.state === "unchanged") {
|
||||
lines.push("已经是这个模式,未做改动。");
|
||||
}
|
||||
if (document.state === "preview") {
|
||||
lines.push(`将切换到:${MODE_SUMMARY[document.target]}`);
|
||||
for (const effect of document.effects)
|
||||
lines.push(`- ${EFFECT_TEXT[effect]}`);
|
||||
lines.push(`未执行任何改动。确认后运行:${document.next_step}`);
|
||||
}
|
||||
if (document.state === "committed") {
|
||||
lines.push("已切换。");
|
||||
for (const effect of document.effects)
|
||||
lines.push(`- ${EFFECT_TEXT[effect]}`);
|
||||
}
|
||||
return `${lines.join("\n")}\n`;
|
||||
}
|
||||
|
||||
export type EvolveCommand = {
|
||||
agentId?: string;
|
||||
maxBatches?: number;
|
||||
wait: boolean;
|
||||
};
|
||||
|
||||
/** `evolve [--agent-id <id>] [--max-batches <n>] [--wait]`; null = usage. */
|
||||
export function parseEvolveCommand(
|
||||
values: readonly string[],
|
||||
): EvolveCommand | null {
|
||||
const parsed: EvolveCommand = { wait: false };
|
||||
const rest = values.filter((value) => value !== "--json");
|
||||
for (let index = 0; index < rest.length; index += 1) {
|
||||
const flag = rest[index];
|
||||
if (flag === "--wait" && !parsed.wait) {
|
||||
parsed.wait = true;
|
||||
} else if (flag === "--agent-id" && parsed.agentId === undefined) {
|
||||
const value = rest[index + 1];
|
||||
if (!value || !/^[A-Za-z0-9._:-]{1,128}$/.test(value)) return null;
|
||||
parsed.agentId = value;
|
||||
index += 1;
|
||||
} else if (flag === "--max-batches" && parsed.maxBatches === undefined) {
|
||||
const value = Number(rest[index + 1]);
|
||||
if (!Number.isInteger(value) || value < 1 || value > 1000) return null;
|
||||
parsed.maxBatches = value;
|
||||
index += 1;
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
export type EvolveDocument = {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
command: "evolve";
|
||||
} & (
|
||||
| { state: "running" | "succeeded" | "failed"; run: Record<string, unknown> }
|
||||
| { state: "rejected" | "unknown" | "unavailable"; code: string }
|
||||
);
|
||||
|
||||
const RUN_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
|
||||
|
||||
function evolveRun(value: unknown): Record<string, unknown> | null {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
const run = value as Record<string, unknown>;
|
||||
if (
|
||||
typeof run.run_id !== "string" ||
|
||||
!RUN_ID.test(run.run_id) ||
|
||||
!["running", "succeeded", "failed"].includes(String(run.state))
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return run;
|
||||
}
|
||||
|
||||
export async function runEvolveCommand(
|
||||
command: EvolveCommand,
|
||||
options: {
|
||||
userId?: string;
|
||||
reader: CoreReader;
|
||||
operations: CoreOperations;
|
||||
sleep?: (ms: number) => Promise<void>;
|
||||
pollMs?: number;
|
||||
},
|
||||
): Promise<EvolveDocument> {
|
||||
const base = {
|
||||
schema_version: 1 as const,
|
||||
product: PRODUCT,
|
||||
command: "evolve" as const,
|
||||
};
|
||||
const body: Record<string, unknown> = {};
|
||||
if (options.userId !== undefined) body.user_id = options.userId;
|
||||
if (command.agentId !== undefined) body.agent_id = command.agentId;
|
||||
if (command.maxBatches !== undefined) body.max_batches = command.maxBatches;
|
||||
const started = await options.operations.post<unknown>(
|
||||
"/api/v2/evolve",
|
||||
body,
|
||||
);
|
||||
if (started.state !== "committed") {
|
||||
return { ...base, state: started.state, code: started.code };
|
||||
}
|
||||
let run = evolveRun(started.data);
|
||||
if (run === null)
|
||||
return { ...base, state: "unknown", code: "response_invalid" };
|
||||
const sleep =
|
||||
options.sleep ??
|
||||
((ms: number) => new Promise((done) => setTimeout(done, ms)));
|
||||
while (command.wait && run.state === "running") {
|
||||
await sleep(options.pollMs ?? 2_000);
|
||||
const read = await options.reader<unknown>(
|
||||
`/api/v2/evolve/${run.run_id}`,
|
||||
options.userId === undefined ? {} : { user_id: options.userId },
|
||||
);
|
||||
if (read.state !== "ok") {
|
||||
return { ...base, state: "unavailable", code: read.code };
|
||||
}
|
||||
const next = evolveRun(read.data);
|
||||
if (next === null)
|
||||
return { ...base, state: "unknown", code: "response_invalid" };
|
||||
run = next;
|
||||
}
|
||||
return {
|
||||
...base,
|
||||
state: run.state as "running" | "succeeded" | "failed",
|
||||
run: projectRun(run),
|
||||
};
|
||||
}
|
||||
|
||||
/** Counts and codes; SDK result bodies stay in the Core receipt. */
|
||||
function projectRun(run: Record<string, unknown>): Record<string, unknown> {
|
||||
const keep = [
|
||||
"run_id",
|
||||
"user_id",
|
||||
"agent_id",
|
||||
"state",
|
||||
"trigger",
|
||||
"started_at",
|
||||
"finished_at",
|
||||
"rounds",
|
||||
"tasks_processed",
|
||||
"batch_failures",
|
||||
"cycle_status",
|
||||
"should_continue",
|
||||
"continuation_reason",
|
||||
"remaining_clusters",
|
||||
"remaining_actionable_clusters",
|
||||
"remaining_fresh_facts",
|
||||
"capped",
|
||||
"elapsed_ms",
|
||||
"error_code",
|
||||
"failure_kind",
|
||||
];
|
||||
return Object.fromEntries(
|
||||
keep.filter((key) => key in run).map((key) => [key, run[key]]),
|
||||
);
|
||||
}
|
||||
|
||||
export function evolveExitCode(document: EvolveDocument): number {
|
||||
return document.state === "running" || document.state === "succeeded" ? 0 : 1;
|
||||
}
|
||||
|
||||
export function renderEvolveText(document: EvolveDocument): string {
|
||||
if ("code" in document) {
|
||||
return document.code === "memory_mode_not_ultra"
|
||||
? "进化只在 ultra 模式下可用。切换:memhub settings mode ultra --yes\n"
|
||||
: `进化未能开始:${document.code}\n`;
|
||||
}
|
||||
const run = document.run;
|
||||
if (document.state === "running") {
|
||||
return `进化已开始(${run.run_id}),在后台运行;加 --wait 等待结果。\n`;
|
||||
}
|
||||
if (document.state === "failed") {
|
||||
return `进化失败(${run.run_id}):${String(run.error_code ?? "unknown")}\n`;
|
||||
}
|
||||
return `进化完成(${run.run_id}):${run.rounds} 轮,处理 ${run.tasks_processed} 项${
|
||||
run.cycle_status ? `,周期 ${run.cycle_status}` : ""
|
||||
}${run.capped ? ",已到轮数上限,可再次运行继续" : ""}\n`;
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { run } from "./cli.js";
|
||||
|
||||
const transport = vi.hoisted(() => ({
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
put: vi.fn(),
|
||||
patch: vi.fn(),
|
||||
delete: vi.fn(),
|
||||
}));
|
||||
vi.mock("axios", () => ({
|
||||
default: {
|
||||
create: vi.fn(() => transport),
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
isAxiosError: () => false,
|
||||
},
|
||||
}));
|
||||
|
||||
const runtime = vi.hoisted(() => ({
|
||||
inspectRuntimeStatus: vi.fn(),
|
||||
discoverRuntimeAttachment: vi.fn(),
|
||||
}));
|
||||
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
|
||||
...runtime,
|
||||
}));
|
||||
|
||||
const VERSIONS = {
|
||||
user_id: "wife",
|
||||
request_id: "r-versions",
|
||||
elapsed_ms: 2,
|
||||
chains: [
|
||||
{
|
||||
memory_id: "l0-head",
|
||||
content: "The user's name is Fei.",
|
||||
status: "active",
|
||||
memory_at: null,
|
||||
gmt_created: 1_790_000_100,
|
||||
basic_info: { name: "Fei" },
|
||||
versions: [
|
||||
{
|
||||
memory_id: "l0-head",
|
||||
content: "The user's name is Fei.",
|
||||
memory_at: null,
|
||||
gmt_created: 1_790_000_100,
|
||||
current: true,
|
||||
},
|
||||
{
|
||||
memory_id: "l0-old",
|
||||
content: "The user's name is 费天翔.",
|
||||
memory_at: null,
|
||||
gmt_created: 1_790_000_000,
|
||||
current: false,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
for (const method of Object.values(transport)) method.mockReset();
|
||||
runtime.inspectRuntimeStatus.mockResolvedValue({
|
||||
state: "running",
|
||||
code: "runtime_verified",
|
||||
instanceId: "instance-1",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
version: "1.2.3",
|
||||
});
|
||||
runtime.discoverRuntimeAttachment.mockResolvedValue({
|
||||
instanceId: "instance-1",
|
||||
apiToken: "synthetic-runtime-token",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
servicePid: 4242,
|
||||
health: {
|
||||
status: "ok",
|
||||
service: "one-memory-core",
|
||||
version: "1.2.3",
|
||||
instance_id: "instance-1",
|
||||
},
|
||||
});
|
||||
transport.get.mockImplementation(async (url: string) =>
|
||||
url === "/api/v2/user-profile/versions"
|
||||
? { status: 200, data: VERSIONS }
|
||||
: { status: 404, data: {} },
|
||||
);
|
||||
});
|
||||
|
||||
describe("memhub profile --versions", () => {
|
||||
it("reads the L0 version chain for the named user", async () => {
|
||||
const result = await capturedRun([
|
||||
"profile",
|
||||
"--versions",
|
||||
"--user-id",
|
||||
"wife",
|
||||
"--json",
|
||||
]);
|
||||
expect(result.code).toBe(0);
|
||||
expect(transport.get).toHaveBeenCalledWith(
|
||||
"/api/v2/user-profile/versions",
|
||||
{ params: { user_id: "wife" } },
|
||||
);
|
||||
const document = JSON.parse(result.stdout);
|
||||
expect(document.state).toBe("ok");
|
||||
expect(document.chains[0].versions).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("marks the current version in text and names no user for local", async () => {
|
||||
const result = await capturedRun(["profile", "--versions"]);
|
||||
expect(result.code).toBe(0);
|
||||
const [url, config] = transport.get.mock.calls[0] ?? [];
|
||||
expect(url).toBe("/api/v2/user-profile/versions");
|
||||
expect(config?.params?.user_id).toBeUndefined();
|
||||
expect(result.stdout).toContain("* l0-head");
|
||||
expect(result.stdout).toContain(" l0-old");
|
||||
expect(result.stdout).toContain("费天翔");
|
||||
});
|
||||
|
||||
it("refuses an unknown flag and a repeated --versions", async () => {
|
||||
expect((await capturedRun(["profile", "--version-chain"])).code).toBe(2);
|
||||
expect(
|
||||
(await capturedRun(["profile", "--versions", "--versions"])).code,
|
||||
).toBe(2);
|
||||
expect(transport.get).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("reports a malformed answer as invalid", async () => {
|
||||
transport.get.mockResolvedValue({ status: 200, data: { chains: "x" } });
|
||||
const result = await capturedRun(["profile", "--versions", "--json"]);
|
||||
expect(result.code).toBe(1);
|
||||
expect(JSON.parse(result.stdout).code).toBe("response_invalid");
|
||||
});
|
||||
});
|
||||
|
||||
async function capturedRun(arguments_: string[]) {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const stdoutSpy = vi
|
||||
.spyOn(process.stdout, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stdout += String(chunk);
|
||||
return true;
|
||||
});
|
||||
const stderrSpy = vi
|
||||
.spyOn(process.stderr, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stderr += String(chunk);
|
||||
return true;
|
||||
});
|
||||
try {
|
||||
return {
|
||||
code: await run(arguments_, {
|
||||
ONE_MEMORY_DATA_DIR: "/synthetic/data",
|
||||
}),
|
||||
stdout,
|
||||
stderr,
|
||||
};
|
||||
} finally {
|
||||
stdoutSpy.mockRestore();
|
||||
stderrSpy.mockRestore();
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,11 @@ import {
|
||||
validateProcessing,
|
||||
validateSources,
|
||||
} from "./core-responses.js";
|
||||
import {
|
||||
describePipeline,
|
||||
type PipelineSummary,
|
||||
pipelineLine,
|
||||
} from "./status-report.js";
|
||||
import { PRODUCT } from "./version.js";
|
||||
|
||||
type ReadState =
|
||||
@@ -84,6 +89,8 @@ export interface ProgressDocument {
|
||||
summary_state: ReadState;
|
||||
summary: Record<string, unknown> | null;
|
||||
};
|
||||
/** The SDK mode and last evolve per user; absent on an older Core. */
|
||||
pipeline?: PipelineSummary | null;
|
||||
}
|
||||
|
||||
export async function readSources(
|
||||
@@ -283,6 +290,10 @@ export async function readProgress(
|
||||
summary:
|
||||
processing.state === "ok" ? projectProcessing(processing.data) : null,
|
||||
},
|
||||
pipeline:
|
||||
processingRead.state === "ok"
|
||||
? describePipeline(processingRead.data)
|
||||
: null,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -353,6 +364,18 @@ export function renderProgressText(document: ProgressDocument): string {
|
||||
? `记忆整理:待处理 ${document.generation.summary.pending} · 进行中 ${document.generation.summary.running} · 等待模型 ${document.generation.summary.waiting_for_model} · 失败 ${document.generation.summary.failed}`
|
||||
: "记忆整理:暂时无法读取",
|
||||
);
|
||||
if (document.pipeline) {
|
||||
lines.push(`记忆模式:${pipelineLine(document.pipeline)}`);
|
||||
for (const [userId, user] of Object.entries(
|
||||
document.pipeline.evolve_users,
|
||||
)) {
|
||||
lines.push(
|
||||
` 进化 ${userId}:${user.last_run_at ?? "尚未进化"}${
|
||||
user.last_error ? `(失败:${user.last_error})` : ""
|
||||
}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (document.imports.output_truncated) {
|
||||
lines.push("这里只显示汇总;完整记录请使用 --json。");
|
||||
}
|
||||
|
||||
@@ -57,6 +57,9 @@ export interface StatusDocument {
|
||||
broker?: BackgroundHealth["broker"];
|
||||
};
|
||||
sections: Record<string, Section>;
|
||||
/** R10: the SDK mode and System 2 state; null when the Runtime is not read
|
||||
* or is older than the setting. */
|
||||
pipeline?: PipelineSummary | null;
|
||||
summary: {
|
||||
state: "ok" | "partial";
|
||||
degraded: string[];
|
||||
@@ -64,6 +67,99 @@ export interface StatusDocument {
|
||||
};
|
||||
}
|
||||
|
||||
export interface PipelineSummary {
|
||||
mode: "lite" | "pro" | "ultra";
|
||||
configured_mode: "lite" | "pro" | "ultra";
|
||||
forced_by_env: boolean;
|
||||
last_evolve_at: string | null;
|
||||
evolve_schedule: { enabled: boolean; time: string } | null;
|
||||
evolve_running: number;
|
||||
evolve_users: Record<
|
||||
string,
|
||||
{
|
||||
last_run_at: string | null;
|
||||
last_error: string | null;
|
||||
state: string | null;
|
||||
}
|
||||
>;
|
||||
graph_build: string | null;
|
||||
/** SDK features (`memory_features`) whose value differs from the default. */
|
||||
features: Record<string, boolean>;
|
||||
}
|
||||
|
||||
const MODES = new Set(["lite", "pro", "ultra"]);
|
||||
|
||||
/** Projection of `pipeline` in `/api/v1/memory-processing`; null if absent or malformed. */
|
||||
export function describePipeline(value: unknown): PipelineSummary | null {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
const pipeline = (value as { pipeline?: unknown }).pipeline;
|
||||
if (!pipeline || typeof pipeline !== "object") return null;
|
||||
const item = pipeline as Record<string, unknown>;
|
||||
if (
|
||||
!MODES.has(String(item.mode)) ||
|
||||
!MODES.has(String(item.configured_mode))
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
const evolve =
|
||||
item.evolve && typeof item.evolve === "object"
|
||||
? (item.evolve as Record<string, unknown>)
|
||||
: {};
|
||||
const schedule =
|
||||
evolve.schedule && typeof evolve.schedule === "object"
|
||||
? (evolve.schedule as Record<string, unknown>)
|
||||
: null;
|
||||
const users: PipelineSummary["evolve_users"] = {};
|
||||
if (evolve.users && typeof evolve.users === "object") {
|
||||
for (const [userId, raw] of Object.entries(
|
||||
evolve.users as Record<string, unknown>,
|
||||
)) {
|
||||
const user =
|
||||
raw && typeof raw === "object" ? (raw as Record<string, unknown>) : {};
|
||||
const result =
|
||||
user.last_result && typeof user.last_result === "object"
|
||||
? (user.last_result as Record<string, unknown>)
|
||||
: {};
|
||||
users[userId] = {
|
||||
last_run_at:
|
||||
typeof user.last_run_at === "string" ? user.last_run_at : null,
|
||||
last_error:
|
||||
typeof user.last_error === "string" ? user.last_error : null,
|
||||
state: typeof result.state === "string" ? result.state : null,
|
||||
};
|
||||
}
|
||||
}
|
||||
const features: Record<string, boolean> = {};
|
||||
if (item.features && typeof item.features === "object") {
|
||||
for (const [name, value] of Object.entries(
|
||||
item.features as Record<string, unknown>,
|
||||
)) {
|
||||
if (typeof value === "boolean") features[name] = value;
|
||||
}
|
||||
}
|
||||
const graph =
|
||||
item.graph_build && typeof item.graph_build === "object"
|
||||
? (item.graph_build as Record<string, unknown>)
|
||||
: {};
|
||||
return {
|
||||
mode: item.mode as PipelineSummary["mode"],
|
||||
configured_mode: item.configured_mode as PipelineSummary["mode"],
|
||||
forced_by_env: item.forced_by_env === true,
|
||||
last_evolve_at:
|
||||
typeof item.last_evolve_at === "string" ? item.last_evolve_at : null,
|
||||
evolve_schedule:
|
||||
schedule &&
|
||||
typeof schedule.enabled === "boolean" &&
|
||||
typeof schedule.time === "string"
|
||||
? { enabled: schedule.enabled, time: schedule.time }
|
||||
: null,
|
||||
evolve_running: Array.isArray(evolve.running) ? evolve.running.length : 0,
|
||||
evolve_users: users,
|
||||
graph_build: typeof graph.state === "string" ? graph.state : null,
|
||||
features,
|
||||
};
|
||||
}
|
||||
|
||||
const SECTION_NAMES = [
|
||||
"sources",
|
||||
"sync",
|
||||
@@ -97,6 +193,7 @@ export async function composeStatusReport(options: {
|
||||
attachFailure?: string;
|
||||
}): Promise<StatusDocument> {
|
||||
const sections: Record<string, Section> = {};
|
||||
let pipeline: PipelineSummary | null = null;
|
||||
if (options.reader === null) {
|
||||
const state: SectionState =
|
||||
options.attachFailure === undefined ? "not_checked" : "unavailable";
|
||||
@@ -118,6 +215,8 @@ export async function composeStatusReport(options: {
|
||||
validateProcessing,
|
||||
describeProcessing,
|
||||
);
|
||||
pipeline =
|
||||
processing.state === "ok" ? describePipeline(processing.data) : null;
|
||||
sections.model = section(model, validateModelProfiles, describeModel);
|
||||
sections.memories = section(
|
||||
memories,
|
||||
@@ -169,6 +268,7 @@ export async function composeStatusReport(options: {
|
||||
},
|
||||
bridge,
|
||||
sections,
|
||||
pipeline,
|
||||
summary: {
|
||||
state:
|
||||
degraded.length === 0 && notChecked.length === 0 ? "ok" : "partial",
|
||||
@@ -205,6 +305,7 @@ export function renderStatusText(
|
||||
`${SECTION_LABELS[name]}:${sectionLine(report.sections[name])}`,
|
||||
);
|
||||
}
|
||||
if (report.pipeline) lines.push(`记忆模式:${pipelineLine(report.pipeline)}`);
|
||||
if (report.summary.state === "partial") {
|
||||
const notes = [
|
||||
report.summary.degraded.length > 0
|
||||
@@ -220,6 +321,36 @@ export function renderStatusText(
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
export function pipelineLine(pipeline: PipelineSummary): string {
|
||||
const parts: string[] = [pipeline.mode];
|
||||
if (pipeline.forced_by_env)
|
||||
parts.push("(ONE_MEMORY_GRAPH_STORE 强制 ultra)");
|
||||
if (pipeline.mode === "ultra") {
|
||||
const schedule = pipeline.evolve_schedule;
|
||||
parts.push(
|
||||
schedule?.enabled
|
||||
? `,每日 ${schedule.time} 自动进化`
|
||||
: ",自动进化已关闭",
|
||||
);
|
||||
parts.push(
|
||||
pipeline.last_evolve_at
|
||||
? `,上次进化 ${pipeline.last_evolve_at}`
|
||||
: ",尚未进化",
|
||||
);
|
||||
if (pipeline.evolve_running > 0)
|
||||
parts.push(`,进行中 ${pipeline.evolve_running}`);
|
||||
}
|
||||
const changed = Object.entries(pipeline.features);
|
||||
if (changed.length > 0) {
|
||||
parts.push(
|
||||
`;SDK 功能:${changed
|
||||
.map(([name, value]) => `${name}=${value ? "开" : "关"}`)
|
||||
.join("、")}`,
|
||||
);
|
||||
}
|
||||
return parts.join("");
|
||||
}
|
||||
|
||||
function runtimeLine(report: StatusDocument): string {
|
||||
if (report.runtime.state !== "running") {
|
||||
return report.runtime.state === "stopped" ? "未运行" : "暂时无法确认";
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -5,6 +5,7 @@ from typing import Annotated, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||
|
||||
from one_memory.sdk_search_options import SdkSearchOptions
|
||||
from one_memory.user_scope import DEFAULT_USER_ID, UserId
|
||||
|
||||
ContextKind = Literal["raw", "memory"]
|
||||
@@ -49,12 +50,17 @@ class ContextSearchRequest(BaseModel):
|
||||
return self
|
||||
|
||||
|
||||
class SdkContextSearchRequest(ContextSearchRequest):
|
||||
class SdkContextSearchRequest(ContextSearchRequest, SdkSearchOptions):
|
||||
"""`POST /api/v2/context/search`: the three SDK recall lanes are set apart.
|
||||
|
||||
`limit` is the normal lane's top-k and the cap on returned items;
|
||||
`schema_limit` and `intention_limit` are the SDK's schema (L6) and
|
||||
proactive intention (L7) lanes. A caller's values are passed through.
|
||||
|
||||
The `SdkSearchOptions` fields reach the memory lane's `client.search`;
|
||||
the raw lane takes the floor and the source, session, agent and time
|
||||
filters (never wider than the caller's raw authority). Absent, both
|
||||
lanes make the call they always made.
|
||||
"""
|
||||
|
||||
limit: int = Field(default=10, ge=1, le=20, strict=True)
|
||||
|
||||
@@ -24,14 +24,57 @@ from one_memory.hy_runtime_config import HY_MEMORY_READER
|
||||
from one_memory.hy_sdk_runtime import sdk_relevance_floor
|
||||
from one_memory.hy_sdk_types import (
|
||||
HySdkMemoryRecord,
|
||||
parse_sdk_lane_search,
|
||||
parse_sdk_raw_search,
|
||||
parse_sdk_search,
|
||||
)
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.sdk_search_options import SdkSearchFilters, SearchLane
|
||||
from one_memory.semantic_deadline import SemanticMemoryHit
|
||||
from one_memory.user_scope import owns
|
||||
|
||||
CALL_EVENT_RESPONSE_BUDGET_MS = 250
|
||||
|
||||
#: Pass-through filters the raw lane honours. Scene, layers, reader and the
|
||||
#: schema/working floors describe memory lanes; the raw lane is always the
|
||||
#: chat store's L1 raw layer.
|
||||
_RAW_LANE_FILTERS = ("created_after", "session_ids", "agent_ids", "exclude_sources", "rerank")
|
||||
|
||||
|
||||
def raw_lane_sources(
|
||||
allowed: Sequence[str] | frozenset[str] | None,
|
||||
requested: Sequence[str] | None,
|
||||
) -> list[str] | None:
|
||||
"""The raw lane's source filter: the caller's, never wider than its authority.
|
||||
|
||||
`allowed` empty or None means every source (the call Core always made).
|
||||
None back means nothing is left to search.
|
||||
"""
|
||||
allowed_list = list(allowed or ())
|
||||
if requested is None:
|
||||
return allowed_list
|
||||
chosen = [
|
||||
source for source in dict.fromkeys(requested) if not allowed_list or source in allowed_list
|
||||
]
|
||||
return chosen or None
|
||||
|
||||
|
||||
def raw_lane_overrides(value: SdkSearchFilters) -> dict[str, object]:
|
||||
"""The caller's raw-lane overrides: its floor and its row filters."""
|
||||
overrides = value.filter_kwargs()
|
||||
found = {name: overrides[name] for name in _RAW_LANE_FILTERS if name in overrides}
|
||||
floor = value.lane_floor("raw")
|
||||
if floor is not None:
|
||||
found["min_score"] = floor
|
||||
return found
|
||||
|
||||
|
||||
def hit_namespace(hit: SemanticMemoryHit) -> dict[str, str]:
|
||||
"""The `get` namespace of a hit: nothing for chat, kind and variant otherwise."""
|
||||
if hit.kind in {"experience", "fact"} and hit.variant is not None:
|
||||
return {"kind": hit.kind, "variant": hit.variant}
|
||||
return {}
|
||||
|
||||
|
||||
def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
"""SDK-native context: no Evidence/Governance ids or fabricated revisions."""
|
||||
@@ -67,26 +110,36 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
client = await client_handle.client()
|
||||
except Exception:
|
||||
raise HTTPException(503, "sdk_context_unavailable") from None
|
||||
memory_mode = getattr(request.app.state, "memory_mode", None)
|
||||
lite = callable(memory_mode) and memory_mode() == "lite"
|
||||
|
||||
async def raw_lane() -> tuple[SdkRawContextItem, ...]:
|
||||
if "raw" not in value.kinds or not authority.raw_allowed:
|
||||
return ()
|
||||
sources = raw_lane_sources(authority.raw_sources, value.include_sources)
|
||||
if sources is None:
|
||||
return ()
|
||||
raw = await asyncio.to_thread(
|
||||
client.search,
|
||||
value.query,
|
||||
user_ids=[value.user_id],
|
||||
agent_ids=[],
|
||||
include_sources=list(authority.raw_sources or ()),
|
||||
limit=value.limit,
|
||||
min_score=0.0,
|
||||
schema_min_score=0.0,
|
||||
schema_limit=0,
|
||||
intention_limit=0,
|
||||
scene="chat",
|
||||
reader=HY_MEMORY_READER,
|
||||
layers=["l1_raw"],
|
||||
**{
|
||||
"user_ids": [value.user_id],
|
||||
"agent_ids": [],
|
||||
"include_sources": sources,
|
||||
"limit": value.limit,
|
||||
"min_score": 0.0,
|
||||
"schema_min_score": 0.0,
|
||||
"schema_limit": 0,
|
||||
"intention_limit": 0,
|
||||
"scene": "chat",
|
||||
"reader": HY_MEMORY_READER,
|
||||
"layers": ["l1_raw"],
|
||||
**raw_lane_overrides(value),
|
||||
},
|
||||
)
|
||||
hits = tuple(
|
||||
hit for hit in parse_sdk_raw_search(raw) if value.passes_floor("raw", hit.score)
|
||||
)
|
||||
hits = parse_sdk_raw_search(raw)
|
||||
return tuple(
|
||||
SdkRawContextItem(
|
||||
raw_id=hit.raw_id,
|
||||
@@ -104,25 +157,41 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
)
|
||||
|
||||
async def memory_lane() -> tuple[SdkMemoryContextItem, ...]:
|
||||
if "memory" not in value.kinds or not authority.memory_allowed:
|
||||
# lite extracts nothing: its search answers raw turns, which the
|
||||
# raw lane already reads under the raw authority.
|
||||
if "memory" not in value.kinds or not authority.memory_allowed or lite:
|
||||
return ()
|
||||
overrides = value.search_overrides()
|
||||
raw = await asyncio.to_thread(
|
||||
client.search,
|
||||
value.query,
|
||||
user_ids=[value.user_id],
|
||||
agent_ids=[],
|
||||
limit=value.limit,
|
||||
min_score=sdk_relevance_floor(),
|
||||
schema_min_score=sdk_relevance_floor(),
|
||||
schema_limit=value.schema_limit,
|
||||
intention_limit=value.intention_limit,
|
||||
scene="normal",
|
||||
reader=HY_MEMORY_READER,
|
||||
**{
|
||||
"user_ids": [value.user_id],
|
||||
"agent_ids": [],
|
||||
"limit": value.limit,
|
||||
"min_score": sdk_relevance_floor(),
|
||||
"schema_min_score": sdk_relevance_floor(),
|
||||
"schema_limit": value.schema_limit,
|
||||
"intention_limit": value.intention_limit,
|
||||
"scene": "normal",
|
||||
"reader": HY_MEMORY_READER,
|
||||
**overrides,
|
||||
},
|
||||
)
|
||||
hits = (
|
||||
tuple(
|
||||
hit
|
||||
for hit in parse_sdk_lane_search(
|
||||
raw, error_code="hy_memory_search_shape_unknown"
|
||||
)
|
||||
if value.passes_floor(cast(SearchLane, hit.lane), hit.score)
|
||||
)
|
||||
if overrides
|
||||
else parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
)
|
||||
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
items: list[SdkMemoryContextItem] = []
|
||||
for hit in hits:
|
||||
record = await asyncio.to_thread(client.get, hit.memory_id)
|
||||
record = await asyncio.to_thread(client.get, hit.memory_id, **hit_namespace(hit))
|
||||
memory = None if record is None else HySdkMemoryRecord.from_sdk(record)
|
||||
# `get` takes no user: a hit is kept only if it is the caller's.
|
||||
if (
|
||||
|
||||
@@ -71,6 +71,16 @@ def graph_store_enabled() -> bool:
|
||||
return raw != "none"
|
||||
|
||||
|
||||
def effective_memory_mode(configured: str = "pro") -> str:
|
||||
"""The SDK mode clients are built with: the setting, unless dev forces ultra.
|
||||
|
||||
`ONE_MEMORY_GRAPH_STORE` (any value but empty or `none`) predates the
|
||||
setting and keeps working for development: it forces ultra whatever the
|
||||
configured mode. Without it the configured mode (default pro) applies.
|
||||
"""
|
||||
return "ultra" if graph_store_enabled() else configured
|
||||
|
||||
|
||||
def inspect_graph_width_mismatch(config: Any) -> GraphWidthMismatch | None:
|
||||
"""Return a mismatch without renaming or deleting anything."""
|
||||
db_path = Path(config.graph_store.db_path)
|
||||
@@ -83,11 +93,12 @@ def inspect_profile_graph_width(
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
mode: str | None = None,
|
||||
) -> GraphWidthMismatch | None:
|
||||
"""Inspect the graph path and width selected by the active profile."""
|
||||
del api_key
|
||||
return _inspect_graph_width(
|
||||
_profile_graph_path(data_dir),
|
||||
_profile_graph_path(data_dir, mode),
|
||||
_profile_embedding_width(profile),
|
||||
)
|
||||
|
||||
@@ -98,11 +109,12 @@ def decide_profile_graph_width(
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
decision: GraphWidthDecision,
|
||||
mode: str | None = None,
|
||||
) -> GraphWidthOutcome:
|
||||
"""Apply the user's decision to the active profile's graph."""
|
||||
del api_key
|
||||
config = _GraphAdminConfig(
|
||||
graph_store=_GraphStoreConfig(str(_profile_graph_path(data_dir))),
|
||||
graph_store=_GraphStoreConfig(str(_profile_graph_path(data_dir, mode))),
|
||||
embedder=_EmbeddingWidthConfig(_profile_embedding_width(profile)),
|
||||
)
|
||||
return apply_graph_width_decision(
|
||||
@@ -240,8 +252,9 @@ def _inspect_graph_width(
|
||||
)
|
||||
|
||||
|
||||
def _profile_graph_path(data_dir: Path) -> Path:
|
||||
name = "graph" if graph_store_enabled() else "graph-disabled"
|
||||
def _profile_graph_path(data_dir: Path, mode: str | None = None) -> Path:
|
||||
ultra = mode == "ultra" if mode is not None else graph_store_enabled()
|
||||
name = "graph" if ultra else "graph-disabled"
|
||||
return data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / name
|
||||
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
import threading
|
||||
from collections.abc import Callable, Coroutine
|
||||
from collections.abc import Callable, Coroutine, Iterable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, ClassVar
|
||||
@@ -31,12 +31,18 @@ from one_memory.embedding_assets import (
|
||||
)
|
||||
from one_memory.hy_graph_admin import (
|
||||
GraphWidthMismatchError,
|
||||
graph_store_enabled,
|
||||
effective_memory_mode,
|
||||
inspect_graph_width_mismatch,
|
||||
)
|
||||
from one_memory.hy_sdk_types import HySdkRuntimeError
|
||||
from one_memory.local_embedding import LocalEmbedder
|
||||
from one_memory.log_file import confine_hy_sdk_logging
|
||||
from one_memory.memory_features import (
|
||||
FEATURES,
|
||||
FEATURES_BY_NAME,
|
||||
FeatureOverrides,
|
||||
feature_overrides,
|
||||
)
|
||||
from one_memory.model_profiles import (
|
||||
ModelProfileSnapshot,
|
||||
ProviderKind,
|
||||
@@ -170,11 +176,17 @@ class SdkClientHandle:
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
mode: str | None = None,
|
||||
features: FeatureOverrides = (),
|
||||
) -> None:
|
||||
self._profile = profile
|
||||
self._api_key = api_key
|
||||
self._data_dir = data_dir
|
||||
self._http_transport_factory = http_transport_factory
|
||||
#: The SDK mode this handle's client is built with, fixed for its life.
|
||||
self.mode = mode or effective_memory_mode()
|
||||
#: The SDK features set on its config (`memory_features`), fixed likewise.
|
||||
self.features = features
|
||||
self._lease: ClientLease | None = None
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
@@ -189,6 +201,8 @@ class SdkClientHandle:
|
||||
api_key=self._api_key,
|
||||
data_dir=self._data_dir,
|
||||
http_transport_factory=self._http_transport_factory,
|
||||
mode=self.mode,
|
||||
features=self.features,
|
||||
)
|
||||
return self._lease.client
|
||||
|
||||
@@ -255,13 +269,48 @@ def sdk_relevance_floor() -> float:
|
||||
return min(1.0, max(0.0, value))
|
||||
|
||||
|
||||
def sdk_feature_defaults() -> dict[str, bool]:
|
||||
"""Each `memory_features` feature's SDK default, as a fresh config has it.
|
||||
|
||||
That is the SDK's code default unless the process environment sets the
|
||||
SDK's own variable (`FeatureSpec.env`), exactly what an unset feature
|
||||
leaves on a client.
|
||||
"""
|
||||
config = MemoryConfig()
|
||||
return {
|
||||
spec.name: bool(getattr(getattr(config, spec.section), spec.field)) for spec in FEATURES
|
||||
}
|
||||
|
||||
|
||||
def apply_sdk_features(
|
||||
config: MemoryConfig, features: Mapping[str, bool] | Iterable[tuple[str, bool]]
|
||||
) -> MemoryConfig:
|
||||
"""Set each feature a person chose on its `MemoryConfig` field.
|
||||
|
||||
Every exposed feature is a config field the SDK reads from the config at
|
||||
call time, never from `os.environ`, so this is per client and
|
||||
deterministic. Nothing here touches `vector_store`, so the collection
|
||||
identity never depends on a feature.
|
||||
"""
|
||||
values = dict(features.items() if isinstance(features, Mapping) else features)
|
||||
for name, value in feature_overrides(values):
|
||||
spec = FEATURES_BY_NAME[name]
|
||||
setattr(getattr(config, spec.section), spec.field, value)
|
||||
return config
|
||||
|
||||
|
||||
def build_sdk_config(
|
||||
profile: ModelProfileSnapshot,
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
features: Mapping[str, bool] | Iterable[tuple[str, bool]] = (),
|
||||
) -> MemoryConfig:
|
||||
"""Map a model snapshot to SDK models, endpoints, backends, and local paths."""
|
||||
"""Map a model snapshot to SDK models, endpoints, backends, and local paths.
|
||||
|
||||
`features` (`memory_features`) are set on top; with none the config is
|
||||
exactly what it was before the setting existed.
|
||||
"""
|
||||
generation_endpoint, generation_key = _generation_access(profile, api_key)
|
||||
embedding_provider = profile.embedding_provider_kind or profile.provider_kind
|
||||
embedding_service_is_local = embedding_provider is ProviderKind.LOCAL_ONNX
|
||||
@@ -307,7 +356,7 @@ def build_sdk_config(
|
||||
if thinking_off:
|
||||
llm["extra_body"] = thinking_off
|
||||
|
||||
return MemoryConfig.from_dict(
|
||||
config = MemoryConfig.from_dict(
|
||||
{
|
||||
"llm": llm,
|
||||
"embedder": {
|
||||
@@ -344,6 +393,7 @@ def build_sdk_config(
|
||||
"coding": {"db_path": str(root / "coding.sqlite3")},
|
||||
}
|
||||
)
|
||||
return apply_sdk_features(config, features)
|
||||
|
||||
|
||||
def acquire_client(
|
||||
@@ -352,9 +402,15 @@ def acquire_client(
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
mode: str | None = None,
|
||||
features: FeatureOverrides = (),
|
||||
) -> ClientLease:
|
||||
"""Construct a public SDK client and return its explicit runtime lease.
|
||||
|
||||
`mode` is the SDK pipeline mode (lite / pro / ultra); None keeps the
|
||||
pre-setting default (ultra when `ONE_MEMORY_GRAPH_STORE` is set, else pro).
|
||||
`features` are the `memory_features` set on its config; none keeps every
|
||||
SDK default.
|
||||
Waits while `hold_stores_offline` holds this home's stores.
|
||||
"""
|
||||
key = data_dir.resolve()
|
||||
@@ -368,6 +424,8 @@ def acquire_client(
|
||||
api_key=api_key,
|
||||
data_dir=data_dir,
|
||||
http_transport_factory=http_transport_factory,
|
||||
mode=mode or effective_memory_mode(),
|
||||
features=features,
|
||||
)
|
||||
finally:
|
||||
with _STORE_GATE:
|
||||
@@ -406,8 +464,10 @@ def _acquire_client(
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
http_transport_factory: Callable[[], Any] | None,
|
||||
mode: str = "pro",
|
||||
features: FeatureOverrides = (),
|
||||
) -> ClientLease:
|
||||
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir)
|
||||
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir, features=features)
|
||||
config.http_transport_factory = http_transport_factory
|
||||
mismatch = inspect_graph_width_mismatch(config)
|
||||
if mismatch is not None:
|
||||
@@ -423,7 +483,7 @@ def _acquire_client(
|
||||
with constructing:
|
||||
client = HyMemoryClient(
|
||||
config=config,
|
||||
mode="ultra" if graph_store_enabled() else "pro",
|
||||
mode=mode,
|
||||
runtime=runtime,
|
||||
embedding_service=embedding_service,
|
||||
)
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal, NamedTuple, cast
|
||||
@@ -391,6 +392,8 @@ class HySdkRawRecord(BaseModel):
|
||||
memory_at: int | None = None
|
||||
gmt_created: int | None = None
|
||||
custom: dict[str, object] = Field(default_factory=dict)
|
||||
#: active, or shadow once pro extracted it; None when the SDK gave none.
|
||||
status: str | None = None
|
||||
|
||||
@classmethod
|
||||
def from_sdk(cls, raw: object) -> HySdkRawRecord:
|
||||
@@ -417,11 +420,34 @@ class HySdkRawRecord(BaseModel):
|
||||
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
|
||||
),
|
||||
custom=dict(custom),
|
||||
status=str(raw["status"]) if isinstance(raw.get("status"), str) else None,
|
||||
)
|
||||
except (KeyError, TypeError, ValueError) as error:
|
||||
raise HySdkReadError("hy_memory_raw_shape_unknown") from error
|
||||
|
||||
|
||||
_RAW_TURN_LINE = re.compile(r"^\[(user|assistant|tool|system)\]:\s?(.*)$")
|
||||
|
||||
|
||||
def raw_turn_messages(content: str) -> tuple[tuple[str, str], ...]:
|
||||
"""The `(role, text)` messages of one SDK raw turn.
|
||||
|
||||
The SDK stores a turn as `[user]: …\\n[assistant]: …`; a line without a
|
||||
role prefix continues the message above it. Content that names no role
|
||||
at all is one message of unknown role (`""`).
|
||||
"""
|
||||
messages: list[list[str]] = []
|
||||
for line in content.splitlines():
|
||||
match = _RAW_TURN_LINE.match(line)
|
||||
if match is not None:
|
||||
messages.append([match.group(1), match.group(2)])
|
||||
elif messages:
|
||||
messages[-1][1] += "\n" + line
|
||||
else:
|
||||
messages.append(["", line])
|
||||
return tuple((role, text) for role, text in messages)
|
||||
|
||||
|
||||
class HySdkRawHit(HySdkRawRecord):
|
||||
score: float = Field(ge=0, le=1)
|
||||
|
||||
@@ -516,6 +542,96 @@ class HySdkUserProfile(BaseModel):
|
||||
)
|
||||
|
||||
|
||||
class HySdkProfileVersion(BaseModel):
|
||||
"""One version on an L0 basic-info chain."""
|
||||
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
memory_id: str
|
||||
content: str
|
||||
#: Conversation time as the SDK renders it on a chain (epoch or text).
|
||||
memory_at: int | str | None = None
|
||||
gmt_created: int | None = None
|
||||
#: The chain head: the version the profile shows now.
|
||||
current: bool = False
|
||||
|
||||
|
||||
class HySdkProfileChain(BaseModel):
|
||||
"""One L0 basic-info head and its versions, latest first."""
|
||||
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
memory_id: str
|
||||
content: str
|
||||
status: str
|
||||
memory_at: int | None = None
|
||||
gmt_created: int | None = None
|
||||
#: The structured fields this head holds (`basic_info_kv`).
|
||||
basic_info: dict[str, object] = Field(default_factory=dict)
|
||||
versions: tuple[HySdkProfileVersion, ...]
|
||||
|
||||
|
||||
class HySdkProfileVersions(BaseModel):
|
||||
"""`get_user_basic_info`: the long-term profile's L0 version chains."""
|
||||
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
request_id: str
|
||||
chains: tuple[HySdkProfileChain, ...]
|
||||
elapsed_ms: float = Field(ge=0)
|
||||
|
||||
@classmethod
|
||||
def from_sdk(cls, raw: object) -> HySdkProfileVersions:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError("hy_memory_profile_versions_shape_unknown")
|
||||
request_id = raw.get("request_id")
|
||||
rows = raw.get("memories")
|
||||
elapsed_ms = raw.get("elapsed_ms")
|
||||
if (
|
||||
not isinstance(request_id, str)
|
||||
or not isinstance(rows, list)
|
||||
or not isinstance(elapsed_ms, int | float)
|
||||
or isinstance(elapsed_ms, bool)
|
||||
):
|
||||
raise HySdkReadError("hy_memory_profile_versions_shape_unknown")
|
||||
chains: list[HySdkProfileChain] = []
|
||||
for row in rows:
|
||||
if not isinstance(row, Mapping) or not isinstance(row.get("memory_id"), str):
|
||||
raise HySdkReadError("hy_memory_profile_versions_shape_unknown")
|
||||
head_id = str(row["memory_id"])
|
||||
chain = row.get("evolution_chain")
|
||||
entries = chain if isinstance(chain, list) and chain else [{**row, "node_id": head_id}]
|
||||
versions = tuple(
|
||||
HySdkProfileVersion(
|
||||
memory_id=str(entry.get("node_id") or entry.get("memory_id")),
|
||||
content=str(entry.get("content") or ""),
|
||||
memory_at=(
|
||||
entry.get("memory_at")
|
||||
if isinstance(entry.get("memory_at"), int | str)
|
||||
and not isinstance(entry.get("memory_at"), bool)
|
||||
else None
|
||||
),
|
||||
gmt_created=_epoch(entry.get("gmt_created")),
|
||||
current=str(entry.get("node_id") or entry.get("memory_id")) == head_id,
|
||||
)
|
||||
for entry in entries
|
||||
if isinstance(entry, Mapping) and (entry.get("node_id") or entry.get("memory_id"))
|
||||
)
|
||||
basic_info = row.get("basic_info_kv")
|
||||
chains.append(
|
||||
HySdkProfileChain(
|
||||
memory_id=head_id,
|
||||
content=str(row.get("content") or ""),
|
||||
status=str(row.get("status") or "active"),
|
||||
memory_at=_epoch(row.get("memory_at")),
|
||||
gmt_created=_epoch(row.get("gmt_created")),
|
||||
basic_info=dict(basic_info) if isinstance(basic_info, Mapping) else {},
|
||||
versions=versions,
|
||||
)
|
||||
)
|
||||
return cls(request_id=request_id, chains=tuple(chains), elapsed_ms=float(elapsed_ms))
|
||||
|
||||
|
||||
def parse_sdk_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit, ...]:
|
||||
rows = _reply_rows(raw, error_code)
|
||||
found: list[SemanticMemoryHit] = []
|
||||
@@ -542,6 +658,120 @@ def parse_sdk_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit
|
||||
return tuple(found)
|
||||
|
||||
|
||||
def parse_sdk_lane_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit, ...]:
|
||||
"""Every scene's search reply as hits that name their lane and namespace.
|
||||
|
||||
`parse_sdk_search` reads the chat reply only and forgets which lane a hit
|
||||
came from. A caller may now pick the scene and a floor per lane, so this
|
||||
reads all three reply shapes:
|
||||
|
||||
- chat: `memories` as `{schema, proactive, normal}` (or one list, normal);
|
||||
- working/coding: `facts` and `experiences` of that variant;
|
||||
- all: one fused `memories` list whose rows carry `scene`, `kind` and
|
||||
their own lane's score (`lane_score`). The fused `score` is a rank
|
||||
fusion (1/(60+rank)), not a relevance, so the lane score is reported.
|
||||
|
||||
Highest score first, first occurrence of an id kept.
|
||||
"""
|
||||
if not isinstance(raw, Mapping):
|
||||
raise HySdkReadError(error_code)
|
||||
scene = raw.get("scene")
|
||||
found: list[SemanticMemoryHit] = []
|
||||
if scene == "all":
|
||||
rows = raw.get("memories")
|
||||
if not isinstance(rows, list) or any(not isinstance(row, Mapping) for row in rows):
|
||||
raise HySdkReadError(error_code)
|
||||
for row in rows:
|
||||
hit = _lane_hit(row, scene=str(row.get("scene") or "chat"), score_key="lane_score")
|
||||
if hit is not None:
|
||||
found.append(hit)
|
||||
elif scene in ("productivity", "working", "coding"):
|
||||
variant = raw.get("variant") if raw.get("variant") in {"working", "coding"} else None
|
||||
lane = variant or (scene if scene in {"working", "coding"} else "working")
|
||||
for key, kind in (("facts", "fact"), ("experiences", "experience")):
|
||||
rows = raw.get(key) or []
|
||||
if not isinstance(rows, list) or any(not isinstance(row, Mapping) for row in rows):
|
||||
raise HySdkReadError(error_code)
|
||||
for row in rows:
|
||||
hit = _lane_hit({**row, "kind": kind}, scene=lane, score_key="score")
|
||||
if hit is not None:
|
||||
found.append(hit)
|
||||
else:
|
||||
memories = raw.get("memories")
|
||||
if isinstance(memories, Mapping):
|
||||
if set(memories) != {"schema", "proactive", "normal"}:
|
||||
raise HySdkReadError(error_code)
|
||||
channels = tuple(memories.items())
|
||||
elif isinstance(memories, list):
|
||||
channels = (("normal", memories),)
|
||||
else:
|
||||
raise HySdkReadError(error_code)
|
||||
for lane, rows in channels:
|
||||
if not isinstance(rows, list) or any(not isinstance(row, Mapping) for row in rows):
|
||||
raise HySdkReadError(error_code)
|
||||
for row in rows:
|
||||
hit = _lane_hit(row, scene="chat", score_key="score", lane=str(lane))
|
||||
if hit is not None:
|
||||
found.append(hit)
|
||||
ordered: list[SemanticMemoryHit] = []
|
||||
seen: set[tuple[str, str, str | None]] = set()
|
||||
for hit in sorted(found, key=lambda value: value.score, reverse=True):
|
||||
identity = (hit.memory_id, hit.kind, hit.variant)
|
||||
if identity in seen:
|
||||
continue
|
||||
seen.add(identity)
|
||||
ordered.append(hit)
|
||||
return tuple(ordered)
|
||||
|
||||
|
||||
def _lane_hit(
|
||||
row: Mapping[str, object],
|
||||
*,
|
||||
scene: str,
|
||||
score_key: str,
|
||||
lane: str | None = None,
|
||||
) -> SemanticMemoryHit | None:
|
||||
kind = str(row.get("kind")) if row.get("kind") in {"experience", "fact"} else "chat"
|
||||
if kind == "chat":
|
||||
memory_id = row.get("memory_id") or row.get("id")
|
||||
variant = None
|
||||
hit_lane = lane or "normal"
|
||||
else:
|
||||
id_key = "experience_id" if kind == "experience" else "fact_id"
|
||||
memory_id = row.get("memory_id") or row.get(id_key)
|
||||
variant = scene if scene in {"working", "coding"} else None
|
||||
if variant is None:
|
||||
return None
|
||||
hit_lane = variant
|
||||
if not isinstance(memory_id, str) or not memory_id:
|
||||
return None
|
||||
value = row.get(score_key)
|
||||
if not isinstance(value, int | float) or isinstance(value, bool):
|
||||
value = row.get("score")
|
||||
score = (
|
||||
max(0.0, min(1.0, float(value)))
|
||||
if isinstance(value, int | float) and not isinstance(value, bool)
|
||||
else 0.0
|
||||
)
|
||||
if kind != "chat":
|
||||
return SemanticMemoryHit(
|
||||
memory_id=memory_id, score=score, lane=hit_lane, kind=kind, variant=variant
|
||||
)
|
||||
content = row.get("content")
|
||||
member_ids, formatting_complete = _formatted_member_ids(row)
|
||||
return SemanticMemoryHit(
|
||||
memory_id=memory_id,
|
||||
score=score,
|
||||
formatted_content=(
|
||||
content.strip() if isinstance(content, str) and content.strip() else None
|
||||
),
|
||||
member_ids=member_ids,
|
||||
sdk_formatted=True,
|
||||
formatting_complete=formatting_complete,
|
||||
lane=hit_lane,
|
||||
)
|
||||
|
||||
|
||||
class LibraryHit(NamedTuple):
|
||||
"""One keyword match in a library search window, with its namespace."""
|
||||
|
||||
|
||||
@@ -28,7 +28,10 @@ from one_memory.hy_sdk_types import LibraryHit
|
||||
DEFAULT_WINDOW_TTL_SECONDS = 300.0
|
||||
DEFAULT_MAX_WINDOWS = 32
|
||||
|
||||
WindowKey = tuple[str, str | None, tuple[str, ...], str, str]
|
||||
#: (query, source, statuses, scene, user, other filters as sorted name/values pairs)
|
||||
WindowKey = tuple[
|
||||
str, str | None, tuple[str, ...], str, str, tuple[tuple[str, tuple[str, ...]], ...]
|
||||
]
|
||||
|
||||
|
||||
def window_key(
|
||||
@@ -37,8 +40,12 @@ def window_key(
|
||||
statuses: list[str],
|
||||
scene: str = "all",
|
||||
user_id: str = "local",
|
||||
filters: dict[str, list[str]] | None = None,
|
||||
) -> WindowKey:
|
||||
return (query, source, tuple(sorted(set(statuses))), scene, user_id)
|
||||
extra = tuple(
|
||||
sorted((name, tuple(sorted(set(values)))) for name, values in (filters or {}).items())
|
||||
)
|
||||
return (query, source, tuple(sorted(set(statuses))), scene, user_id, extra)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
||||
@@ -0,0 +1,618 @@
|
||||
"""System 2 for ultra: evolve runs, their receipts, the nightly schedule, graph backfill.
|
||||
|
||||
hy-memory 2.0.9's `client.evolve(user_id, agent_id, max_batches)` is ultra
|
||||
only, holds no lock, and has no scheduler: MemHub owns all three.
|
||||
|
||||
- A run is asynchronous: `start` answers a receipt at once and the run goes
|
||||
on in the background (evolve makes model calls across the user's whole
|
||||
library and can take minutes, longer than any HTTP timeout). One run per
|
||||
user at a time: asking again while one runs returns that run. A run holds
|
||||
the write coordinator's per-user exclusivity, so it never overlaps that
|
||||
user's write batch, and loops while the SDK answers `should_continue`, up
|
||||
to `EVOLVE_MAX_ROUNDS`.
|
||||
- The schedule runs every user that has memories once a day at the
|
||||
configured local time, and catches up after the machine slept through it.
|
||||
- Switching to ultra backfills each user's existing facts into the graph
|
||||
once (`build_memory_graph`).
|
||||
|
||||
What is kept on disk is counts and stable codes only, never memory text:
|
||||
`<data_dir>/evolve-state.json`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from collections.abc import Awaitable, Callable, Iterable, Mapping
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from tempfile import NamedTemporaryFile
|
||||
from typing import Any, Literal, Protocol
|
||||
from uuid import uuid4
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.diagnostic_log import diagnostic_event
|
||||
from one_memory.memory_server.config_state import EvolveSchedule
|
||||
from one_memory.memory_server.durable_queue import safe_error_code
|
||||
from one_memory.sdk_write_processor import PRODUCT_AGENT_ID
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Rounds one run makes while the SDK says there is more to do.
|
||||
EVOLVE_MAX_ROUNDS = 20
|
||||
#: How often the schedule looks at the clock; also bounds catch-up after wake.
|
||||
SCHEDULE_CHECK_SECONDS = 300.0
|
||||
#: Receipts kept in memory for `GET /api/v2/evolve/{run_id}`.
|
||||
_KEPT_RUNS = 64
|
||||
_STABLE_CODE = re.compile(r"[a-z][a-z0-9_]{0,95}")
|
||||
_COUNTS = (
|
||||
"remaining_clusters",
|
||||
"remaining_actionable_clusters",
|
||||
"remaining_fresh_facts",
|
||||
)
|
||||
|
||||
RunState = Literal["running", "succeeded", "failed"]
|
||||
Trigger = Literal["manual", "scheduled"]
|
||||
|
||||
|
||||
class EvolveModeError(RuntimeError):
|
||||
"""Evolve asked for outside ultra."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__("memory_mode_not_ultra")
|
||||
|
||||
|
||||
class EvolveRunReceipt(BaseModel):
|
||||
"""One evolve run. Counts and codes; `results` only while in memory."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
run_id: str
|
||||
user_id: str
|
||||
agent_id: str
|
||||
max_batches: int | None = None
|
||||
trigger: Trigger = "manual"
|
||||
state: RunState = "running"
|
||||
started_at: datetime
|
||||
finished_at: datetime | None = None
|
||||
rounds: int = 0
|
||||
tasks_processed: int = 0
|
||||
batch_failures: int = 0
|
||||
cycle_status: str | None = None
|
||||
should_continue: bool | None = None
|
||||
continuation_reason: str | None = None
|
||||
remaining_clusters: int | None = None
|
||||
remaining_actionable_clusters: int | None = None
|
||||
remaining_fresh_facts: int | None = None
|
||||
#: True when the round cap stopped a run the SDK would have continued.
|
||||
capped: bool = False
|
||||
elapsed_ms: float = 0.0
|
||||
error_code: str | None = None
|
||||
failure_kind: str | None = None
|
||||
#: The last round's SDK `request_id`, for matching SDK logs.
|
||||
request_id: str | None = None
|
||||
#: The last round's SDK `results` and `cross_domain_sweeper`. Kept only
|
||||
#: in this receipt, never written to disk or diagnostics.
|
||||
results: Any = None
|
||||
cross_domain_sweeper: Any = None
|
||||
|
||||
def summary(self) -> dict[str, object]:
|
||||
"""What is persisted and shown in status: no SDK result bodies."""
|
||||
return self.model_dump(mode="json", exclude={"results", "cross_domain_sweeper"})
|
||||
|
||||
|
||||
class UserEvolveState(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
last_run_at: datetime | None = None
|
||||
last_result: dict[str, object] | None = None
|
||||
last_error: str | None = None
|
||||
|
||||
|
||||
class GraphBuildUser(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
state: Literal["pending", "running", "succeeded", "failed"] = "pending"
|
||||
memories_scanned: int | None = None
|
||||
nodes: int | None = None
|
||||
error_code: str | None = None
|
||||
|
||||
|
||||
class GraphBuildState(BaseModel):
|
||||
"""The one-time backfill of existing facts into the graph after → ultra."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
state: Literal["idle", "running", "succeeded", "failed"] = "idle"
|
||||
started_at: datetime | None = None
|
||||
finished_at: datetime | None = None
|
||||
users: dict[str, GraphBuildUser] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class EvolveStateDocument(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
schema_version: int = Field(default=1, ge=1, le=1)
|
||||
users: dict[str, UserEvolveState] = Field(default_factory=dict)
|
||||
#: When the schedule first ran in ultra: the "last run" of a user that
|
||||
#: never ran, so enabling ultra at noon does not evolve everyone at once.
|
||||
schedule_anchor: datetime | None = None
|
||||
graph_build: GraphBuildState = Field(default_factory=GraphBuildState)
|
||||
|
||||
|
||||
class EvolveStateStore:
|
||||
"""The only writer of `<data_dir>/evolve-state.json`."""
|
||||
|
||||
def __init__(self, data_dir: Path) -> None:
|
||||
self.path = data_dir / "evolve-state.json"
|
||||
self._lock = threading.RLock()
|
||||
self._document = self._read()
|
||||
|
||||
def document(self) -> EvolveStateDocument:
|
||||
with self._lock:
|
||||
return self._document.model_copy(deep=True)
|
||||
|
||||
def user(self, user_id: str) -> UserEvolveState:
|
||||
with self._lock:
|
||||
return self._document.users.get(user_id, UserEvolveState()).model_copy()
|
||||
|
||||
def record_run(self, receipt: EvolveRunReceipt) -> None:
|
||||
with self._lock:
|
||||
users = dict(self._document.users)
|
||||
users[receipt.user_id] = UserEvolveState(
|
||||
last_run_at=receipt.started_at,
|
||||
last_result=receipt.summary(),
|
||||
last_error=receipt.error_code,
|
||||
)
|
||||
self._save(self._document.model_copy(update={"users": users}))
|
||||
|
||||
def anchor(self, now: datetime) -> datetime:
|
||||
"""The schedule anchor, set to `now` the first time it is asked."""
|
||||
with self._lock:
|
||||
if self._document.schedule_anchor is None:
|
||||
self._save(self._document.model_copy(update={"schedule_anchor": now}))
|
||||
assert self._document.schedule_anchor is not None
|
||||
return self._document.schedule_anchor
|
||||
|
||||
def graph_build(self) -> GraphBuildState:
|
||||
with self._lock:
|
||||
return self._document.graph_build.model_copy(deep=True)
|
||||
|
||||
def set_graph_build(self, state: GraphBuildState) -> None:
|
||||
with self._lock:
|
||||
self._save(self._document.model_copy(update={"graph_build": state}))
|
||||
|
||||
def _save(self, document: EvolveStateDocument) -> None:
|
||||
payload = json.dumps(document.model_dump(mode="json"), indent=2, sort_keys=True) + "\n"
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary: Path | None = None
|
||||
try:
|
||||
with NamedTemporaryFile(
|
||||
mode="w",
|
||||
encoding="utf-8",
|
||||
dir=self.path.parent,
|
||||
prefix=".evolve-state.",
|
||||
suffix=".tmp",
|
||||
delete=False,
|
||||
) as stream:
|
||||
temporary = Path(stream.name)
|
||||
stream.write(payload)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.chmod(temporary, 0o600)
|
||||
os.replace(temporary, self.path)
|
||||
except OSError:
|
||||
if temporary is not None:
|
||||
temporary.unlink(missing_ok=True)
|
||||
# Status is best-effort: a run must not fail because its record did.
|
||||
logger.warning("evolve_state_write_failed")
|
||||
self._document = document
|
||||
|
||||
def _read(self) -> EvolveStateDocument:
|
||||
try:
|
||||
return EvolveStateDocument.model_validate_json(self.path.read_text(encoding="utf-8"))
|
||||
except FileNotFoundError:
|
||||
return EvolveStateDocument()
|
||||
except (OSError, ValueError):
|
||||
# A damaged status file only loses history; start a fresh one.
|
||||
logger.warning("evolve_state_unreadable")
|
||||
return EvolveStateDocument()
|
||||
|
||||
|
||||
class EvolveCoordinator(Protocol):
|
||||
"""The write coordinator surface evolve and the graph backfill use."""
|
||||
|
||||
def exclusive_for_evolve(self, user_id: str) -> Any: ...
|
||||
|
||||
def use_client(self) -> Any: ...
|
||||
|
||||
|
||||
class EvolveService:
|
||||
"""Start, serialize and record evolve runs."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
coordinator: EvolveCoordinator,
|
||||
state: EvolveStateStore,
|
||||
memory_mode: Callable[[], str],
|
||||
max_rounds: int = EVOLVE_MAX_ROUNDS,
|
||||
now: Callable[[], datetime] = lambda: datetime.now(UTC),
|
||||
) -> None:
|
||||
self.coordinator = coordinator
|
||||
self.state = state
|
||||
self.memory_mode = memory_mode
|
||||
self.max_rounds = max_rounds
|
||||
self._now = now
|
||||
self._runs: dict[str, EvolveRunReceipt] = {}
|
||||
self._tasks: dict[str, asyncio.Task[None]] = {}
|
||||
self._active: dict[str, str] = {}
|
||||
|
||||
def start(
|
||||
self,
|
||||
user_id: str,
|
||||
*,
|
||||
agent_id: str | None = None,
|
||||
max_batches: int | None = None,
|
||||
trigger: Trigger = "manual",
|
||||
) -> EvolveRunReceipt:
|
||||
"""Start a run, or return the one already running for this user."""
|
||||
if self.memory_mode() != "ultra":
|
||||
raise EvolveModeError()
|
||||
running = self._active.get(user_id)
|
||||
if running is not None:
|
||||
return self._runs[running]
|
||||
receipt = EvolveRunReceipt(
|
||||
run_id=str(uuid4()),
|
||||
user_id=user_id,
|
||||
agent_id=agent_id or PRODUCT_AGENT_ID,
|
||||
max_batches=max_batches,
|
||||
trigger=trigger,
|
||||
started_at=self._now(),
|
||||
)
|
||||
self._remember(receipt)
|
||||
self._active[user_id] = receipt.run_id
|
||||
self._tasks[receipt.run_id] = asyncio.create_task(
|
||||
self._run(receipt.run_id), name="one-memory-evolve"
|
||||
)
|
||||
return receipt
|
||||
|
||||
def get(self, run_id: str) -> EvolveRunReceipt | None:
|
||||
return self._runs.get(run_id)
|
||||
|
||||
def running(self) -> tuple[EvolveRunReceipt, ...]:
|
||||
return tuple(self._runs[run_id] for run_id in self._active.values())
|
||||
|
||||
async def wait(self, run_id: str) -> EvolveRunReceipt | None:
|
||||
task = self._tasks.get(run_id)
|
||||
if task is not None:
|
||||
await asyncio.shield(task)
|
||||
return self._runs.get(run_id)
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
tasks = tuple(self._tasks.values())
|
||||
for task in tasks:
|
||||
task.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
self._tasks.clear()
|
||||
|
||||
def _remember(self, receipt: EvolveRunReceipt) -> None:
|
||||
self._runs[receipt.run_id] = receipt
|
||||
while len(self._runs) > _KEPT_RUNS:
|
||||
oldest = next(iter(self._runs))
|
||||
if oldest in self._active.values():
|
||||
break
|
||||
del self._runs[oldest]
|
||||
|
||||
async def _run(self, run_id: str) -> None:
|
||||
receipt = self._runs[run_id]
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="memory.evolve",
|
||||
event="memory_evolve.started",
|
||||
outcome="started",
|
||||
details={"trigger": receipt.trigger, "max_batches": receipt.max_batches},
|
||||
)
|
||||
try:
|
||||
async with (
|
||||
self.coordinator.exclusive_for_evolve(receipt.user_id),
|
||||
self.coordinator.use_client() as client,
|
||||
):
|
||||
while receipt.rounds < self.max_rounds:
|
||||
result = await asyncio.to_thread(
|
||||
client.evolve,
|
||||
receipt.user_id,
|
||||
agent_id=receipt.agent_id,
|
||||
max_batches=receipt.max_batches,
|
||||
)
|
||||
receipt = _folded(receipt, result)
|
||||
self._runs[run_id] = receipt
|
||||
if receipt.state == "failed" or not receipt.should_continue:
|
||||
break
|
||||
else:
|
||||
receipt = receipt.model_copy(update={"capped": True})
|
||||
if receipt.state == "running":
|
||||
receipt = receipt.model_copy(update={"state": "succeeded"})
|
||||
except asyncio.CancelledError:
|
||||
receipt = receipt.model_copy(
|
||||
update={"state": "failed", "error_code": "memory_evolve_cancelled"}
|
||||
)
|
||||
raise
|
||||
except Exception as error: # noqa: BLE001 - recorded as a stable code
|
||||
code = safe_error_code(error)
|
||||
receipt = receipt.model_copy(
|
||||
update={
|
||||
"state": "failed",
|
||||
"error_code": _stable(code, "hy_memory_evolve_failed"),
|
||||
"failure_kind": type(error).__name__,
|
||||
}
|
||||
)
|
||||
finally:
|
||||
receipt = receipt.model_copy(update={"finished_at": self._now()})
|
||||
self._runs[run_id] = receipt
|
||||
self._active.pop(receipt.user_id, None)
|
||||
self._tasks.pop(run_id, None)
|
||||
await asyncio.to_thread(self.state.record_run, receipt)
|
||||
diagnostic_event(
|
||||
level="info" if receipt.state == "succeeded" else "warning",
|
||||
component="memory.evolve",
|
||||
event="memory_evolve.finished",
|
||||
outcome="succeeded" if receipt.state == "succeeded" else "failed",
|
||||
details={
|
||||
"trigger": receipt.trigger,
|
||||
"rounds": receipt.rounds,
|
||||
"tasks_processed": receipt.tasks_processed,
|
||||
"batch_failures": receipt.batch_failures,
|
||||
"capped": receipt.capped,
|
||||
"elapsed_ms": round(receipt.elapsed_ms, 1),
|
||||
"error_code": receipt.error_code,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _folded(receipt: EvolveRunReceipt, result: object) -> EvolveRunReceipt:
|
||||
"""One more evolve round's result added to the run."""
|
||||
if not isinstance(result, Mapping):
|
||||
return receipt.model_copy(
|
||||
update={"state": "failed", "error_code": "hy_memory_evolve_shape_unknown"}
|
||||
)
|
||||
failures = result.get("batch_failures")
|
||||
update: dict[str, object] = {
|
||||
"rounds": receipt.rounds + 1,
|
||||
"tasks_processed": receipt.tasks_processed + _int(result.get("tasks_processed")),
|
||||
"batch_failures": receipt.batch_failures
|
||||
+ (len(failures) if isinstance(failures, list | tuple) else _int(failures)),
|
||||
"elapsed_ms": receipt.elapsed_ms + _float(result.get("elapsed_ms")),
|
||||
"should_continue": result.get("should_continue") is True,
|
||||
"cycle_status": _text(result.get("cycle_status")),
|
||||
"continuation_reason": _text(result.get("continuation_reason")),
|
||||
"request_id": _text(result.get("request_id")),
|
||||
"results": result.get("results"),
|
||||
"cross_domain_sweeper": result.get("cross_domain_sweeper"),
|
||||
}
|
||||
for name in _COUNTS:
|
||||
value = result.get(name)
|
||||
update[name] = value if isinstance(value, int) and not isinstance(value, bool) else None
|
||||
if result.get("success") is not True:
|
||||
code = result.get("error_code")
|
||||
kind = result.get("failure_kind")
|
||||
update.update(
|
||||
state="failed",
|
||||
should_continue=False,
|
||||
error_code=(
|
||||
code
|
||||
if isinstance(code, str) and _STABLE_CODE.fullmatch(code)
|
||||
else "hy_memory_evolve_rejected"
|
||||
),
|
||||
failure_kind=kind if isinstance(kind, str) and _STABLE_CODE.fullmatch(kind) else None,
|
||||
)
|
||||
return receipt.model_copy(update=update)
|
||||
|
||||
|
||||
def _stable(code: str, fallback: str) -> str:
|
||||
return code if _STABLE_CODE.fullmatch(code) else fallback
|
||||
|
||||
|
||||
def _int(value: object) -> int:
|
||||
return value if isinstance(value, int) and not isinstance(value, bool) else 0
|
||||
|
||||
|
||||
def _float(value: object) -> float:
|
||||
if isinstance(value, int | float) and not isinstance(value, bool):
|
||||
return float(value)
|
||||
return 0.0
|
||||
|
||||
|
||||
def _text(value: object) -> str | None:
|
||||
return value[:128] if isinstance(value, str) else None
|
||||
|
||||
|
||||
def evolve_due(
|
||||
now: datetime,
|
||||
schedule: EvolveSchedule,
|
||||
last_run_at: datetime | None,
|
||||
anchor: datetime,
|
||||
) -> bool:
|
||||
"""Whether a user's nightly evolve is due at `now` (aware, local time).
|
||||
|
||||
Due once the day's slot (`schedule.time`, local) has passed and the last
|
||||
run (or, for a user who never ran, the anchor) is older than that slot.
|
||||
A last run more than 24 hours old is due at once as well: a machine asleep
|
||||
at the slot catches up when it wakes, even before the next day's slot.
|
||||
"""
|
||||
if not schedule.enabled:
|
||||
return False
|
||||
local = now.astimezone()
|
||||
hour, minute = (int(part) for part in schedule.time.split(":"))
|
||||
slot = local.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||
reference = last_run_at or anchor
|
||||
if local >= slot and reference < slot:
|
||||
return True
|
||||
return last_run_at is not None and local - last_run_at > timedelta(hours=24)
|
||||
|
||||
|
||||
class EvolveScheduler:
|
||||
"""Run evolve nightly for every user that has memories (ultra only)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
service: EvolveService,
|
||||
schedule: Callable[[], EvolveSchedule],
|
||||
users_with_memories: Callable[[], Awaitable[Iterable[str]]],
|
||||
now: Callable[[], datetime] = lambda: datetime.now().astimezone(),
|
||||
check_seconds: float = SCHEDULE_CHECK_SECONDS,
|
||||
) -> None:
|
||||
self.service = service
|
||||
self.schedule = schedule
|
||||
self.users_with_memories = users_with_memories
|
||||
self._now = now
|
||||
self.check_seconds = check_seconds
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
if self._task is None or self._task.done():
|
||||
self._task = asyncio.create_task(self._loop(), name="one-memory-evolve-schedule")
|
||||
|
||||
async def tick(self) -> tuple[str, ...]:
|
||||
"""Run every due user once, one after another; the users run."""
|
||||
schedule = self.schedule()
|
||||
if not schedule.enabled or self.service.memory_mode() != "ultra":
|
||||
return ()
|
||||
now = self._now()
|
||||
anchor = await asyncio.to_thread(self.service.state.anchor, now)
|
||||
ran: list[str] = []
|
||||
for user_id in await self.users_with_memories():
|
||||
last = self.service.state.user(user_id).last_run_at
|
||||
if not evolve_due(now, schedule, last, anchor):
|
||||
continue
|
||||
try:
|
||||
receipt = self.service.start(user_id, trigger="scheduled")
|
||||
except EvolveModeError:
|
||||
break
|
||||
await self.service.wait(receipt.run_id)
|
||||
ran.append(user_id)
|
||||
if ran:
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="memory.evolve",
|
||||
event="memory_evolve.schedule_ran",
|
||||
outcome="succeeded",
|
||||
details={"users": len(ran)},
|
||||
)
|
||||
return tuple(ran)
|
||||
|
||||
async def _loop(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
await self.tick()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as error: # noqa: BLE001 - the loop must survive
|
||||
logger.warning("evolve_schedule_tick_failed error_type=%s", type(error).__name__)
|
||||
await asyncio.sleep(self.check_seconds)
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
if self._task is None:
|
||||
return
|
||||
self._task.cancel()
|
||||
await asyncio.gather(self._task, return_exceptions=True)
|
||||
self._task = None
|
||||
|
||||
|
||||
class MemoryGraphBackfill:
|
||||
"""After a switch to ultra, project each user's existing facts into the graph once."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
coordinator: EvolveCoordinator,
|
||||
state: EvolveStateStore,
|
||||
now: Callable[[], datetime] = lambda: datetime.now(UTC),
|
||||
) -> None:
|
||||
self.coordinator = coordinator
|
||||
self.state = state
|
||||
self._now = now
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
|
||||
@property
|
||||
def running(self) -> bool:
|
||||
return self._task is not None and not self._task.done()
|
||||
|
||||
def start(self, user_ids: Iterable[str]) -> GraphBuildState:
|
||||
if self.running:
|
||||
return self.state.graph_build()
|
||||
users = tuple(dict.fromkeys(user_ids))
|
||||
state = GraphBuildState(
|
||||
state="running",
|
||||
started_at=self._now(),
|
||||
users={user_id: GraphBuildUser() for user_id in users},
|
||||
)
|
||||
self.state.set_graph_build(state)
|
||||
self._task = asyncio.create_task(self._run(users), name="one-memory-graph-backfill")
|
||||
return state
|
||||
|
||||
async def wait(self) -> None:
|
||||
if self._task is not None:
|
||||
await asyncio.shield(self._task)
|
||||
|
||||
async def _run(self, users: tuple[str, ...]) -> None:
|
||||
state = self.state.graph_build()
|
||||
for user_id in users:
|
||||
state.users[user_id] = GraphBuildUser(state="running")
|
||||
await asyncio.to_thread(self.state.set_graph_build, state.model_copy(deep=True))
|
||||
try:
|
||||
async with (
|
||||
self.coordinator.exclusive_for_evolve(user_id),
|
||||
self.coordinator.use_client() as client,
|
||||
):
|
||||
result = await asyncio.to_thread(
|
||||
client.build_memory_graph,
|
||||
user_id,
|
||||
agent_id=None,
|
||||
limit=10000,
|
||||
clear_first=False,
|
||||
)
|
||||
ok = isinstance(result, Mapping) and result.get("success") is True
|
||||
state.users[user_id] = GraphBuildUser(
|
||||
state="succeeded" if ok else "failed",
|
||||
memories_scanned=_int(result.get("memories_scanned"))
|
||||
if isinstance(result, Mapping)
|
||||
else None,
|
||||
nodes=_int(result.get("nodes")) if isinstance(result, Mapping) else None,
|
||||
error_code=None if ok else "hy_memory_graph_build_rejected",
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as error: # noqa: BLE001 - recorded as a stable code
|
||||
code = safe_error_code(error)
|
||||
state.users[user_id] = GraphBuildUser(
|
||||
state="failed",
|
||||
error_code=_stable(code, "hy_memory_graph_build_failed"),
|
||||
)
|
||||
failed = any(user.state == "failed" for user in state.users.values())
|
||||
state.state = "failed" if failed else "succeeded"
|
||||
state.finished_at = self._now()
|
||||
await asyncio.to_thread(self.state.set_graph_build, state)
|
||||
diagnostic_event(
|
||||
level="warning" if failed else "info",
|
||||
component="memory.evolve",
|
||||
event="memory_graph_backfill.finished",
|
||||
outcome="failed" if failed else "succeeded",
|
||||
details={
|
||||
"users": len(state.users),
|
||||
"failed_users": sum(user.state == "failed" for user in state.users.values()),
|
||||
},
|
||||
)
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
if self._task is None:
|
||||
return
|
||||
self._task.cancel()
|
||||
await asyncio.gather(self._task, return_exceptions=True)
|
||||
self._task = None
|
||||
@@ -0,0 +1,262 @@
|
||||
"""SDK retrieval and extraction features a person may turn on or off.
|
||||
|
||||
MemHub hard-codes none of these. Each is one `MemoryConfig` field that the SDK
|
||||
reads from its config object (never from `os.environ` at call time), so
|
||||
`hy_sdk_runtime.build_sdk_config` sets it per client and a client's features
|
||||
are fixed for its life, like its mode. An unset feature keeps the SDK default,
|
||||
so with nothing set the SDK config is byte for byte what it was before this
|
||||
setting existed.
|
||||
|
||||
The setting is one global document in `config.json` (`memory_features`),
|
||||
beside `memory_mode`; management stays global, not per user. A change
|
||||
rebuilds the SDK handles the way a mode change does (#218 use count).
|
||||
|
||||
What is deliberately left out, and why, is in
|
||||
docs/contracts/memory-features-v1.md. In short: flags the SDK reads straight
|
||||
from the process environment (the System 2 schema dedup/compaction lane, the
|
||||
balanced-catalog and specific-supplement lanes, the hybrid reader's
|
||||
`MEMORY_HYBRID_V2_*` knobs) cannot be scoped to one client; flags MemHub never
|
||||
exercises (query rewrite needs `queries=[...]`, session narratives need
|
||||
`finalize_session`); flags that are dangerous or internal (graph write, local
|
||||
cross-encoder download, access write-back); and everything MemHub already
|
||||
controls elsewhere (pipeline log, trace, memory operations, telemetry, mode).
|
||||
|
||||
No SDK import here: only `hy_sdk_runtime` and `hy_sdk_migration` import it.
|
||||
`config_state` imports this module, so it imports nothing from there.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, StrictBool
|
||||
|
||||
MemoryFeatureName = Literal[
|
||||
"graph_rerank",
|
||||
"same_series_timeline",
|
||||
"episode_recall",
|
||||
"session_episodes",
|
||||
"episode_summary",
|
||||
"intention",
|
||||
"few_shot",
|
||||
]
|
||||
#: Read features change search; write features change what extraction stores.
|
||||
FeatureSide = Literal["read", "write"]
|
||||
#: none: no extra call. extra_query: one more store query per search.
|
||||
#: llm_call: extra generation-model calls. prompt_tokens: no extra call, but
|
||||
#: a longer extraction prompt.
|
||||
FeatureCost = Literal["none", "extra_query", "llm_call", "prompt_tokens"]
|
||||
#: Why an enabled feature does nothing right now (`active: false`).
|
||||
FeatureReason = Literal["disabled", "requires_pro", "requires_ultra", "requires_graph_rerank"]
|
||||
#: A client's features as a hashable key: sorted (name, value) pairs, set ones only.
|
||||
type FeatureOverrides = tuple[tuple[str, bool], ...]
|
||||
|
||||
_MODE_RANK: dict[str, int] = {"lite": 0, "pro": 1, "ultra": 2}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FeatureSpec:
|
||||
name: MemoryFeatureName
|
||||
#: `MemoryConfig` section and field the value is written to.
|
||||
section: Literal["graph_store", "recall", "llm"]
|
||||
field: str
|
||||
#: The SDK's own env var for the same field; only its default comes from it.
|
||||
env: str
|
||||
side: FeatureSide
|
||||
#: Lowest mode the feature acts in.
|
||||
requires_mode: Literal["pro", "ultra"]
|
||||
#: Another feature that must be active for this one to act.
|
||||
requires_feature: MemoryFeatureName | None
|
||||
cost: FeatureCost
|
||||
description: str
|
||||
|
||||
@property
|
||||
def sdk_field(self) -> str:
|
||||
return f"{self.section}.{self.field}"
|
||||
|
||||
|
||||
FEATURES: tuple[FeatureSpec, ...] = (
|
||||
FeatureSpec(
|
||||
name="graph_rerank",
|
||||
section="graph_store",
|
||||
field="graph_rerank_enabled",
|
||||
env="MEMORY_GRAPH_RERANK_ENABLED",
|
||||
side="read",
|
||||
requires_mode="ultra",
|
||||
requires_feature=None,
|
||||
cost="extra_query",
|
||||
description=(
|
||||
"Usage A: propagate support over knowledge-graph edges inside the "
|
||||
"recall pool, so memories linked to strong hits rank higher. Needs "
|
||||
"the graph store (ultra). No model call; one graph query per search."
|
||||
),
|
||||
),
|
||||
FeatureSpec(
|
||||
name="same_series_timeline",
|
||||
section="recall",
|
||||
field="same_series_timeline_enabled",
|
||||
env="MEMORY_SAME_SERIES_TIMELINE_ENABLED",
|
||||
side="read",
|
||||
requires_mode="ultra",
|
||||
requires_feature="graph_rerank",
|
||||
cost="none",
|
||||
description=(
|
||||
"Keep memories joined by a same-series graph edge together, in time "
|
||||
"order, in the results. Uses the edges graph rerank reads, so it "
|
||||
"needs graph_rerank (ultra). No model call."
|
||||
),
|
||||
),
|
||||
FeatureSpec(
|
||||
name="episode_recall",
|
||||
section="recall",
|
||||
field="episode_recall_enabled",
|
||||
env="MEMORY_EPISODE_RECALL_ENABLED",
|
||||
side="read",
|
||||
requires_mode="pro",
|
||||
requires_feature=None,
|
||||
cost="extra_query",
|
||||
description=(
|
||||
"Usage B: pull in memories from the same episode as a hit and score "
|
||||
"them on tag-keyword overlap, content overlap and time affinity. No "
|
||||
"graph needed (pro or ultra). No model call; one store query per "
|
||||
"search. Helps only memories that carry an episode_id."
|
||||
),
|
||||
),
|
||||
FeatureSpec(
|
||||
name="session_episodes",
|
||||
section="recall",
|
||||
field="session_episode_enabled",
|
||||
env="MEMORY_SESSION_EPISODE_ENABLED",
|
||||
side="write",
|
||||
requires_mode="pro",
|
||||
requires_feature=None,
|
||||
cost="none",
|
||||
description=(
|
||||
"Episode write side: memories extracted from one conversation share "
|
||||
"one stable episode_id, instead of the default rule (a new episode "
|
||||
"after 30 minutes idle or 50 memories). No model call."
|
||||
),
|
||||
),
|
||||
FeatureSpec(
|
||||
name="episode_summary",
|
||||
section="recall",
|
||||
field="episode_summary_enabled",
|
||||
env="MEMORY_EPISODE_SUMMARY_ENABLED",
|
||||
side="write",
|
||||
requires_mode="pro",
|
||||
requires_feature=None,
|
||||
cost="llm_call",
|
||||
description=(
|
||||
"When an episode closes (the idle-gap rule), summarize it once into "
|
||||
"an episode memory. One model call per closed episode of 3 or more "
|
||||
"memories."
|
||||
),
|
||||
),
|
||||
FeatureSpec(
|
||||
name="intention",
|
||||
section="llm",
|
||||
field="enable_intention",
|
||||
env="MEMORY_ENABLE_INTENTION",
|
||||
side="write",
|
||||
requires_mode="pro",
|
||||
requires_feature=None,
|
||||
cost="none",
|
||||
description=(
|
||||
"Extract the user's stated plans and intentions (L7) along with "
|
||||
"facts. On by default; off drops them from the extraction prompt. "
|
||||
"No extra call."
|
||||
),
|
||||
),
|
||||
FeatureSpec(
|
||||
name="few_shot",
|
||||
section="llm",
|
||||
field="few_shot_enabled",
|
||||
env="MEMORY_FEW_SHOT_ENABLED",
|
||||
side="write",
|
||||
requires_mode="pro",
|
||||
requires_feature=None,
|
||||
cost="prompt_tokens",
|
||||
description=(
|
||||
"Append worked examples to the extraction and reconcile prompts. "
|
||||
"Can help a weaker model; no extra call, but every extraction "
|
||||
"sends more input tokens."
|
||||
),
|
||||
),
|
||||
)
|
||||
FEATURE_NAMES: tuple[MemoryFeatureName, ...] = tuple(spec.name for spec in FEATURES)
|
||||
FEATURES_BY_NAME: dict[str, FeatureSpec] = {spec.name: spec for spec in FEATURES}
|
||||
|
||||
|
||||
class MemoryFeatures(BaseModel):
|
||||
"""`config.json` `memory_features`, and the `PUT` body.
|
||||
|
||||
Every key is optional; absent (or null) means the SDK default. In a `PUT`
|
||||
an absent key is left as it is and null resets it. Unknown keys and
|
||||
non-boolean values are refused (422).
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
|
||||
|
||||
graph_rerank: StrictBool | None = None
|
||||
same_series_timeline: StrictBool | None = None
|
||||
episode_recall: StrictBool | None = None
|
||||
session_episodes: StrictBool | None = None
|
||||
episode_summary: StrictBool | None = None
|
||||
intention: StrictBool | None = None
|
||||
few_shot: StrictBool | None = None
|
||||
|
||||
def overrides(self) -> FeatureOverrides:
|
||||
"""The features a person set, as the client key."""
|
||||
return feature_overrides(
|
||||
{name: value for name, value in self.model_dump().items() if value is not None}
|
||||
)
|
||||
|
||||
def merged(self, update: MemoryFeatures) -> MemoryFeatures:
|
||||
"""This document with the keys `update` names: a value sets, null resets."""
|
||||
values = self.model_dump()
|
||||
for name in update.model_fields_set:
|
||||
values[name] = getattr(update, name)
|
||||
return MemoryFeatures(**{key: value for key, value in values.items() if value is not None})
|
||||
|
||||
|
||||
def feature_overrides(values: Mapping[str, bool]) -> FeatureOverrides:
|
||||
unknown = set(values) - set(FEATURES_BY_NAME)
|
||||
if unknown:
|
||||
raise ValueError("unknown_memory_feature")
|
||||
return tuple(sorted((name, bool(value)) for name, value in values.items()))
|
||||
|
||||
|
||||
def effective_features(
|
||||
overrides: FeatureOverrides, defaults: Mapping[str, bool]
|
||||
) -> dict[str, bool]:
|
||||
"""Every feature's value: what a person set, else the SDK default."""
|
||||
values: dict[str, bool] = {spec.name: bool(defaults.get(spec.name, False)) for spec in FEATURES}
|
||||
values.update(dict(overrides))
|
||||
return values
|
||||
|
||||
|
||||
def feature_activity(name: str, effective: Mapping[str, bool], mode: str) -> FeatureReason | None:
|
||||
"""None when the feature acts in this mode; else why it does not."""
|
||||
spec = FEATURES_BY_NAME[name]
|
||||
if not effective.get(name, False):
|
||||
return "disabled"
|
||||
if _MODE_RANK.get(mode, 0) < _MODE_RANK[spec.requires_mode]:
|
||||
return "requires_ultra" if spec.requires_mode == "ultra" else "requires_pro"
|
||||
if spec.requires_feature is not None and (
|
||||
feature_activity(spec.requires_feature, effective, mode) is not None
|
||||
):
|
||||
return "requires_graph_rerank"
|
||||
return None
|
||||
|
||||
|
||||
def non_default_features(
|
||||
overrides: FeatureOverrides, defaults: Mapping[str, bool]
|
||||
) -> dict[str, bool]:
|
||||
"""The features whose value differs from the SDK default, for status."""
|
||||
return {
|
||||
name: value
|
||||
for name, value in effective_features(overrides, defaults).items()
|
||||
if value != bool(defaults.get(name, False))
|
||||
}
|
||||
@@ -13,9 +13,11 @@ import threading
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from tempfile import NamedTemporaryFile
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, StrictBool, field_validator
|
||||
|
||||
from one_memory.memory_features import MemoryFeatures
|
||||
from one_memory.memory_queue import ExtractionWindowPreset
|
||||
|
||||
_SOURCE_ID = re.compile(r"^[a-z0-9][a-z0-9-]{0,127}$")
|
||||
@@ -27,6 +29,31 @@ class SourceConfig(BaseModel):
|
||||
sync_enabled: StrictBool = False
|
||||
|
||||
|
||||
#: The SDK pipeline mode (hy-memory `HyMemoryClient(mode=...)`):
|
||||
#: lite keeps raw conversations only, with no model call; pro extracts
|
||||
#: memories (System 1); ultra adds the knowledge graph and evolve (System 2).
|
||||
MemoryMode = Literal["lite", "pro", "ultra"]
|
||||
MEMORY_MODES: tuple[MemoryMode, ...] = ("lite", "pro", "ultra")
|
||||
_SCHEDULE_TIME = re.compile(r"^(?:[01][0-9]|2[0-3]):[0-5][0-9]$")
|
||||
|
||||
|
||||
class EvolveSchedule(BaseModel):
|
||||
"""When ultra runs evolve for every user: once a day, local time."""
|
||||
|
||||
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
|
||||
|
||||
enabled: StrictBool = True
|
||||
#: `HH:MM`, 24-hour, the machine's local time.
|
||||
time: str = "03:00"
|
||||
|
||||
@field_validator("time")
|
||||
@classmethod
|
||||
def valid_time(cls, value: str) -> str:
|
||||
if _SCHEDULE_TIME.fullmatch(value) is None:
|
||||
raise ValueError("invalid_evolve_schedule_time")
|
||||
return value
|
||||
|
||||
|
||||
class ConfigDocument(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
|
||||
|
||||
@@ -39,6 +66,13 @@ class ConfigDocument(BaseModel):
|
||||
#: needs no re-entry of a key.
|
||||
sedimentation_enabled: StrictBool = True
|
||||
extraction_window_preset: ExtractionWindowPreset = ExtractionWindowPreset.THREE_MONTHS
|
||||
#: The SDK mode every client is built with. `ONE_MEMORY_GRAPH_STORE` (dev)
|
||||
#: still forces ultra over it; see `hy_graph_admin.effective_memory_mode`.
|
||||
memory_mode: MemoryMode = "pro"
|
||||
evolve_schedule: EvolveSchedule = Field(default_factory=EvolveSchedule)
|
||||
#: SDK retrieval and extraction features a person set; an unset one keeps
|
||||
#: the SDK default. See `one_memory.memory_features`.
|
||||
memory_features: MemoryFeatures = Field(default_factory=MemoryFeatures)
|
||||
|
||||
@field_validator("sources")
|
||||
@classmethod
|
||||
@@ -61,6 +95,9 @@ class ConfigFileError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
_WRITTEN_WHEN_SET = frozenset({"memory_mode", "evolve_schedule", "memory_features"})
|
||||
|
||||
|
||||
class ConfigStore:
|
||||
"""The only writer of ``<data_dir>/config.json`` inside Core."""
|
||||
|
||||
@@ -107,6 +144,45 @@ class ConfigStore:
|
||||
self._document = updated
|
||||
return updated
|
||||
|
||||
def memory_mode(self) -> MemoryMode:
|
||||
with self._lock:
|
||||
return self._document.memory_mode
|
||||
|
||||
def set_memory_mode(self, mode: MemoryMode) -> ConfigDocument:
|
||||
if mode not in MEMORY_MODES:
|
||||
raise ValueError("invalid_memory_mode")
|
||||
with self._lock:
|
||||
updated = self._document.model_copy(update={"memory_mode": mode})
|
||||
self._write(updated)
|
||||
self._document = updated
|
||||
return updated
|
||||
|
||||
def evolve_schedule(self) -> EvolveSchedule:
|
||||
with self._lock:
|
||||
return self._document.evolve_schedule
|
||||
|
||||
def set_evolve_schedule(self, schedule: EvolveSchedule) -> ConfigDocument:
|
||||
if not isinstance(schedule, EvolveSchedule):
|
||||
raise ValueError("invalid_evolve_schedule")
|
||||
with self._lock:
|
||||
updated = self._document.model_copy(update={"evolve_schedule": schedule})
|
||||
self._write(updated)
|
||||
self._document = updated
|
||||
return updated
|
||||
|
||||
def memory_features(self) -> MemoryFeatures:
|
||||
with self._lock:
|
||||
return self._document.memory_features
|
||||
|
||||
def set_memory_features(self, features: MemoryFeatures) -> ConfigDocument:
|
||||
if not isinstance(features, MemoryFeatures):
|
||||
raise ValueError("invalid_memory_features")
|
||||
with self._lock:
|
||||
updated = self._document.model_copy(update={"memory_features": features})
|
||||
self._write(updated)
|
||||
self._document = updated
|
||||
return updated
|
||||
|
||||
def migrate_extraction_window_preset(self, preset: ExtractionWindowPreset) -> ConfigDocument:
|
||||
"""Adopt the legacy SQL value only when config.json has no value."""
|
||||
if not isinstance(preset, ExtractionWindowPreset):
|
||||
@@ -152,9 +228,16 @@ class ConfigStore:
|
||||
raise ConfigFileError("config_invalid") from error
|
||||
|
||||
def _write(self, document: ConfigDocument) -> None:
|
||||
# Settings added after 1.0.5 are written only once a person chose
|
||||
# them, so an untouched install's file stays what older builds read.
|
||||
unset = {name for name in _WRITTEN_WHEN_SET if name not in document.model_fields_set}
|
||||
values = document.model_dump(mode="json", exclude=unset)
|
||||
if "memory_features" in values:
|
||||
# Only the features a person set; an absent one is the SDK default.
|
||||
values["memory_features"] = document.memory_features.model_dump(exclude_none=True)
|
||||
payload = (
|
||||
json.dumps(
|
||||
document.model_dump(mode="json"),
|
||||
values,
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
|
||||
@@ -2563,6 +2563,8 @@ async def execute_job(
|
||||
in {
|
||||
"model_content_source_disabled",
|
||||
"model_sedimentation_disabled",
|
||||
# Evolve holds this user; the job runs once it is done.
|
||||
"memory_evolve_in_progress",
|
||||
}
|
||||
and not hy_invoked
|
||||
):
|
||||
|
||||
@@ -14,6 +14,7 @@ from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.hy_sdk_types import HySdkMemoryRecord
|
||||
from one_memory.memory_governance import GovernedMemory, MemoryStatus
|
||||
from one_memory.sdk_search_options import SdkSearchOptions
|
||||
from one_memory.semantic_deadline import (
|
||||
SemanticMemoryHit,
|
||||
SemanticSearch,
|
||||
@@ -50,7 +51,15 @@ class TruncationBudget(StrEnum):
|
||||
TIMEOUT = "timeout_budget"
|
||||
|
||||
|
||||
class RecallRequest(BaseModel):
|
||||
class RecallRequest(SdkSearchOptions):
|
||||
"""`POST /api/v1/recall`.
|
||||
|
||||
The `SdkSearchOptions` fields, `schema_limit` and `intention_limit` pass
|
||||
through to the SDK memory search (and the floor and row filters to the
|
||||
raw fallback). Absent, recall makes the search it always made: the
|
||||
relevance floor, schema and intention lanes sized to `limit`.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
query: str = Field(min_length=1, max_length=2_000)
|
||||
@@ -70,6 +79,17 @@ class RecallRequest(BaseModel):
|
||||
timeout_ms: int = Field(default=2_000, ge=50, le=30_000)
|
||||
#: Whose memories to recall. Omitted, it is `local`, the App's library.
|
||||
user_id: UserId = DEFAULT_USER_ID
|
||||
#: The SDK's schema (L6) and proactive intention (L7) lanes.
|
||||
schema_limit: int | None = Field(default=None, ge=0, le=20)
|
||||
intention_limit: int | None = Field(default=None, ge=0, le=20)
|
||||
|
||||
def search_overrides(self) -> Mapping[str, object]:
|
||||
values = dict(super().search_overrides())
|
||||
if self.schema_limit is not None:
|
||||
values["schema_limit"] = self.schema_limit
|
||||
if self.intention_limit is not None:
|
||||
values["intention_limit"] = self.intention_limit
|
||||
return values
|
||||
|
||||
|
||||
class RecallHit(BaseModel):
|
||||
|
||||
@@ -6,7 +6,7 @@ import asyncio
|
||||
import json
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable
|
||||
from contextlib import aclosing
|
||||
from typing import Any, Literal, Protocol
|
||||
from typing import Any, Literal, Protocol, cast
|
||||
|
||||
import httpx
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
@@ -14,7 +14,7 @@ from pydantic import BaseModel, ConfigDict
|
||||
from one_memory.http_client import create_async_client
|
||||
from one_memory.hy_runtime_config import HY_MEMORY_READER
|
||||
from one_memory.hy_sdk_runtime import sdk_relevance_floor
|
||||
from one_memory.hy_sdk_types import HySdkMemoryRecord, parse_sdk_search
|
||||
from one_memory.hy_sdk_types import HySdkMemoryRecord, parse_sdk_lane_search, parse_sdk_search
|
||||
from one_memory.memory_qa import (
|
||||
MemoryAnswerError,
|
||||
_DeferredSecretStore,
|
||||
@@ -31,6 +31,7 @@ from one_memory.model_profiles import (
|
||||
ProviderKind,
|
||||
attach_disable_thinking,
|
||||
)
|
||||
from one_memory.sdk_search_options import SdkSearchFilters, SearchLane
|
||||
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
|
||||
from one_memory.user_scope import owns
|
||||
|
||||
@@ -96,22 +97,39 @@ class SdkMemoryAnswerer:
|
||||
question: str,
|
||||
*,
|
||||
user_id: str = "local",
|
||||
filters: SdkSearchFilters | None = None,
|
||||
) -> tuple[SdkAskMemory, ...]:
|
||||
"""The user's active memories nearest the question.
|
||||
|
||||
`filters` are a caller's pass-through search options; absent, the
|
||||
search is the one ask always made.
|
||||
"""
|
||||
client = await self.client_handle.client()
|
||||
overrides = filters.search_overrides() if filters is not None else {}
|
||||
raw = await asyncio.to_thread(
|
||||
client.search,
|
||||
question,
|
||||
user_ids=[user_id],
|
||||
agent_ids=[],
|
||||
limit=12,
|
||||
min_score=sdk_relevance_floor(),
|
||||
schema_min_score=sdk_relevance_floor(),
|
||||
schema_limit=12,
|
||||
intention_limit=12,
|
||||
scene="normal",
|
||||
reader=HY_MEMORY_READER,
|
||||
**{
|
||||
"user_ids": [user_id],
|
||||
"agent_ids": [],
|
||||
"limit": 12,
|
||||
"min_score": sdk_relevance_floor(),
|
||||
"schema_min_score": sdk_relevance_floor(),
|
||||
"schema_limit": 12,
|
||||
"intention_limit": 12,
|
||||
"scene": "normal",
|
||||
"reader": HY_MEMORY_READER,
|
||||
**overrides,
|
||||
},
|
||||
)
|
||||
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
if filters is not None and overrides:
|
||||
hits = tuple(
|
||||
hit
|
||||
for hit in parse_sdk_lane_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
if filters.passes_floor(cast(SearchLane, hit.lane), hit.score)
|
||||
)
|
||||
else:
|
||||
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
|
||||
found: list[SdkAskMemory] = []
|
||||
seen: set[str] = set()
|
||||
for hit in hits:
|
||||
@@ -134,8 +152,9 @@ class SdkMemoryAnswerer:
|
||||
question: str,
|
||||
*,
|
||||
user_id: str = "local",
|
||||
filters: SdkSearchFilters | None = None,
|
||||
) -> SdkMemoryAnswer:
|
||||
candidates = await self.retrieve(question, user_id=user_id)
|
||||
candidates = await self.retrieve(question, user_id=user_id, filters=filters)
|
||||
if not candidates:
|
||||
return SdkMemoryAnswer(answer=_NO_GROUNDING, memories=(), mode="empty")
|
||||
profile = await self._active_profile()
|
||||
@@ -166,8 +185,9 @@ class SdkMemoryAnswerer:
|
||||
question: str,
|
||||
*,
|
||||
user_id: str = "local",
|
||||
filters: SdkSearchFilters | None = None,
|
||||
) -> AsyncIterator[dict[str, object]]:
|
||||
candidates = await self.retrieve(question, user_id=user_id)
|
||||
candidates = await self.retrieve(question, user_id=user_id, filters=filters)
|
||||
await self._require_current(candidates)
|
||||
yield {"type": "recall", "memories": candidates}
|
||||
if not candidates:
|
||||
|
||||
@@ -5,8 +5,8 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import AsyncIterator, Callable, Iterable
|
||||
from contextlib import AsyncExitStack, asynccontextmanager, suppress
|
||||
from collections.abc import AsyncIterator, Callable
|
||||
from contextlib import AsyncExitStack, asynccontextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -33,16 +33,6 @@ from one_memory.semantic_deadline import SemanticMemoryHit
|
||||
_STABLE_CODE = re.compile(r"[a-z][a-z0-9_]{0,95}")
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
RAW_RETENTION_DAYS = 3
|
||||
RAW_RETENTION_CHECK_SECONDS = 6 * 60 * 60
|
||||
_RAW_STATUSES = (
|
||||
"active",
|
||||
"superseded",
|
||||
"shadow",
|
||||
"negated",
|
||||
"conflicted",
|
||||
"archived",
|
||||
)
|
||||
|
||||
|
||||
class SdkMemoryRuntime:
|
||||
@@ -56,84 +46,17 @@ class SdkMemoryRuntime:
|
||||
write_queue: DurableWriteQueue,
|
||||
extraction_settings: ExtractionSettingsStore,
|
||||
data_dir: Path,
|
||||
user_ids: Callable[[], Iterable[str]] = lambda: ("local",),
|
||||
memory_mode: Callable[[], str] = lambda: "pro",
|
||||
) -> None:
|
||||
self.coordinator = coordinator
|
||||
#: Every user whose raw layer is kept to the retention window.
|
||||
self.user_ids = user_ids
|
||||
#: The SDK mode, for the graph store's path (`hy_graph_admin`).
|
||||
self.memory_mode = memory_mode
|
||||
self.read_handle = read_handle
|
||||
self.write_queue = write_queue
|
||||
self.extraction_settings = extraction_settings
|
||||
self.data_dir = data_dir
|
||||
self._raw_retention_task: asyncio.Task[None] | None = None
|
||||
|
||||
def start_raw_retention(self) -> None:
|
||||
if self._raw_retention_task is None or self._raw_retention_task.done():
|
||||
self._raw_retention_task = asyncio.create_task(
|
||||
self._raw_retention_loop(),
|
||||
name="one-memory-sdk-raw-retention",
|
||||
)
|
||||
|
||||
async def prune_expired_raw(self, *, now: datetime | None = None) -> int:
|
||||
deleted = 0
|
||||
for user_id in dict.fromkeys(await asyncio.to_thread(lambda: tuple(self.user_ids()))):
|
||||
deleted += await self._prune_expired_raw_for(user_id, now=now)
|
||||
return deleted
|
||||
|
||||
async def _prune_expired_raw_for(self, user_id: str, *, now: datetime | None) -> int:
|
||||
cutoff = (now or datetime.now(UTC)) - timedelta(days=RAW_RETENTION_DAYS)
|
||||
cutoff_timestamp = int(cutoff.timestamp())
|
||||
client = await self.read_handle.client()
|
||||
deleted = 0
|
||||
while True:
|
||||
page = await asyncio.to_thread(
|
||||
client.list_memories,
|
||||
user_id=user_id,
|
||||
layers=["l1_raw"],
|
||||
statuses=list(_RAW_STATUSES),
|
||||
order="asc",
|
||||
limit=500,
|
||||
offset=0,
|
||||
)
|
||||
bucket = page.get("vdb") if isinstance(page, dict) else None
|
||||
rows = bucket.get("memories") if isinstance(bucket, dict) else None
|
||||
if not isinstance(rows, list) or not rows:
|
||||
return deleted
|
||||
expired = [
|
||||
row
|
||||
for row in rows
|
||||
if isinstance(row, dict)
|
||||
and isinstance(row.get("memory_id"), str)
|
||||
and int(row.get("memory_at") or row.get("gmt_created") or 0) < cutoff_timestamp
|
||||
]
|
||||
if not expired:
|
||||
return deleted
|
||||
removed_page = 0
|
||||
for row in expired:
|
||||
result = await asyncio.to_thread(client.delete, row["memory_id"])
|
||||
if isinstance(result, dict) and result.get("success") is True:
|
||||
count = int(result.get("deleted_count") or 0)
|
||||
deleted += count
|
||||
removed_page += count
|
||||
if removed_page == 0:
|
||||
return deleted
|
||||
|
||||
async def _raw_retention_loop(self) -> None:
|
||||
while True:
|
||||
await asyncio.sleep(RAW_RETENTION_CHECK_SECONDS)
|
||||
try:
|
||||
# Its own task, so this pass's use of the read client ends
|
||||
# with it instead of with this loop (see `sdk_read_handle`).
|
||||
removed = await asyncio.create_task(self.prune_expired_raw())
|
||||
if removed:
|
||||
logger.info("sdk_raw_retention_deleted count=%d", removed)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as error:
|
||||
logger.warning(
|
||||
"sdk_raw_retention_failed error_type=%s",
|
||||
type(error).__name__,
|
||||
)
|
||||
# MemHub keeps every raw the SDK stores, in every mode: there is no
|
||||
# retention window (owner decision; the SDK itself never deletes raw).
|
||||
|
||||
async def semantic_recall(
|
||||
self,
|
||||
@@ -233,6 +156,7 @@ class SdkMemoryRuntime:
|
||||
profile,
|
||||
api_key=secret,
|
||||
data_dir=self.data_dir,
|
||||
mode=self.memory_mode(),
|
||||
)
|
||||
|
||||
async def decide_graph_width(
|
||||
@@ -249,6 +173,7 @@ class SdkMemoryRuntime:
|
||||
api_key=secret,
|
||||
data_dir=self.data_dir,
|
||||
decision=decision,
|
||||
mode=self.memory_mode(),
|
||||
)
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -387,12 +312,7 @@ class SdkMemoryRuntime:
|
||||
return None
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
if self._raw_retention_task is None:
|
||||
return
|
||||
self._raw_retention_task.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await self._raw_retention_task
|
||||
self._raw_retention_task = None
|
||||
return None
|
||||
|
||||
|
||||
def _agent_ids(raw: object) -> tuple[str, ...]:
|
||||
|
||||
@@ -10,14 +10,21 @@ is closed once no task still uses it, as `SdkWriteCoordinator` does for writes
|
||||
(#218). A use is the task that asked for the client: callers keep their
|
||||
`client = await handle.client()` shape, and the use ends with that task (one
|
||||
HTTP request, one retention pass).
|
||||
|
||||
The memory mode (lite / pro / ultra) is fixed per SDK client, so a mode change
|
||||
rebuilds and retires the same way, and so do the SDK features
|
||||
(`memory_features`): handles are keyed by (profile, mode, features). Neither
|
||||
mode nor features names a collection; both are separate client parameters.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from one_memory.hy_graph_admin import effective_memory_mode
|
||||
from one_memory.hy_sdk_runtime import (
|
||||
BUILTIN_RETRIEVAL_PROFILE_ID,
|
||||
SdkClientHandle,
|
||||
@@ -25,6 +32,7 @@ from one_memory.hy_sdk_runtime import (
|
||||
retrieval_view,
|
||||
)
|
||||
from one_memory.hy_sdk_types import HySdkRuntimeError
|
||||
from one_memory.memory_features import FeatureOverrides
|
||||
from one_memory.model_profiles import ModelProfileStore, ProviderKind
|
||||
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
|
||||
|
||||
@@ -40,12 +48,16 @@ class ActiveProfileReadHandle(SdkClientHandle):
|
||||
profiles: ModelProfileStore,
|
||||
secrets: SecretStore,
|
||||
data_dir: Path,
|
||||
memory_mode: Callable[[], str] = effective_memory_mode,
|
||||
features: Callable[[], FeatureOverrides] = lambda: (),
|
||||
) -> None:
|
||||
super().__init__(builtin_retrieval_snapshot(), api_key=None, data_dir=data_dir)
|
||||
self.profiles = profiles
|
||||
self.secrets = secrets
|
||||
self.data_dir = data_dir
|
||||
self._handles: dict[tuple[str, int], SdkClientHandle] = {}
|
||||
self.memory_mode = memory_mode
|
||||
self.current_features = features
|
||||
self._handles: dict[tuple[str, int, str, FeatureOverrides], SdkClientHandle] = {}
|
||||
#: Tasks in flight per handle; a retired handle closes at zero.
|
||||
self._uses: dict[SdkClientHandle, int] = {}
|
||||
self._retired: set[SdkClientHandle] = set()
|
||||
@@ -53,7 +65,8 @@ class ActiveProfileReadHandle(SdkClientHandle):
|
||||
self._handle_lock = asyncio.Lock()
|
||||
|
||||
async def client(self) -> Any:
|
||||
identity = await asyncio.to_thread(self.profiles.active_identity) or _BUILTIN
|
||||
profile = await asyncio.to_thread(self.profiles.active_identity) or _BUILTIN
|
||||
identity = (*profile, self.memory_mode(), self.current_features())
|
||||
async with self._handle_lock:
|
||||
handle = self._handles.get(identity)
|
||||
if handle is None:
|
||||
@@ -95,10 +108,16 @@ class ActiveProfileReadHandle(SdkClientHandle):
|
||||
self._closing.add(closing)
|
||||
closing.add_done_callback(self._closing.discard)
|
||||
|
||||
def _build(self, identity: tuple[str, int]) -> SdkClientHandle:
|
||||
def _build(self, key: tuple[str, int, str, FeatureOverrides]) -> SdkClientHandle:
|
||||
profile_id, revision, mode, features = key
|
||||
identity = (profile_id, revision)
|
||||
if identity == _BUILTIN:
|
||||
return SdkClientHandle(
|
||||
builtin_retrieval_snapshot(), api_key=None, data_dir=self.data_dir
|
||||
builtin_retrieval_snapshot(),
|
||||
api_key=None,
|
||||
data_dir=self.data_dir,
|
||||
mode=mode,
|
||||
features=features,
|
||||
)
|
||||
profile = self.profiles.snapshot(*identity)
|
||||
view = retrieval_view(profile)
|
||||
@@ -111,4 +130,6 @@ class ActiveProfileReadHandle(SdkClientHandle):
|
||||
raise HySdkRuntimeError("secure_storage_unavailable") from None
|
||||
if secret is None:
|
||||
raise HySdkRuntimeError("model_secret_unavailable")
|
||||
return SdkClientHandle(view, api_key=secret, data_dir=self.data_dir)
|
||||
return SdkClientHandle(
|
||||
view, api_key=secret, data_dir=self.data_dir, mode=mode, features=features
|
||||
)
|
||||
@@ -0,0 +1,179 @@
|
||||
"""SDK search parameters a host may pass through to `client.search`.
|
||||
|
||||
`POST /api/v2/context/search`, `POST /api/v1/recall` and the v2 ask routes
|
||||
used to fix every SDK search knob, so a host that wanted a stricter relevance
|
||||
floor filtered the answer itself. These fields are the public `search`
|
||||
parameters of hy-memory 2.0.9 that are safe to hand a caller. Every field is
|
||||
optional: absent, the route keeps exactly the call it made before.
|
||||
|
||||
Never exposed: `user_ids` (a route searches its one scoped `user_id`),
|
||||
`embed_override` and `query_embedding` (they bypass the configured embedder),
|
||||
and `request_id` (Core's own correlation id).
|
||||
|
||||
A caller's floor is also applied to what Core returns, not only handed to the
|
||||
SDK: a hit below the floor that names its lane is dropped (`passes_floor`).
|
||||
With `min_score` alone that floor covers every lane, so `min_score=0.9`
|
||||
answers only results scoring at least 0.9.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, StrictBool, model_validator
|
||||
|
||||
#: The SDK's `MemoryLayer` values (hy-memory 2.0.9).
|
||||
MemoryLayerName = Literal[
|
||||
"l0_basic_info",
|
||||
"l1_raw",
|
||||
"l2_fact",
|
||||
"l3_summary",
|
||||
"l4_identity",
|
||||
"l5_knowledge",
|
||||
"l6_schema",
|
||||
"l7_intention",
|
||||
]
|
||||
MEMORY_LAYERS: tuple[str, ...] = (
|
||||
"l0_basic_info",
|
||||
"l1_raw",
|
||||
"l2_fact",
|
||||
"l3_summary",
|
||||
"l4_identity",
|
||||
"l5_knowledge",
|
||||
"l6_schema",
|
||||
"l7_intention",
|
||||
)
|
||||
#: The SDK's `MemoryStatus` values.
|
||||
MEMORY_STATUSES: tuple[str, ...] = (
|
||||
"active",
|
||||
"superseded",
|
||||
"negated",
|
||||
"conflicted",
|
||||
"shadow",
|
||||
"archived",
|
||||
)
|
||||
SearchScene = Literal["chat", "working", "coding", "all"]
|
||||
#: Which recall lane produced a hit, for the floor that applies to it.
|
||||
SearchLane = Literal["normal", "schema", "proactive", "working", "coding", "raw"]
|
||||
|
||||
Score = Annotated[float, Field(ge=0.0, le=1.0)]
|
||||
SourceName = Annotated[str, Field(min_length=1, max_length=128)]
|
||||
ScopeId = Annotated[str, Field(min_length=1, max_length=512)]
|
||||
|
||||
|
||||
class SdkSearchFilters(BaseModel):
|
||||
"""Filters and floors every search route accepts (recall, context, ask)."""
|
||||
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
#: Chat lane floor; with nothing more specific, every lane's.
|
||||
min_score: Score | None = None
|
||||
#: Schema (L6) lane floor; defaults to `min_score` when only that is set.
|
||||
schema_min_score: Score | None = None
|
||||
#: False skips the SDK's reranking stages for this call.
|
||||
rerank: StrictBool | None = None
|
||||
include_sources: tuple[SourceName, ...] | None = Field(default=None, max_length=64)
|
||||
exclude_sources: tuple[SourceName, ...] | None = Field(default=None, max_length=64)
|
||||
#: Unix seconds: only memories created at or after it.
|
||||
created_after: float | None = Field(default=None, ge=0)
|
||||
session_ids: tuple[ScopeId, ...] | None = Field(default=None, max_length=64)
|
||||
agent_ids: tuple[ScopeId, ...] | None = Field(default=None, max_length=64)
|
||||
#: Only these layers (pro/ultra; lite always searches raw).
|
||||
layers: tuple[MemoryLayerName, ...] | None = Field(default=None, min_length=1, max_length=8)
|
||||
|
||||
def filter_kwargs(self) -> dict[str, object]:
|
||||
"""The SDK keyword arguments this request sets, and only those."""
|
||||
values: dict[str, object] = {}
|
||||
for name in (
|
||||
"min_score",
|
||||
"schema_min_score",
|
||||
"rerank",
|
||||
"created_after",
|
||||
):
|
||||
value = getattr(self, name)
|
||||
if value is not None:
|
||||
values[name] = value
|
||||
for name in ("include_sources", "exclude_sources", "session_ids", "agent_ids", "layers"):
|
||||
value = getattr(self, name)
|
||||
if value is not None:
|
||||
values[name] = list(dict.fromkeys(value))
|
||||
if self.min_score is not None and self.schema_min_score is None:
|
||||
values["schema_min_score"] = self.min_score
|
||||
return values
|
||||
|
||||
def lane_floor(self, lane: SearchLane) -> float | None:
|
||||
"""The caller's floor for one lane, or None when the caller set none."""
|
||||
if lane == "schema":
|
||||
return self.schema_min_score if self.schema_min_score is not None else self.min_score
|
||||
return self.min_score
|
||||
|
||||
def passes_floor(self, lane: SearchLane, score: float) -> bool:
|
||||
floor = self.lane_floor(lane)
|
||||
return floor is None or score >= floor
|
||||
|
||||
def search_overrides(self) -> Mapping[str, object]:
|
||||
return self.filter_kwargs()
|
||||
|
||||
|
||||
class SdkSearchOptions(SdkSearchFilters):
|
||||
"""Every SDK search knob a recall or context caller may set."""
|
||||
|
||||
#: working/coding fact-layer floor (SDK default 0.30).
|
||||
working_fact_min_score: Score | None = None
|
||||
#: working/coding experience-layer floor (SDK default fact floor + 0.10).
|
||||
working_experience_min_score: Score | None = None
|
||||
reader: Literal["legacy", "hybrid"] | None = None
|
||||
#: chat (the default), working, coding, or all three fused.
|
||||
scene: SearchScene | None = None
|
||||
#: agentic spends one model call to classify and rewrite the query.
|
||||
search_mode: Literal["default", "agentic"] | None = None
|
||||
#: working/coding lanes only; `branch` needs `workspace_id`.
|
||||
workspace_id: ScopeId | None = None
|
||||
branch: ScopeId | None = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def branch_needs_workspace(self) -> SdkSearchOptions:
|
||||
if self.branch is not None and self.workspace_id is None:
|
||||
raise ValueError("branch requires workspace_id")
|
||||
return self
|
||||
|
||||
def search_overrides(self) -> Mapping[str, object]:
|
||||
values = self.filter_kwargs()
|
||||
for name in (
|
||||
"working_fact_min_score",
|
||||
"working_experience_min_score",
|
||||
"reader",
|
||||
"scene",
|
||||
"search_mode",
|
||||
"workspace_id",
|
||||
"branch",
|
||||
):
|
||||
value = getattr(self, name)
|
||||
if value is not None:
|
||||
values[name] = value
|
||||
if self.min_score is not None and self.working_fact_min_score is None:
|
||||
values["working_fact_min_score"] = self.min_score
|
||||
if self.min_score is not None and self.working_experience_min_score is None:
|
||||
values["working_experience_min_score"] = self.min_score
|
||||
return values
|
||||
|
||||
def lane_floor(self, lane: SearchLane) -> float | None:
|
||||
if lane in {"working", "coding"}:
|
||||
explicit = [
|
||||
value
|
||||
for value in (self.working_fact_min_score, self.working_experience_min_score)
|
||||
if value is not None
|
||||
]
|
||||
# Facts and experiences share one score field on a fused hit, so
|
||||
# the looser of the two floors is the one Core can enforce.
|
||||
return min(explicit) if explicit else self.min_score
|
||||
return super().lane_floor(lane)
|
||||
|
||||
|
||||
def split_csv(values: list[str] | None) -> list[str]:
|
||||
"""Query values given repeated (`a&a`) or comma separated (`a,b`)."""
|
||||
found: list[str] = []
|
||||
for value in values or ():
|
||||
found.extend(part.strip() for part in value.split(",") if part.strip())
|
||||
return list(dict.fromkeys(found))
|
||||
@@ -9,9 +9,11 @@ from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from one_memory.hy_sdk_runtime import SdkClientHandle
|
||||
from one_memory.hy_graph_admin import effective_memory_mode
|
||||
from one_memory.hy_sdk_runtime import SdkClientHandle, builtin_retrieval_snapshot
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord
|
||||
from one_memory.input_sanitization import sanitize_model_input
|
||||
from one_memory.memory_features import FeatureOverrides
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ModelProfileStore
|
||||
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
|
||||
|
||||
@@ -31,8 +33,11 @@ class SdkWriteReceipt:
|
||||
class SdkWriteProcessor:
|
||||
"""Turn one durable job into exactly one public SDK write call."""
|
||||
|
||||
def __init__(self, client_handle: SdkClientHandle) -> None:
|
||||
def __init__(self, client_handle: SdkClientHandle, *, lite: bool = False) -> None:
|
||||
self._client_handle = client_handle
|
||||
#: lite makes no model call: `add_extracted` would still call one, so a
|
||||
#: direct memory note is stored as one raw user turn through `add`.
|
||||
self._lite = lite
|
||||
|
||||
async def process_sdk_job(
|
||||
self,
|
||||
@@ -64,7 +69,7 @@ class SdkWriteProcessor:
|
||||
# waits and retries instead of being left `unknown`.
|
||||
raise RuntimeError("hy_memory_client_unavailable") from error
|
||||
|
||||
if extracted:
|
||||
if extracted and not self._lite:
|
||||
contents = [sanitize_model_input(record.verbatim) for record in records]
|
||||
await before_hy_invocation()
|
||||
result = await asyncio.to_thread(
|
||||
@@ -81,12 +86,15 @@ class SdkWriteProcessor:
|
||||
else:
|
||||
messages = [
|
||||
{
|
||||
"role": record.role,
|
||||
"role": "user" if extracted else record.role,
|
||||
"content": sanitize_model_input(record.verbatim),
|
||||
}
|
||||
for record in records
|
||||
if record.content_kind is EvidenceContentKind.RAW_CONVERSATION
|
||||
and record.role in _MESSAGE_ROLES
|
||||
if extracted
|
||||
or (
|
||||
record.content_kind is EvidenceContentKind.RAW_CONVERSATION
|
||||
and record.role in _MESSAGE_ROLES
|
||||
)
|
||||
]
|
||||
if not messages:
|
||||
raise RuntimeError("hy_memory_task_without_inputs")
|
||||
@@ -138,6 +146,16 @@ class SdkWriteCoordinator:
|
||||
from under a write still running on it: the SDK cleared its stores and
|
||||
the SharedRuntime closed mid-`add`, the job failed with AttributeError
|
||||
after HY was invoked, and it was left `unknown`.
|
||||
|
||||
Handles are keyed by profile, memory mode and the SDK features
|
||||
(`memory_features`), so a mode or feature change retires handles the same
|
||||
way. In lite no generation model is needed: without an
|
||||
active profile the built-in local embedder writes raw turns.
|
||||
|
||||
`exclusive_for_evolve(user_id)` keeps evolve and that user's writes
|
||||
apart: it waits for the user's write in flight, and while it runs that
|
||||
user's jobs are refused before the SDK is called
|
||||
(`memory_evolve_in_progress`), so the drain puts them back.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -148,13 +166,24 @@ class SdkWriteCoordinator:
|
||||
data_dir: Any,
|
||||
transport_factory: Callable[[ModelProfileSnapshot], Callable[[], Any] | None],
|
||||
sedimentation_enabled: Callable[[], bool],
|
||||
memory_mode: Callable[[], str] = effective_memory_mode,
|
||||
memory_features: Callable[[], FeatureOverrides] = lambda: (),
|
||||
) -> None:
|
||||
self.profiles = profiles
|
||||
self.secrets = secrets
|
||||
self.data_dir = data_dir
|
||||
self.transport_factory = transport_factory
|
||||
self.sedimentation_enabled = sedimentation_enabled
|
||||
self._handles: dict[tuple[str, int], SdkClientHandle] = {}
|
||||
self.memory_mode = memory_mode
|
||||
self.memory_features = memory_features
|
||||
self._handles: dict[tuple[str, int, str, FeatureOverrides], SdkClientHandle] = {}
|
||||
#: Users evolve (or the graph backfill) is running for, the writes in
|
||||
#: flight per user, and one exclusive run per user at a time. Writes
|
||||
#: of one user still run side by side; only evolve waits for them.
|
||||
self._evolving: set[str] = set()
|
||||
self._writing: dict[str, int] = {}
|
||||
self._writes_idle = asyncio.Condition()
|
||||
self._exclusive: dict[str, asyncio.Lock] = {}
|
||||
#: Uses in flight per handle; a retired handle closes at zero.
|
||||
self._uses: dict[SdkClientHandle, int] = {}
|
||||
self._retired: set[SdkClientHandle] = set()
|
||||
@@ -169,16 +198,70 @@ class SdkWriteCoordinator:
|
||||
) -> SdkWriteReceipt:
|
||||
if not self.sedimentation_enabled():
|
||||
raise RuntimeError("model_sedimentation_disabled")
|
||||
async with self._active_handle() as handle:
|
||||
return await SdkWriteProcessor(handle).process_sdk_job(
|
||||
records,
|
||||
request_id=request_id,
|
||||
before_hy_invocation=before_hy_invocation,
|
||||
)
|
||||
user_id = records[0].user_id if records else "local"
|
||||
if user_id in self._evolving:
|
||||
# Before any SDK call: the drain returns the job to pending.
|
||||
raise RuntimeError("memory_evolve_in_progress")
|
||||
self._writing[user_id] = self._writing.get(user_id, 0) + 1
|
||||
try:
|
||||
async with self._active_handle() as handle:
|
||||
return await SdkWriteProcessor(
|
||||
handle, lite=getattr(handle, "mode", None) == "lite"
|
||||
).process_sdk_job(
|
||||
records,
|
||||
request_id=request_id,
|
||||
before_hy_invocation=before_hy_invocation,
|
||||
)
|
||||
finally:
|
||||
remaining = self._writing.get(user_id, 1) - 1
|
||||
if remaining > 0:
|
||||
self._writing[user_id] = remaining
|
||||
else:
|
||||
self._writing.pop(user_id, None)
|
||||
async with self._writes_idle:
|
||||
self._writes_idle.notify_all()
|
||||
|
||||
async def inquire_write_outcome(
|
||||
self, request_id: str
|
||||
) -> SdkWriteReceipt | None:
|
||||
async def retire_stale_handles(self) -> None:
|
||||
"""After a mode or feature change: retire handles built otherwise.
|
||||
|
||||
The idle ones close now. A handle still writing is closed when its
|
||||
last use ends (#218), so a write that started in the old mode, or
|
||||
with the old features, finishes in it.
|
||||
"""
|
||||
build = (self.memory_mode(), self.memory_features())
|
||||
async with self._lock:
|
||||
stale = {key: handle for key, handle in self._handles.items() if key[2:] != build}
|
||||
for key in stale:
|
||||
del self._handles[key]
|
||||
self._retired.update(stale.values())
|
||||
unused = {value for value in self._retired if not self._uses.get(value)}
|
||||
self._retired -= unused
|
||||
await asyncio.gather(*(value.close() for value in unused))
|
||||
|
||||
#: The name #253 introduced; a mode change is one kind of rebuild.
|
||||
retire_stale_modes = retire_stale_handles
|
||||
|
||||
def evolving(self, user_id: str) -> bool:
|
||||
"""Whether evolve holds this user, so their queued writes should wait."""
|
||||
return user_id in self._evolving
|
||||
|
||||
@asynccontextmanager
|
||||
async def exclusive_for_evolve(self, user_id: str) -> AsyncIterator[None]:
|
||||
"""Hold one user's writes off while evolve runs for them.
|
||||
|
||||
New writes of the user are refused from here on; the ones already in
|
||||
flight are waited for before the body runs.
|
||||
"""
|
||||
async with self._exclusive.setdefault(user_id, asyncio.Lock()):
|
||||
self._evolving.add(user_id)
|
||||
try:
|
||||
async with self._writes_idle:
|
||||
await self._writes_idle.wait_for(lambda: not self._writing.get(user_id))
|
||||
yield
|
||||
finally:
|
||||
self._evolving.discard(user_id)
|
||||
|
||||
async def inquire_write_outcome(self, request_id: str) -> SdkWriteReceipt | None:
|
||||
async with self._active_handle() as handle:
|
||||
return await SdkWriteProcessor(handle).inquire_write_outcome(request_id)
|
||||
|
||||
@@ -204,9 +287,12 @@ class SdkWriteCoordinator:
|
||||
@asynccontextmanager
|
||||
async def _active_handle(self) -> AsyncIterator[SdkClientHandle]:
|
||||
"""Use the active profile's handle; retire the others, closing the unused."""
|
||||
identity, profile, secret = await asyncio.to_thread(
|
||||
self._read_active_credentials
|
||||
mode = self.memory_mode()
|
||||
features = self.memory_features()
|
||||
profile_identity, profile, secret = await asyncio.to_thread(
|
||||
self._read_active_credentials, mode
|
||||
)
|
||||
identity = (*profile_identity, mode, features)
|
||||
async with self._lock:
|
||||
handle = self._handles.get(identity)
|
||||
if handle is None:
|
||||
@@ -215,6 +301,8 @@ class SdkWriteCoordinator:
|
||||
api_key=secret,
|
||||
data_dir=self.data_dir,
|
||||
http_transport_factory=self.transport_factory(profile),
|
||||
mode=mode,
|
||||
features=features,
|
||||
)
|
||||
self._handles[identity] = handle
|
||||
self._retired.update(value for key, value in self._handles.items() if key != identity)
|
||||
@@ -239,9 +327,14 @@ class SdkWriteCoordinator:
|
||||
|
||||
def _read_active_credentials(
|
||||
self,
|
||||
mode: str = "pro",
|
||||
) -> tuple[tuple[str, int], ModelProfileSnapshot, str | None]:
|
||||
identity = self.profiles.active_identity()
|
||||
if identity is None:
|
||||
if mode == "lite":
|
||||
# lite stores raw turns with one embedding and calls no model.
|
||||
builtin = builtin_retrieval_snapshot()
|
||||
return (builtin.profile_id, builtin.revision), builtin, None
|
||||
raise RuntimeError("model_profile_not_active")
|
||||
profile = self.profiles.snapshot(*identity)
|
||||
try:
|
||||
|
||||
@@ -27,6 +27,11 @@ class SemanticMemoryHit:
|
||||
member_ids: tuple[str, ...] = ()
|
||||
sdk_formatted: bool = False
|
||||
formatting_complete: bool = False
|
||||
#: The recall lane that produced the hit (`sdk_search_options.SearchLane`)
|
||||
#: and the namespace to read it back from. Defaults are a chat memory.
|
||||
lane: str = "normal"
|
||||
kind: str = "chat"
|
||||
variant: str | None = None
|
||||
|
||||
def as_dict(self) -> dict[str, object]:
|
||||
return {
|
||||
|
||||
@@ -159,8 +159,10 @@
|
||||
| `test_memory_citation_query_plan.py` | `contract` | Correlated citation checks must look links up by memory, not scan by 'active'. |
|
||||
| `test_memory_content_import.py` | `integration` | 覆盖 existing memory preview keeps unknown author and exact original、invalid existing memory records never leave a partial upload、memory content requires explicit mode and rejects vendor guessing 等 5 个场景。 |
|
||||
| `test_memory_evidence_freshness.py` | `contract` | 新鲜度拆除后:读路径恒为 current;证据修订 / 畸形 proof / segment ref 不再藏记忆或 503。 |
|
||||
| `test_memory_features.py` | `contract` | The SDK retrieval and extraction features MemHub lets a person turn on. |
|
||||
| `test_memory_freshness_integration_review.py` | `integration` | Independent synthetic read-path review; no lifespan, model or user database. |
|
||||
| `test_memory_library_all_scenes.py` | `integration` | The v2 memory library across chat, working and coding memories. |
|
||||
| `test_memory_mode_and_evolve.py` | `contract` | The memory mode (lite / pro / ultra), evolve and its nightly schedule. |
|
||||
| `test_memory_qa.py` | `contract` | 覆盖 search terms keep meaningful single chinese character、streamed answer emits recall then incremental grounded text、governance change during generation stops the old answer 等 14 个场景。 |
|
||||
| `test_memory_qa_api.py` | `integration` | 覆盖 ask stream endpoint emits recall deltas and final、ask stream endpoint requires the local token 场景。 |
|
||||
| `test_memory_qa_permissions.py` | `contract` | 覆盖 initial refusal precedes all secret access and network、invalid authorization result fails before secret access、authorization receives exact question and only outbound grounding 等 16 个场景。 |
|
||||
@@ -248,6 +250,7 @@
|
||||
| `test_schema_gate.py` | `contract` | 覆盖 fresh database records the build schema、future schema refuses to open、legacy database without a ledger is left untouched 场景。 |
|
||||
| `test_schema_migrations.py` | `contract` | 覆盖 evidence store adds source ordinal to previous schema、import store migrates previous job and error schema、old active job index lets a sync sit beside paused history 等 4 个场景。 |
|
||||
| `test_schema_ownership_gate.py` | `contract` | A1-03/A1-04: schema ownership, and the hard part of telling DDL from reads. |
|
||||
| `test_sdk_api_surface.py` | `contract` | The SDK surface MemHub exposes: search pass-through, layers, raw, v2 aliases. |
|
||||
| `test_sdk_client_construction.py` | `contract` | SDK clients of one home are constructed one at a time. |
|
||||
| `test_sdk_data_plane_migration.py` | `contract` | SDK-authority migration ledger, aliases and atomic cutover marker. |
|
||||
| `test_sdk_memory_qa.py` | `contract` | 覆盖 sdk ask propagates sdk search failure without legacy fallback、sdk ask revalidates memory before returning extract、sdk ask uses native sdk memory without governance fields 场景。 |
|
||||
@@ -311,4 +314,4 @@
|
||||
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
|
||||
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
|
||||
|
||||
共 306 个测试文件。
|
||||
共 309 个测试文件。
|
||||
@@ -14,8 +14,8 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
|
||||
EXPECTED_TEST_FILE_COUNT = 306
|
||||
EXPECTED_TEST_FILES_SHA256 = "e616dcf9fea288bf15a4445045f7b4278517f3304091c04fffd55be80308d481"
|
||||
EXPECTED_TEST_FILE_COUNT = 309
|
||||
EXPECTED_TEST_FILES_SHA256 = "a7d6ed33175f618d2d40f401d447a5553e1347ffd51dd878370c0fc84d13396d"
|
||||
|
||||
# Pure transformations, validation models, parity checks and state machines.
|
||||
# This list is intentionally small: uncertain files default to contract.
|
||||
|
||||
@@ -164,7 +164,7 @@ def test_metadata_repair_never_overlaps_an_sdk_client(tmp_path, monkeypatch):
|
||||
events.append("repair_finished")
|
||||
return MetadataRepairResult(None, 0, 0, 0)
|
||||
|
||||
def open_client(profile, *, api_key, data_dir, http_transport_factory):
|
||||
def open_client(profile, *, api_key, data_dir, http_transport_factory, mode="pro", features=()):
|
||||
events.append("client_opened")
|
||||
return ClientLease(object(), data_dir.resolve())
|
||||
|
||||
|
||||
@@ -0,0 +1,499 @@
|
||||
"""The SDK retrieval and extraction features MemHub lets a person turn on.
|
||||
|
||||
Graph rerank (usage A), episode recall (usage B), the episode write side and
|
||||
the extraction switches are `MemoryConfig` fields. MemHub hard-codes none of
|
||||
them: one global `memory_features` document sets them per client in
|
||||
`build_sdk_config`, an unset one keeps the SDK default (so nothing set is the
|
||||
config of before, byte for byte), and a change rebuilds the handles the way a
|
||||
mode change does. A feature never touches the vector collection identity.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import dataclasses
|
||||
import json
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import SecretStr, ValidationError
|
||||
|
||||
from one_memory import hy_sdk_runtime, sdk_write_processor
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
from one_memory.memory_features import (
|
||||
FEATURE_NAMES,
|
||||
FEATURES,
|
||||
FEATURES_BY_NAME,
|
||||
MemoryFeatures,
|
||||
effective_features,
|
||||
feature_activity,
|
||||
)
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
|
||||
TOKEN = "memory-features-test"
|
||||
|
||||
#: What MemHub ran with before the setting existed, on a clean environment.
|
||||
TODAY = {
|
||||
"graph_rerank": False,
|
||||
"same_series_timeline": False,
|
||||
"episode_recall": False,
|
||||
"session_episodes": False,
|
||||
"episode_summary": False,
|
||||
"intention": True,
|
||||
"few_shot": False,
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_sdk_env(monkeypatch):
|
||||
"""The SDK's own env vars feed its defaults; tests start without them."""
|
||||
for spec in FEATURES:
|
||||
monkeypatch.delenv(spec.env, raising=False)
|
||||
monkeypatch.delenv("ONE_MEMORY_GRAPH_STORE", raising=False)
|
||||
|
||||
|
||||
def _remote_profile() -> ModelProfileSnapshot:
|
||||
return ModelProfileSnapshot(
|
||||
profile_id="remote",
|
||||
revision=3,
|
||||
provider_kind=ProviderKind.OPENAI_COMPATIBLE,
|
||||
base_url="https://models.example.test/v1",
|
||||
model="chat-model",
|
||||
embedding_provider_kind=ProviderKind.OPENAI_COMPATIBLE,
|
||||
embedding_model="embed-model",
|
||||
embedding_dimensions=1024,
|
||||
timeout_seconds=60,
|
||||
max_concurrency=1,
|
||||
context_window=None,
|
||||
has_api_key=True,
|
||||
# Since #255 the embedding endpoint is its own field, never inherited.
|
||||
embedding_base_url="https://models.example.test/v1",
|
||||
)
|
||||
|
||||
|
||||
PROFILES = (hy_sdk_runtime.builtin_retrieval_snapshot(), _remote_profile())
|
||||
|
||||
|
||||
def _config(tmp_path: Path, profile: ModelProfileSnapshot, features=()):
|
||||
return hy_sdk_runtime.build_sdk_config(
|
||||
profile, api_key="secret-key", data_dir=tmp_path, features=features
|
||||
)
|
||||
|
||||
|
||||
def _field(config, name: str) -> bool:
|
||||
spec = FEATURES_BY_NAME[name]
|
||||
return getattr(getattr(config, spec.section), spec.field)
|
||||
|
||||
|
||||
# --- the catalog and the document ------------------------------------------
|
||||
|
||||
|
||||
def test_the_document_names_exactly_the_catalog() -> None:
|
||||
assert tuple(MemoryFeatures.model_fields) == FEATURE_NAMES
|
||||
assert set(TODAY) == set(FEATURE_NAMES)
|
||||
# Each feature is a MemoryConfig field the SDK has; no env var is written.
|
||||
defaults = hy_sdk_runtime.sdk_feature_defaults()
|
||||
assert set(defaults) == set(FEATURE_NAMES)
|
||||
|
||||
|
||||
def test_the_defaults_are_todays_behaviour() -> None:
|
||||
assert hy_sdk_runtime.sdk_feature_defaults() == TODAY
|
||||
assert MemoryFeatures().overrides() == ()
|
||||
assert effective_features((), TODAY) == TODAY
|
||||
|
||||
|
||||
def test_the_sdk_env_var_still_moves_the_default_of_an_unset_feature(monkeypatch) -> None:
|
||||
monkeypatch.setenv("MEMORY_EPISODE_RECALL_ENABLED", "true")
|
||||
assert hy_sdk_runtime.sdk_feature_defaults()["episode_recall"] is True
|
||||
|
||||
|
||||
def test_unknown_or_non_boolean_values_are_refused() -> None:
|
||||
with pytest.raises(ValidationError):
|
||||
MemoryFeatures.model_validate({"graph_rerank": True, "local_rerank": True})
|
||||
with pytest.raises(ValidationError):
|
||||
MemoryFeatures.model_validate({"graph_rerank": "true"})
|
||||
with pytest.raises(ValidationError):
|
||||
MemoryFeatures.model_validate({"graph_rerank": 1})
|
||||
with pytest.raises(ValueError, match="unknown_memory_feature"):
|
||||
hy_sdk_runtime.build_sdk_config(
|
||||
PROFILES[0], api_key=None, data_dir=Path("/tmp/unused"), features={"nope": True}
|
||||
)
|
||||
|
||||
|
||||
def test_merge_sets_resets_and_leaves_absent_keys_alone() -> None:
|
||||
saved = MemoryFeatures(graph_rerank=True, few_shot=True)
|
||||
update = MemoryFeatures.model_validate({"few_shot": None, "episode_recall": True})
|
||||
merged = saved.merged(update)
|
||||
assert merged.overrides() == (("episode_recall", True), ("graph_rerank", True))
|
||||
assert saved.merged(MemoryFeatures()) == saved
|
||||
|
||||
|
||||
def test_the_setting_persists_and_an_untouched_install_writes_none(tmp_path) -> None:
|
||||
store = ConfigStore(tmp_path)
|
||||
store.set_sync("cursor", True)
|
||||
assert "memory_features" not in json.loads((tmp_path / "config.json").read_text())
|
||||
store.set_memory_features(MemoryFeatures(graph_rerank=True, intention=False))
|
||||
saved = json.loads((tmp_path / "config.json").read_text())
|
||||
# Only what a person set; an absent key is the SDK default.
|
||||
assert saved["memory_features"] == {"graph_rerank": True, "intention": False}
|
||||
assert ConfigStore(tmp_path).memory_features().overrides() == (
|
||||
("graph_rerank", True),
|
||||
("intention", False),
|
||||
)
|
||||
(tmp_path / "config.json").write_text(json.dumps({"memory_features": {"bogus": True}}))
|
||||
with pytest.raises(Exception, match="config_invalid"):
|
||||
ConfigStore(tmp_path)
|
||||
|
||||
|
||||
# --- how settings reach the SDK ----------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("profile", PROFILES, ids=("builtin", "remote"))
|
||||
def test_each_feature_reaches_the_built_memory_config(tmp_path, profile) -> None:
|
||||
for spec in FEATURES:
|
||||
flipped = not TODAY[spec.name]
|
||||
config = _config(tmp_path, profile, {spec.name: flipped})
|
||||
assert _field(config, spec.name) is flipped, spec.name
|
||||
for other in FEATURES:
|
||||
if other.name != spec.name:
|
||||
assert _field(config, other.name) is TODAY[other.name], (spec.name, other.name)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("profile", PROFILES, ids=("builtin", "remote"))
|
||||
def test_nothing_set_is_the_config_of_before_byte_for_byte(tmp_path, profile) -> None:
|
||||
before = repr(dataclasses.asdict(_config(tmp_path, profile)))
|
||||
assert repr(dataclasses.asdict(_config(tmp_path, profile, ()))) == before
|
||||
assert repr(dataclasses.asdict(_config(tmp_path, profile, {}))) == before
|
||||
# Setting every feature to its default value changes nothing either.
|
||||
assert repr(dataclasses.asdict(_config(tmp_path, profile, TODAY))) == before
|
||||
|
||||
|
||||
@pytest.mark.parametrize("profile", PROFILES, ids=("builtin", "remote"))
|
||||
def test_features_never_change_the_collection_identity(tmp_path, profile) -> None:
|
||||
def identity(config):
|
||||
return (
|
||||
dataclasses.asdict(config.vector_store),
|
||||
config.embedder.model,
|
||||
config.embedder.embedding_dims,
|
||||
config.graph_store.db_path,
|
||||
config.cache.db_path,
|
||||
)
|
||||
|
||||
baseline = identity(_config(tmp_path, profile))
|
||||
for values in (
|
||||
{name: True for name in FEATURE_NAMES},
|
||||
{name: False for name in FEATURE_NAMES},
|
||||
{"graph_rerank": True, "episode_recall": True},
|
||||
):
|
||||
assert identity(_config(tmp_path, profile, values)) == baseline
|
||||
|
||||
|
||||
def test_acquire_client_builds_the_client_with_the_features(tmp_path, monkeypatch) -> None:
|
||||
built: list[tuple[str, bool, bool]] = []
|
||||
|
||||
class Client:
|
||||
def __init__(self, *, config, mode, runtime, embedding_service):
|
||||
built.append(
|
||||
(
|
||||
mode,
|
||||
config.graph_store.graph_rerank_enabled,
|
||||
config.recall.episode_recall_enabled,
|
||||
)
|
||||
)
|
||||
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(hy_sdk_runtime, "HyMemoryClient", Client)
|
||||
monkeypatch.setattr(hy_sdk_runtime, "_acquire_runtime", lambda config, key: object())
|
||||
monkeypatch.setattr(hy_sdk_runtime, "_release_runtime", lambda key: None)
|
||||
monkeypatch.setattr(hy_sdk_runtime, "inspect_graph_width_mismatch", lambda config: None)
|
||||
profile = hy_sdk_runtime.builtin_retrieval_snapshot()
|
||||
hy_sdk_runtime.acquire_client(profile, api_key=None, data_dir=tmp_path, mode="pro").close()
|
||||
hy_sdk_runtime.acquire_client(
|
||||
profile,
|
||||
api_key=None,
|
||||
data_dir=tmp_path,
|
||||
mode="ultra",
|
||||
features=(("episode_recall", True), ("graph_rerank", True)),
|
||||
).close()
|
||||
assert built == [("pro", False, False), ("ultra", True, True)]
|
||||
|
||||
|
||||
# --- active in the current mode ---------------------------------------------
|
||||
|
||||
|
||||
def test_a_feature_needing_a_higher_mode_is_reported_inactive() -> None:
|
||||
on = effective_features((("graph_rerank", True), ("episode_recall", True)), TODAY)
|
||||
assert feature_activity("graph_rerank", on, "pro") == "requires_ultra"
|
||||
assert feature_activity("graph_rerank", on, "ultra") is None
|
||||
# Usage B needs no graph: it acts in pro, not in lite (nothing extracted).
|
||||
assert feature_activity("episode_recall", on, "pro") is None
|
||||
assert feature_activity("episode_recall", on, "lite") == "requires_pro"
|
||||
assert feature_activity("intention", TODAY, "lite") == "requires_pro"
|
||||
assert feature_activity("few_shot", TODAY, "ultra") == "disabled"
|
||||
series = effective_features((("same_series_timeline", True),), TODAY)
|
||||
assert feature_activity("same_series_timeline", series, "ultra") == "requires_graph_rerank"
|
||||
series = effective_features((("same_series_timeline", True), ("graph_rerank", True)), TODAY)
|
||||
assert feature_activity("same_series_timeline", series, "pro") == "requires_ultra"
|
||||
assert feature_activity("same_series_timeline", series, "ultra") is None
|
||||
|
||||
|
||||
# --- rebuilding handles -------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeHandle:
|
||||
def __init__(self, mode: str, features=(), profile_id: str = "p") -> None:
|
||||
self.mode = mode
|
||||
self.features = features
|
||||
self.profile_id = profile_id
|
||||
self.closed = False
|
||||
|
||||
async def client(self):
|
||||
return self
|
||||
|
||||
async def close(self):
|
||||
self.closed = True
|
||||
|
||||
|
||||
def test_the_read_handle_rebuilds_its_client_when_the_features_change(
|
||||
monkeypatch, tmp_path
|
||||
) -> None:
|
||||
"""Reads follow the active profile (#255), the mode and the features."""
|
||||
from one_memory import sdk_read_handle
|
||||
|
||||
features = [()]
|
||||
made: list[_FakeHandle] = []
|
||||
|
||||
def handle_for(profile, *, api_key, data_dir, mode=None, features=()):
|
||||
handle = _FakeHandle(mode or "pro", features, profile.profile_id)
|
||||
made.append(handle)
|
||||
return handle
|
||||
|
||||
monkeypatch.setattr(sdk_read_handle, "SdkClientHandle", handle_for)
|
||||
|
||||
class _NoProfile:
|
||||
def active_identity(self):
|
||||
return None
|
||||
|
||||
reads = sdk_read_handle.ActiveProfileReadHandle(
|
||||
profiles=_NoProfile(), # type: ignore[arg-type]
|
||||
secrets=None, # type: ignore[arg-type]
|
||||
data_dir=tmp_path,
|
||||
memory_mode=lambda: "pro",
|
||||
features=lambda: features[0],
|
||||
)
|
||||
hold = asyncio.Event()
|
||||
holding = asyncio.Event()
|
||||
|
||||
async def read():
|
||||
return await reads.client()
|
||||
|
||||
async def slow_read():
|
||||
client = await reads.client()
|
||||
holding.set()
|
||||
await hold.wait()
|
||||
return client
|
||||
|
||||
async def scenario():
|
||||
in_flight = asyncio.create_task(slow_read())
|
||||
await asyncio.wait_for(holding.wait(), 5)
|
||||
old = made[0]
|
||||
assert (await asyncio.create_task(read())) is old
|
||||
features[0] = (("episode_recall", True),)
|
||||
try:
|
||||
fresh = await asyncio.create_task(read())
|
||||
assert fresh.features == (("episode_recall", True),) and fresh.mode == "pro"
|
||||
# A read that took the old client is still using it.
|
||||
assert not old.closed
|
||||
finally:
|
||||
hold.set()
|
||||
await in_flight
|
||||
for _ in range(10):
|
||||
await asyncio.sleep(0)
|
||||
assert old.closed and not fresh.closed
|
||||
await reads.close()
|
||||
assert fresh.closed
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
class _Profiles:
|
||||
def active_identity(self):
|
||||
return ("profile-a", 1)
|
||||
|
||||
def snapshot(self, profile_id, revision):
|
||||
return SimpleNamespace(profile_id=profile_id, has_api_key=False)
|
||||
|
||||
def resolve_snapshot_secret(self, profile_id, revision, secrets):
|
||||
return None
|
||||
|
||||
|
||||
def test_a_feature_change_keeps_a_busy_write_handle_until_its_write_ends(monkeypatch) -> None:
|
||||
features = [()]
|
||||
opened: list[_FakeHandle] = []
|
||||
|
||||
def handle(profile, **kwargs):
|
||||
value = _FakeHandle(kwargs["mode"], kwargs["features"], profile.profile_id)
|
||||
opened.append(value)
|
||||
return value
|
||||
|
||||
monkeypatch.setattr(sdk_write_processor, "SdkClientHandle", handle)
|
||||
coordinator = sdk_write_processor.SdkWriteCoordinator(
|
||||
profiles=_Profiles(), # type: ignore[arg-type]
|
||||
secrets=None, # type: ignore[arg-type]
|
||||
data_dir=None,
|
||||
transport_factory=lambda _profile: None,
|
||||
sedimentation_enabled=lambda: True,
|
||||
memory_mode=lambda: "pro",
|
||||
memory_features=lambda: features[0],
|
||||
)
|
||||
|
||||
async def scenario():
|
||||
async with coordinator.use_client() as client:
|
||||
assert client.features == ()
|
||||
features[0] = (("few_shot", True),)
|
||||
await coordinator.retire_stale_handles()
|
||||
# In flight: it finishes with the features it started with (#218).
|
||||
assert not opened[0].closed
|
||||
assert opened[0].closed
|
||||
async with coordinator.use_client() as client:
|
||||
assert client.features == (("few_shot", True),)
|
||||
features[0] = ()
|
||||
await coordinator.retire_stale_handles()
|
||||
assert opened[1].closed
|
||||
await coordinator.close()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
# --- the HTTP surface -----------------------------------------------------------
|
||||
|
||||
|
||||
def application(tmp_path: Path):
|
||||
settings = Settings(
|
||||
data_dir=tmp_path / "data",
|
||||
source_home=tmp_path,
|
||||
api_token=SecretStr(TOKEN),
|
||||
imports_enabled=False,
|
||||
automatic_memory_processing=False,
|
||||
live_sync_enabled=False,
|
||||
)
|
||||
return create_app(settings, secret_store=InMemorySecretStore())
|
||||
|
||||
|
||||
def run_app(tmp_path, monkeypatch, scenario) -> None:
|
||||
async def main() -> None:
|
||||
app = application(tmp_path)
|
||||
|
||||
@asynccontextmanager
|
||||
async def write_client():
|
||||
yield SimpleNamespace()
|
||||
|
||||
async with app.router.lifespan_context(app):
|
||||
monkeypatch.setattr(app.state.sdk_write_coordinator, "use_client", write_client)
|
||||
async with AsyncClient(
|
||||
transport=ASGITransport(app=app),
|
||||
base_url="http://test",
|
||||
headers={"Authorization": f"Bearer {TOKEN}"},
|
||||
) as client:
|
||||
await scenario(app, client)
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
|
||||
def _by_name(body) -> dict[str, dict]:
|
||||
return {item["name"]: item for item in body["features"]}
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_features_api_get_put_reset_and_status(tmp_path, monkeypatch) -> None:
|
||||
async def scenario(app, client):
|
||||
current = (await client.get("/api/v1/settings/memory-features")).json()
|
||||
assert current["mode"] == "pro" and current["non_default"] == {}
|
||||
listed = _by_name(current)
|
||||
assert list(listed) == list(FEATURE_NAMES)
|
||||
assert {name: item["enabled"] for name, item in listed.items()} == TODAY
|
||||
assert all(item["configured"] is None for item in listed.values())
|
||||
rerank = listed["graph_rerank"]
|
||||
assert rerank["default"] is False and rerank["active"] is False
|
||||
assert rerank["reason"] == "disabled" and rerank["requires_mode"] == "ultra"
|
||||
assert rerank["sdk_field"] == "graph_store.graph_rerank_enabled"
|
||||
assert rerank["env"] == "MEMORY_GRAPH_RERANK_ENABLED"
|
||||
assert listed["intention"]["active"] is True
|
||||
assert listed["episode_summary"]["cost"] == "llm_call"
|
||||
reads = app.state.sdk_client_handle
|
||||
# The read handle is keyed by (profile, mode, features): what it
|
||||
# builds next is what these return.
|
||||
assert reads.current_features() == ()
|
||||
|
||||
changed = await client.put(
|
||||
"/api/v1/settings/memory-features",
|
||||
json={"graph_rerank": True, "episode_recall": True},
|
||||
)
|
||||
assert changed.status_code == 200, changed.text
|
||||
body = changed.json()
|
||||
assert body["changed"] == ["graph_rerank", "episode_recall"]
|
||||
listed = _by_name(body)
|
||||
# Enabled but the graph store is ultra only: reported, not hidden.
|
||||
assert listed["graph_rerank"]["enabled"] is True
|
||||
assert listed["graph_rerank"]["active"] is False
|
||||
assert listed["graph_rerank"]["reason"] == "requires_ultra"
|
||||
assert listed["episode_recall"]["active"] is True
|
||||
assert body["non_default"] == {"graph_rerank": True, "episode_recall": True}
|
||||
# The handles were rebuilt with the new features.
|
||||
expected = (("episode_recall", True), ("graph_rerank", True))
|
||||
assert app.state.sdk_write_coordinator.memory_features() == expected
|
||||
assert reads.current_features() == expected
|
||||
assert reads.memory_mode() == "pro"
|
||||
|
||||
processing = (await client.get("/api/v1/memory-processing")).json()
|
||||
assert processing["pipeline"]["features"] == {
|
||||
"graph_rerank": True,
|
||||
"episode_recall": True,
|
||||
}
|
||||
|
||||
await client.put("/api/v1/settings/memory-mode", json={"mode": "ultra"})
|
||||
await app.state.memory_graph_backfill.wait()
|
||||
ultra = _by_name((await client.get("/api/v1/settings/memory-features")).json())
|
||||
assert ultra["graph_rerank"]["active"] is True
|
||||
assert ultra["graph_rerank"]["reason"] is None
|
||||
|
||||
again = (
|
||||
await client.put("/api/v1/settings/memory-features", json={"graph_rerank": True})
|
||||
).json()
|
||||
assert again["changed"] == []
|
||||
assert reads.current_features() == expected
|
||||
|
||||
reset = (
|
||||
await client.put(
|
||||
"/api/v1/settings/memory-features",
|
||||
json={"graph_rerank": None, "episode_recall": None},
|
||||
)
|
||||
).json()
|
||||
assert reset["changed"] == ["graph_rerank", "episode_recall"]
|
||||
assert reset["non_default"] == {}
|
||||
assert reads.current_features() == ()
|
||||
saved = json.loads((tmp_path / "data" / "config.json").read_text())
|
||||
assert saved["memory_features"] == {}
|
||||
|
||||
for bad in (
|
||||
{"local_rerank": True},
|
||||
{"graph_rerank": "true"},
|
||||
{"graph_rerank": 1},
|
||||
{"features": {"graph_rerank": True}},
|
||||
):
|
||||
response = await client.put("/api/v1/settings/memory-features", json=bad)
|
||||
assert response.status_code == 422, bad
|
||||
assert (
|
||||
await client.get("/api/v1/settings/memory-features", params={"user_id": "x"})
|
||||
).status_code == 422
|
||||
assert (await client.get("/api/v1/settings/memory-features")).json()["non_default"] == {}
|
||||
|
||||
run_app(tmp_path, monkeypatch, scenario)
|
||||
@@ -0,0 +1,861 @@
|
||||
"""The memory mode (lite / pro / ultra), evolve and its nightly schedule.
|
||||
|
||||
Item 5: the mode is a saved setting (default pro, `ONE_MEMORY_GRAPH_STORE`
|
||||
still forces ultra), it decides the mode every SDK client is built with, and
|
||||
a change rebuilds the handles. lite needs no generation model, never calls
|
||||
`add_extracted`, and keeps every raw. Item 6 (P1-3 / R7): `POST
|
||||
/api/v2/evolve` is ultra only, one run per user at a time, never beside that
|
||||
user's write, and loops while the SDK says `should_continue`; the schedule
|
||||
runs nightly and catches up after sleep. R10: the status carries
|
||||
`pipeline.mode` and the last evolve.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import SecretStr
|
||||
|
||||
from one_memory import hy_sdk_runtime, sdk_write_processor
|
||||
from one_memory.api import _sdk_client_handle, create_app, memory_mode_effects
|
||||
from one_memory.config import Settings
|
||||
from one_memory.hy_graph_admin import effective_memory_mode
|
||||
from one_memory.imports.models import (
|
||||
EvidenceContentKind,
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
materialize_write_input,
|
||||
)
|
||||
from one_memory.memory_evolution import (
|
||||
EvolveModeError,
|
||||
EvolveScheduler,
|
||||
EvolveService,
|
||||
EvolveStateStore,
|
||||
MemoryGraphBackfill,
|
||||
evolve_due,
|
||||
)
|
||||
from one_memory.memory_server.config_state import ConfigStore, EvolveSchedule
|
||||
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
|
||||
TOKEN = "memory-mode-test"
|
||||
|
||||
|
||||
# --- the setting and the client mode -----------------------------------------
|
||||
|
||||
|
||||
def test_mode_defaults_to_pro_and_the_env_still_forces_ultra(tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.delenv("ONE_MEMORY_GRAPH_STORE", raising=False)
|
||||
store = ConfigStore(tmp_path)
|
||||
assert store.memory_mode() == "pro"
|
||||
assert effective_memory_mode(store.memory_mode()) == "pro"
|
||||
store.set_memory_mode("lite")
|
||||
assert ConfigStore(tmp_path).memory_mode() == "lite"
|
||||
assert json.loads((tmp_path / "config.json").read_text())["memory_mode"] == "lite"
|
||||
monkeypatch.setenv("ONE_MEMORY_GRAPH_STORE", "kuzu")
|
||||
assert effective_memory_mode("lite") == "ultra"
|
||||
monkeypatch.setenv("ONE_MEMORY_GRAPH_STORE", "none")
|
||||
assert effective_memory_mode("lite") == "lite"
|
||||
with pytest.raises(ValueError):
|
||||
store.set_memory_mode("max") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_an_untouched_install_writes_no_new_settings(tmp_path) -> None:
|
||||
store = ConfigStore(tmp_path)
|
||||
store.set_sync("cursor", True)
|
||||
saved = json.loads((tmp_path / "config.json").read_text())
|
||||
assert "memory_mode" not in saved and "evolve_schedule" not in saved
|
||||
|
||||
|
||||
def test_acquire_client_builds_the_sdk_client_in_the_given_mode(tmp_path, monkeypatch) -> None:
|
||||
built: list[str] = []
|
||||
|
||||
class Client:
|
||||
def __init__(self, *, config, mode, runtime, embedding_service):
|
||||
built.append(mode)
|
||||
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(hy_sdk_runtime, "HyMemoryClient", Client)
|
||||
monkeypatch.setattr(hy_sdk_runtime, "_acquire_runtime", lambda config, key: object())
|
||||
monkeypatch.setattr(hy_sdk_runtime, "_release_runtime", lambda key: None)
|
||||
monkeypatch.setattr(hy_sdk_runtime, "inspect_graph_width_mismatch", lambda config: None)
|
||||
monkeypatch.delenv("ONE_MEMORY_GRAPH_STORE", raising=False)
|
||||
profile = hy_sdk_runtime.builtin_retrieval_snapshot()
|
||||
for mode in ("lite", "pro", "ultra"):
|
||||
hy_sdk_runtime.acquire_client(profile, api_key=None, data_dir=tmp_path, mode=mode).close()
|
||||
hy_sdk_runtime.acquire_client(profile, api_key=None, data_dir=tmp_path).close()
|
||||
monkeypatch.setenv("ONE_MEMORY_GRAPH_STORE", "kuzu")
|
||||
hy_sdk_runtime.acquire_client(profile, api_key=None, data_dir=tmp_path).close()
|
||||
assert built == ["lite", "pro", "ultra", "pro", "ultra"]
|
||||
|
||||
|
||||
def test_the_read_handle_rebuilds_on_a_mode_or_profile_change(monkeypatch, tmp_path) -> None:
|
||||
"""Reads follow the active profile's embedding (#255) and the memory mode."""
|
||||
from one_memory import sdk_read_handle
|
||||
|
||||
modes = ["pro"]
|
||||
made: list[_FakeHandle] = []
|
||||
|
||||
def handle_for(profile, *, api_key, data_dir, mode=None, features=()):
|
||||
handle = _FakeHandle(mode or "pro", profile.profile_id)
|
||||
made.append(handle)
|
||||
return handle
|
||||
|
||||
monkeypatch.setattr(sdk_read_handle, "SdkClientHandle", handle_for)
|
||||
profiles = _Profiles(identity=None)
|
||||
reads = sdk_read_handle.ActiveProfileReadHandle(
|
||||
profiles=profiles, # type: ignore[arg-type]
|
||||
secrets=None, # type: ignore[arg-type]
|
||||
data_dir=tmp_path,
|
||||
memory_mode=lambda: modes[0],
|
||||
)
|
||||
|
||||
async def read():
|
||||
return await reads.client()
|
||||
|
||||
async def scenario():
|
||||
first = await asyncio.create_task(read())
|
||||
assert (first.mode, first.profile_id) == ("pro", "builtin-retrieval")
|
||||
modes[0] = "lite"
|
||||
# A read still in flight on the old client keeps it open.
|
||||
hold = asyncio.Event()
|
||||
holding_client = asyncio.Event()
|
||||
|
||||
async def holding():
|
||||
await reads.client()
|
||||
holding_client.set()
|
||||
await hold.wait()
|
||||
|
||||
modes[0] = "pro"
|
||||
holder = asyncio.create_task(holding())
|
||||
await asyncio.wait_for(holding_client.wait(), 5)
|
||||
modes[0] = "lite"
|
||||
try:
|
||||
assert (await asyncio.create_task(read())).mode == "lite"
|
||||
assert not made[0].closed
|
||||
finally:
|
||||
hold.set()
|
||||
await holder
|
||||
for _ in range(10):
|
||||
await asyncio.sleep(0)
|
||||
assert made[0].closed
|
||||
assert [item.mode for item in made] == ["pro", "lite"]
|
||||
await reads.close()
|
||||
assert made[1].closed
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
class _FakeHandle:
|
||||
def __init__(self, mode: str, profile_id: str = "p") -> None:
|
||||
self.mode = mode
|
||||
self.profile_id = profile_id
|
||||
self.closed = False
|
||||
|
||||
async def client(self):
|
||||
return self
|
||||
|
||||
async def close(self):
|
||||
self.closed = True
|
||||
|
||||
|
||||
class _Profiles:
|
||||
def __init__(self, identity=("profile-a", 1)) -> None:
|
||||
self.identity = identity
|
||||
|
||||
def active_identity(self):
|
||||
return self.identity
|
||||
|
||||
def snapshot(self, profile_id, revision):
|
||||
return SimpleNamespace(profile_id=profile_id, has_api_key=False)
|
||||
|
||||
def resolve_snapshot_secret(self, profile_id, revision, secrets):
|
||||
return None
|
||||
|
||||
|
||||
def _coordinator(monkeypatch, modes, profiles=None):
|
||||
opened: list[_FakeHandle] = []
|
||||
|
||||
def handle(profile, **kwargs):
|
||||
value = _FakeHandle(kwargs.get("mode") or "pro", profile.profile_id)
|
||||
opened.append(value)
|
||||
return value
|
||||
|
||||
monkeypatch.setattr(sdk_write_processor, "SdkClientHandle", handle)
|
||||
coordinator = sdk_write_processor.SdkWriteCoordinator(
|
||||
profiles=profiles or _Profiles(), # type: ignore[arg-type]
|
||||
secrets=None, # type: ignore[arg-type]
|
||||
data_dir=None,
|
||||
transport_factory=lambda _profile: None,
|
||||
sedimentation_enabled=lambda: True,
|
||||
memory_mode=lambda: modes[0],
|
||||
)
|
||||
return coordinator, opened
|
||||
|
||||
|
||||
def test_a_mode_change_retires_idle_write_handles_and_keeps_a_busy_one(monkeypatch) -> None:
|
||||
modes = ["pro"]
|
||||
coordinator, opened = _coordinator(monkeypatch, modes)
|
||||
|
||||
async def scenario():
|
||||
async with coordinator.use_client() as client:
|
||||
assert client.mode == "pro"
|
||||
modes[0] = "ultra"
|
||||
await coordinator.retire_stale_modes()
|
||||
# Still in use: closed only when this use ends (#218).
|
||||
assert not opened[0].closed
|
||||
assert opened[0].closed
|
||||
async with coordinator.use_client() as client:
|
||||
assert client.mode == "ultra"
|
||||
modes[0] = "lite"
|
||||
await coordinator.retire_stale_modes()
|
||||
assert opened[1].closed
|
||||
await coordinator.close()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_lite_needs_no_generation_model(monkeypatch) -> None:
|
||||
modes = ["lite"]
|
||||
coordinator, opened = _coordinator(monkeypatch, modes, _Profiles(identity=None))
|
||||
|
||||
async def scenario():
|
||||
async with coordinator.use_client() as client:
|
||||
assert client.mode == "lite"
|
||||
assert client.profile_id == hy_sdk_runtime.BUILTIN_RETRIEVAL_PROFILE_ID
|
||||
modes[0] = "pro"
|
||||
with pytest.raises(RuntimeError, match="model_profile_not_active"):
|
||||
async with coordinator.use_client():
|
||||
pass
|
||||
await coordinator.close()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def _record(text: str, *, extracted: bool, user_id: str = "local", role: str = "user"):
|
||||
record = materialize_write_input(
|
||||
EvidenceInput(
|
||||
user_id=user_id,
|
||||
source_id="one-memory-manual" if extracted else "hermes",
|
||||
conversation_id="session-1",
|
||||
message_id=str(uuid4()),
|
||||
role=role,
|
||||
written_by=WrittenBy.PERSON,
|
||||
created_at=datetime(2026, 10, 1, tzinfo=UTC),
|
||||
verbatim=text,
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
)
|
||||
if extracted:
|
||||
record = record.model_copy(update={"content_kind": EvidenceContentKind.MEMORY_CONTENT})
|
||||
return record
|
||||
|
||||
|
||||
class _WritingSdk:
|
||||
def __init__(self) -> None:
|
||||
self.calls: list[tuple[str, object]] = []
|
||||
|
||||
def add(self, messages, **kwargs):
|
||||
self.calls.append(("add", messages))
|
||||
return {"success": True, "raw_ids": ["raw-1"]}
|
||||
|
||||
def add_extracted(self, contents, **kwargs):
|
||||
self.calls.append(("add_extracted", contents))
|
||||
return {"success": True, "memory_ids": ["m-1"]}
|
||||
|
||||
|
||||
class _SdkHandle:
|
||||
def __init__(self, sdk, mode="pro") -> None:
|
||||
self.sdk = sdk
|
||||
self.mode = mode
|
||||
|
||||
async def client(self):
|
||||
return self.sdk
|
||||
|
||||
async def close(self):
|
||||
return None
|
||||
|
||||
|
||||
async def _noop() -> None:
|
||||
return None
|
||||
|
||||
|
||||
def test_a_note_in_lite_is_stored_as_raw_and_never_calls_add_extracted() -> None:
|
||||
sdk = _WritingSdk()
|
||||
note = (_record("I prefer green tea", extracted=True),)
|
||||
|
||||
async def scenario():
|
||||
await sdk_write_processor.SdkWriteProcessor(_SdkHandle(sdk), lite=True).process_sdk_job(
|
||||
note, request_id="r-lite", before_hy_invocation=_noop
|
||||
)
|
||||
await sdk_write_processor.SdkWriteProcessor(_SdkHandle(sdk)).process_sdk_job(
|
||||
note, request_id="r-pro", before_hy_invocation=_noop
|
||||
)
|
||||
|
||||
asyncio.run(scenario())
|
||||
assert sdk.calls == [
|
||||
("add", [{"role": "user", "content": "I prefer green tea"}]),
|
||||
("add_extracted", ["I prefer green tea"]),
|
||||
]
|
||||
|
||||
|
||||
def test_a_lite_write_runs_through_the_coordinator_without_a_model(monkeypatch) -> None:
|
||||
sdk = _WritingSdk()
|
||||
coordinator = sdk_write_processor.SdkWriteCoordinator(
|
||||
profiles=_Profiles(identity=None), # type: ignore[arg-type]
|
||||
secrets=None, # type: ignore[arg-type]
|
||||
data_dir=None,
|
||||
transport_factory=lambda _profile: None,
|
||||
sedimentation_enabled=lambda: True,
|
||||
memory_mode=lambda: "lite",
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sdk_write_processor, "SdkClientHandle", lambda profile, **kwargs: _SdkHandle(sdk, "lite")
|
||||
)
|
||||
turn = (
|
||||
_record("hello", extracted=False),
|
||||
_record("hi there", extracted=False, role="assistant"),
|
||||
)
|
||||
receipt = asyncio.run(
|
||||
coordinator.process_sdk_job(turn, request_id="r-1", before_hy_invocation=_noop)
|
||||
)
|
||||
assert receipt.raw_ids == ("raw-1",)
|
||||
assert [name for name, _ in sdk.calls] == ["add"]
|
||||
|
||||
|
||||
def test_no_raw_is_ever_deleted_in_any_mode() -> None:
|
||||
"""Owner decision: MemHub keeps every raw; there is no retention window."""
|
||||
from one_memory import sdk_memory_runtime
|
||||
|
||||
for name in ("prune_expired_raw", "start_raw_retention", "_raw_retention_loop"):
|
||||
assert not hasattr(SdkMemoryRuntime, name), name
|
||||
assert not hasattr(sdk_memory_runtime, "RAW_RETENTION_DAYS")
|
||||
|
||||
|
||||
def test_mode_switch_effects() -> None:
|
||||
assert memory_mode_effects("lite", "pro") == (
|
||||
"earlier_raw_not_extracted",
|
||||
"lite_search_stops",
|
||||
)
|
||||
assert memory_mode_effects("ultra", "pro") == ("higher_mode_memories_hidden", "evolve_stopped")
|
||||
assert memory_mode_effects("pro", "ultra", graph_backfill=True) == (
|
||||
"existing_memories_added_to_graph",
|
||||
)
|
||||
assert memory_mode_effects("pro", "ultra", graph_backfill=False) == (
|
||||
"graph_backfill_already_running",
|
||||
)
|
||||
assert memory_mode_effects("pro", "lite") == ("higher_mode_memories_hidden",)
|
||||
assert memory_mode_effects("pro", "pro") == ()
|
||||
|
||||
|
||||
# --- evolve ---------------------------------------------------------------------
|
||||
|
||||
|
||||
class _EvolveSdk:
|
||||
"""`evolve` answers `should_continue` for the first rounds, then stops."""
|
||||
|
||||
def __init__(self, continuing: int = 2, *, gate: asyncio.Event | None = None) -> None:
|
||||
self.calls: list[dict] = []
|
||||
self.continuing = continuing
|
||||
self.gate = gate
|
||||
self.graph_builds: list[str] = []
|
||||
self.active = 0
|
||||
self.peak = 0
|
||||
|
||||
def evolve(self, user_id, agent_id="default_agent", max_batches=None):
|
||||
self.active += 1
|
||||
self.peak = max(self.peak, self.active)
|
||||
try:
|
||||
self.calls.append(
|
||||
{"user_id": user_id, "agent_id": agent_id, "max_batches": max_batches}
|
||||
)
|
||||
more = len([call for call in self.calls if call["user_id"] == user_id]) <= (
|
||||
self.continuing
|
||||
)
|
||||
return {
|
||||
"success": True,
|
||||
"request_id": f"evolve-{len(self.calls)}",
|
||||
"tasks_processed": 2,
|
||||
"results": {"schemas": ["private text"]},
|
||||
"cross_domain_sweeper": {"merged": 1},
|
||||
"should_continue": more,
|
||||
"continuation_reason": "remaining_clusters" if more else None,
|
||||
"remaining_clusters": 3 if more else 0,
|
||||
"remaining_actionable_clusters": 1 if more else 0,
|
||||
"remaining_fresh_facts": 0,
|
||||
"cycle_status": "partial" if more else "complete",
|
||||
"batch_failures": [],
|
||||
"elapsed_ms": 5.0,
|
||||
}
|
||||
finally:
|
||||
self.active -= 1
|
||||
|
||||
def build_memory_graph(self, user_id, *, agent_id=None, limit=10000, clear_first=False):
|
||||
self.graph_builds.append(user_id)
|
||||
return {"success": True, "memories_scanned": 4, "nodes": 4}
|
||||
|
||||
|
||||
class _Coordinator:
|
||||
"""The real per-user exclusivity over a stand-in client."""
|
||||
|
||||
def __init__(self, sdk, monkeypatch) -> None:
|
||||
self.inner, _ = _coordinator(monkeypatch, ["ultra"])
|
||||
self.sdk = sdk
|
||||
|
||||
def exclusive_for_evolve(self, user_id):
|
||||
return self.inner.exclusive_for_evolve(user_id)
|
||||
|
||||
@asynccontextmanager
|
||||
async def use_client(self):
|
||||
yield self.sdk
|
||||
|
||||
|
||||
def test_evolve_refuses_outside_ultra(tmp_path, monkeypatch) -> None:
|
||||
service = EvolveService(
|
||||
coordinator=_Coordinator(_EvolveSdk(), monkeypatch),
|
||||
state=EvolveStateStore(tmp_path),
|
||||
memory_mode=lambda: "pro",
|
||||
)
|
||||
|
||||
async def scenario():
|
||||
with pytest.raises(EvolveModeError):
|
||||
service.start("local")
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_evolve_loops_while_should_continue_and_records_counts_only(tmp_path, monkeypatch) -> None:
|
||||
sdk = _EvolveSdk(continuing=2)
|
||||
state = EvolveStateStore(tmp_path)
|
||||
service = EvolveService(
|
||||
coordinator=_Coordinator(sdk, monkeypatch), state=state, memory_mode=lambda: "ultra"
|
||||
)
|
||||
|
||||
async def scenario():
|
||||
started = service.start("wife", max_batches=2)
|
||||
assert started.state == "running"
|
||||
done = await service.wait(started.run_id)
|
||||
assert done is not None
|
||||
return done
|
||||
|
||||
done = asyncio.run(scenario())
|
||||
assert done.state == "succeeded" and done.rounds == 3 and done.tasks_processed == 6
|
||||
assert done.cycle_status == "complete" and done.should_continue is False
|
||||
assert {call["agent_id"] for call in sdk.calls} == {"all-apps"}
|
||||
assert {call["max_batches"] for call in sdk.calls} == {2}
|
||||
saved = (tmp_path / "evolve-state.json").read_text()
|
||||
assert "private text" not in saved
|
||||
assert json.loads(saved)["users"]["wife"]["last_result"]["rounds"] == 3
|
||||
|
||||
|
||||
def test_evolve_stops_at_the_round_cap(tmp_path, monkeypatch) -> None:
|
||||
sdk = _EvolveSdk(continuing=100)
|
||||
service = EvolveService(
|
||||
coordinator=_Coordinator(sdk, monkeypatch),
|
||||
state=EvolveStateStore(tmp_path),
|
||||
memory_mode=lambda: "ultra",
|
||||
max_rounds=4,
|
||||
)
|
||||
|
||||
async def scenario():
|
||||
return await service.wait(service.start("local").run_id)
|
||||
|
||||
done = asyncio.run(scenario())
|
||||
assert done is not None and done.rounds == 4 and done.capped is True
|
||||
assert done.state == "succeeded"
|
||||
|
||||
|
||||
def test_one_run_per_user_and_never_beside_that_users_write(tmp_path, monkeypatch) -> None:
|
||||
sdk = _EvolveSdk(continuing=0)
|
||||
coordinator = _Coordinator(sdk, monkeypatch)
|
||||
service = EvolveService(
|
||||
coordinator=coordinator, state=EvolveStateStore(tmp_path), memory_mode=lambda: "ultra"
|
||||
)
|
||||
writing = asyncio.Event()
|
||||
release = asyncio.Event()
|
||||
|
||||
async def fake_write(self, records, *, request_id, before_hy_invocation):
|
||||
writing.set()
|
||||
await release.wait()
|
||||
return sdk_write_processor.SdkWriteReceipt()
|
||||
|
||||
monkeypatch.setattr(sdk_write_processor.SdkWriteProcessor, "process_sdk_job", fake_write)
|
||||
record = (_record("hello", extracted=False, user_id="wife"),)
|
||||
|
||||
async def scenario():
|
||||
inner = coordinator.inner
|
||||
write = asyncio.create_task(
|
||||
inner.process_sdk_job(record, request_id="w", before_hy_invocation=_noop)
|
||||
)
|
||||
await writing.wait()
|
||||
first = service.start("wife")
|
||||
again = service.start("wife")
|
||||
assert again.run_id == first.run_id
|
||||
other = service.start("local")
|
||||
await service.wait(other.run_id)
|
||||
# The other user is not held; wife's run waits for wife's write.
|
||||
assert sdk.calls and {call["user_id"] for call in sdk.calls} == {"local"}
|
||||
assert inner.evolving("wife")
|
||||
with pytest.raises(RuntimeError, match="memory_evolve_in_progress"):
|
||||
await inner.process_sdk_job(record, request_id="w2", before_hy_invocation=_noop)
|
||||
release.set()
|
||||
await write
|
||||
done = await service.wait(first.run_id)
|
||||
assert done is not None and done.state == "succeeded"
|
||||
assert not inner.evolving("wife")
|
||||
assert sdk.peak == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_failed_round_records_its_code(tmp_path, monkeypatch) -> None:
|
||||
class Failing(_EvolveSdk):
|
||||
def evolve(self, user_id, agent_id="default_agent", max_batches=None):
|
||||
return {
|
||||
"success": False,
|
||||
"error_code": "schema_llm_failed",
|
||||
"failure_kind": "llm_error",
|
||||
"error_message": "the provider said something private",
|
||||
}
|
||||
|
||||
state = EvolveStateStore(tmp_path)
|
||||
service = EvolveService(
|
||||
coordinator=_Coordinator(Failing(), monkeypatch), state=state, memory_mode=lambda: "ultra"
|
||||
)
|
||||
done = asyncio.run(service.wait(_start(service, "local")))
|
||||
assert done is not None and done.state == "failed"
|
||||
assert done.error_code == "schema_llm_failed" and done.failure_kind == "llm_error"
|
||||
assert state.user("local").last_error == "schema_llm_failed"
|
||||
assert "private" not in (tmp_path / "evolve-state.json").read_text()
|
||||
|
||||
|
||||
def _start(service: EvolveService, user_id: str) -> str:
|
||||
async def begin() -> str:
|
||||
return service.start(user_id).run_id
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
run_id = loop.run_until_complete(begin())
|
||||
loop.run_until_complete(service.wait(run_id))
|
||||
finally:
|
||||
loop.close()
|
||||
return run_id
|
||||
|
||||
|
||||
# --- the nightly schedule -------------------------------------------------------
|
||||
|
||||
LOCAL = datetime.now().astimezone().tzinfo
|
||||
|
||||
|
||||
def _at(day: int, hour: int, minute: int = 0) -> datetime:
|
||||
return datetime(2026, 11, day, hour, minute, tzinfo=LOCAL)
|
||||
|
||||
|
||||
def test_evolve_is_due_after_the_slot_once_a_day_and_catches_up_after_sleep() -> None:
|
||||
schedule = EvolveSchedule()
|
||||
anchor = _at(1, 12)
|
||||
# Never ran: not at noon on the day ultra was enabled, then at 03:00 next day.
|
||||
assert not evolve_due(_at(1, 13), schedule, None, anchor)
|
||||
assert not evolve_due(_at(2, 2, 59), schedule, None, anchor)
|
||||
assert evolve_due(_at(2, 3, 0), schedule, None, anchor)
|
||||
# Ran at 03:00: not again that day, due again the next night.
|
||||
assert not evolve_due(_at(2, 22), schedule, _at(2, 3, 1), anchor)
|
||||
assert evolve_due(_at(3, 3, 5), schedule, _at(2, 3, 1), anchor)
|
||||
# Asleep at 03:00: a run older than the slot is due on wake, the same day.
|
||||
assert evolve_due(_at(3, 9), schedule, _at(2, 3, 1), anchor)
|
||||
# Last run more than 24h ago is due even before today's slot.
|
||||
assert evolve_due(_at(4, 1), schedule, _at(2, 3, 1), anchor)
|
||||
assert not evolve_due(_at(3, 1), schedule, _at(2, 3, 1), anchor)
|
||||
# Moved and turned off.
|
||||
assert evolve_due(_at(2, 23, 30), EvolveSchedule(time="23:30"), _at(2, 3), anchor)
|
||||
assert not evolve_due(_at(3, 9), EvolveSchedule(enabled=False), _at(1, 3), anchor)
|
||||
|
||||
|
||||
def test_the_scheduler_runs_each_due_user_once_in_ultra_only(tmp_path, monkeypatch) -> None:
|
||||
sdk = _EvolveSdk(continuing=0)
|
||||
state = EvolveStateStore(tmp_path)
|
||||
modes = ["pro"]
|
||||
clock = [_at(1, 12)]
|
||||
service = EvolveService(
|
||||
coordinator=_Coordinator(sdk, monkeypatch),
|
||||
state=state,
|
||||
memory_mode=lambda: modes[0],
|
||||
now=lambda: clock[0],
|
||||
)
|
||||
schedule = [EvolveSchedule()]
|
||||
|
||||
async def users():
|
||||
return ("local", "wife")
|
||||
|
||||
scheduler = EvolveScheduler(
|
||||
service=service,
|
||||
schedule=lambda: schedule[0],
|
||||
users_with_memories=users,
|
||||
now=lambda: clock[0],
|
||||
)
|
||||
|
||||
async def scenario():
|
||||
assert await scheduler.tick() == () # pro: nothing
|
||||
modes[0] = "ultra"
|
||||
assert await scheduler.tick() == () # anchored at noon
|
||||
clock[0] = _at(2, 3, 2)
|
||||
assert await scheduler.tick() == ("local", "wife")
|
||||
assert await scheduler.tick() == () # once a night
|
||||
clock[0] = _at(3, 8) # slept through 03:00
|
||||
assert await scheduler.tick() == ("local", "wife")
|
||||
schedule[0] = EvolveSchedule(enabled=False)
|
||||
clock[0] = _at(5, 8)
|
||||
assert await scheduler.tick() == ()
|
||||
|
||||
asyncio.run(scenario())
|
||||
assert [call["user_id"] for call in sdk.calls] == ["local", "wife", "local", "wife"]
|
||||
assert state.user("wife").last_result["trigger"] == "scheduled"
|
||||
|
||||
|
||||
def test_the_graph_backfill_runs_once_per_user(tmp_path, monkeypatch) -> None:
|
||||
sdk = _EvolveSdk()
|
||||
state = EvolveStateStore(tmp_path)
|
||||
backfill = MemoryGraphBackfill(coordinator=_Coordinator(sdk, monkeypatch), state=state)
|
||||
|
||||
async def scenario():
|
||||
backfill.start(["local", "wife", "local"])
|
||||
await backfill.wait()
|
||||
|
||||
asyncio.run(scenario())
|
||||
assert sdk.graph_builds == ["local", "wife"]
|
||||
built = state.graph_build()
|
||||
assert built.state == "succeeded"
|
||||
assert {user: value.state for user, value in built.users.items()} == {
|
||||
"local": "succeeded",
|
||||
"wife": "succeeded",
|
||||
}
|
||||
|
||||
|
||||
# --- the HTTP surface -----------------------------------------------------------
|
||||
|
||||
|
||||
def application(tmp_path: Path):
|
||||
settings = Settings(
|
||||
data_dir=tmp_path / "data",
|
||||
source_home=tmp_path,
|
||||
api_token=SecretStr(TOKEN),
|
||||
imports_enabled=False,
|
||||
automatic_memory_processing=False,
|
||||
live_sync_enabled=False,
|
||||
)
|
||||
return create_app(settings, secret_store=InMemorySecretStore())
|
||||
|
||||
|
||||
def client_for(app) -> AsyncClient:
|
||||
return AsyncClient(
|
||||
transport=ASGITransport(app=app),
|
||||
base_url="http://test",
|
||||
headers={"Authorization": f"Bearer {TOKEN}"},
|
||||
)
|
||||
|
||||
|
||||
class _ReadSdk(_EvolveSdk):
|
||||
def memory_stats(self, *, user_id, statuses=None):
|
||||
return {"total": 1, "scenes": {}}
|
||||
|
||||
|
||||
def run_app(tmp_path, monkeypatch, scenario) -> None:
|
||||
monkeypatch.delenv("ONE_MEMORY_GRAPH_STORE", raising=False)
|
||||
|
||||
async def main() -> None:
|
||||
app = application(tmp_path)
|
||||
sdk = _ReadSdk(continuing=1)
|
||||
|
||||
@asynccontextmanager
|
||||
async def write_client():
|
||||
yield sdk
|
||||
|
||||
async with app.router.lifespan_context(app):
|
||||
monkeypatch.setattr(app.state.sdk_write_coordinator, "use_client", write_client)
|
||||
async with client_for(app) as client:
|
||||
await scenario(app, sdk, client)
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_mode_api_get_put_and_its_effects(tmp_path, monkeypatch) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
current = (await client.get("/api/v1/settings/memory-mode")).json()
|
||||
assert current["mode"] == current["configured_mode"] == "pro"
|
||||
assert current["forced_by_env"] is False
|
||||
assert app.state.sdk_write_coordinator.memory_mode() == "pro"
|
||||
|
||||
lite = await client.put("/api/v1/settings/memory-mode", json={"mode": "lite"})
|
||||
assert lite.status_code == 200, lite.text
|
||||
body = lite.json()
|
||||
assert body["mode"] == "lite" and body["previous_mode"] == "pro"
|
||||
assert body["effects"] == ["higher_mode_memories_hidden"]
|
||||
assert app.state.sdk_write_coordinator.memory_mode() == "lite"
|
||||
assert _sdk_client_handle(app).memory_mode() == "lite"
|
||||
# lite claims queued work without a generation model.
|
||||
from one_memory.api import _sedimentation_claim_enabled
|
||||
|
||||
assert app.state.model_profile_store.active_identity() is None
|
||||
app.state.history_import_gate_released = True
|
||||
assert _sedimentation_claim_enabled(app, "one-memory-manual")
|
||||
|
||||
ultra = await client.put("/api/v1/settings/memory-mode", json={"mode": "ultra"})
|
||||
assert ultra.status_code == 200, ultra.text
|
||||
assert "existing_memories_added_to_graph" in ultra.json()["effects"]
|
||||
await app.state.memory_graph_backfill.wait()
|
||||
assert sdk.graph_builds == ["local"]
|
||||
reread = (await client.get("/api/v1/settings/memory-mode")).json()
|
||||
assert reread["graph_build"]["state"] == "succeeded"
|
||||
|
||||
for bad in ({"mode": "max"}, {"mode": "pro", "x": 1}):
|
||||
assert (await client.put("/api/v1/settings/memory-mode", json=bad)).status_code == 422
|
||||
assert (
|
||||
await client.get("/api/v1/settings/memory-mode", params={"user_id": "x"})
|
||||
).status_code == 422
|
||||
|
||||
monkeypatch.setenv("ONE_MEMORY_GRAPH_STORE", "kuzu")
|
||||
await client.put("/api/v1/settings/memory-mode", json={"mode": "pro"})
|
||||
forced = (await client.get("/api/v1/settings/memory-mode")).json()
|
||||
assert forced["mode"] == "ultra" and forced["configured_mode"] == "pro"
|
||||
assert forced["forced_by_env"] is True
|
||||
|
||||
run_app(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_evolve_api_is_ultra_only_async_and_scoped_per_user(tmp_path, monkeypatch) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
refused = await client.post("/api/v2/evolve", json={})
|
||||
assert refused.status_code == 409
|
||||
assert refused.json()["detail"] == "memory_mode_not_ultra"
|
||||
|
||||
await client.put("/api/v1/settings/memory-mode", json={"mode": "ultra"})
|
||||
await app.state.memory_graph_backfill.wait()
|
||||
started = await client.post(
|
||||
"/api/v2/evolve", json={"user_id": "wife", "agent_id": "all-apps", "max_batches": 2}
|
||||
)
|
||||
assert started.status_code == 202, started.text
|
||||
run = started.json()
|
||||
assert run["state"] == "running" and run["user_id"] == "wife"
|
||||
await app.state.evolve_service.wait(run["run_id"])
|
||||
path = f"/api/v2/evolve/{run['run_id']}"
|
||||
assert (await client.get(path)).status_code == 404
|
||||
done = (await client.get(path, params={"user_id": "wife"})).json()
|
||||
assert done["state"] == "succeeded" and done["rounds"] == 2
|
||||
assert sdk.calls[0] == {"user_id": "wife", "agent_id": "all-apps", "max_batches": 2}
|
||||
|
||||
overview = (await client.get("/api/v2/evolve", params={"user_id": "wife"})).json()
|
||||
assert set(overview["users"]) == {"wife"}
|
||||
assert overview["schedule"] == {"enabled": True, "time": "03:00"}
|
||||
|
||||
processing = (await client.get("/api/v1/memory-processing")).json()
|
||||
pipeline = processing["pipeline"]
|
||||
assert pipeline["mode"] == "ultra"
|
||||
assert pipeline["last_evolve_at"] is not None
|
||||
assert pipeline["evolve"]["users"]["wife"]["last_result"]["state"] == "succeeded"
|
||||
|
||||
for bad in (
|
||||
{"user_id": "a:b"},
|
||||
{"max_batches": 0},
|
||||
{"from_beginning": True},
|
||||
):
|
||||
assert (await client.post("/api/v2/evolve", json=bad)).status_code == 422
|
||||
|
||||
moved = await client.put(
|
||||
"/api/v1/settings/evolve-schedule", json={"time": "04:30", "enabled": False}
|
||||
)
|
||||
assert moved.json() == {"enabled": False, "time": "04:30"}
|
||||
assert (
|
||||
await client.put("/api/v1/settings/evolve-schedule", json={"time": "25:00"})
|
||||
).status_code == 422
|
||||
|
||||
run_app(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_a_job_written_in_lite_stays_done_and_a_pending_job_takes_the_new_mode(
|
||||
tmp_path, monkeypatch
|
||||
) -> None:
|
||||
"""lite to pro reprocesses nothing: the mode is read when a job runs."""
|
||||
from one_memory.memory_server.durable_queue import DurableWriteDrain, execute_job
|
||||
|
||||
modes = ["lite"]
|
||||
seen: list[str] = []
|
||||
|
||||
class Sdk:
|
||||
def __init__(self, mode):
|
||||
self.mode = mode
|
||||
|
||||
def add(self, messages, **kwargs):
|
||||
seen.append(self.mode)
|
||||
return {"success": True, "raw_ids": [f"raw-{len(seen)}"]}
|
||||
|
||||
monkeypatch.setattr(
|
||||
sdk_write_processor,
|
||||
"SdkClientHandle",
|
||||
lambda profile, **kwargs: _SdkHandle(Sdk(kwargs["mode"]), kwargs["mode"]),
|
||||
)
|
||||
coordinator = sdk_write_processor.SdkWriteCoordinator(
|
||||
profiles=_Profiles(), # type: ignore[arg-type]
|
||||
secrets=None, # type: ignore[arg-type]
|
||||
data_dir=None,
|
||||
transport_factory=lambda _profile: None,
|
||||
sedimentation_enabled=lambda: True,
|
||||
memory_mode=lambda: modes[0],
|
||||
)
|
||||
|
||||
async def never(_job_id):
|
||||
return None
|
||||
|
||||
async def drain_once(queue) -> int:
|
||||
drain = DurableWriteDrain(queue, execute=never, lease=timedelta(seconds=30))
|
||||
claimed = drain.claim(limit=8)
|
||||
for job_id in claimed:
|
||||
await execute_job(
|
||||
job_id,
|
||||
queue=queue,
|
||||
supervisor=coordinator,
|
||||
evidence=None, # type: ignore[arg-type]
|
||||
)
|
||||
return len(claimed)
|
||||
|
||||
async def scenario(app, _sdk, client):
|
||||
queue = app.state.memory_write_queue
|
||||
for request_id in ("turn-lite", "turn-pro"):
|
||||
written = await client.post(
|
||||
"/api/v2/memory-writes",
|
||||
json={
|
||||
"request_id": request_id,
|
||||
"session_id": f"s-{request_id}",
|
||||
"source_id": "one-memory-manual",
|
||||
"messages": [
|
||||
{"role": "user", "content": request_id},
|
||||
{"role": "assistant", "content": "ok"},
|
||||
],
|
||||
},
|
||||
)
|
||||
assert written.status_code == 202, written.text
|
||||
if request_id == "turn-lite":
|
||||
assert await drain_once(queue) == 1
|
||||
modes[0] = "pro"
|
||||
await coordinator.retire_stale_modes()
|
||||
assert await drain_once(queue) == 1
|
||||
assert await drain_once(queue) == 0
|
||||
assert queue.get("turn-lite").state == queue.get("turn-pro").state == "completed"
|
||||
assert seen == ["lite", "pro"]
|
||||
await coordinator.close()
|
||||
|
||||
run_app(tmp_path, monkeypatch, scenario)
|
||||
@@ -0,0 +1,744 @@
|
||||
"""The SDK surface MemHub exposes: search pass-through, layers, raw, v2 aliases.
|
||||
|
||||
P2-1 (R8): `context/search`, `recall` and the v2 ask routes pass the public
|
||||
`client.search` knobs through, validated, and keep a caller's floor on what
|
||||
they return. P2-2 (R9): the library honours `layers`, `statuses`, `scene` and
|
||||
the source/session/agent filters, and counts memories per layer. P2-4: raw
|
||||
lists the SDK's L1 raw turns, active and shadow. P2-3: v2 aliases of the
|
||||
source and import routes, per user. Profile versions read the L0 chain.
|
||||
|
||||
Every route stays scoped to its one `user_id`; the stand-in client records
|
||||
each call so a test can see exactly what reached the SDK.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import sqlite3
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import SecretStr
|
||||
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
from one_memory.imports.sources.registry import build_source_catalog
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
|
||||
TOKEN = "sdk-surface-test"
|
||||
|
||||
CHAT_ROWS = {
|
||||
"local": [
|
||||
("m-high", "local likes green tea", 0.95, "l2_fact", "hermes"),
|
||||
("m-mid", "local walks at dawn", 0.6, "l2_fact", "codex"),
|
||||
],
|
||||
"wife": [("w-high", "wife likes coffee", 0.92, "l2_fact", "hermes")],
|
||||
}
|
||||
RAW_ROWS = {
|
||||
"local": [
|
||||
{
|
||||
"memory_id": "raw-1",
|
||||
"content": "[user]: 我叫费天翔\n[assistant]: 好的,记住了。",
|
||||
"status": "shadow",
|
||||
"session_id": "s-1",
|
||||
"source": "hermes",
|
||||
"memory_at": 1_790_000_000,
|
||||
"gmt_created": 1_790_000_100,
|
||||
},
|
||||
{
|
||||
"memory_id": "raw-2",
|
||||
"content": "[user]: 你好",
|
||||
"status": "active",
|
||||
"session_id": "s-2",
|
||||
"source": "codex",
|
||||
"memory_at": 1_790_100_000,
|
||||
"gmt_created": 1_790_100_100,
|
||||
},
|
||||
],
|
||||
"wife": [
|
||||
{
|
||||
"memory_id": "raw-w",
|
||||
"content": "[user]: I am the wife",
|
||||
"status": "active",
|
||||
"session_id": "s-w",
|
||||
"source": "hermes",
|
||||
"memory_at": 1_790_000_000,
|
||||
"gmt_created": 1_790_000_000,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _chat(user_id: str, memory_id: str, content: str, layer: str, source: str) -> dict:
|
||||
return {
|
||||
"memory_id": memory_id,
|
||||
"content": content,
|
||||
"layer": layer,
|
||||
"status": "active",
|
||||
"user_id": user_id,
|
||||
"agent_id": "all-apps",
|
||||
"session_id": "s-1",
|
||||
"source": source,
|
||||
"tags": [],
|
||||
"gmt_created": 1_790_000_000,
|
||||
}
|
||||
|
||||
|
||||
class RecordingSdk:
|
||||
"""The public client surface, recording every call's keyword arguments."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.calls: list[tuple[str, dict]] = []
|
||||
|
||||
def _record(self, name: str, kwargs: dict) -> None:
|
||||
self.calls.append((name, dict(kwargs)))
|
||||
|
||||
def last(self, name: str) -> dict:
|
||||
return [kwargs for called, kwargs in self.calls if called == name][-1]
|
||||
|
||||
def all(self, name: str) -> list[dict]:
|
||||
return [kwargs for called, kwargs in self.calls if called == name]
|
||||
|
||||
# The SDK answers rows below a caller's floor in some lanes (a schema
|
||||
# floor, a fused rank score), so Core must filter what it returns.
|
||||
def search(self, query, **kwargs):
|
||||
self._record("search", {"query": query, **kwargs})
|
||||
[user_id] = kwargs["user_ids"]
|
||||
if kwargs.get("layers") == ["l1_raw"]:
|
||||
hits = [
|
||||
{**row, "user_id": user_id, "agent_id": "all-apps", "score": score}
|
||||
for row, score in zip(RAW_ROWS.get(user_id, []), (0.97, 0.3), strict=False)
|
||||
]
|
||||
return {"request_id": "s", "memories": hits}
|
||||
rows = [
|
||||
{**_chat(user_id, mid, text, layer, source), "score": score}
|
||||
for mid, text, score, layer, source in CHAT_ROWS.get(user_id, [])
|
||||
]
|
||||
scene = kwargs.get("scene")
|
||||
if scene == "all":
|
||||
fused = [
|
||||
{**row, "scene": "chat", "kind": "chat", "lane_score": row["score"], "score": 0.016}
|
||||
for row in rows
|
||||
]
|
||||
return {"request_id": "s", "scene": "all", "memories": fused, "lanes": {}, "errors": {}}
|
||||
if scene in {"working", "coding"}:
|
||||
return {
|
||||
"request_id": "s",
|
||||
"scene": "productivity",
|
||||
"variant": scene,
|
||||
"facts": [
|
||||
{"fact_id": "f-1", "statement": "use uv", "score": 0.91, "user_id": user_id}
|
||||
],
|
||||
"experiences": [],
|
||||
}
|
||||
return {
|
||||
"request_id": "s",
|
||||
"scene": "normal",
|
||||
"memories": {"normal": rows, "schema": [], "proactive": []},
|
||||
}
|
||||
|
||||
def get(self, memory_id, kind=None, variant=None):
|
||||
self._record("get", {"memory_id": memory_id, "kind": kind, "variant": variant})
|
||||
if kind == "fact":
|
||||
return {
|
||||
"kind": "fact",
|
||||
"variant": variant,
|
||||
"fact_id": memory_id,
|
||||
"statement": "use uv",
|
||||
"status": "active",
|
||||
"user_id": "local",
|
||||
}
|
||||
for user_id, rows in CHAT_ROWS.items():
|
||||
for mid, text, _score, layer, source in rows:
|
||||
if mid == memory_id:
|
||||
return {**_chat(user_id, mid, text, layer, source), "kind": "chat"}
|
||||
return None
|
||||
|
||||
def list_memories(self, **kwargs):
|
||||
self._record("list_memories", kwargs)
|
||||
user_id = kwargs["user_id"]
|
||||
layers = kwargs.get("layers")
|
||||
if layers == ["l1_raw"]:
|
||||
statuses = set(kwargs.get("statuses") or ["active"])
|
||||
rows = [
|
||||
{**row, "layer": "l1_raw", "user_id": user_id, "agent_id": "all-apps"}
|
||||
for row in RAW_ROWS.get(user_id, [])
|
||||
if row["status"] in statuses
|
||||
and (
|
||||
not kwargs.get("include_sources") or row["source"] in kwargs["include_sources"]
|
||||
)
|
||||
and (not kwargs.get("session_ids") or row["session_id"] in kwargs["session_ids"])
|
||||
]
|
||||
return {"vdb": {"memories": rows, "total": len(rows)}}
|
||||
rows = [_chat(user_id, *row[:2], *row[3:]) for row in CHAT_ROWS.get(user_id, [])]
|
||||
if layers:
|
||||
rows = [row for row in rows if row["layer"] in layers]
|
||||
if "l0_basic_info" in layers:
|
||||
rows.append(_chat(user_id, "l0-1", "name is Fei", "l0_basic_info", "hermes"))
|
||||
if kwargs.get("scene") == "all":
|
||||
return {
|
||||
"scene": "all",
|
||||
"memories": [{**row, "scene": "chat", "kind": "chat"} for row in rows],
|
||||
"total": len(rows),
|
||||
"totals": {"chat": len(rows), "working": 0, "coding": 0},
|
||||
"errors": {},
|
||||
}
|
||||
return {"vdb": {"memories": rows, "total": len(rows)}}
|
||||
|
||||
def memory_stats(self, *, user_id, statuses=None):
|
||||
return {"total": 0, "request_id": "stats", "elapsed_ms": 1.0}
|
||||
|
||||
def get_user_basic_info(self, *, user_id):
|
||||
self._record("get_user_basic_info", {"user_id": user_id})
|
||||
return {
|
||||
"request_id": f"basic-{user_id}",
|
||||
"elapsed_ms": 1.5,
|
||||
"memories": [
|
||||
{
|
||||
"memory_id": f"{user_id}-l0-head",
|
||||
"content": "The user's name is Fei.",
|
||||
"layer": "l0_basic_info",
|
||||
"status": "active",
|
||||
"memory_at": 1_790_000_100,
|
||||
"gmt_created": 1_790_000_100,
|
||||
"tags": [],
|
||||
"basic_info_kv": {"name": "Fei"},
|
||||
"evolution_chain": [
|
||||
{
|
||||
"node_id": f"{user_id}-l0-head",
|
||||
"content": "The user's name is Fei.",
|
||||
"memory_at": "2026-10-31 10:00",
|
||||
"gmt_created": 1_790_000_100,
|
||||
"layer": "l0_basic_info",
|
||||
},
|
||||
{
|
||||
"node_id": f"{user_id}-l0-old",
|
||||
"content": "The user's name is 费天翔.",
|
||||
"memory_at": "2026-10-30 10:00",
|
||||
"gmt_created": 1_790_000_000,
|
||||
"layer": "l0_basic_info",
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class Handle:
|
||||
def __init__(self, sdk: RecordingSdk) -> None:
|
||||
self.sdk = sdk
|
||||
|
||||
async def client(self) -> RecordingSdk:
|
||||
return self.sdk
|
||||
|
||||
async def close(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def application(tmp_path: Path, *, home: Path | None = None):
|
||||
settings = Settings(
|
||||
data_dir=tmp_path / "data",
|
||||
source_home=home or tmp_path,
|
||||
api_token=SecretStr(TOKEN),
|
||||
imports_enabled=home is not None,
|
||||
automatic_memory_processing=False,
|
||||
live_sync_enabled=False,
|
||||
)
|
||||
return create_app(
|
||||
settings,
|
||||
secret_store=InMemorySecretStore(),
|
||||
source_catalog=(
|
||||
build_source_catalog(home=home, environment={}, platform="darwin")
|
||||
if home is not None
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def client_for(app) -> AsyncClient:
|
||||
return AsyncClient(
|
||||
transport=ASGITransport(app=app),
|
||||
base_url="http://test",
|
||||
headers={"Authorization": f"Bearer {TOKEN}"},
|
||||
)
|
||||
|
||||
|
||||
async def _model_ready(_app) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def run_with_sdk(tmp_path, monkeypatch, scenario) -> None:
|
||||
async def main() -> None:
|
||||
app = application(tmp_path)
|
||||
sdk = RecordingSdk()
|
||||
|
||||
@asynccontextmanager
|
||||
async def write_client():
|
||||
yield sdk
|
||||
|
||||
async with app.router.lifespan_context(app):
|
||||
app.state.sdk_client_handle = Handle(sdk)
|
||||
monkeypatch.setattr(app.state.sdk_write_coordinator, "use_client", write_client)
|
||||
monkeypatch.setattr("one_memory.api._recall_model_ready", _model_ready)
|
||||
async with client_for(app) as client:
|
||||
await scenario(app, sdk, client)
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
|
||||
PASS_THROUGH = {
|
||||
"min_score": 0.5,
|
||||
"schema_min_score": 0.6,
|
||||
"working_fact_min_score": 0.3,
|
||||
"working_experience_min_score": 0.45,
|
||||
"schema_limit": 4,
|
||||
"intention_limit": 2,
|
||||
"rerank": False,
|
||||
"reader": "legacy",
|
||||
"scene": "chat",
|
||||
"search_mode": "agentic",
|
||||
"include_sources": ["hermes", "codex"],
|
||||
"exclude_sources": ["cursor"],
|
||||
"created_after": 1_780_000_000,
|
||||
"session_ids": ["s-1"],
|
||||
"agent_ids": ["all-apps"],
|
||||
"workspace_id": "ws-1",
|
||||
"branch": "main",
|
||||
"layers": ["l2_fact", "l0_basic_info"],
|
||||
}
|
||||
#: How each field reaches `client.search` (lists stay lists).
|
||||
EXPECTED_SEARCH = {**PASS_THROUGH}
|
||||
|
||||
|
||||
def _memory_search(sdk: RecordingSdk) -> dict:
|
||||
return [call for call in sdk.all("search") if call.get("layers") != ["l1_raw"]][-1]
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_every_context_search_option_reaches_the_sdk_and_absent_ones_keep_the_old_call(
|
||||
tmp_path, monkeypatch
|
||||
) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
baseline = await client.post(
|
||||
"/api/v2/context/search", json={"query": "tea", "kinds": ["memory"]}
|
||||
)
|
||||
assert baseline.status_code == 200, baseline.text
|
||||
before = _memory_search(sdk)
|
||||
assert before["min_score"] == before["schema_min_score"] == 0.35
|
||||
assert before["scene"] == "normal" and before["reader"] == "hybrid"
|
||||
assert before["schema_limit"] == 3 and before["intention_limit"] == 1
|
||||
for absent in ("rerank", "search_mode", "layers", "session_ids", "created_after"):
|
||||
assert absent not in before
|
||||
|
||||
response = await client.post(
|
||||
"/api/v2/context/search",
|
||||
json={"query": "tea", "kinds": ["memory"], **PASS_THROUGH},
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
sent = _memory_search(sdk)
|
||||
for name, value in EXPECTED_SEARCH.items():
|
||||
assert sent[name] == value, name
|
||||
assert sent["user_ids"] == ["local"]
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_every_recall_and_ask_option_reaches_the_sdk(tmp_path, monkeypatch) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
recalled = await client.post(
|
||||
"/api/v1/recall", json={"query": "tea", "user_id": "wife", **PASS_THROUGH}
|
||||
)
|
||||
assert recalled.status_code == 200, recalled.text
|
||||
sent = _memory_search(sdk)
|
||||
for name, value in EXPECTED_SEARCH.items():
|
||||
assert sent[name] == value, name
|
||||
assert sent["user_ids"] == ["wife"]
|
||||
|
||||
ask_options = {
|
||||
name: value
|
||||
for name, value in PASS_THROUGH.items()
|
||||
if name
|
||||
not in {
|
||||
"working_fact_min_score",
|
||||
"working_experience_min_score",
|
||||
"schema_limit",
|
||||
"intention_limit",
|
||||
"reader",
|
||||
"scene",
|
||||
"search_mode",
|
||||
"workspace_id",
|
||||
"branch",
|
||||
}
|
||||
}
|
||||
asked = await client.post(
|
||||
"/api/v2/memories/ask", json={"question": "tea?", "user_id": "wife", **ask_options}
|
||||
)
|
||||
assert asked.status_code == 200, asked.text
|
||||
sent = _memory_search(sdk)
|
||||
for name, value in ask_options.items():
|
||||
assert sent[name] == value, name
|
||||
assert sent["user_ids"] == ["wife"] and sent["scene"] == "normal"
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_min_score_is_enforced_on_what_core_returns(tmp_path, monkeypatch) -> None:
|
||||
"""Acceptance item 9: `min_score=0.9` answers only results scoring ≥ 0.9."""
|
||||
|
||||
async def scenario(app, sdk, client):
|
||||
context = await client.post(
|
||||
"/api/v2/context/search", json={"query": "tea", "min_score": 0.9}
|
||||
)
|
||||
assert context.status_code == 200, context.text
|
||||
items = context.json()["items"]
|
||||
assert items and all(item["score"] >= 0.9 for item in items)
|
||||
assert {item.get("memory_id") or item.get("raw_id") for item in items} == {
|
||||
"m-high",
|
||||
"raw-1",
|
||||
}
|
||||
raw_call = sdk.last("search") if sdk.last("search").get("layers") else None
|
||||
assert raw_call is None or raw_call["min_score"] == 0.9
|
||||
|
||||
# The fused scene ranks by 1/(60+rank); the lane's own score is the floor's.
|
||||
fused = await client.post(
|
||||
"/api/v2/context/search",
|
||||
json={"query": "tea", "kinds": ["memory"], "scene": "all", "min_score": 0.9},
|
||||
)
|
||||
assert [item["memory_id"] for item in fused.json()["items"]] == ["m-high"]
|
||||
assert fused.json()["items"][0]["score"] == 0.95
|
||||
|
||||
recalled = await client.post("/api/v1/recall", json={"query": "tea", "min_score": 0.9})
|
||||
assert recalled.status_code == 200, recalled.text
|
||||
assert [hit["memory_id"] for hit in recalled.json()["hits"]] == ["m-high"]
|
||||
|
||||
# A working-scene fact is read back from its own namespace.
|
||||
working = await client.post(
|
||||
"/api/v2/context/search",
|
||||
json={"query": "uv", "kinds": ["memory"], "scene": "working", "min_score": 0.9},
|
||||
)
|
||||
assert [item["memory_id"] for item in working.json()["items"]] == ["f-1"]
|
||||
assert sdk.last("get") == {"memory_id": "f-1", "kind": "fact", "variant": "working"}
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
@pytest.mark.parametrize(
|
||||
"extra",
|
||||
[
|
||||
{"min_score": 1.5},
|
||||
{"min_score": -0.1},
|
||||
{"schema_min_score": "high"},
|
||||
{"scene": "normal"},
|
||||
{"search_mode": "fast"},
|
||||
{"reader": "vector"},
|
||||
{"layers": ["l9_nothing"]},
|
||||
{"layers": []},
|
||||
{"branch": "main"},
|
||||
{"rerank": "no"},
|
||||
{"created_after": -1},
|
||||
{"user_ids": ["wife"]},
|
||||
{"embed_override": {"model": "x"}},
|
||||
{"query_embedding": [0.1]},
|
||||
{"request_id": "r-1"},
|
||||
],
|
||||
)
|
||||
def test_out_of_range_and_unexposed_search_fields_answer_422(tmp_path, extra) -> None:
|
||||
async def scenario():
|
||||
app = application(tmp_path)
|
||||
async with app.router.lifespan_context(app), client_for(app) as client:
|
||||
for path, body in (
|
||||
("/api/v2/context/search", {"query": "q"}),
|
||||
("/api/v1/recall", {"query": "q"}),
|
||||
("/api/v2/memories/ask", {"question": "q"}),
|
||||
):
|
||||
response = await client.post(path, json={**body, **extra})
|
||||
assert response.status_code == 422, (path, response.text)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_raw_lane_takes_the_callers_sources_only_inside_its_authority(
|
||||
tmp_path, monkeypatch
|
||||
) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
response = await client.post(
|
||||
"/api/v2/context/search",
|
||||
json={
|
||||
"query": "tea",
|
||||
"kinds": ["raw"],
|
||||
"include_sources": ["hermes"],
|
||||
"session_ids": ["s-1"],
|
||||
"created_after": 1_700_000_000,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
raw_call = sdk.last("search")
|
||||
assert raw_call["layers"] == ["l1_raw"]
|
||||
assert raw_call["include_sources"] == ["hermes"]
|
||||
assert raw_call["session_ids"] == ["s-1"]
|
||||
assert raw_call["created_after"] == 1_700_000_000
|
||||
# Memory-lane knobs never reach the raw search.
|
||||
assert raw_call["scene"] == "chat" and raw_call["schema_limit"] == 0
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_library_honours_layers_statuses_and_filters(tmp_path, monkeypatch) -> None:
|
||||
"""Acceptance item 10: `layer=l0_basic_info` returns only L0 records."""
|
||||
|
||||
async def scenario(app, sdk, client):
|
||||
page = await client.get("/api/v2/memory-library", params={"layer": "l0_basic_info"})
|
||||
assert page.status_code == 200, page.text
|
||||
assert {memory["layer"] for memory in page.json()["memories"]} == {"l0_basic_info"}
|
||||
sent = sdk.last("list_memories")
|
||||
assert sent["layers"] == ["l0_basic_info"] and sent["scene"] is None
|
||||
|
||||
await client.get(
|
||||
"/api/v2/memory-library",
|
||||
params=[
|
||||
("layers", "l2_fact,l6_schema"),
|
||||
("layers", "l7_intention"),
|
||||
("statuses", "active,superseded"),
|
||||
("session_ids", "s-1"),
|
||||
("agent_ids", "all-apps"),
|
||||
("exclude_sources", "cursor"),
|
||||
("user_id", "wife"),
|
||||
],
|
||||
)
|
||||
sent = sdk.last("list_memories")
|
||||
assert sent["layers"] == ["l2_fact", "l6_schema", "l7_intention"]
|
||||
assert sent["statuses"] == ["active", "superseded"]
|
||||
assert sent["session_ids"] == ["s-1"] and sent["agent_ids"] == ["all-apps"]
|
||||
assert sent["exclude_sources"] == ["cursor"] and sent["user_id"] == "wife"
|
||||
|
||||
# Without a layer the page is what it always was.
|
||||
await client.get("/api/v2/memory-library")
|
||||
sent = sdk.last("list_memories")
|
||||
assert sent["scene"] == "all" and "layers" not in sent
|
||||
|
||||
searched = await client.get(
|
||||
"/api/v2/memory-library", params={"query": "tea", "layers": "l2_fact"}
|
||||
)
|
||||
assert searched.status_code == 200, searched.text
|
||||
assert sdk.last("search")["layers"] == ["l2_fact"]
|
||||
|
||||
for params in (
|
||||
{"layers": "l9_nothing"},
|
||||
{"statuses": "gone"},
|
||||
{"layers": "l2_fact", "scene": "working"},
|
||||
):
|
||||
refused = await client.get("/api/v2/memory-library", params=params)
|
||||
assert refused.status_code == 422, (params, refused.text)
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_layer_counts_read_each_layer_for_one_user(tmp_path, monkeypatch) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
counts = await client.get("/api/v2/memory-library/layers", params={"user_id": "wife"})
|
||||
assert counts.status_code == 200, counts.text
|
||||
body = counts.json()
|
||||
assert body["layers"]["l1_raw"] == 1
|
||||
assert body["layers"]["l2_fact"] == 1
|
||||
assert body["layers"]["l0_basic_info"] == 1
|
||||
assert body["total"] == sum(body["layers"].values())
|
||||
calls = sdk.all("list_memories")
|
||||
assert {call["user_id"] for call in calls} == {"wife"}
|
||||
[raw_call] = [call for call in calls if call["layers"] == ["l1_raw"]]
|
||||
assert raw_call["statuses"] == ["active", "shadow"]
|
||||
assert (
|
||||
await client.get("/api/v2/memory-library/layers", params={"layer": "x"})
|
||||
).status_code == 422
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_raw_lists_active_and_shadow_turns_with_roles_and_hashes(tmp_path, monkeypatch) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
page = await client.get("/api/v2/raw")
|
||||
assert page.status_code == 200, page.text
|
||||
body = page.json()
|
||||
assert body["total"] == 2 and len(body["items"]) == 2
|
||||
sent = sdk.last("list_memories")
|
||||
assert sent["layers"] == ["l1_raw"] and sent["statuses"] == ["active", "shadow"]
|
||||
first = next(item for item in body["items"] if item["raw_id"] == "raw-1")
|
||||
assert first["status"] == "shadow"
|
||||
assert [message["role"] for message in first["messages"]] == ["user", "assistant"]
|
||||
assert first["messages"][0]["content"] == "我叫费天翔"
|
||||
assert first["role"] is None
|
||||
expected = hashlib.sha256(first["content"].encode()).hexdigest()
|
||||
assert first["content_hash"] == f"sha256:{expected}"
|
||||
assert first["session_id"] == "s-1" and first["source"] == "hermes"
|
||||
assert first["created"].startswith("2026-")
|
||||
second = next(item for item in body["items"] if item["raw_id"] == "raw-2")
|
||||
assert second["role"] == "user"
|
||||
|
||||
await client.get(
|
||||
"/api/v2/raw",
|
||||
params=[("source", "hermes,codex"), ("session_id", "s-1"), ("order", "asc")],
|
||||
)
|
||||
sent = sdk.last("list_memories")
|
||||
assert sent["include_sources"] == ["hermes", "codex"]
|
||||
assert sent["session_ids"] == ["s-1"] and sent["order"] == "asc"
|
||||
|
||||
windowed = (await client.get("/api/v2/raw", params={"after": 1_790_050_000})).json()
|
||||
assert windowed["time_filtered"] is True
|
||||
assert [item["raw_id"] for item in windowed["items"]] == ["raw-2"]
|
||||
|
||||
wife = (await client.get("/api/v2/raw", params={"user_id": "wife"})).json()
|
||||
assert [item["raw_id"] for item in wife["items"]] == ["raw-w"]
|
||||
assert sdk.last("list_memories")["user_id"] == "wife"
|
||||
assert (await client.get("/api/v2/raw", params={"statuses": "bogus"})).status_code == 422
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_profile_versions_read_the_l0_chain_for_one_user(tmp_path, monkeypatch) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
response = await client.get("/api/v2/user-profile/versions", params={"user_id": "wife"})
|
||||
assert response.status_code == 200, response.text
|
||||
body = response.json()
|
||||
assert body["user_id"] == "wife"
|
||||
assert sdk.last("get_user_basic_info") == {"user_id": "wife"}
|
||||
[chain] = body["chains"]
|
||||
assert chain["memory_id"] == "wife-l0-head"
|
||||
assert chain["basic_info"] == {"name": "Fei"}
|
||||
assert [(v["memory_id"], v["current"]) for v in chain["versions"]] == [
|
||||
("wife-l0-head", True),
|
||||
("wife-l0-old", False),
|
||||
]
|
||||
local = (await client.get("/api/v2/user-profile/versions")).json()
|
||||
assert local["user_id"] == "local"
|
||||
assert sdk.last("get_user_basic_info") == {"user_id": "local"}
|
||||
refused = await client.get("/api/v2/user-profile/versions", params={"owner": "wife"})
|
||||
assert refused.status_code == 422
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_graph_takes_layers_and_session_filters(tmp_path, monkeypatch) -> None:
|
||||
async def scenario(app, sdk, client):
|
||||
response = await client.get(
|
||||
"/api/v2/memories/graph",
|
||||
params={"layers": "l2_fact", "session_ids": "s-1", "user_id": "wife"},
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
sent = sdk.last("list_memories")
|
||||
assert sent["layers"] == ["l2_fact"] and sent["scene"] is None
|
||||
assert sent["session_ids"] == ["s-1"] and sent["user_id"] == "wife"
|
||||
|
||||
run_with_sdk(tmp_path, monkeypatch, scenario)
|
||||
|
||||
|
||||
def _hermes_state_db(root: Path, text: str) -> None:
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
connection = sqlite3.connect(root / "state.db")
|
||||
connection.execute(
|
||||
"CREATE TABLE messages (id INTEGER PRIMARY KEY, session_id TEXT, role TEXT,"
|
||||
" content TEXT, timestamp REAL, tool_name TEXT, active INTEGER)"
|
||||
)
|
||||
connection.executemany(
|
||||
"INSERT INTO messages VALUES (?, ?, ?, ?, 1790000000, NULL, 1)",
|
||||
[(1, "session-1", "user", text), (2, "session-1", "assistant", "ok")],
|
||||
)
|
||||
connection.commit()
|
||||
connection.close()
|
||||
|
||||
|
||||
@pytest.mark.fresh_sdk
|
||||
def test_v2_source_and_import_aliases_behave_like_v1_per_user(tmp_path) -> None:
|
||||
async def scenario():
|
||||
home = tmp_path / "home"
|
||||
hermes = home / ".hermes"
|
||||
wife = hermes / "profiles" / "wife"
|
||||
_hermes_state_db(hermes, "I am the husband")
|
||||
_hermes_state_db(wife, "I am the wife")
|
||||
app = application(tmp_path, home=home)
|
||||
async with app.router.lifespan_context(app), client_for(app) as client:
|
||||
location = (await client.post("/api/v1/sources/hermes/location")).json()
|
||||
local = await client.post(
|
||||
"/api/v1/sources/connect",
|
||||
json={
|
||||
"source_ids": ["hermes"],
|
||||
"request_id": str(uuid4()),
|
||||
"source_roots": {"hermes": location["root_id"]},
|
||||
"enable_sync": True,
|
||||
},
|
||||
)
|
||||
assert local.status_code == 200, local.text
|
||||
connected = await client.post(
|
||||
"/api/v1/sources/connect",
|
||||
json={
|
||||
"source_ids": ["hermes"],
|
||||
"request_id": str(uuid4()),
|
||||
"user_id": "wife",
|
||||
"root": str(wife),
|
||||
},
|
||||
)
|
||||
assert connected.status_code == 200, connected.text
|
||||
for job in (*local.json()["jobs"], *connected.json()["jobs"]):
|
||||
await app.state.import_supervisor.wait(job["job_id"])
|
||||
|
||||
v1 = (await client.get("/api/v1/sources")).json()
|
||||
v2 = (await client.get("/api/v2/sources")).json()
|
||||
assert v1 == v2
|
||||
wife_sources = (await client.get("/api/v2/sources", params={"user_id": "wife"})).json()
|
||||
assert [source["source_id"] for source in wife_sources] == ["hermes"]
|
||||
assert wife_sources[0]["authorized"] is True
|
||||
assert (await client.get("/api/v2/sources", params={"u": "x"})).status_code == 422
|
||||
|
||||
created = await client.post(
|
||||
"/api/v2/import-jobs",
|
||||
json={"source_id": "hermes", "operation": "sync", "user_id": "wife"},
|
||||
)
|
||||
assert created.status_code == 200, created.text
|
||||
wife_job = created.json()
|
||||
assert wife_job["user_id"] == "wife" and wife_job["source_id"] == "hermes"
|
||||
await app.state.import_supervisor.wait(wife_job["job_id"])
|
||||
refused = await client.post(
|
||||
"/api/v2/import-jobs",
|
||||
json={"source_id": "claude-code", "user_id": "wife"},
|
||||
)
|
||||
assert refused.status_code in {404, 422}
|
||||
unknown = await client.post(
|
||||
"/api/v2/import-jobs",
|
||||
json={"source_id": "hermes", "from_beginning": True},
|
||||
)
|
||||
assert unknown.status_code == 422
|
||||
|
||||
listed = (await client.get("/api/v2/import-jobs", params={"user_id": "wife"})).json()
|
||||
assert {job["user_id"] for job in listed} == {"wife"}
|
||||
assert (await client.get("/api/v2/import-jobs")).json() == (
|
||||
await client.get("/api/v1/import-jobs")
|
||||
).json()
|
||||
path = f"/api/v2/import-jobs/{wife_job['job_id']}"
|
||||
assert (await client.get(path)).status_code == 404
|
||||
own = await client.get(path, params={"user_id": "wife"})
|
||||
assert own.status_code == 200 and own.json()["job_id"] == wife_job["job_id"]
|
||||
|
||||
synced = await client.post("/api/v2/sources/hermes/sync", json={"user_id": "wife"})
|
||||
v1_synced = await client.post("/api/v1/sources/hermes/sync")
|
||||
assert synced.status_code == v1_synced.status_code, (synced.text, v1_synced.text)
|
||||
assert synced.status_code == 200
|
||||
assert synced.json()["user_id"] == "wife"
|
||||
assert v1_synced.json()["user_id"] == "local"
|
||||
other = await client.post("/api/v2/sources/claude-code/sync", json={"user_id": "wife"})
|
||||
assert other.status_code in {404, 422}
|
||||
assert (
|
||||
await client.post("/api/v2/sources/hermes/sync", json={"owner": "wife"})
|
||||
).status_code == 422
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -6,7 +6,7 @@ import io
|
||||
import json
|
||||
import sqlite3
|
||||
from contextlib import closing
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
@@ -30,7 +30,6 @@ from one_memory.sdk_data_plane_migration import (
|
||||
MemoryAuthority,
|
||||
SdkDataPlaneMigrationStore,
|
||||
)
|
||||
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
|
||||
from one_memory.sdk_migration_bundle import create_sdk_migration_bundle
|
||||
from one_memory.sdk_migration_runner import (
|
||||
PREPARED_PHASE,
|
||||
@@ -1350,57 +1349,6 @@ def test_raw_http_dto_uses_the_unified_memory_list_shape():
|
||||
assert page.raw[0].custom == {"role": "user"}
|
||||
|
||||
|
||||
def test_sdk_raw_retention_keeps_only_the_latest_three_days():
|
||||
now = datetime(2026, 10, 23, tzinfo=UTC)
|
||||
|
||||
class Client:
|
||||
def __init__(self):
|
||||
self.rows = [
|
||||
{
|
||||
"memory_id": "old",
|
||||
"memory_at": int((now - timedelta(days=4)).timestamp()),
|
||||
},
|
||||
{
|
||||
"memory_id": "recent",
|
||||
"memory_at": int((now - timedelta(days=2)).timestamp()),
|
||||
},
|
||||
]
|
||||
|
||||
def list_memories(self, **_kwargs):
|
||||
return {
|
||||
"vdb": {
|
||||
"memories": list(self.rows),
|
||||
"total": len(self.rows),
|
||||
"limit": 500,
|
||||
"offset": 0,
|
||||
}
|
||||
}
|
||||
|
||||
def delete(self, memory_id):
|
||||
before = len(self.rows)
|
||||
self.rows = [row for row in self.rows if row["memory_id"] != memory_id]
|
||||
return {
|
||||
"success": True,
|
||||
"deleted_count": before - len(self.rows),
|
||||
}
|
||||
|
||||
client = Client()
|
||||
|
||||
class Handle:
|
||||
async def client(self):
|
||||
return client
|
||||
|
||||
runtime = SdkMemoryRuntime(
|
||||
coordinator=None,
|
||||
read_handle=Handle(),
|
||||
write_queue=None,
|
||||
extraction_settings=None,
|
||||
data_dir=Path("."),
|
||||
)
|
||||
assert asyncio.run(runtime.prune_expired_raw(now=now)) == 1
|
||||
assert [row["memory_id"] for row in client.rows] == ["recent"]
|
||||
|
||||
|
||||
# The UI asks which memory data plane is authoritative and must get the
|
||||
# answer the API gates use.
|
||||
|
||||
|
||||
@@ -140,7 +140,9 @@ class _Recorder:
|
||||
self.opened: list[tuple[str, int | None, str | None]] = []
|
||||
self.closed: list[str] = []
|
||||
|
||||
def __call__(self, profile, *, api_key, data_dir, http_transport_factory=None):
|
||||
def __call__(
|
||||
self, profile, *, api_key, data_dir, http_transport_factory=None, mode="pro", features=()
|
||||
):
|
||||
from one_memory.hy_sdk_runtime import build_sdk_config
|
||||
|
||||
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir)
|
||||
|
||||
@@ -130,7 +130,7 @@ This exception does not extend to VDB, Coding, prompt, reconcile, or lifecycle.
|
||||
|
||||
- Storage backends:仅 SDK config/env(Chroma/FAISS/Qdrant、Kuzu/Neo4j、cache/history sqlite|mysql、coding sqlite 等)。
|
||||
- Queue:pending `memory_write_jobs` 只表示尚未调用 SDK 的待处理请求,因此没有 SDK raw/memory ID。worker 单次调用 SDK `add()`;SDK 内部完成 raw 写入与 extraction/reconciliation,成功后 MemHub 仅将返回的 raw ID / memory IDs 记作 job outcome。
|
||||
- Raw retention:SDK L1 raw 只保留最近三天,由 Runtime 的 `prune_expired_raw` 执行;每个完整 turn 只保存真人 user 与 assistant,tool 不进入 raw。高层 Chat/Coding/Working memory 不受该窗口影响。
|
||||
- Raw retention:MemHub 不删除任何 SDK L1 raw,任何模式都全部保留(SDK 自身也不删 raw);每个完整 turn 只保存真人 user 与 assistant,tool 不进入 raw。记忆模式(lite / pro / ultra)与 evolve 见 `docs/contracts/memory-mode-evolve-v1.md`;SDK 可选检索与提取功能(graph rerank、episode recall 等,全局设置,默认保持 SDK 默认值)见 `docs/contracts/memory-features-v1.md`。
|
||||
- Recall:Host Bridge / Core 先做宿主安装凭据与 tool grant,再调 **SDK search/get**。当前产品没有逐 memory 的 A-harness→B-harness 可见性开关,默认允许跨 harness 搜索;未来若增加来源限制,直接透传 SDK `source` / `include_sources` / `exclude_sources`,不在 MemHub 建 memory ACL。
|
||||
- Context wire:cutover marker 提交前只使用 legacy Context v1;提交后 Host Bridge 只使用 SDK-native Context v2。两种 schema 原样校验、原样返回,不把 SDK item 补造成 v1 的 `revision`、`evidence_ids`、`paths` 或 `adoption`,已选择的 v2 路径失败时也不回退 legacy 权威。
|
||||
- Ask wire:marker 前保留 Governance Ask v1;marker 后 Web / Host Bridge 只调用 SDK Ask v2。v2 的引用只含 SDK 原生 `memory_id/content/layer/status/source/gmt_created`,生成过程中重复 `get` 校验同一 SDK 快照,不补造 revision、Evidence 或 scopes,SDK 失败不回退 v1。
|
||||
|
||||
@@ -142,6 +142,48 @@ node apps/cli/dist/one-memory.cjs write status <request_id> --user-id wife
|
||||
- 不带 `--user-id`(或写 `--user-id local`)时请求里不出现 `user_id`,和以前完全一样。
|
||||
- 用户之间互相隔离:读、改、删另一个用户的记忆 ID 返回「未找到」;write 的 request id 也只属于提交它的用户。
|
||||
|
||||
### 画像版本(`profile --versions`)
|
||||
|
||||
```bash
|
||||
node apps/cli/dist/one-memory.cjs profile --versions # L0 画像版本链,最新在前,* 为当前版本
|
||||
node apps/cli/dist/one-memory.cjs profile --versions --user-id wife --json
|
||||
```
|
||||
|
||||
用于找出覆盖了正确值的错误版本(例如 `用户` / `The user`),再用
|
||||
`memories delete <memory-id> --yes` 删除;对应 `GET /api/v2/user-profile/versions`。
|
||||
|
||||
### 记忆模式与进化(`settings mode` / `evolve`)
|
||||
|
||||
```bash
|
||||
node apps/cli/dist/one-memory.cjs settings mode # 当前模式
|
||||
node apps/cli/dist/one-memory.cjs settings mode ultra # 只显示切换计划,不改动
|
||||
node apps/cli/dist/one-memory.cjs settings mode ultra --yes # 切换
|
||||
node apps/cli/dist/one-memory.cjs evolve --user-id wife --wait # 立即进化(仅 ultra)
|
||||
```
|
||||
|
||||
- 切换不迁移数据:已处理的对话不会重做,排队中的和新的按新模式处理;正在写的一批按旧模式写完。
|
||||
- 任何模式都保留全部原始对话。
|
||||
- `status --json` 与 `progress --json` 输出 `pipeline.mode`、每晚进化时间和每个用户的上次进化。
|
||||
- 契约见 `docs/contracts/memory-mode-evolve-v1.md`。
|
||||
|
||||
### SDK 可选功能(`settings features`)
|
||||
|
||||
```bash
|
||||
node apps/cli/dist/one-memory.cjs settings features # 列出全部功能、默认值、当前模式下是否生效
|
||||
node apps/cli/dist/one-memory.cjs settings features set episode_recall=true session_episodes=true # 只显示计划
|
||||
node apps/cli/dist/one-memory.cjs settings features set episode_recall=true session_episodes=true --yes # 生效
|
||||
node apps/cli/dist/one-memory.cjs settings features reset episode_recall # 恢复默认
|
||||
node apps/cli/dist/one-memory.cjs settings features reset # 全部恢复默认
|
||||
```
|
||||
|
||||
- 全局设置,不分用户;没设置的功能保持 SDK 默认值,什么都不设时与以前完全一样(App 不变)。
|
||||
- 可设:`graph_rerank`(图谱重排,仅 ultra)、`same_series_timeline`(需 graph_rerank)、`episode_recall`(片段召回,pro 即可)、`session_episodes`(片段写入)、`episode_summary`(片段摘要,每个片段一次模型调用)、`intention`(默认开)、`few_shot`。
|
||||
- 片段召回只对带片段的记忆有用:同时开 `session_episodes`,之后提取的记忆才按会话归入片段。
|
||||
- 在当前模式下不生效的功能照样保存,并注明原因(如 `graph_rerank` 在 pro 下「需要 ultra」)。
|
||||
- 修改会重建 SDK 客户端:正在写的一批按旧设置写完,之后的写入和检索用新设置;不迁移数据,不改集合名。
|
||||
- `status --json` / `progress --json` 的 `pipeline.features` 列出与默认值不同的功能。
|
||||
- 契约见 `docs/contracts/memory-features-v1.md`。
|
||||
|
||||
## 8. 召回(recall)
|
||||
|
||||
`recall` 两条路径(raw / memory)此前已真机验证。文本输出会标明每条是「原始对话摘录」
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
# SDK retrieval and extraction features v1
|
||||
|
||||
Status: Core + CLI implemented. Stacks on the memory-mode contract
|
||||
(`memory-mode-evolve-v1.md`). The App keeps today's behaviour and has no
|
||||
control for this yet; the copy in section 9 is there if it adds one.
|
||||
|
||||
MemHub hard-codes none of the SDK's optional features. A person can turn the
|
||||
ones below on or off through the CLI or the API. Nobody has to: an unset
|
||||
feature keeps the SDK default, and with nothing set the SDK config is byte for
|
||||
byte what MemHub built before this setting existed.
|
||||
|
||||
All routes need the Runtime bearer token. Unknown query parameters and body
|
||||
fields answer `422 {"detail": "request_validation_failed"}`, like every other
|
||||
settings route. The setting is global, like the memory mode: it applies to
|
||||
every user's SDK client and takes no `user_id`.
|
||||
|
||||
## 1. Inventory
|
||||
|
||||
These are the hy-memory flags (`hy_memory/config.py` and the modules that read
|
||||
the environment directly) that change retrieval or extraction quality. Cost:
|
||||
"none" means no extra call; "query" means one more local store query per
|
||||
search; "LLM" means extra generation-model calls; "tokens" means a longer
|
||||
prompt and no extra call.
|
||||
|
||||
### Exposed
|
||||
|
||||
| Name | SDK field (env) | Default | Mode | Cost |
|
||||
|---|---|---|---|---|
|
||||
| `graph_rerank` | `graph_store.graph_rerank_enabled` (`MEMORY_GRAPH_RERANK_ENABLED`) | off | ultra (graph store) | query (graph edges) |
|
||||
| `same_series_timeline` | `recall.same_series_timeline_enabled` (`MEMORY_SAME_SERIES_TIMELINE_ENABLED`) | off | ultra + `graph_rerank` | none |
|
||||
| `episode_recall` | `recall.episode_recall_enabled` (`MEMORY_EPISODE_RECALL_ENABLED`) | off | pro or ultra; no graph | query (episode members) |
|
||||
| `session_episodes` | `recall.session_episode_enabled` (`MEMORY_SESSION_EPISODE_ENABLED`) | off | pro or ultra | none |
|
||||
| `episode_summary` | `recall.episode_summary_enabled` (`MEMORY_EPISODE_SUMMARY_ENABLED`) | off | pro or ultra | LLM: one call per closed episode of 3+ memories |
|
||||
| `intention` | `llm.enable_intention` (`MEMORY_ENABLE_INTENTION`) | **on** | pro or ultra | none (off shortens the prompt) |
|
||||
| `few_shot` | `llm.few_shot_enabled` (`MEMORY_FEW_SHOT_ENABLED`) | off | pro or ultra | tokens (extraction and reconcile) |
|
||||
|
||||
- **Graph rerank (usage A).** It propagates support over the graph edges
|
||||
inside the recall pool (`reader_base._apply_graph_rerank`). It needs a graph
|
||||
store, which MemHub opens only in ultra. In pro and lite it is reported
|
||||
`active: false, reason: "requires_ultra"`.
|
||||
- **Episode recall (usage B).** It adds memories from the same episode as a
|
||||
hit and scores them on three channels: tag-keyword overlap, content overlap
|
||||
and temporal affinity. It needs no graph, so it works in pro (issue #121).
|
||||
It helps only memories that carry an `episode_id`.
|
||||
- Memories that System 1 adds through reconcile get an idle-gap
|
||||
`episode_id` whenever the SDK cache is available, which MemHub's SQLite
|
||||
cache is. A new episode starts after 30 minutes idle or 50 memories.
|
||||
- `session_episodes` replaces that with one stable episode per conversation.
|
||||
- Memories stored before the change keep the id they had.
|
||||
- Raw turns from lite, and notes written before episodes existed, carry none.
|
||||
- **Episode write side.** `session_episodes` and `episode_summary` are the
|
||||
write half of episodes, and both are off by default. Usage B only helps once
|
||||
episodes are being written. The SDK's session narrative (L3, `finalize_session`)
|
||||
is a separate feature and is not exposed (section 2).
|
||||
- **`intention`** is the only feature that is on by default. Turning it off
|
||||
stops L7 intention memories from being extracted.
|
||||
|
||||
### Left out, and why
|
||||
|
||||
| Flag(s) | Why it is not exposed |
|
||||
|---|---|
|
||||
| `recall.entity_store_enabled` (`MEMORY_ENTITY_STORE_ENABLED`) | Entity extraction needs spaCy and `en_core_web_sm`, which MemHub does not ship, so it would silently do nothing. It also writes a new `{collection}_entities` collection, and MemHub's collection identity has to stay fixed. |
|
||||
| `llm.enable_query_rewrite` (`MEMORY_ENABLE_QUERY_REWRITE`) | It only affects default-mode searches that pass `queries=[...]`, and MemHub never sends that. Per-request rewrite already exists as `search_mode: "agentic"` (#252). |
|
||||
| `session_episode_context_enabled`, `dynamic_episode_enabled`, `series_episode_enabled`, `episode_capsule_context_enabled`, `episode_member_expansion_enabled` | They read or write the L3 session narrative that `finalize_session` builds, and MemHub never calls it. When dynamic or series finalize fails, the SDK fails the write with a non-retryable 409 partial result, which would leave MemHub jobs `unknown`. |
|
||||
| `local_rerank_enabled` (+ model / device) | It downloads a cross-encoder model on first use. That is a network fetch MemHub's packaged, offline runtime and zero-collection rules do not allow. |
|
||||
| `strength_enabled`, `access_tracking_enabled` | They write access counts back to memory rows on every search: a read that mutates. Internal. |
|
||||
| `structure_metadata_enabled`, `retrieval_clauses_*`, `supplemental_propositions_enabled`, `hierarchy_bundle_*`, `multi_seed_bundle_*`, `tag_index_nominee_*`, `dynamic_entity_tag_embed_enabled` | Experimental retrieval lanes, each with many tuning parameters. The SDK marks them as still under gray validation, and some depend on System 2 output. Internal tuning, not a person's choice. |
|
||||
| `graph_store.memory_graph_write`, `path_enabled` (default on) | Turning them off breaks graph rerank and evolve. Dangerous. |
|
||||
| `hierarchy_topic_write`, `temporal_spine`, `topic_summary_enabled` | Experimental graph write lanes ("stage 5"); topic summary also costs LLM calls. Not validated for users. |
|
||||
| Schema lane and catalog / supplement (`MEMORY_EVOLING_SCHEMA_DEDUP_ENABLED`, `MEMORY_EVOLING_SCHEMA_COMPACTION_ENABLED`, `MEMORY_EVOLING_BALANCED_CATALOG_ENABLED`, `MEMORY_EVOLING_SPECIFIC_SUPPLEMENT_ENABLED`, `MEMORY_USER_PROFILE_*`, `MEMORY_HYBRID_V2_*_L0_ENABLED`) | The SDK reads them from `os.environ`, not from `MemoryConfig`. Some are read once at import (`system2_tools`), others on each call for the whole process. They cannot be scoped to one client, and setting them in Core's environment would need a restart and would leak into every client. Section 3. |
|
||||
| The hybrid reader's env knobs (`MEMORY_HYBRID_V2_*`, `MEMORY_TOPIC_*_ENABLED`, `MEMORY_*_BUNDLE_ENABLED`, `MEMORY_RERANK_*`) | Same reason: read from the process environment on each search. |
|
||||
| `RECONCILE_ENABLED`, `SWEEPER_ENABLED`, `MEMORY_CROSS_DOMAIN_ENABLE` | Read at import time. Turning reconcile off stores every extraction without dedup. Dangerous. |
|
||||
| `coding.chain_block_enabled`, `coding.verbatim_fallback_enabled` | Coding-scene options. The verbatim fallback stores assistant and tool traces as facts. |
|
||||
| `MEMORY_ENABLE_GRAPH` (graph in pro) | That is the memory mode's job (`settings mode`). |
|
||||
| `MEMORY_LOG_LLM_CONTENT`, pipeline log, trace, memory operations, metrics / telemetry | Diagnostics. MemHub controls them elsewhere (`diagnostic-logging.md`). |
|
||||
|
||||
## 2. The setting
|
||||
|
||||
`config.json` gains an optional `memory_features` object next to
|
||||
`memory_mode`. It is written only once a person sets a feature, and then it
|
||||
holds only the features they set:
|
||||
|
||||
```json
|
||||
{"memory_mode": "pro", "memory_features": {"episode_recall": true, "session_episodes": true}}
|
||||
```
|
||||
|
||||
The values are strict booleans and unknown keys are refused, so a hand-edited
|
||||
file with an unknown feature fails to load (`config_invalid`), like any other
|
||||
malformed setting.
|
||||
|
||||
## 3. How a setting reaches the SDK
|
||||
|
||||
- Every exposed feature is a `MemoryConfig` field that the SDK reads from the
|
||||
config object when it is called. None of them is read from `os.environ` at
|
||||
call time.
|
||||
- `hy_sdk_runtime.build_sdk_config(..., features=...)` sets those fields on
|
||||
each client's config after `MemoryConfig.from_dict`. The result is
|
||||
deterministic per client.
|
||||
- The SDK's own env vars (the second column of the table) still decide the
|
||||
default of a feature nobody set. That is exactly what happened before the
|
||||
setting existed. A setting always wins over the env var.
|
||||
- MemHub writes no env var. The flags the SDK reads straight from the
|
||||
environment are the ones left out above.
|
||||
- Each client's features are fixed when it is built, like its mode. Both the
|
||||
write handles (`SdkWriteCoordinator`) and the read handle
|
||||
(`ActiveProfileReadHandle`) are keyed by (profile, revision, mode,
|
||||
features).
|
||||
- A change runs `retire_stale_handles`, the #218 use count. An idle write
|
||||
handle closes at once, and a write in flight finishes with the features it
|
||||
started with. The next read builds a client with the new key. The old read
|
||||
client closes once no request still uses it.
|
||||
- Nothing stored is migrated. A feature never touches `vector_store`, the
|
||||
embedder or any store path, so the collection identity is the same with any
|
||||
features (tested).
|
||||
|
||||
## 4. `GET /api/v1/settings/memory-features`
|
||||
|
||||
```json
|
||||
{
|
||||
"mode": "pro",
|
||||
"features": [
|
||||
{
|
||||
"name": "graph_rerank",
|
||||
"enabled": true,
|
||||
"configured": true,
|
||||
"default": false,
|
||||
"active": false,
|
||||
"reason": "requires_ultra",
|
||||
"side": "read",
|
||||
"requires_mode": "ultra",
|
||||
"requires_feature": null,
|
||||
"cost": "extra_query",
|
||||
"sdk_field": "graph_store.graph_rerank_enabled",
|
||||
"env": "MEMORY_GRAPH_RERANK_ENABLED",
|
||||
"description": "Usage A: propagate support over knowledge-graph edges ..."
|
||||
}
|
||||
],
|
||||
"non_default": {"graph_rerank": true}
|
||||
}
|
||||
```
|
||||
|
||||
- `enabled` is what clients are built with: `configured` if set, else `default`.
|
||||
- `configured` is null when unset.
|
||||
- `default` is the SDK default: its code default, or its own env var when set.
|
||||
- `active` is false with a `reason` when the feature does nothing right now.
|
||||
The reason is one of `disabled`, `requires_pro` (lite), `requires_ultra`, or
|
||||
`requires_graph_rerank` (for `same_series_timeline`).
|
||||
- `cost` is one of `none`, `extra_query`, `llm_call`, `prompt_tokens`.
|
||||
- `features` always lists every feature, in the table's order.
|
||||
|
||||
## 5. `PUT /api/v1/settings/memory-features`
|
||||
|
||||
The body has any of the feature names as keys:
|
||||
|
||||
- `true` or `false` sets that feature.
|
||||
- `null` resets it to the default.
|
||||
- An absent key is left as it is.
|
||||
- Anything else answers `422`, including a non-boolean, `"true"`, `1`, or an
|
||||
unknown name.
|
||||
|
||||
```json
|
||||
{"graph_rerank": true, "episode_recall": true, "few_shot": null}
|
||||
```
|
||||
|
||||
The answer is the state of section 4, plus `changed`: the features whose
|
||||
effective value changed, in the table's order. When the set of configured
|
||||
features changes, the SDK clients are rebuilt (section 3). That includes a
|
||||
reset to a value equal to the default, after which the client carries no
|
||||
override at all.
|
||||
|
||||
## 6. Status
|
||||
|
||||
`pipeline` in `GET /api/v1/memory-processing` (memory-mode contract §7) gains
|
||||
`features`, which lists the features whose effective value differs from the
|
||||
default. It is `{}` while every feature is at its default:
|
||||
|
||||
```json
|
||||
{"pipeline": {"mode": "pro", "...": "...", "features": {"episode_recall": true, "intention": false}}}
|
||||
```
|
||||
|
||||
`memhub status --json` and `memhub progress --json` carry it as
|
||||
`pipeline.features`. The text status appends `;SDK 功能:episode_recall=开、…`
|
||||
to the memory-mode line.
|
||||
|
||||
## 7. CLI
|
||||
|
||||
```bash
|
||||
memhub settings features # list them
|
||||
memhub settings features set episode_recall=true session_episodes=true # plan only
|
||||
memhub settings features set episode_recall=true session_episodes=true --yes
|
||||
memhub settings features reset few_shot # back to the default
|
||||
memhub settings features reset # every set feature
|
||||
```
|
||||
|
||||
`set` prints its plan and needs `--yes`, like `settings mode`, because some
|
||||
features spend model calls. `reset` only returns features to their defaults,
|
||||
so it applies at once. Unknown names or values other than `true` / `false` are
|
||||
a usage error (exit 2) before the Runtime is asked.
|
||||
|
||||
## 8. Rules kept
|
||||
|
||||
- Global management, as for the mode: no `user_id`; reads stay scoped per
|
||||
user as before.
|
||||
- Strict parameters: unknown body fields and query parameters answer 422.
|
||||
- The mode semantics of #253 are unchanged. A feature needing a higher mode is
|
||||
stored and reported inactive; it is not refused.
|
||||
|
||||
## 9. Copy the UI can reuse
|
||||
|
||||
| Key | zh-CN | en | es |
|
||||
|---|---|---|---|
|
||||
| title | 高级检索与提取 | Advanced retrieval and extraction | Recuperación y extracción avanzadas |
|
||||
| graph_rerank | 图谱重排:沿知识图谱的关联提升相关记忆的排名(仅 ultra)。 | Graph rerank: ranks memories higher when the knowledge graph links them to strong matches (ultra only). | Reordenación por grafo: sube en la lista las memorias que el grafo de conocimiento relaciona con buenas coincidencias (solo ultra). |
|
||||
| same_series_timeline | 同系列时间线:同一系列的记忆按时间放在一起(需图谱重排)。 | Same-series timeline: keeps memories of one series together in time order (needs graph rerank). | Línea temporal de la serie: mantiene juntas y en orden temporal las memorias de una misma serie (requiere la reordenación por grafo). |
|
||||
| episode_recall | 片段召回:补入与命中记忆同一片段的记忆,并按标签、内容和时间打分。 | Episode recall: adds memories from the same episode as a match and scores them by tags, content and time. | Recuperación por episodio: añade memorias del mismo episodio que una coincidencia y las puntúa por etiquetas, contenido y tiempo. |
|
||||
| session_episodes | 片段写入:同一会话中提取的记忆归入同一片段。 | Episodes per conversation: memories extracted from one conversation share one episode. | Episodios por conversación: las memorias extraídas de una conversación comparten un episodio. |
|
||||
| episode_summary | 片段摘要:片段结束时总结一次(每个片段一次模型调用)。 | Episode summaries: each finished episode is summarized once (one model call per episode). | Resúmenes de episodio: cada episodio terminado se resume una vez (una llamada al modelo por episodio). |
|
||||
| intention | 意图提取:随事实一起记住你提到的计划和意图。 | Intentions: remembers the plans and intentions you mention, along with facts. | Intenciones: recuerda los planes e intenciones que mencionas, junto con los hechos. |
|
||||
| few_shot | 提取示例:在提取提示中加入示例,可帮助较弱的模型(输入更长)。 | Extraction examples: adds examples to the extraction prompt, which can help a weaker model (longer input). | Ejemplos de extracción: añade ejemplos a la instrucción de extracción, lo que puede ayudar a un modelo más débil (entrada más larga). |
|
||||
| default | 默认 | Default | Predeterminado |
|
||||
| requires_ultra | 当前模式下不生效,需要 ultra。 | Has no effect in this mode; needs ultra. | No tiene efecto en este modo; requiere ultra. |
|
||||
| requires_pro | lite 模式下不生效。 | Has no effect in lite. | No tiene efecto en lite. |
|
||||
| requires_graph_rerank | 需要先开启图谱重排。 | Turn on graph rerank first. | Activa primero la reordenación por grafo. |
|
||||
| episodes_hint | 片段召回只对带片段的记忆有用;开启片段写入后,新提取的记忆才会按会话归入片段。 | Episode recall only helps memories that belong to an episode; with episodes per conversation on, new memories are grouped by conversation. | La recuperación por episodio solo ayuda a las memorias que pertenecen a un episodio; con los episodios por conversación activados, las memorias nuevas se agrupan por conversación. |
|
||||
| cost_llm | 会增加模型调用 | Uses extra model calls | Usa llamadas adicionales al modelo |
|
||||
| applied | 正在进行的写入按原设置完成,之后的写入和检索使用新设置。 | A write in progress finishes with the old settings; later writes and searches use the new ones. | Una escritura en curso termina con la configuración anterior; las siguientes escrituras y búsquedas usan la nueva. |
|
||||
@@ -0,0 +1,357 @@
|
||||
# Memory mode, evolve and the nightly schedule v1
|
||||
|
||||
Status: Core + CLI implemented (PR B of the SDK-surface batch). The App
|
||||
controls are built separately against this contract.
|
||||
|
||||
All routes need the Runtime bearer token. Unknown query parameters and body
|
||||
fields answer `422 {"detail": "request_validation_failed"}`, like every
|
||||
memory route. `user_id` is optional (default `local`) and must match
|
||||
`^[A-Za-z0-9._-]{1,64}$`. The App never sends it.
|
||||
|
||||
## 1. What the modes are
|
||||
|
||||
| Mode | SDK client | What happens to a conversation | What search reads |
|
||||
|---|---|---|---|
|
||||
| `lite` | `mode="lite"` | Stored as one raw turn with one embedding. No model call; no generation model needs to be configured. | Raw turns only (the SDK forces `L1_RAW`). |
|
||||
| `pro` (default) | `mode="pro"` | Stored as raw, then a model extracts memories (System 1). The raw becomes SHADOW once extraction succeeds. | Extracted memories (facts, profile, intentions). |
|
||||
| `ultra` | `mode="ultra"` | Same as pro, and new facts are also projected into the knowledge graph. Every night, evolve runs System 2 for every user. | Same as pro, plus the graph-backed lanes below. |
|
||||
|
||||
The product difference between pro and ultra is the nightly evolve. MemHub ultra
|
||||
uses SDK `mode="ultra"` because `client.evolve` refuses any other mode. Compared
|
||||
with pro, SDK ultra (hy-memory 2.0.9) also turns on the following, so the owner
|
||||
can judge their cost:
|
||||
|
||||
- **The graph store.** Kuzu is always opened in ultra. Client init fails if the
|
||||
graph cannot open, so `PUT ultra` answers `409 graph_store_unavailable` when
|
||||
`kuzu` is missing. In pro it opens only when `MEMORY_ENABLE_GRAPH` is set, and
|
||||
MemHub does not set it. Writes project new L2 facts into the graph.
|
||||
- **The schema lane (L6_SCHEMA).** The search's schema channel runs only when
|
||||
a graph store exists, so in MemHub only in ultra. It returns the schemas that
|
||||
evolve wrote (`schema_limit`, `schema_min_score`).
|
||||
- **Graph-aware ranking on the normal lane.** With a graph store, search adds
|
||||
graph-entity (MENTIONS) boosts when the query names known entities and no
|
||||
source filter is set. The heavier graph stages need the graph *and* their
|
||||
own env flag, and every flag defaults off, so none of them run in MemHub:
|
||||
graph rerank and episode expansion (`MEMORY_GRAPH_RERANK_ENABLED`), topic
|
||||
graph entry, same-series bundle, hierarchy bundle.
|
||||
- **The profile.** `get_user_profile` appends the balanced catalog and the
|
||||
specific supplement from the graph, when the SDK's balanced-catalog flag is
|
||||
on.
|
||||
|
||||
`ONE_MEMORY_GRAPH_STORE` (development only) still forces ultra whatever the
|
||||
setting says. Any value other than empty or `none` does this. The API reports
|
||||
it as `forced_by_env: true`, with `configured_mode` showing the saved choice.
|
||||
|
||||
**Raw is never deleted.** In every mode MemHub keeps every raw turn the SDK
|
||||
stored, and the SDK never deletes raw itself. There is no retention window.
|
||||
|
||||
## 2. Switching mode: what changes and what does not
|
||||
|
||||
The SDK migrates nothing when the mode changes. MemHub applies these rules,
|
||||
and they are the owner's:
|
||||
|
||||
- The mode is read **when a queued job runs**, not when it was queued.
|
||||
- A job that already completed, in any mode, stays completed. It is never
|
||||
reclaimed or reprocessed.
|
||||
- Jobs still waiting (pending, assembling, waiting) and new input run in
|
||||
the new mode.
|
||||
- **A batch in flight finishes in the mode it started in.** Its SDK client is
|
||||
retired only when its last use ends (the #218 use count). The next claim
|
||||
uses the new mode.
|
||||
- **Search follows the new mode at once.** The read client is rebuilt on the
|
||||
next request (the active-profile read handle is keyed by profile and mode); the client it replaces closes once no request still uses it.
|
||||
- **lite → pro/ultra:**
|
||||
- what lite stored stays raw and is not extracted;
|
||||
- lite's raw-only search stops at once;
|
||||
- nothing is re-ingested or backfilled.
|
||||
- **pro/ultra → lite:** memories made by the higher mode stay on disk, but lite
|
||||
search does not serve them. Nothing is deleted.
|
||||
- **ultra → pro/lite:** the nightly evolve stops.
|
||||
- **→ ultra:**
|
||||
- every known user's existing facts are added to the graph once
|
||||
(`build_memory_graph(user_id, limit=10000, clear_first=False)`);
|
||||
- this runs in the background, one user at a time, never beside that
|
||||
user's write;
|
||||
- progress is reported in `graph_build`.
|
||||
- Vector collection names never depend on the mode. The mode is a separate
|
||||
client parameter, not part of the embedding-space identity.
|
||||
|
||||
## 3. `GET /api/v1/settings/memory-mode`
|
||||
|
||||
No parameters. `200` response:
|
||||
|
||||
```json
|
||||
{
|
||||
"mode": "pro",
|
||||
"configured_mode": "pro",
|
||||
"forced_by_env": false,
|
||||
"modes": ["lite", "pro", "ultra"],
|
||||
"graph_build": {
|
||||
"state": "idle",
|
||||
"started_at": null,
|
||||
"finished_at": null,
|
||||
"users": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `mode` is what SDK clients are built with now; `configured_mode` is the
|
||||
saved setting.
|
||||
- `graph_build.state` is `idle | running | succeeded | failed`. Each entry in
|
||||
`graph_build.users[<user_id>]` is
|
||||
`{state: pending|running|succeeded|failed, memories_scanned, nodes, error_code}`.
|
||||
|
||||
## 4. `PUT /api/v1/settings/memory-mode`
|
||||
|
||||
Body: `{"mode": "lite" | "pro" | "ultra"}`. `200` response: the GET shape,
|
||||
plus these fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"previous_mode": "pro",
|
||||
"effects": ["existing_memories_added_to_graph"]
|
||||
}
|
||||
```
|
||||
|
||||
`effects` is empty when the effective mode did not change. That happens when
|
||||
the mode was already set, or when `ONE_MEMORY_GRAPH_STORE` forces ultra; the
|
||||
saved setting still changes in that case. Otherwise `effects` lists these, in
|
||||
this order:
|
||||
|
||||
| Effect | When | Meaning |
|
||||
|---|---|---|
|
||||
| `earlier_raw_not_extracted` | from lite | What lite stored stays raw; it is not extracted. |
|
||||
| `lite_search_stops` | from lite | lite's raw-only search ends now; search reads memories. |
|
||||
| `higher_mode_memories_hidden` | to a lower mode | Memories of the higher mode stay stored but are not shown. |
|
||||
| `evolve_stopped` | from ultra | The nightly evolve stops. |
|
||||
| `existing_memories_added_to_graph` | to ultra | The one-time graph backfill started (see `graph_build`). |
|
||||
| `graph_backfill_already_running` | to ultra | A backfill from an earlier switch is still running; no second one started. |
|
||||
|
||||
Errors:
|
||||
|
||||
- `409 {"detail": "graph_store_unavailable"}`: `ultra` was asked for, but the
|
||||
graph store package is not installed. Nothing changed.
|
||||
- `422 request_validation_failed`: an unknown mode or an extra field.
|
||||
|
||||
Show the planned effects before calling PUT. Clients can compute them with the
|
||||
table above: the CLI (`memhub settings mode <m>` without `--yes`) and Core
|
||||
(`memory_mode_effects`) use the same rule.
|
||||
|
||||
## 5. Evolve
|
||||
|
||||
### `POST /api/v2/evolve` → `202`
|
||||
|
||||
Body (every field optional):
|
||||
|
||||
```json
|
||||
{"user_id": "local", "agent_id": "all-apps", "max_batches": 3}
|
||||
```
|
||||
|
||||
- `agent_id` is 1..128 characters. The default is `all-apps`, the agent
|
||||
MemHub writes every memory under.
|
||||
- `max_batches` is 1..1000. The default is the SDK's own.
|
||||
|
||||
The response is the run receipt below, with `state: "running"`. The run goes
|
||||
on in the background.
|
||||
|
||||
- **One run per user.** Asking while that user's run is going answers the same
|
||||
receipt, again `202`.
|
||||
- **Never beside that user's writes.** The run waits for the user's write in
|
||||
flight. While it runs, that user's queued jobs wait: they are returned to
|
||||
pending before any SDK call. Other users are not held.
|
||||
- **Loops while the SDK answers `should_continue: true`**, up to 20 rounds. A
|
||||
run stopped by that cap has `capped: true`; start it again to continue.
|
||||
|
||||
Errors:
|
||||
|
||||
- `409 {"detail": "memory_mode_not_ultra"}`: the effective mode is not ultra.
|
||||
- `422`: a malformed `user_id`, `max_batches` outside its range, or an extra
|
||||
field.
|
||||
|
||||
### Run receipt
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": "uuid",
|
||||
"user_id": "local",
|
||||
"agent_id": "all-apps",
|
||||
"max_batches": null,
|
||||
"trigger": "manual",
|
||||
"state": "succeeded",
|
||||
"started_at": "2026-11-01T03:00:00Z",
|
||||
"finished_at": "2026-11-01T03:00:42Z",
|
||||
"rounds": 2,
|
||||
"tasks_processed": 5,
|
||||
"batch_failures": 0,
|
||||
"cycle_status": "complete",
|
||||
"should_continue": false,
|
||||
"continuation_reason": null,
|
||||
"remaining_clusters": 0,
|
||||
"remaining_actionable_clusters": 0,
|
||||
"remaining_fresh_facts": 0,
|
||||
"capped": false,
|
||||
"elapsed_ms": 41873.2,
|
||||
"error_code": null,
|
||||
"failure_kind": null,
|
||||
"request_id": "sdk-request-id",
|
||||
"results": {},
|
||||
"cross_domain_sweeper": {}
|
||||
}
|
||||
```
|
||||
|
||||
- `trigger` is `manual` or `scheduled`.
|
||||
- `state` is `running`, `succeeded` or `failed`.
|
||||
- `cycle_status` is the SDK's `partial` or `complete`.
|
||||
- `error_code` and `failure_kind` are stable codes on failure. Examples:
|
||||
`memory_evolve_cancelled` (the Runtime stopped mid-run),
|
||||
`hy_memory_evolve_rejected` (the SDK answered `success: false` without a
|
||||
code), or the SDK's own `error_code`. Provider messages never appear.
|
||||
- `results` and `cross_domain_sweeper` are the last round's SDK bodies. They
|
||||
live in memory only and are never written to disk or diagnostics.
|
||||
|
||||
### `GET /api/v2/evolve/{run_id}?user_id=` → receipt
|
||||
|
||||
Returns `404 {"detail": "evolve_run_not_found"}` in two cases:
|
||||
|
||||
- the run belongs to another user (answered exactly like a missing run);
|
||||
- the Runtime restarted since the run. Receipts live in memory, and the last
|
||||
64 are kept.
|
||||
|
||||
### `GET /api/v2/evolve?user_id=` → overview
|
||||
|
||||
```json
|
||||
{
|
||||
"mode": "ultra",
|
||||
"schedule": {"enabled": true, "time": "03:00"},
|
||||
"users": {
|
||||
"local": {
|
||||
"last_run_at": "2026-11-01T03:00:00Z",
|
||||
"last_result": {"state": "succeeded", "rounds": 2, "...": "receipt without results"},
|
||||
"last_error": null
|
||||
}
|
||||
},
|
||||
"running": []
|
||||
}
|
||||
```
|
||||
|
||||
- Without `user_id` the overview covers every user; with it, that one user.
|
||||
- `users` survives restarts. It is kept in `<data_dir>/evolve-state.json`,
|
||||
counts and codes only.
|
||||
|
||||
## 6. Nightly schedule
|
||||
|
||||
### `GET /api/v1/settings/evolve-schedule`
|
||||
|
||||
Returns `{"enabled": true, "time": "03:00"}`. `time` is `HH:MM`, 24-hour, in
|
||||
the machine's local time.
|
||||
|
||||
### `PUT /api/v1/settings/evolve-schedule`
|
||||
|
||||
Body: `{"enabled"?: bool, "time"?: "HH:MM"}`. An absent field keeps its
|
||||
value. The response is the saved schedule. A bad time answers `422`.
|
||||
|
||||
How the schedule runs:
|
||||
|
||||
- It runs only in ultra.
|
||||
- It checks every 5 minutes. A user's run is due when either holds:
|
||||
- today's slot has passed, and the user's last run is older than that slot;
|
||||
- the user's last run is more than 24 hours old.
|
||||
|
||||
The second rule is how a machine that slept through 03:00 catches up when
|
||||
it wakes.
|
||||
- A user who never ran is measured from the first time the schedule ran in
|
||||
ultra, not from the epoch. So switching to ultra at noon does not evolve
|
||||
everyone at once; their first run is at the next slot.
|
||||
- Users run one after another.
|
||||
|
||||
The schedule runs every user that has memories. The SDK cannot list users, so
|
||||
Core takes them from its own records:
|
||||
|
||||
- `local`;
|
||||
- every user holding a source grant (a Hermes profile);
|
||||
- every user whose turns or notes passed the write queue.
|
||||
|
||||
A user is kept when `memory_stats` reports any memory or fact for them.
|
||||
|
||||
## 7. Status fields (R10)
|
||||
|
||||
`GET /api/v1/memory-processing` has a new optional field, `pipeline`:
|
||||
|
||||
```json
|
||||
{
|
||||
"pipeline": {
|
||||
"mode": "ultra",
|
||||
"configured_mode": "ultra",
|
||||
"forced_by_env": false,
|
||||
"evolve": {"...": "the overview of section 5"},
|
||||
"last_evolve_at": "2026-11-01T03:00:00Z",
|
||||
"graph_build": {"...": "as in section 3"},
|
||||
"features": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`features` lists the SDK features whose value differs from the default (empty
|
||||
while every feature is at its default); see `memory-features-v1.md`.
|
||||
|
||||
`memhub status --json` adds a top-level projection of it:
|
||||
|
||||
```json
|
||||
{
|
||||
"pipeline": {
|
||||
"mode": "ultra",
|
||||
"configured_mode": "ultra",
|
||||
"forced_by_env": false,
|
||||
"last_evolve_at": "...",
|
||||
"evolve_schedule": {"enabled": true, "time": "03:00"},
|
||||
"evolve_running": 0,
|
||||
"evolve_users": {
|
||||
"local": {"last_run_at": "...", "last_error": null, "state": "succeeded"}
|
||||
},
|
||||
"graph_build": "succeeded",
|
||||
"features": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`memhub progress --json` carries the same `pipeline`. Both are `null` against
|
||||
an older Core.
|
||||
|
||||
## 8. CLI
|
||||
|
||||
```bash
|
||||
memhub settings mode # show the current mode
|
||||
memhub settings mode lite # print the plan, change nothing
|
||||
memhub settings mode lite --yes # switch
|
||||
memhub evolve [--user-id <id>] [--agent-id <id>] [--max-batches <n>] [--wait] [--json]
|
||||
memhub settings features [set <name>=<true|false> ... [--yes] | reset [<name> ...]]
|
||||
```
|
||||
|
||||
The SDK's optional retrieval and extraction features (graph rerank, episode
|
||||
recall, the episode write side, …) are a second global setting beside the
|
||||
mode. A change rebuilds the SDK handles the same way. Contract:
|
||||
`memory-features-v1.md`.
|
||||
|
||||
## 9. Copy the UI can reuse
|
||||
|
||||
| Key | zh-CN | en | es |
|
||||
|---|---|---|---|
|
||||
| title | 记忆模式 | Memory mode | Modo de memoria |
|
||||
| lite | lite:不调用模型,保留全部原始对话,只在原始对话里检索。 | lite: no model calls. Every original conversation is kept, and search reads only those conversations. | lite: sin llamadas al modelo. Se conservan todas las conversaciones originales y la búsqueda solo las consulta a ellas. |
|
||||
| pro | pro:由模型从对话中提取记忆(System 1)。 | pro: a model extracts memories from your conversations (System 1). | pro: un modelo extrae memorias de tus conversaciones (System 1). |
|
||||
| ultra | ultra:在 pro 之上每晚自动整合记忆(System 2,evolve),并使用知识图谱。 | ultra: pro plus nightly memory consolidation (System 2, evolve), using a knowledge graph. | ultra: pro más una consolidación nocturna de las memorias (System 2, evolve), con un grafo de conocimiento. |
|
||||
| confirm | 切换到 {mode} 前请确认: | Before switching to {mode}, please confirm: | Antes de cambiar a {mode}, confirma lo siguiente: |
|
||||
| earlier_raw_not_extracted | lite 下已处理完的对话不会再被提取;只有尚未处理的和之后的新对话会由模型整理成记忆。 | Conversations already handled in lite are not extracted again; only conversations still waiting, and new ones, become memories. | Las conversaciones ya procesadas en lite no se vuelven a extraer; solo las pendientes y las nuevas se convierten en memorias. |
|
||||
| lite_search_stops | 只在原始对话里的检索会立即停止,改为检索记忆。 | Search over original conversations stops now; search reads memories instead. | La búsqueda en las conversaciones originales se detiene ahora; se buscará en las memorias. |
|
||||
| higher_mode_memories_hidden | 更高模式生成的记忆仍然保存,但在这个模式下不会显示。 | Memories created by a higher mode stay stored but are not shown in this mode. | Las memorias creadas por un modo superior se conservan, pero no se muestran en este modo. |
|
||||
| evolve_stopped | 每晚的自动整合(System 2)会停止。 | Nightly consolidation (System 2) stops. | La consolidación nocturna (System 2) se detiene. |
|
||||
| existing_memories_added_to_graph | 已有记忆会在后台加入知识图谱。 | Your existing memories are added to the knowledge graph in the background. | Tus memorias actuales se añaden al grafo de conocimiento en segundo plano. |
|
||||
| graph_backfill_already_running | 知识图谱正在补建中。 | The knowledge graph is still being built. | El grafo de conocimiento todavía se está construyendo. |
|
||||
| raw_kept | 原始对话全部保留,任何模式都不删除。 | Every original conversation is kept, in every mode. | Se conservan todas las conversaciones originales, en cualquier modo. |
|
||||
| graph_build_running | 正在把已有记忆加入知识图谱… | Adding existing memories to the knowledge graph… | Añadiendo las memorias actuales al grafo de conocimiento… |
|
||||
| forced_by_env | 开发环境变量 ONE_MEMORY_GRAPH_STORE 正在强制使用 ultra。 | The development variable ONE_MEMORY_GRAPH_STORE is forcing ultra. | La variable de desarrollo ONE_MEMORY_GRAPH_STORE está forzando ultra. |
|
||||
| graph_store_unavailable | 这个安装缺少知识图谱组件,无法切换到 ultra。 | This installation lacks the knowledge-graph component, so it cannot switch to ultra. | A esta instalación le falta el componente del grafo de conocimiento; no puede cambiar a ultra. |
|
||||
| schedule | 每天 {time} 自动整合 | Consolidates daily at {time} | Consolida cada día a las {time} |
|
||||
| last_evolve | 上次整合:{time} | Last consolidation: {time} | Última consolidación: {time} |
|
||||
| never_evolved | 尚未整合 | Not consolidated yet | Aún no consolidado |
|
||||
| evolve_not_ultra | 记忆整合只在 ultra 模式下可用。 | Consolidation is available in ultra mode only. | La consolidación solo está disponible en el modo ultra. |
|
||||
@@ -26,8 +26,7 @@ and no raw turn, and does not compare the mirror with the SDK:
|
||||
cannot be trusted to decide what the SDK should contain;
|
||||
- the product has no user editing of memories in practice, so the mirror holds
|
||||
no user intent the SDK lacks;
|
||||
- SDK L1 raw is retained for three days by the Runtime's own
|
||||
`prune_expired_raw`, not by the migration.
|
||||
- SDK L1 raw is kept: neither the migration nor the Runtime deletes raw.
|
||||
|
||||
"Already complete" holds only if the new Runtime opens the same stores. Builds
|
||||
before #145 named a profile's vector collections `one_memory_<12 hex>…`; #145
|
||||
|
||||
@@ -636,6 +636,16 @@ export interface SdkRawRecord {
|
||||
memory_at: number | null;
|
||||
gmt_created: number | null;
|
||||
custom: Record<string, unknown>;
|
||||
/** active, or shadow once pro extracted it. */
|
||||
status?: string | null;
|
||||
/** The turn's messages, parsed from the SDK's `[role]: text` lines. */
|
||||
messages?: { role: string; content: string }[];
|
||||
/** The one role of a single-message turn; null for a mixed turn. */
|
||||
role?: string | null;
|
||||
/** `sha256:<hex>` of the stored content, computed by Core. */
|
||||
content_hash?: string;
|
||||
/** ISO time the SDK stored the raw. */
|
||||
created?: string | null;
|
||||
}
|
||||
|
||||
export interface SdkRawPage {
|
||||
@@ -643,6 +653,37 @@ export interface SdkRawPage {
|
||||
total: number;
|
||||
limit: number;
|
||||
offset: number;
|
||||
/** A time range was applied to this page in Core (the SDK list has none). */
|
||||
time_filtered?: boolean;
|
||||
}
|
||||
|
||||
/** `GET /api/v2/memory-library/layers`: memories per SDK layer. */
|
||||
export interface SdkLayerCounts {
|
||||
layers: Record<string, number>;
|
||||
total: number;
|
||||
statuses: string[] | null;
|
||||
}
|
||||
|
||||
/** `GET /api/v2/user-profile/versions`: the L0 basic-info version chains. */
|
||||
export interface SdkProfileVersions {
|
||||
user_id: string;
|
||||
request_id: string;
|
||||
elapsed_ms: number;
|
||||
chains: {
|
||||
memory_id: string;
|
||||
content: string;
|
||||
status: string;
|
||||
memory_at: number | null;
|
||||
gmt_created: number | null;
|
||||
basic_info: Record<string, unknown>;
|
||||
versions: {
|
||||
memory_id: string;
|
||||
content: string;
|
||||
memory_at: number | string | null;
|
||||
gmt_created: number | null;
|
||||
current: boolean;
|
||||
}[];
|
||||
}[];
|
||||
}
|
||||
|
||||
export interface FileImportReceipt {
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "hy-memory"
|
||||
version = "2.0.9"
|
||||
version = "2.0.10"
|
||||
description = "HY Memory - Industrial-grade dual-system cognitive memory framework"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.9"
|
||||
|
||||
@@ -0,0 +1,376 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
raw 在写入时就裁剪好;历史拼接不再二次裁剪。
|
||||
|
||||
规则(hy_memory/utils/raw_trim.py):
|
||||
- user / system 完整保留
|
||||
- assistant 与对方真人(非已知 role 的 label)strip 后超过 500 字符截成前 500 + "..."
|
||||
- tool 头尾保留、中段替换为 "[...N bytes omitted by truncate...]"
|
||||
(上限 coding.tool_result_max_bytes)
|
||||
- 全部幂等
|
||||
|
||||
覆盖:
|
||||
1. 规则本身 + 幂等(含 coding preproc 的两个截断函数)
|
||||
2. lite:L1_RAW 存的是裁剪后的文本(tool 头尾截断 + 标记),带 raw_trimmed 标记;
|
||||
审计 history 的 ADD 记录同样裁剪
|
||||
3. coding_trace 存的 tool result 已头尾截断
|
||||
4. pro:extractor 输入仍是完整 messages;历史 = 已存 raw 直接拼接;
|
||||
没有标记的旧 raw 仍按同一规则兼容裁剪
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sqlite3
|
||||
import types
|
||||
import uuid
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import pytest
|
||||
|
||||
from hy_memory.coding.preproc import (
|
||||
truncate_assistant_text,
|
||||
truncate_tool_result_text,
|
||||
)
|
||||
from hy_memory.models.memory import MemoryLayer, MemoryNode, MemoryStatus
|
||||
from hy_memory.pipelines.base import ChatMessage, WriteRequest
|
||||
from hy_memory.utils.raw_trim import (
|
||||
DEFAULT_RAW_ASSISTANT_MAX_CHARS,
|
||||
RAW_TRIMMED_KEY,
|
||||
trim_raw_body,
|
||||
trim_raw_messages,
|
||||
)
|
||||
|
||||
_MARKER = "bytes omitted by truncate...]"
|
||||
_LONG_USER = "我把整段日志贴给你:" + "U" * 5000
|
||||
_LONG_ASSISTANT = "A" * 1200
|
||||
_LONG_TOOL = "H" * 3000 + "MIDDLE" + "T" * 3000
|
||||
|
||||
|
||||
def _messages() -> List[ChatMessage]:
|
||||
return [
|
||||
ChatMessage(role="system", content="S" * 900),
|
||||
ChatMessage(role="user", content=_LONG_USER),
|
||||
ChatMessage(role="assistant", content=_LONG_ASSISTANT),
|
||||
ChatMessage(role="tool", content=_LONG_TOOL, tool_call_id="c1", tool_name="Bash"),
|
||||
ChatMessage(role="assistant", name="John", content="J" * 800),
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 1. 规则
|
||||
|
||||
|
||||
class TestTrimRule:
|
||||
def test_user_kept_in_full(self):
|
||||
assert trim_raw_body("user", _LONG_USER) == _LONG_USER
|
||||
|
||||
def test_system_kept_in_full(self):
|
||||
assert trim_raw_body("system", "S" * 900) == "S" * 900
|
||||
|
||||
def test_long_assistant_cut_at_limit(self):
|
||||
out = trim_raw_body("assistant", _LONG_ASSISTANT)
|
||||
assert out == "A" * DEFAULT_RAW_ASSISTANT_MAX_CHARS + "..."
|
||||
|
||||
def test_short_assistant_untouched(self):
|
||||
assert trim_raw_body("assistant", " short ") == " short "
|
||||
|
||||
def test_other_speaker_cut_like_assistant(self):
|
||||
assert trim_raw_body("John", "J" * 800) == "J" * 500 + "..."
|
||||
|
||||
def test_tool_head_tail_with_marker(self):
|
||||
out = trim_raw_body("tool", _LONG_TOOL, tool_max_bytes=2048)
|
||||
assert out.startswith("H" * 100) and out.endswith("T" * 100)
|
||||
assert "MIDDLE" not in out and _MARKER in out
|
||||
assert len(out.encode("utf-8")) <= 2048
|
||||
|
||||
def test_messages_trimmed_without_mutating_input(self):
|
||||
msgs = _messages()
|
||||
out = trim_raw_messages(msgs, tool_max_bytes=512)
|
||||
assert [m.content for m in msgs][1] == _LONG_USER
|
||||
assert msgs[2].content == _LONG_ASSISTANT, "入参不能被修改"
|
||||
assert out[1].content == _LONG_USER
|
||||
assert out[2].content == "A" * 500 + "..."
|
||||
assert _MARKER in out[3].content and out[3].tool_call_id == "c1"
|
||||
assert out[4].content == "J" * 500 + "..." and out[4].name == "John"
|
||||
|
||||
@pytest.mark.parametrize("max_bytes", [48, 64, 200, 512, 2048])
|
||||
def test_idempotent(self, max_bytes):
|
||||
once = trim_raw_messages(_messages(), tool_max_bytes=max_bytes)
|
||||
twice = trim_raw_messages(once, tool_max_bytes=max_bytes)
|
||||
assert [m.content for m in twice] == [m.content for m in once]
|
||||
for label in ("assistant", "John", "tool", "user", "system"):
|
||||
for body in (_LONG_ASSISTANT, _LONG_TOOL, "中文" * 2000, " x " * 400):
|
||||
t1 = trim_raw_body(label, body, tool_max_bytes=max_bytes)
|
||||
assert trim_raw_body(label, t1, tool_max_bytes=max_bytes) == t1
|
||||
|
||||
def test_preproc_truncations_idempotent(self):
|
||||
t = truncate_tool_result_text(_LONG_TOOL, max_bytes=64)
|
||||
assert truncate_tool_result_text(t, max_bytes=64) == t
|
||||
a = truncate_assistant_text("x" * 10000, head_chars=30, tail_chars=30)
|
||||
assert truncate_assistant_text(a, head_chars=30, tail_chars=30) == a
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 2. lite 端到端
|
||||
|
||||
|
||||
def _lite_config(tmp_path):
|
||||
from hy_memory import MemoryConfig
|
||||
|
||||
return MemoryConfig.from_dict({
|
||||
"vector_store": {
|
||||
"provider": "chroma",
|
||||
"collection_name": f"test_raw_trim_{uuid.uuid4().hex[:8]}",
|
||||
"persist_directory": str(tmp_path / "chroma"),
|
||||
"embedding_dims": 8,
|
||||
},
|
||||
"embedder": {"embedding_dims": 8},
|
||||
"cache": {"backend": "sqlite", "sqlite_path": str(tmp_path / "cache.db")},
|
||||
"history": {"enable": True, "db_path": str(tmp_path / "history.db")},
|
||||
"coding": {"enable": True, "tool_result_max_bytes": 512},
|
||||
})
|
||||
|
||||
|
||||
def _install_fixed_embed(client):
|
||||
vector = [1.0] + [0.0] * 7
|
||||
|
||||
async def one(_self, _text, use_cache=True):
|
||||
return list(vector)
|
||||
|
||||
async def one_meta(_self, _text, use_cache=True):
|
||||
return list(vector), {"cache_hit": False, "provider": "test"}
|
||||
|
||||
async def batch(_self, texts, use_cache=True):
|
||||
return [list(vector) for _ in texts]
|
||||
|
||||
s = client._embed_service
|
||||
s.embed = types.MethodType(one, s)
|
||||
s.embed_queued = types.MethodType(one, s)
|
||||
s.embed_with_meta = types.MethodType(one_meta, s)
|
||||
s.embed_queued_with_meta = types.MethodType(one_meta, s)
|
||||
s.embed_batch = types.MethodType(batch, s)
|
||||
|
||||
|
||||
def test_lite_stores_trimmed_raw(tmp_path, monkeypatch):
|
||||
from hy_memory import HyMemoryClient
|
||||
|
||||
monkeypatch.setenv("MEMORY_DATA_DIR", str(tmp_path / "data"))
|
||||
monkeypatch.setenv("MEMORY_LOG_DIR", str(tmp_path / "logs"))
|
||||
monkeypatch.setenv("MEMORY_PIPELINE_TRACE_ENABLED", "false")
|
||||
client = HyMemoryClient(_lite_config(tmp_path), mode="lite")
|
||||
try:
|
||||
_install_fixed_embed(client)
|
||||
resp = client.add(
|
||||
[
|
||||
{"role": "user", "content": _LONG_USER},
|
||||
{"role": "assistant", "content": _LONG_ASSISTANT,
|
||||
"tool_calls": [{"id": "c1", "type": "function",
|
||||
"function": {"name": "Bash", "arguments": "{}"}}]},
|
||||
{"role": "tool", "tool_call_id": "c1", "content": _LONG_TOOL},
|
||||
],
|
||||
user_id="u1",
|
||||
)
|
||||
assert resp["success"] is True, resp
|
||||
|
||||
async def _raw_nodes():
|
||||
return await client._vector_store.list_by_user(
|
||||
user_id="u1", layers=[MemoryLayer.L1_RAW],
|
||||
with_vectors=False, allow_raw_layer=True,
|
||||
)
|
||||
|
||||
nodes = client._loop_thread.run(_raw_nodes())
|
||||
assert len(nodes) == 1
|
||||
content = nodes[0].content
|
||||
assert f"[user]: {_LONG_USER}" in content, "user 必须完整保留"
|
||||
assert f"[assistant]: {'A' * 500}...\n" in content
|
||||
tool_line = content.split("[tool]: ", 1)[1]
|
||||
assert tool_line.startswith("H") and tool_line.endswith("T")
|
||||
assert _MARKER in tool_line and "MIDDLE" not in tool_line
|
||||
assert len(tool_line.encode("utf-8")) <= 512
|
||||
assert (nodes[0].custom or {}).get(RAW_TRIMMED_KEY) is True
|
||||
|
||||
conn = sqlite3.connect(str(tmp_path / "history.db"))
|
||||
try:
|
||||
(audit,) = conn.execute(
|
||||
"SELECT new_memory FROM memory_history WHERE event='ADD'"
|
||||
).fetchone()
|
||||
finally:
|
||||
conn.close()
|
||||
assert _LONG_USER in audit and "MIDDLE" not in audit
|
||||
assert "A" * 501 not in audit
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 3. coding_trace
|
||||
|
||||
|
||||
def test_coding_trace_stores_truncated_tool_result(tmp_path):
|
||||
from hy_memory.coding.writer import CodingWriter
|
||||
|
||||
captured: Dict[str, Any] = {}
|
||||
|
||||
class _Store:
|
||||
config = types.SimpleNamespace(
|
||||
coding=types.SimpleNamespace(tool_result_max_bytes=512),
|
||||
extractor=types.SimpleNamespace(history_turns=5),
|
||||
)
|
||||
|
||||
async def get_recent_trace_messages(self, **kw):
|
||||
return []
|
||||
|
||||
async def append_trace(self, messages, **kw):
|
||||
captured["messages"] = messages
|
||||
|
||||
class _Extractor:
|
||||
async def extract(self, messages, **kw):
|
||||
return [], []
|
||||
|
||||
w = CodingWriter.__new__(CodingWriter)
|
||||
w.variant = "working"
|
||||
w.store = _Store()
|
||||
w.extractor = _Extractor()
|
||||
asyncio.run(w.write(_messages(), user_id="u1"))
|
||||
|
||||
stored = {m.role: m.content for m in captured["messages"] if m.role != "assistant"}
|
||||
assert stored["user"] == _LONG_USER
|
||||
assert _MARKER in stored["tool"] and "MIDDLE" not in stored["tool"]
|
||||
assert len(stored["tool"].encode("utf-8")) <= 512
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 4. pro writer
|
||||
|
||||
|
||||
class _Store:
|
||||
supports_fulltext = False
|
||||
|
||||
def __init__(self):
|
||||
self.nodes: Dict[str, MemoryNode] = {}
|
||||
|
||||
async def initialize(self):
|
||||
pass
|
||||
|
||||
async def upsert(self, node: MemoryNode) -> str:
|
||||
node.node_id = node.node_id or uuid.uuid4().hex
|
||||
if node.gmt_created is None:
|
||||
node.gmt_created = datetime.now()
|
||||
self.nodes[node.node_id] = node
|
||||
return node.node_id
|
||||
|
||||
async def get_by_id(self, node_id):
|
||||
return self.nodes.get(node_id)
|
||||
|
||||
async def list_by_user(self, *a, layers=None, **kw):
|
||||
return [
|
||||
n for n in self.nodes.values()
|
||||
if not layers or n.layer in layers
|
||||
]
|
||||
|
||||
async def search(self, *a, **kw):
|
||||
return []
|
||||
|
||||
|
||||
class _Embed:
|
||||
async def embed_queued_with_meta(self, text):
|
||||
return [0.0] * 8, {"provider": "test"}
|
||||
|
||||
async def embed_queued(self, text):
|
||||
return [0.0] * 8
|
||||
|
||||
async def embed_batch(self, texts):
|
||||
return [[0.0] * 8 for _ in texts]
|
||||
|
||||
|
||||
class _CapturingAgent:
|
||||
def __init__(self):
|
||||
self.calls: List[Dict[str, Any]] = []
|
||||
|
||||
async def process_add(self, **kwargs):
|
||||
from tests.test_shadow_l1_raw_on_empty_extract import _FakeAgentResult
|
||||
|
||||
self.calls.append(kwargs)
|
||||
return _FakeAgentResult(success=True, extracted_info={"memory": [], "intentions": []})
|
||||
|
||||
|
||||
def _pro_writer(store):
|
||||
from hy_memory.config import MemoryConfig
|
||||
from hy_memory.pipelines.writer import MemoryWriter
|
||||
|
||||
w = MemoryWriter(MemoryConfig(), embed_service=_Embed(), vector_store=store, cache=None)
|
||||
w._mem_agent = _CapturingAgent()
|
||||
w._vector_store = store
|
||||
|
||||
async def _empty_tags(*a, **kw):
|
||||
return []
|
||||
|
||||
w._collect_existing_tags = _empty_tags
|
||||
return w
|
||||
|
||||
|
||||
def _write(w, msgs: List[ChatMessage]):
|
||||
req = WriteRequest(
|
||||
content="", messages=msgs, user_id="u1", agent_id="default_agent",
|
||||
session_id="s1", content_type="messages",
|
||||
extra={"agent_mode": "full", "mode": "pro"},
|
||||
request_id=uuid.uuid4().hex,
|
||||
)
|
||||
return asyncio.run(w.write(req))
|
||||
|
||||
|
||||
def test_pro_extraction_input_unchanged_and_history_concatenates():
|
||||
store = _Store()
|
||||
w = _pro_writer(store)
|
||||
turn1 = [
|
||||
ChatMessage(role="user", content=_LONG_USER),
|
||||
ChatMessage(role="assistant", content=_LONG_ASSISTANT),
|
||||
]
|
||||
turn2 = [
|
||||
ChatMessage(role="user", content="第二轮"),
|
||||
ChatMessage(role="assistant", content="B" * 900),
|
||||
]
|
||||
r1 = _write(w, turn1)
|
||||
assert r1.memory_id
|
||||
r2 = _write(w, turn2)
|
||||
assert r2.memory_id
|
||||
|
||||
calls = w._mem_agent.calls
|
||||
assert len(calls) == 2
|
||||
# extractor 收到的仍是完整 messages(没有被落库裁剪影响)
|
||||
for call, msgs in zip(calls, (turn1, turn2)):
|
||||
assert json.loads(call["content"]) == [
|
||||
{"role": m.role, "content": m.content} for m in msgs
|
||||
]
|
||||
|
||||
raw1 = store.nodes[r1.memory_id]
|
||||
assert raw1.content == f"[user]: {_LONG_USER}\n[assistant]: {'A' * 500}..."
|
||||
assert raw1.custom.get(RAW_TRIMMED_KEY) is True
|
||||
# 历史 = 已存 raw 直接拼接(第一次写入时没有历史)
|
||||
assert calls[0]["history_context"] == ""
|
||||
assert calls[1]["history_context"] == raw1.content
|
||||
|
||||
|
||||
def test_history_does_not_retrim_new_raw_but_trims_legacy_raw():
|
||||
"""新 raw(带标记)原样拼接;旧 raw(无标记)按同一规则兼容裁剪。"""
|
||||
from hy_memory.pipelines.writer import MemoryWriter
|
||||
|
||||
store = _Store()
|
||||
base = datetime.now()
|
||||
# 新 raw:内容里刻意放一条超过 500 的 assistant——带标记就不应再被裁
|
||||
new_content = "[user]: 新问题\n[assistant]: " + "N" * 700
|
||||
legacy_content = "[user]: 旧问题\n[assistant]: " + "O" * 700 + "\n[tool]: " + _LONG_TOOL
|
||||
store.nodes["legacy"] = MemoryNode(
|
||||
node_id="legacy", user_id="u1", agent_id="default_agent",
|
||||
layer=MemoryLayer.L1_RAW, content=legacy_content,
|
||||
status=MemoryStatus.ACTIVE, gmt_created=base - timedelta(seconds=2),
|
||||
)
|
||||
store.nodes["new"] = MemoryNode(
|
||||
node_id="new", user_id="u1", agent_id="default_agent",
|
||||
layer=MemoryLayer.L1_RAW, content=new_content,
|
||||
status=MemoryStatus.ACTIVE, gmt_created=base - timedelta(seconds=1),
|
||||
custom={RAW_TRIMMED_KEY: True},
|
||||
)
|
||||
w = MemoryWriter.__new__(MemoryWriter)
|
||||
history = asyncio.run(w._get_recent_history(store, "u1", "default_agent", max_turns=20))
|
||||
|
||||
assert history.endswith(new_content), "新 raw 必须原样拼接"
|
||||
assert "[assistant]: " + "O" * 500 + "...\n" in history
|
||||
assert "MIDDLE" not in history and _MARKER in history
|
||||
@@ -0,0 +1,642 @@
|
||||
"""Re-embed into a new embedding space: vector stores, scan, diff and merge.
|
||||
|
||||
The switch model: writes are paused, every vector is computed again with the
|
||||
new model into a new space, the source is never touched. The graph half lives
|
||||
in ``test_reembed_graph.py``.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import random
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from hy_memory import (
|
||||
EmbeddingDimensionMismatchError,
|
||||
MemoryConfig,
|
||||
ReembedCancelled,
|
||||
VectorSpace,
|
||||
diff_spaces,
|
||||
drop_vector_space,
|
||||
plan_reembed,
|
||||
reembed_text,
|
||||
reembed_vector_space,
|
||||
scan_vector_spaces,
|
||||
space_identity,
|
||||
vector_space_names,
|
||||
verify_reembed,
|
||||
)
|
||||
from hy_memory.config import EmbedderConfig
|
||||
from hy_memory.core.embed_service import EmbedService
|
||||
from hy_memory.models.memory import MemoryLayer, MemoryNode, MemoryStatus
|
||||
from hy_memory.reembed import VECTOR_ROLES, parse_collection_name
|
||||
|
||||
|
||||
class FakeEmbedder:
|
||||
"""Deterministic vectors: same (model, text) -> same unit vector."""
|
||||
|
||||
def __init__(self, model: str, dims: int, *, wrong_width_after: int = -1):
|
||||
self.model = model
|
||||
self.dims = dims
|
||||
self.calls = []
|
||||
self.texts = 0
|
||||
self.wrong_width_after = wrong_width_after
|
||||
|
||||
def vector(self, text: str):
|
||||
seed = int(hashlib.sha256(f"{self.model}\0{text}".encode()).hexdigest()[:16], 16)
|
||||
rng = random.Random(seed)
|
||||
values = [rng.gauss(0.0, 1.0) for _ in range(self.dims)]
|
||||
norm = math.sqrt(sum(v * v for v in values)) or 1.0
|
||||
return [v / norm for v in values]
|
||||
|
||||
async def embed_batch(self, texts, use_cache=True):
|
||||
self.calls.append(list(texts))
|
||||
out = []
|
||||
for text in texts:
|
||||
self.texts += 1
|
||||
vector = self.vector(text)
|
||||
if 0 <= self.wrong_width_after < self.texts:
|
||||
vector = vector + [0.0]
|
||||
out.append(vector)
|
||||
return out
|
||||
|
||||
async def embed(self, text, use_cache=True):
|
||||
return self.vector(text)
|
||||
|
||||
async def embed_queued(self, text, use_cache=True):
|
||||
return self.vector(text)
|
||||
|
||||
def get_embedding_dims(self):
|
||||
return self.dims
|
||||
|
||||
|
||||
def _client(path):
|
||||
import chromadb
|
||||
return chromadb.PersistentClient(path=str(path))
|
||||
|
||||
|
||||
def _space(path, base, dims, model=None):
|
||||
return VectorSpace(base_name=base, dims=dims, persist_directory=str(path), model=model)
|
||||
|
||||
|
||||
def _custom_state():
|
||||
return {
|
||||
"memory_kind": "canonical_state",
|
||||
"state_key": "home_city",
|
||||
"state_value": "Berlin",
|
||||
"state_retrieval_text": "user lives in Berlin (home city)",
|
||||
}
|
||||
|
||||
|
||||
async def _seed(path, base, dims, *, model="old-model", users=("alice", "bob"), per_user=7):
|
||||
"""Write a realistic source space through the SDK's own stores."""
|
||||
from hy_memory.data.vector_store_chroma import ChromaVectorStore
|
||||
|
||||
old = FakeEmbedder(model, dims)
|
||||
cfg = MemoryConfig()
|
||||
cfg.vector_store.provider = "chroma"
|
||||
cfg.vector_store.persist_directory = str(path)
|
||||
cfg.vector_store.collection_name = base
|
||||
cfg.vector_store.embedding_dims = dims
|
||||
store = ChromaVectorStore(cfg)
|
||||
await store.initialize()
|
||||
nodes = []
|
||||
for user in users:
|
||||
for i in range(per_user):
|
||||
layer = [MemoryLayer.L1_RAW, MemoryLayer.L2_FACT, MemoryLayer.L3_SUMMARY][i % 3]
|
||||
node = MemoryNode(
|
||||
user_id=user, agent_id="a", session_id=f"s{i % 2}",
|
||||
layer=layer, content=f"{user} fact {i}",
|
||||
status=MemoryStatus.SUPERSEDED if i == 4 else MemoryStatus.ACTIVE,
|
||||
tags=["t1", f"t{i}"],
|
||||
custom={"k": i, "nested": {"x": [1, 2]}},
|
||||
)
|
||||
if i == 4:
|
||||
node.is_latest = False
|
||||
node.superseded_by = ["later"]
|
||||
if i == 5:
|
||||
node.custom = _custom_state()
|
||||
node.embedding = old.vector(node.content)
|
||||
nodes.append(node)
|
||||
await store.upsert_batch(nodes)
|
||||
entities = await store._get_entity_collection()
|
||||
entities.add(
|
||||
ids=["e1", "e2"],
|
||||
embeddings=[old.vector("Berlin"), old.vector("Dylan")],
|
||||
documents=["Berlin", "Dylan"],
|
||||
metadatas=[{"user_id": "alice", "label": "GPE"}, {"user_id": "bob", "label": "PERSON"}],
|
||||
)
|
||||
await store.close()
|
||||
client = _client(path)
|
||||
names = vector_space_names(_space(path, base, dims))
|
||||
for role, doc in (("coding_keys", "fix the flaky test"), ("working_facts_cv", "repo: uses uv")):
|
||||
coll = client.get_or_create_collection(names[role], metadata={"hnsw:space": "cosine"})
|
||||
coll.add(
|
||||
ids=[f"{role}:0"],
|
||||
embeddings=[old.vector(doc)],
|
||||
documents=[doc],
|
||||
metadatas=[{"user_id": "alice", "layer": "l1_raw",
|
||||
"custom": json.dumps({"coding_key_text": doc})}],
|
||||
)
|
||||
return names
|
||||
|
||||
|
||||
def _snapshot(path, names):
|
||||
client = _client(path)
|
||||
out = {}
|
||||
for role, name in names.items():
|
||||
try:
|
||||
coll = client.get_collection(name)
|
||||
except Exception:
|
||||
continue
|
||||
got = coll.get(include=["documents", "metadatas", "embeddings"])
|
||||
out[role] = sorted(
|
||||
(rid, doc, json.dumps(meta, sort_keys=True), [round(float(x), 6) for x in emb])
|
||||
for rid, doc, meta, emb in zip(got["ids"], got["documents"], got["metadatas"], got["embeddings"])
|
||||
)
|
||||
return hashlib.sha256(json.dumps(out, sort_keys=True).encode()).hexdigest()
|
||||
|
||||
|
||||
def _records(path, name):
|
||||
got = _client(path).get_collection(name).get(include=["documents", "metadatas", "embeddings"])
|
||||
return {
|
||||
rid: (doc, meta, list(emb))
|
||||
for rid, doc, meta, emb in zip(got["ids"], got["documents"], got["metadatas"], got["embeddings"])
|
||||
}
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# naming
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
class TestVectorSpaceNames:
|
||||
@pytest.mark.parametrize("dims", [512, None])
|
||||
async def test_names_equal_what_the_stores_open(self, tmp_path, dims):
|
||||
from hy_memory.coding.store import CodingMemoryStore
|
||||
from hy_memory.data.vector_store_chroma import ChromaVectorStore
|
||||
|
||||
cfg = MemoryConfig()
|
||||
cfg.vector_store.provider = "chroma"
|
||||
cfg.vector_store.persist_directory = str(tmp_path / "v")
|
||||
cfg.vector_store.collection_name = "memhub_abc123"
|
||||
cfg.vector_store.embedding_dims = dims
|
||||
cfg.history.db_path = str(tmp_path / "history.sqlite3")
|
||||
store = ChromaVectorStore(cfg)
|
||||
await store.initialize()
|
||||
await store._get_entity_collection()
|
||||
coding = []
|
||||
for variant in ("working", "coding"):
|
||||
cs = CodingMemoryStore(cfg, FakeEmbedder("m", dims or 8), variant=variant)
|
||||
await cs.initialize()
|
||||
coding.append(cs)
|
||||
opened = {c.name for c in _client(tmp_path / "v").list_collections()}
|
||||
names = vector_space_names(cfg)
|
||||
assert set(names) == set(VECTOR_ROLES)
|
||||
assert set(names.values()) == opened
|
||||
suffix = f"_{dims}" if dims else ""
|
||||
assert names["memory"] == f"memhub_abc123{suffix}"
|
||||
assert names["coding_keys_cv"] == f"memhub_abc123_coding_keys_cv{suffix}"
|
||||
await store.close()
|
||||
for cs in coding:
|
||||
await cs.close()
|
||||
|
||||
def test_explicit_base_name_and_space(self, tmp_path):
|
||||
cfg = MemoryConfig()
|
||||
cfg.vector_store.provider = "chroma"
|
||||
cfg.vector_store.embedding_dims = 1024
|
||||
names = vector_space_names(cfg, base_name="staging_x")
|
||||
assert names["memory"] == "staging_x_1024"
|
||||
assert names["entities"] == "staging_x_1024_entities"
|
||||
assert names["working_facts"] == "staging_x_working_facts_1024"
|
||||
assert vector_space_names(_space(tmp_path, "b", 512))["memory"] == "b_512"
|
||||
|
||||
def test_parse_collection_name_round_trips(self):
|
||||
for dims in (512, None):
|
||||
for role, name in vector_space_names(VectorSpace("memhub_9f", dims)).items():
|
||||
assert parse_collection_name(name) == ("memhub_9f", dims, role)
|
||||
assert parse_collection_name("memhub_9f_512_tag_index") is None
|
||||
|
||||
def test_space_identity_is_model_and_width_only(self):
|
||||
a = space_identity("BAAI/bge-small-zh-v1.5", 512)
|
||||
b = space_identity(" BAAI/bge-small-zh-v1.5 ", 512)
|
||||
assert a == b and a.key == "BAAI/bge-small-zh-v1.5#512"
|
||||
assert a.digest != space_identity("BAAI/bge-small-zh-v1.5", 1024).digest
|
||||
with pytest.raises(ValueError):
|
||||
space_identity("", 512)
|
||||
|
||||
def test_non_chroma_provider_is_not_supported(self):
|
||||
cfg = MemoryConfig()
|
||||
cfg.vector_store.provider = "qdrant"
|
||||
with pytest.raises(NotImplementedError):
|
||||
vector_space_names(cfg)
|
||||
|
||||
|
||||
class TestReembedText:
|
||||
def test_canonical_state_uses_state_retrieval_text(self):
|
||||
meta = {"content": "住在柏林", "custom": json.dumps(_custom_state())}
|
||||
assert reembed_text("memory", "住在柏林", meta) == "user lives in Berlin (home city)"
|
||||
|
||||
def test_retrieval_text_when_present(self):
|
||||
meta = {"custom": json.dumps({"retrieval_text": " pattern: plans ahead "})}
|
||||
assert reembed_text("memory", "画像", meta) == "pattern: plans ahead"
|
||||
|
||||
def test_document_otherwise_with_content_fallback(self):
|
||||
assert reembed_text("memory", "plain fact", {"custom": "{}"}) == "plain fact"
|
||||
assert reembed_text("memory", None, {"content": "from metadata"}) == "from metadata"
|
||||
assert reembed_text("memory", "doc", {"custom": "not json"}) == "doc"
|
||||
|
||||
def test_coding_key_and_working_fact_use_the_document(self):
|
||||
coding = {"custom": json.dumps({"coding_key_text": "fix flaky test",
|
||||
"retrieval_text": "ignored outside memory"})}
|
||||
assert reembed_text("coding_keys", "fix flaky test", coding) == "fix flaky test"
|
||||
assert reembed_text("working_facts", "repo: uses uv", {}) == "repo: uses uv"
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# re-embed
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"src_dims,tgt_dims,src_model,tgt_model",
|
||||
[(512, 512, "old-model", "new-model"), (512, 1024, "old", "new"), (1024, 512, "old", "new")],
|
||||
)
|
||||
async def test_reembed_copies_everything_and_never_touches_source(
|
||||
tmp_path, src_dims, tgt_dims, src_model, tgt_model,
|
||||
):
|
||||
path = tmp_path / "v"
|
||||
names = await _seed(path, "memhub_src", src_dims, model=src_model)
|
||||
before = _snapshot(path, names)
|
||||
source = _space(path, "memhub_src", src_dims)
|
||||
target = _space(path, "memhub_dst", tgt_dims, model=tgt_model)
|
||||
new = FakeEmbedder(tgt_model, tgt_dims)
|
||||
|
||||
plan = await plan_reembed(source, target)
|
||||
assert plan.per_role["memory"] == 14 and plan.per_role["entities"] == 2
|
||||
assert plan.total_vectors == 18 == plan.estimated_vectors
|
||||
assert plan.source_dims == src_dims and plan.target_dims == tgt_dims
|
||||
|
||||
seen = []
|
||||
report = await reembed_vector_space(
|
||||
source, target, embedding=new, batch_size=4, on_progress=seen.append,
|
||||
)
|
||||
assert report.embedded == 18 and report.reused == 0
|
||||
assert seen and seen[-1].done == seen[-1].total
|
||||
|
||||
assert _snapshot(path, names) == before # source unchanged, byte for byte
|
||||
target_names = vector_space_names(target)
|
||||
for role in ("memory", "entities", "coding_keys", "working_facts_cv"):
|
||||
src = _records(path, names[role])
|
||||
dst = _records(path, target_names[role])
|
||||
assert set(src) == set(dst)
|
||||
for rid, (doc, meta, _) in src.items():
|
||||
assert dst[rid][0] == doc
|
||||
assert dst[rid][1] == meta # status, is_latest, superseded_by, timestamps, custom
|
||||
assert len(dst[rid][2]) == tgt_dims
|
||||
assert dst[rid][2] == pytest.approx(new.vector(reembed_text(role, doc, meta)), abs=1e-5)
|
||||
memory = _records(path, target_names["memory"])
|
||||
layers = {meta["layer"] for _, meta, _ in memory.values()}
|
||||
users = {meta["user_id"] for _, meta, _ in memory.values()}
|
||||
assert "l1_raw" in layers and users == {"alice", "bob"}
|
||||
statuses = {meta["status"] for _, meta, _ in memory.values()}
|
||||
assert statuses == {"active", "superseded"}
|
||||
state = [r for r in memory.values() if "canonical_state" in r[1]["custom"]]
|
||||
assert state and state[0][2] == pytest.approx(
|
||||
new.vector("user lives in Berlin (home city)"), abs=1e-5,
|
||||
)
|
||||
meta = _client(path).get_collection(target_names["memory"]).metadata
|
||||
assert meta["hnsw:space"] == "cosine" and meta["hy_embedding_model"] == tgt_model
|
||||
|
||||
verified = await verify_reembed(source, target, embedding=new)
|
||||
assert verified.ok, verified.problems
|
||||
assert verified.per_user == {"alice": 10, "bob": 8}
|
||||
assert verified.self_recall_samples == 18 and verified.self_recall_hits == 18
|
||||
|
||||
|
||||
async def test_resume_after_kill_only_embeds_the_rest(tmp_path):
|
||||
path = tmp_path / "v"
|
||||
await _seed(path, "src", 512, per_user=20)
|
||||
source, target = _space(path, "src", 512), _space(path, "dst", 1024)
|
||||
|
||||
class Killed(Exception):
|
||||
pass
|
||||
|
||||
first = FakeEmbedder("new", 1024)
|
||||
batches = {"n": 0}
|
||||
|
||||
def _kill():
|
||||
batches["n"] += 1
|
||||
if batches["n"] > 3:
|
||||
raise Killed()
|
||||
return False
|
||||
|
||||
with pytest.raises(Killed):
|
||||
await reembed_vector_space(
|
||||
source, target, embedding=first, batch_size=5, max_inflight=1,
|
||||
roles=["memory"], cancel=_kill,
|
||||
)
|
||||
assert first.texts == 15
|
||||
second = FakeEmbedder("new", 1024)
|
||||
report = await reembed_vector_space(
|
||||
source, target, embedding=second, batch_size=5, max_inflight=1, roles=["memory"],
|
||||
)
|
||||
assert second.texts == 40 - 15
|
||||
assert report.per_role["memory"].reused == 15
|
||||
# The end state equals an uninterrupted run: every source record, verbatim,
|
||||
# with the vector of its own text.
|
||||
resumed = _records(path, "dst_1024")
|
||||
src = _records(path, "src_512")
|
||||
assert set(resumed) == set(src)
|
||||
for rid, (doc, meta, vector) in resumed.items():
|
||||
assert (doc, meta) == src[rid][:2]
|
||||
assert vector == pytest.approx(second.vector(reembed_text("memory", doc, meta)), abs=1e-5)
|
||||
assert (await verify_reembed(source, target, roles=["memory"], samples=0)).ok
|
||||
|
||||
|
||||
async def test_cancel_at_batch_boundary_leaves_no_partial_batch(tmp_path):
|
||||
path = tmp_path / "v"
|
||||
await _seed(path, "src", 512)
|
||||
calls = {"n": 0}
|
||||
|
||||
def _cancel():
|
||||
calls["n"] += 1
|
||||
return calls["n"] > 2
|
||||
|
||||
with pytest.raises(ReembedCancelled) as info:
|
||||
await reembed_vector_space(
|
||||
_space(path, "src", 512), _space(path, "dst", 512),
|
||||
embedding=FakeEmbedder("new", 512), batch_size=3, max_inflight=1,
|
||||
roles=["memory"], cancel=_cancel,
|
||||
)
|
||||
assert info.value.report.per_role["memory"].embedded == 6
|
||||
assert _client(path).get_collection("dst_512").count() == 6
|
||||
|
||||
|
||||
async def test_wrong_width_raises_before_any_upsert(tmp_path):
|
||||
path = tmp_path / "v"
|
||||
await _seed(path, "src", 512)
|
||||
bad = FakeEmbedder("new", 1024, wrong_width_after=2)
|
||||
with pytest.raises(EmbeddingDimensionMismatchError):
|
||||
await reembed_vector_space(
|
||||
_space(path, "src", 512), _space(path, "dst", 1024),
|
||||
embedding=bad, batch_size=4, roles=["memory"],
|
||||
)
|
||||
assert _client(path).get_collection("dst_1024").count() == 0
|
||||
declared_wrong = FakeEmbedder("new", 768)
|
||||
with pytest.raises(EmbeddingDimensionMismatchError):
|
||||
await reembed_vector_space(
|
||||
_space(path, "src", 512), _space(path, "dst", 1024), embedding=declared_wrong,
|
||||
)
|
||||
|
||||
|
||||
async def test_refuses_to_write_into_the_source_collection(tmp_path):
|
||||
path = tmp_path / "v"
|
||||
await _seed(path, "src", 512)
|
||||
with pytest.raises(ValueError):
|
||||
await reembed_vector_space(
|
||||
_space(path, "src", 512), _space(path, "src", 512),
|
||||
embedding=FakeEmbedder("new", 512),
|
||||
)
|
||||
|
||||
|
||||
async def test_verify_detects_missing_extra_drift_and_width(tmp_path):
|
||||
path = tmp_path / "v"
|
||||
await _seed(path, "src", 512)
|
||||
source, target = _space(path, "src", 512), _space(path, "dst", 512)
|
||||
new = FakeEmbedder("new", 512)
|
||||
await reembed_vector_space(source, target, embedding=new, roles=["memory"])
|
||||
coll = _client(path).get_collection("dst_512")
|
||||
got = coll.get(include=["documents", "metadatas"])
|
||||
victim, drifted = got["ids"][0], got["ids"][1]
|
||||
coll.delete(ids=[victim])
|
||||
meta = dict(got["metadatas"][1])
|
||||
meta["status"] = "archived"
|
||||
coll.update(ids=[drifted], metadatas=[meta])
|
||||
coll.add(ids=["stranger"], embeddings=[new.vector("x")], documents=["x"], metadatas=[{"user_id": "eve"}])
|
||||
result = await verify_reembed(source, target, embedding=new, roles=["memory"], samples=5)
|
||||
assert not result.ok
|
||||
counts = result.per_role["memory"]
|
||||
assert (counts["missing"], counts["extra"], counts["mismatched"]) == (1, 1, 1)
|
||||
assert any("eve" in p for p in result.problems)
|
||||
|
||||
wide = _space(path, "dst", 1024)
|
||||
wide_coll = _client(path).get_or_create_collection("dst_1024", metadata={"hnsw:space": "cosine"})
|
||||
src = _records(path, "src_512")
|
||||
wide_coll.add(
|
||||
ids=list(src),
|
||||
embeddings=[[0.0] * 512 for _ in src],
|
||||
documents=[v[0] for v in src.values()],
|
||||
metadatas=[v[1] for v in src.values()],
|
||||
)
|
||||
result = await verify_reembed(source, wide, roles=["memory"], samples=0)
|
||||
assert result.per_role["memory"]["bad_dims"] == len(src) and not result.ok
|
||||
|
||||
|
||||
async def test_drop_vector_space_refuses_live_collections(tmp_path):
|
||||
from hy_memory.data.vector_store_chroma import ChromaVectorStore
|
||||
|
||||
path = tmp_path / "v"
|
||||
await _seed(path, "src", 512)
|
||||
cfg = MemoryConfig()
|
||||
cfg.vector_store.provider = "chroma"
|
||||
cfg.vector_store.persist_directory = str(path)
|
||||
cfg.vector_store.collection_name = "src"
|
||||
cfg.vector_store.embedding_dims = 512
|
||||
live = ChromaVectorStore(cfg)
|
||||
await live.initialize()
|
||||
with pytest.raises(RuntimeError):
|
||||
await drop_vector_space(_space(path, "src", 512))
|
||||
await live.close()
|
||||
dropped = await drop_vector_space(_space(path, "src", 512))
|
||||
assert set(dropped) == {"src_512", "src_512_entities", "src_coding_keys_512", "src_working_facts_cv_512"}
|
||||
assert not {c.name for c in _client(path).list_collections()}
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# scan, diff, merge
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
async def _two_spaces(path):
|
||||
"""A live 1024 space and a stray 512 space sharing some ids."""
|
||||
from hy_memory.data.vector_store_chroma import ChromaVectorStore
|
||||
|
||||
async def _store(base, dims, model, contents):
|
||||
cfg = MemoryConfig()
|
||||
cfg.vector_store.provider = "chroma"
|
||||
cfg.vector_store.persist_directory = str(path)
|
||||
cfg.vector_store.collection_name = base
|
||||
cfg.vector_store.embedding_dims = dims
|
||||
store = ChromaVectorStore(cfg)
|
||||
await store.initialize()
|
||||
emb = FakeEmbedder(model, dims)
|
||||
nodes = []
|
||||
for node_id, user, content, status in contents:
|
||||
node = MemoryNode(
|
||||
node_id=node_id, user_id=user, agent_id="a", session_id="s",
|
||||
layer=MemoryLayer.L2_FACT, content=content, status=status,
|
||||
)
|
||||
node.embedding = emb.vector(content)
|
||||
nodes.append(node)
|
||||
await store.upsert_batch(nodes)
|
||||
await store.close()
|
||||
|
||||
ids = [f"00000000-0000-0000-0000-00000000000{i}" for i in range(6)]
|
||||
await _store("memhub_old", 512, "bge", [
|
||||
(ids[0], "alice", "shared one", MemoryStatus.ACTIVE),
|
||||
(ids[1], "alice", "shared two (stray wording)", MemoryStatus.ACTIVE),
|
||||
(ids[2], "bob", "only in stray", MemoryStatus.ACTIVE),
|
||||
(ids[3], "bob", "only in stray, deleted", MemoryStatus.SHADOW),
|
||||
])
|
||||
await _store("memhub_new", 1024, "e5", [
|
||||
(ids[0], "alice", "shared one", MemoryStatus.ACTIVE),
|
||||
(ids[1], "alice", "shared two (live wording)", MemoryStatus.ARCHIVED),
|
||||
(ids[4], "carol", "only in live", MemoryStatus.ACTIVE),
|
||||
])
|
||||
return ids
|
||||
|
||||
|
||||
async def test_scan_diff_and_merge(tmp_path):
|
||||
path = tmp_path / "v"
|
||||
ids = await _two_spaces(path)
|
||||
spaces = await scan_vector_spaces(path)
|
||||
by_base = {(s.base_name, s.dims): s for s in spaces}
|
||||
assert set(by_base) == {("memhub_old", 512), ("memhub_new", 1024)}
|
||||
old, live = by_base[("memhub_old", 512)], by_base[("memhub_new", 1024)]
|
||||
assert old.roles["memory"].count == 4 and old.width == 512
|
||||
assert live.roles["memory"].count == 3 and live.width == 1024
|
||||
assert old.per_user == {"alice": 2, "bob": 2}
|
||||
assert live.per_user == {"alice": 2, "carol": 1}
|
||||
|
||||
stray, target = old.space(), _space(path, "memhub_new", 1024, model="e5")
|
||||
diff = await diff_spaces(stray, target)
|
||||
assert set(diff.missing["memory"]) == {ids[2], ids[3]} # status ignored
|
||||
assert diff.total_missing == 2
|
||||
|
||||
live_before = _records(path, "memhub_new_1024")
|
||||
stray_before = _snapshot(path, stray.names())
|
||||
merger = FakeEmbedder("e5", 1024)
|
||||
report = await reembed_vector_space(stray, target, embedding=merger)
|
||||
assert merger.texts == 2 and report.embedded == 2
|
||||
assert report.reused == 1 and report.kept == 1
|
||||
after = _records(path, "memhub_new_1024")
|
||||
assert set(after) == set(live_before) | {ids[2], ids[3]}
|
||||
for rid, record in live_before.items(): # the target wins
|
||||
assert after[rid][:2] == record[:2]
|
||||
assert after[rid][2] == pytest.approx(record[2], abs=1e-6)
|
||||
assert _snapshot(path, stray.names()) == stray_before
|
||||
assert (await diff_spaces(stray, target)).empty
|
||||
|
||||
again = FakeEmbedder("e5", 1024)
|
||||
second = await reembed_vector_space(stray, target, embedding=again)
|
||||
assert again.texts == 0 and second.embedded == 0
|
||||
assert _records(path, "memhub_new_1024").keys() == after.keys()
|
||||
rescanned = {(s.base_name, s.dims): s for s in await scan_vector_spaces(path)}
|
||||
assert rescanned[("memhub_new", 1024)].roles["memory"].count == 5
|
||||
|
||||
|
||||
async def test_scan_of_missing_directory_is_empty(tmp_path):
|
||||
assert await scan_vector_spaces(tmp_path / "nope") == []
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# EmbedService fixes
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
def _fake_raw(n, dims):
|
||||
items = []
|
||||
for i in range(n):
|
||||
item = MagicMock()
|
||||
item.embedding = [0.1] * dims
|
||||
item.index = i
|
||||
items.append(item)
|
||||
resp = MagicMock()
|
||||
resp.model = "m"
|
||||
resp.usage = None
|
||||
resp.data = items
|
||||
raw = MagicMock()
|
||||
raw.status_code = 200
|
||||
raw.headers = {}
|
||||
raw.parse.return_value = resp
|
||||
return raw
|
||||
|
||||
|
||||
def _service(dims=8, **embedder):
|
||||
cfg = MemoryConfig(embedder=EmbedderConfig(
|
||||
provider="openai", model="m", api_key="sk", base_url="https://e.invalid/v1",
|
||||
embedding_dims=dims, max_retries=2, retry_delay=0.0, **embedder,
|
||||
))
|
||||
return EmbedService(cfg)
|
||||
|
||||
|
||||
async def test_embed_batch_chunks_to_batch_size():
|
||||
svc = _service()
|
||||
sizes = []
|
||||
|
||||
async def _create(**kwargs):
|
||||
sizes.append(len(kwargs["input"]))
|
||||
return _fake_raw(len(kwargs["input"]), 8)
|
||||
|
||||
client = MagicMock()
|
||||
client.embeddings.with_raw_response.create = AsyncMock(side_effect=_create)
|
||||
with patch("openai.AsyncOpenAI", return_value=client):
|
||||
vectors = await svc.embed_batch([f"t{i}" for i in range(1000)], use_cache=False)
|
||||
assert len(vectors) == 1000
|
||||
assert len(sizes) == math.ceil(1000 / 64) and max(sizes) == 64
|
||||
|
||||
small = _service(batch_size=10)
|
||||
sizes.clear()
|
||||
with patch("openai.AsyncOpenAI", return_value=client):
|
||||
_, metas = await small.embed_batch_with_meta([f"u{i}" for i in range(25)], use_cache=False)
|
||||
assert sizes == [10, 10, 5]
|
||||
assert [m["batch_index"] for m in metas] == list(range(25))
|
||||
|
||||
|
||||
async def test_429_honours_retry_after():
|
||||
svc = _service()
|
||||
|
||||
class RateLimited(Exception):
|
||||
status_code = 429
|
||||
|
||||
def __init__(self):
|
||||
super().__init__("rate limited")
|
||||
self.response = MagicMock()
|
||||
self.response.headers = {"retry-after": "2"}
|
||||
|
||||
client = MagicMock()
|
||||
client.embeddings.with_raw_response.create = AsyncMock(
|
||||
side_effect=[RateLimited(), _fake_raw(1, 8)],
|
||||
)
|
||||
sleeps = []
|
||||
|
||||
async def _sleep(seconds):
|
||||
sleeps.append(seconds)
|
||||
|
||||
with patch("openai.AsyncOpenAI", return_value=client), \
|
||||
patch("hy_memory.core.embed_service.asyncio.sleep", _sleep):
|
||||
vectors = await svc.embed_batch(["x"], use_cache=False)
|
||||
assert len(vectors) == 1
|
||||
assert len(sleeps) == 1 and 2.0 <= sleeps[0] <= 2.5
|
||||
|
||||
|
||||
async def test_per_call_max_retries():
|
||||
svc = _service()
|
||||
|
||||
class Down(Exception):
|
||||
status_code = 503
|
||||
|
||||
client = MagicMock()
|
||||
client.embeddings.with_raw_response.create = AsyncMock(side_effect=Down("down"))
|
||||
with patch("openai.AsyncOpenAI", return_value=client), pytest.raises(Down):
|
||||
await svc.embed_batch(["x"], use_cache=False, max_retries=5)
|
||||
assert client.embeddings.with_raw_response.create.await_count == 6
|
||||
|
||||
|
||||
async def test_returned_width_is_validated_and_not_retried():
|
||||
svc = _service(dims=8)
|
||||
client = MagicMock()
|
||||
client.embeddings.with_raw_response.create = AsyncMock(return_value=_fake_raw(2, 7))
|
||||
with patch("openai.AsyncOpenAI", return_value=client), \
|
||||
pytest.raises(EmbeddingDimensionMismatchError):
|
||||
await svc.embed_batch(["a", "b"], use_cache=False)
|
||||
assert client.embeddings.with_raw_response.create.await_count == 1
|
||||
@@ -0,0 +1,252 @@
|
||||
"""Re-embed the Kuzu graph into a new embedding space.
|
||||
|
||||
The source graph is opened read-only and never modified. A new graph is built
|
||||
at the target path with the target width: every node and edge is copied, the
|
||||
embedding columns are recomputed from their text, the HNSW index is rebuilt.
|
||||
Runs in CI: kuzu is a core dependency.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from hy_memory import (
|
||||
EmbeddingDimensionMismatchError,
|
||||
MemoryConfig,
|
||||
ReembedCancelled,
|
||||
drop_graph,
|
||||
reembed_graph,
|
||||
verify_graph_reembed,
|
||||
)
|
||||
from hy_memory.models.memory import MemoryLayer, MemoryNode
|
||||
from hy_memory.reembed import hierarchy_topic_label, plan_graph_reembed
|
||||
|
||||
from .test_reembed import FakeEmbedder
|
||||
|
||||
|
||||
def _cfg(path, dims):
|
||||
cfg = MemoryConfig()
|
||||
cfg.graph_store.provider = "kuzu"
|
||||
cfg.graph_store.db_path = str(path)
|
||||
cfg.vector_store.embedding_dims = dims
|
||||
return cfg
|
||||
|
||||
|
||||
async def _open(path, dims):
|
||||
from hy_memory.data.graph_store_kuzu import KuzuGraphStore
|
||||
|
||||
store = KuzuGraphStore(_cfg(path, dims))
|
||||
await store.initialize()
|
||||
assert store._available
|
||||
return store
|
||||
|
||||
|
||||
def _node(content, *, layer=MemoryLayer.L2_FACT, custom=None, user="alice"):
|
||||
return MemoryNode(
|
||||
user_id=user, agent_id="a", session_id="s1",
|
||||
layer=layer, content=content, custom=custom or {},
|
||||
)
|
||||
|
||||
|
||||
async def _seed_graph(path, dims, model="old"):
|
||||
"""A graph with every node and edge type the SDK writes."""
|
||||
old = FakeEmbedder(model, dims)
|
||||
gs = await _open(path, dims)
|
||||
schema = _node(
|
||||
"画像:喜欢提前规划", layer=MemoryLayer.L6_SCHEMA,
|
||||
custom={"schema_type": "basic", "retrieval_text": "plans trips weeks ahead",
|
||||
"_has_beh": True, "beh_abstraction": "prepares early under uncertainty"},
|
||||
)
|
||||
schema._graph_embedding = old.vector("plans trips weeks ahead")
|
||||
core = _node("core: anticipates", layer=MemoryLayer.L6_SCHEMA, custom={"schema_type": "abstract"})
|
||||
core._graph_embedding = old.vector("core: anticipates")
|
||||
stale_beh = _node("beh without text", layer=MemoryLayer.L6_SCHEMA, custom={})
|
||||
stale_beh._graph_embedding = old.vector("beh without text")
|
||||
facts = [_node(f"fact {i}", user="bob" if i % 2 else "alice") for i in range(5)]
|
||||
for node in [schema, core, stale_beh, *facts]:
|
||||
await gs.upsert_memory_node(node)
|
||||
await gs.update_embedding(schema.node_id, beh_embedding=old.vector("prepares early under uncertainty"))
|
||||
await gs.update_embedding(stale_beh.node_id, beh_embedding=old.vector("lost text"))
|
||||
await gs.add_cross_abstracts_to(schema.node_id, core.node_id)
|
||||
ik = facts[0].get_isolation_key()
|
||||
await gs.add_edge(facts[0].node_id, facts[1].node_id, "RELATED_TO", {"relation_type": "co_occurs", "weight": 0.7})
|
||||
await gs.add_temporal_edge(facts[0].node_id, facts[2].node_id, topic_id="t")
|
||||
for fact in facts[:3]:
|
||||
await gs.add_topic_tag(fact.node_id, "travel", ik, embed_service=old)
|
||||
await gs.add_topic_tag(facts[3].node_id, "work", ik) # no embed service: NULL embedding
|
||||
await gs.add_topic_path(
|
||||
facts[0].node_id, ["Travel", "Japan trip"], ik, "alice",
|
||||
node_types=["topic", "series"], embed_service=old,
|
||||
)
|
||||
await gs.add_topic_path(facts[1].node_id, ["Work", "Standups"], ik, "alice", embed_service=old)
|
||||
await gs.add_entity_mention(facts[0].node_id, "Kyoto", "GPE", ik)
|
||||
await gs.ensure_vdbref("raw-1", "l1_raw")
|
||||
await gs.add_derived_from(schema.node_id, "raw-1")
|
||||
await gs.close()
|
||||
return old, {"schema": schema, "core": core, "stale_beh": stale_beh, "facts": facts}
|
||||
|
||||
|
||||
def _digest(path):
|
||||
"""Hash of the Kuzu DB file and any sidecar files next to it."""
|
||||
h = hashlib.sha256()
|
||||
for name in sorted(os.listdir(path.parent)):
|
||||
if name == path.name or name.startswith(path.name + "."):
|
||||
h.update(name.encode())
|
||||
with open(path.parent / name, "rb") as handle:
|
||||
h.update(handle.read())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _read(path, query):
|
||||
import kuzu
|
||||
|
||||
db = kuzu.Database(str(path), read_only=True)
|
||||
conn = kuzu.Connection(db)
|
||||
try:
|
||||
result = conn.execute(query)
|
||||
rows = []
|
||||
while result.has_next():
|
||||
rows.append(result.get_next())
|
||||
return rows
|
||||
finally:
|
||||
conn.close()
|
||||
db.close()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"src_dims,tgt_dims,src_model,tgt_model",
|
||||
[(512, 512, "old", "new"), (512, 1024, "old", "new"), (1024, 512, "old", "new")],
|
||||
)
|
||||
async def test_reembed_graph_copies_everything_at_the_new_width(
|
||||
tmp_path, src_dims, tgt_dims, src_model, tgt_model,
|
||||
):
|
||||
source, target = tmp_path / "graph", tmp_path / "graph-new"
|
||||
_, nodes = await _seed_graph(source, src_dims, model=src_model)
|
||||
before = _digest(source)
|
||||
plan = await plan_graph_reembed(source, target_dims=tgt_dims)
|
||||
assert plan.source_width == src_dims and plan.target_width == tgt_dims
|
||||
assert plan.memory_nodes == 8 and plan.topics == 2 and plan.hierarchy_topics == 4
|
||||
assert plan.memory_embeddings == 3 and plan.beh_embeddings == 2
|
||||
assert plan.topic_embeddings == 1 and plan.hierarchy_topic_embeddings == 4
|
||||
|
||||
new = FakeEmbedder(tgt_model, tgt_dims)
|
||||
seen = []
|
||||
report = await reembed_graph(
|
||||
source, target, embedding=new, target_dims=tgt_dims, batch_size=3, on_progress=seen.append,
|
||||
)
|
||||
assert _digest(source) == before # the source graph is untouched
|
||||
assert not os.path.exists(str(target) + ".reembed-partial")
|
||||
assert report.source_width == src_dims and report.target_width == tgt_dims
|
||||
assert report.node_counts == plan.node_counts
|
||||
assert report.rel_counts == plan.rel_counts
|
||||
assert report.embedded == {
|
||||
"Memory.embedding": 3, "Memory.beh_embedding": 1,
|
||||
"Topic.embedding": 1, "HierarchyTopic.embedding": 4,
|
||||
}
|
||||
assert report.cleared == {"Memory.beh_embedding": 1} # beh vector with no beh_abstraction
|
||||
assert report.hierarchy_labels_exact == 4 and report.hierarchy_labels_inexact == ()
|
||||
assert report.vector_indexes == ("memory_content_idx",)
|
||||
assert seen
|
||||
|
||||
check = await verify_graph_reembed(source, target, target_dims=tgt_dims)
|
||||
assert check.ok, check.problems
|
||||
assert check.vector_index and check.target_width == tgt_dims
|
||||
assert all(src == dst for src, dst in check.node_counts.values())
|
||||
assert all(src == dst for src, dst in check.rel_counts.values())
|
||||
|
||||
schema = nodes["schema"]
|
||||
rows = _read(target, f"MATCH (m:Memory {{node_id: '{schema.node_id}'}}) RETURN m.embedding, m.beh_embedding, m.custom_json, m.content;")
|
||||
emb, beh, custom, content = rows[0]
|
||||
assert len(emb) == tgt_dims
|
||||
assert emb == pytest.approx(new.vector("plans trips weeks ahead"), abs=1e-5)
|
||||
assert beh == pytest.approx(new.vector("prepares early under uncertainty"), abs=1e-5)
|
||||
assert json.loads(custom)["retrieval_text"] == "plans trips weeks ahead" and content == schema.content
|
||||
rows = _read(target, f"MATCH (m:Memory {{node_id: '{nodes['core'].node_id}'}}) RETURN m.embedding;")
|
||||
assert rows[0][0] == pytest.approx(new.vector("core: anticipates"), abs=1e-5)
|
||||
rows = _read(target, f"MATCH (m:Memory {{node_id: '{nodes['facts'][0].node_id}'}}) RETURN m.embedding IS NULL;")
|
||||
assert rows[0][0] is True # a NULL source vector stays NULL
|
||||
topics = dict(_read(target, "MATCH (t:Topic) RETURN t.name, t.embedding;"))
|
||||
assert topics["travel"] == pytest.approx(new.vector("travel"), abs=1e-5)
|
||||
assert topics["work"] is None
|
||||
for subject, path_json, embedding in _read(target, "MATCH (t:HierarchyTopic) RETURN t.subject, t.path_json, t.embedding;"):
|
||||
label, exact = hierarchy_topic_label(subject, path_json)
|
||||
assert exact and embedding == pytest.approx(new.vector(label), abs=1e-5)
|
||||
|
||||
# The rebuilt graph serves the SDK: vector search on the new index works.
|
||||
gs = await _open(target, tgt_dims)
|
||||
try:
|
||||
hits = await gs.vector_search(
|
||||
query_embedding=new.vector("plans trips weeks ahead"),
|
||||
isolation_key=schema.get_isolation_key(), limit=1,
|
||||
)
|
||||
assert hits and hits[0]["node_id"] == schema.node_id
|
||||
width = await gs._run("CALL table_info('Memory') RETURN *;")
|
||||
assert dict((r[1], r[2]) for r in width)["embedding"] == f"FLOAT[{tgt_dims}]"
|
||||
finally:
|
||||
await gs.close()
|
||||
|
||||
|
||||
async def test_source_labels_rebuild_exactly(tmp_path):
|
||||
"""The stored HierarchyTopic vectors were computed from the rebuilt label."""
|
||||
source = tmp_path / "graph"
|
||||
old, _ = await _seed_graph(source, 64)
|
||||
rows = _read(source, "MATCH (t:HierarchyTopic) RETURN t.subject, t.path_json, t.embedding;")
|
||||
assert len(rows) == 4
|
||||
for subject, path_json, embedding in rows:
|
||||
label, exact = hierarchy_topic_label(subject, path_json)
|
||||
assert exact
|
||||
assert embedding == pytest.approx(old.vector(label), abs=1e-6)
|
||||
|
||||
|
||||
def test_legacy_path_json_is_flagged_inexact():
|
||||
label, exact = hierarchy_topic_label("alice", json.dumps(["Travel", "Japan"]), "Japan")
|
||||
assert label == "alice: topic:Travel > facet:Japan" and not exact
|
||||
label, exact = hierarchy_topic_label("alice", "", "Japan")
|
||||
assert label == "alice: Japan" and not exact
|
||||
|
||||
|
||||
async def test_cancel_and_failure_leave_no_target(tmp_path):
|
||||
source, target = tmp_path / "graph", tmp_path / "graph-new"
|
||||
await _seed_graph(source, 64)
|
||||
with pytest.raises(ReembedCancelled):
|
||||
await reembed_graph(
|
||||
source, target, embedding=FakeEmbedder("new", 32), target_dims=32,
|
||||
cancel=lambda: True,
|
||||
)
|
||||
with pytest.raises(EmbeddingDimensionMismatchError):
|
||||
await reembed_graph(
|
||||
source, target, embedding=FakeEmbedder("new", 32, wrong_width_after=1), target_dims=32,
|
||||
)
|
||||
assert not os.path.exists(target)
|
||||
assert not os.path.exists(str(target) + ".reembed-partial")
|
||||
# A restart starts over and succeeds.
|
||||
report = await reembed_graph(source, target, embedding=FakeEmbedder("new", 32), target_dims=32)
|
||||
assert report.node_counts["Memory"] == 8
|
||||
with pytest.raises(FileExistsError):
|
||||
await reembed_graph(source, target, embedding=FakeEmbedder("new", 32), target_dims=32)
|
||||
again = await reembed_graph(
|
||||
source, target, embedding=FakeEmbedder("new", 32), target_dims=32, overwrite=True,
|
||||
)
|
||||
assert again.rel_counts == report.rel_counts
|
||||
|
||||
|
||||
async def test_refuses_open_source_and_drops_only_closed_graphs(tmp_path):
|
||||
source, target = tmp_path / "graph", tmp_path / "graph-new"
|
||||
await _seed_graph(source, 64)
|
||||
live = await _open(source, 64)
|
||||
try:
|
||||
with pytest.raises(RuntimeError):
|
||||
await reembed_graph(source, target, embedding=FakeEmbedder("new", 64), target_dims=64)
|
||||
with pytest.raises(RuntimeError):
|
||||
await drop_graph(source)
|
||||
finally:
|
||||
await live.close()
|
||||
with pytest.raises(ValueError):
|
||||
await reembed_graph(source, source, embedding=FakeEmbedder("new", 64), target_dims=64)
|
||||
await reembed_graph(source, target, embedding=FakeEmbedder("new", 64), target_dims=64)
|
||||
assert await drop_graph(target) is True
|
||||
assert not os.path.exists(target)
|
||||
assert await drop_graph(target) is False
|
||||
assert os.path.exists(source)
|
||||
Reference in new issue
Block a user