Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
35f86dc96d | ||
|
|
7b7fd217a8 | ||
|
|
7efe054e83 | ||
|
|
fdbcd9934d | ||
|
|
3171007b3c |
No files matched your search
@@ -14,6 +14,13 @@ Hy Memory and MemHub are versioned and released independently.
|
|||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
- Serialized each SDK SQLite store's shared connection lifecycle so concurrent
|
||||||
|
history/cache operations cannot race with `close()` and crash the process.
|
||||||
|
- Clarified that `add(metadata=...)` carries legacy SDK processing options,
|
||||||
|
while persistent caller business context belongs in `meta`.
|
||||||
|
- Added an opt-in `get(..., include_raw=True)` path for reading `L1_RAW`
|
||||||
|
memories, including memories written in lite mode. The default remains
|
||||||
|
hidden for backward compatibility.
|
||||||
- Removed MemHub's task-scoped HyPermission store wrappers and global
|
- Removed MemHub's task-scoped HyPermission store wrappers and global
|
||||||
model-content consent switch. The trusted embedded Hy Memory SDK now uses
|
model-content consent switch. The trusted embedded Hy Memory SDK now uses
|
||||||
its native local stores directly.
|
its native local stores directly.
|
||||||
|
|||||||
@@ -106,6 +106,15 @@ result = client.add(
|
|||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`metadata` 与 `meta` 不是同一字段:
|
||||||
|
|
||||||
|
- `metadata` 是兼容旧调用方保留的 SDK 处理选项,会进入内部
|
||||||
|
`WriteRequest.extra`,不保证持久化或随查询结果返回。
|
||||||
|
- `meta` 是调用方的业务上下文,会持久化并在 `search()`、`get()` 等结果中返回。
|
||||||
|
同一记忆被多次合并时,各次 `meta` 会按写入顺序保存在列表中。
|
||||||
|
|
||||||
|
需要以后读回的数据必须放在 `meta`,不要放在 `metadata`。
|
||||||
|
|
||||||
`scene` 的公开取值:
|
`scene` 的公开取值:
|
||||||
|
|
||||||
- `None`:普通文本直接走 chat;messages 含 tool 时执行快速场景判断。
|
- `None`:普通文本直接走 chat;messages 含 tool 时执行快速场景判断。
|
||||||
@@ -263,6 +272,8 @@ catalog 也可以在没有基础 L0 时形成只读的合成画像。
|
|||||||
|
|
||||||
```python
|
```python
|
||||||
memory = client.get("memory_id")
|
memory = client.get("memory_id")
|
||||||
|
# lite 模式写入的是 L1_RAW;按 ID 读取时必须显式选择返回原文。
|
||||||
|
raw_memory = client.get("memory_id", include_raw=True)
|
||||||
result = client.update("memory_id", "更新后的内容")
|
result = client.update("memory_id", "更新后的内容")
|
||||||
result = client.delete("memory_id")
|
result = client.delete("memory_id")
|
||||||
result = client.delete_all(user_id="user_001")
|
result = client.delete_all(user_id="user_001")
|
||||||
@@ -282,6 +293,10 @@ result = client.clone_user("user_001", "user_002", agent_id=None)
|
|||||||
对应异步方法为 `async_get()`、`async_update()`、`async_delete()`、
|
对应异步方法为 `async_get()`、`async_update()`、`async_delete()`、
|
||||||
`async_delete_all()`、`async_list_memories()`、`async_clone_user()`。
|
`async_delete_all()`、`async_list_memories()`、`async_clone_user()`。
|
||||||
|
|
||||||
|
`get()` / `async_get()` 默认继续隐藏 `L1_RAW`,因此不会改变现有调用方的原文访问
|
||||||
|
边界。只有显式传入 `include_raw=True` 才返回该层;这也是 lite 模式按 ID 读取
|
||||||
|
写入内容及其 `meta` 的方式。MemHub 的 HTTP 读取接口不开放此参数。
|
||||||
|
|
||||||
普通 `list_memories()` 返回 `vdb` 桶,并在 Graph 有结果时附带 `graph`;传入
|
普通 `list_memories()` 返回 `vdb` 桶,并在 Graph 有结果时附带 `graph`;传入
|
||||||
`workspace_id` 时返回生产力记忆的 `scene="productivity"` 和 `experiences`。
|
`workspace_id` 时返回生产力记忆的 `scene="productivity"` 和 `experiences`。
|
||||||
|
|
||||||
|
|||||||
+39
-12
@@ -1597,7 +1597,9 @@ class HyMemoryClient:
|
|||||||
session_id: Session ID(三级隔离 key)
|
session_id: Session ID(三级隔离 key)
|
||||||
source: 调用来源,例如 openclaw / cursor / codex;与 agent_id 正交。
|
source: 调用来源,例如 openclaw / cursor / codex;与 agent_id 正交。
|
||||||
写入前执行 strip().lower(),默认 "default"。
|
写入前执行 strip().lower(),默认 "default"。
|
||||||
metadata: 自定义元数据
|
metadata: 旧版 SDK 处理选项,会并入内部 WriteRequest.extra。
|
||||||
|
不保证持久化或随查询返回;调用方需要保存并读回的业务字段
|
||||||
|
请使用 meta。
|
||||||
memory_at: 记忆时间戳(不传则用当前时间),用于导入历史记忆
|
memory_at: 记忆时间戳(不传则用当前时间),用于导入历史记忆
|
||||||
scene: 显式指定写入路由,跳过 LLM 自动判定。
|
scene: 显式指定写入路由,跳过 LLM 自动判定。
|
||||||
- None(默认):含 tool 时 LLM 判 chat/working/coding;
|
- None(默认):含 tool 时 LLM 判 chat/working/coding;
|
||||||
@@ -1623,8 +1625,8 @@ class HyMemoryClient:
|
|||||||
branch: Coding memory 路径专用。分支名。同 workspace_id。chat 路径忽略。
|
branch: Coding memory 路径专用。分支名。同 workspace_id。chat 路径忽略。
|
||||||
meta: 调用方自定义业务 meta,随本次产出的记忆存储、查询时原样返回。
|
meta: 调用方自定义业务 meta,随本次产出的记忆存储、查询时原样返回。
|
||||||
不 embed、不参与召回、不作过滤条件。
|
不 embed、不参与召回、不作过滤条件。
|
||||||
与 metadata 不同:metadata 会并入 extra(含 SDK 内部控制键),
|
与 metadata 不同:metadata 是旧版内部处理选项,
|
||||||
meta 独立存放,只做透传。
|
meta 独立持久化,只做业务透传。
|
||||||
语义为「该次 write 时刻的上下文」,故存为 list——同一节点被
|
语义为「该次 write 时刻的上下文」,故存为 list——同一节点被
|
||||||
多次写入合并时会追加而非覆盖,详见 get() 返回的 meta 字段。
|
多次写入合并时会追加而非覆盖,详见 get() 返回的 meta 字段。
|
||||||
|
|
||||||
@@ -1791,17 +1793,26 @@ class HyMemoryClient:
|
|||||||
query_embedding=query_embedding,
|
query_embedding=query_embedding,
|
||||||
))
|
))
|
||||||
|
|
||||||
def get(self, memory_id: str) -> Optional[Dict[str, Any]]:
|
def get(
|
||||||
|
self,
|
||||||
|
memory_id: str,
|
||||||
|
*,
|
||||||
|
include_raw: bool = False,
|
||||||
|
) -> Optional[Dict[str, Any]]:
|
||||||
"""
|
"""
|
||||||
获取单条记忆。
|
获取单条记忆。
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
memory_id: 记忆 ID
|
memory_id: 记忆 ID
|
||||||
|
include_raw: 是否允许返回 L1_RAW 原始记忆。默认 False,保持原有隐藏行为;
|
||||||
|
lite 模式的记忆均为 L1_RAW,需显式设为 True 才能按 ID 读取。
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
{"memory_id", "content", "layer", "request_id", "elapsed_ms"} 或 None(不存在时)
|
{"memory_id", "content", "layer", "request_id", "elapsed_ms"} 或 None(不存在时)
|
||||||
"""
|
"""
|
||||||
return self._loop_thread.run(self.async_get(memory_id))
|
return self._loop_thread.run(
|
||||||
|
self.async_get(memory_id, include_raw=include_raw)
|
||||||
|
)
|
||||||
|
|
||||||
def update(
|
def update(
|
||||||
self,
|
self,
|
||||||
@@ -3650,12 +3661,21 @@ class HyMemoryClient:
|
|||||||
# ================================================================
|
# ================================================================
|
||||||
|
|
||||||
@_ensure_internal_loop
|
@_ensure_internal_loop
|
||||||
async def async_get(self, memory_id: str) -> Optional[Dict[str, Any]]:
|
async def async_get(
|
||||||
|
self,
|
||||||
|
memory_id: str,
|
||||||
|
*,
|
||||||
|
include_raw: bool = False,
|
||||||
|
) -> Optional[Dict[str, Any]]:
|
||||||
"""
|
"""
|
||||||
获取单条记忆(异步)。
|
获取单条记忆(异步)。
|
||||||
|
|
||||||
VDB + Graph 双路并行查询,哪个有结果返回哪个。
|
VDB + Graph 双路并行查询,哪个有结果返回哪个。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
memory_id: 记忆 ID
|
||||||
|
include_raw: 是否允许返回 L1_RAW 原始记忆,默认 False。
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
{"memory_id", "content", "layer", "source", "request_id", "elapsed_ms"} 或 None
|
{"memory_id", "content", "layer", "source", "request_id", "elapsed_ms"} 或 None
|
||||||
"""
|
"""
|
||||||
@@ -3683,17 +3703,24 @@ class HyMemoryClient:
|
|||||||
|
|
||||||
vdb_node, graph_node = await _asyncio_get.gather(_get_vdb(), _get_graph())
|
vdb_node, graph_node = await _asyncio_get.gather(_get_vdb(), _get_graph())
|
||||||
|
|
||||||
# L1_RAW 不对外可见:按 ID 直取走主键查询、绕过向量库的 layer filter,
|
# L1_RAW 默认不对外可见:按 ID 直取走主键查询、绕过向量库的 layer
|
||||||
# 故在此统一挡掉。async_get 是 GET /v1/memories/{id} 与
|
# filter,故在此统一挡掉。SDK 调用方可通过 include_raw=True 显式读取;
|
||||||
# /v1/memories/batch 两个对外接口的共同必经处,而内部写链路不经过它
|
# HTTP 接口不传该参数,继续保持隐藏。内部写链路不经过此方法。
|
||||||
# (writer 写后校验直接调 vector_store.get_by_id),挡在此处不影响内部。
|
|
||||||
#
|
#
|
||||||
# 置于 gather 之后、埋点之前:命中 raw 仍照常落 GET 埋点(保留可观测性,
|
# 置于 gather 之后、埋点之前:命中 raw 仍照常落 GET 埋点(保留可观测性,
|
||||||
# 便于发现有调用方拿 raw id 试探),只是不把内容返回给调用方。
|
# 便于发现有调用方拿 raw id 试探),只是不把内容返回给调用方。
|
||||||
# 对调用方的表现与"该 id 不存在"完全一致,不额外暴露"存在但被挡"的信息。
|
# 对调用方的表现与"该 id 不存在"完全一致,不额外暴露"存在但被挡"的信息。
|
||||||
if vdb_node is not None and vdb_node.layer == MemoryLayer.L1_RAW:
|
if (
|
||||||
|
not include_raw
|
||||||
|
and vdb_node is not None
|
||||||
|
and vdb_node.layer == MemoryLayer.L1_RAW
|
||||||
|
):
|
||||||
vdb_node = None
|
vdb_node = None
|
||||||
if graph_node is not None and getattr(graph_node, "layer", None) == MemoryLayer.L1_RAW:
|
if (
|
||||||
|
not include_raw
|
||||||
|
and graph_node is not None
|
||||||
|
and getattr(graph_node, "layer", None) == MemoryLayer.L1_RAW
|
||||||
|
):
|
||||||
graph_node = None
|
graph_node = None
|
||||||
|
|
||||||
elapsed = round((time.perf_counter() - t0) * 1000, 2)
|
elapsed = round((time.perf_counter() - t0) * 1000, 2)
|
||||||
|
|||||||
+136
-125
@@ -7,7 +7,7 @@ Agent Memory V2 - SqliteCache
|
|||||||
设计模式跟随 history_store.py:
|
设计模式跟随 history_store.py:
|
||||||
- 标准库 sqlite3 + _run_in_sqlite_pool()
|
- 标准库 sqlite3 + _run_in_sqlite_pool()
|
||||||
- WAL 模式 + check_same_thread=False
|
- WAL 模式 + check_same_thread=False
|
||||||
- threading.Lock 保护写操作
|
- 每实例 threading.RLock 保护共享连接的全部操作与关闭
|
||||||
|
|
||||||
三张表:
|
三张表:
|
||||||
- memory_operations / pipeline_logs / system_metrics: 审计与观测
|
- memory_operations / pipeline_logs / system_metrics: 审计与观测
|
||||||
@@ -122,7 +122,10 @@ class SqliteCache(CacheBase):
|
|||||||
def __init__(self, config: MemoryConfig):
|
def __init__(self, config: MemoryConfig):
|
||||||
self.config = config
|
self.config = config
|
||||||
self._conn: Optional[sqlite3.Connection] = None
|
self._conn: Optional[sqlite3.Connection] = None
|
||||||
self._lock = threading.Lock()
|
# One connection is shared by the executor workers. Use the same lock
|
||||||
|
# for reads, writes, initialization and close so close() cannot free
|
||||||
|
# the connection while another worker is inside sqlite3.
|
||||||
|
self._lock = threading.RLock()
|
||||||
|
|
||||||
# 读取 db_path
|
# 读取 db_path
|
||||||
cache_cfg = getattr(config, "cache", None)
|
cache_cfg = getattr(config, "cache", None)
|
||||||
@@ -144,35 +147,37 @@ class SqliteCache(CacheBase):
|
|||||||
logger.debug(f"SqliteCache initialized: {self._db_path}")
|
logger.debug(f"SqliteCache initialized: {self._db_path}")
|
||||||
|
|
||||||
def _init_sync(self) -> None:
|
def _init_sync(self) -> None:
|
||||||
db_dir = os.path.dirname(self._db_path)
|
with self._lock:
|
||||||
if db_dir:
|
db_dir = os.path.dirname(self._db_path)
|
||||||
os.makedirs(db_dir, exist_ok=True)
|
if db_dir:
|
||||||
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
|
os.makedirs(db_dir, exist_ok=True)
|
||||||
self._conn.row_factory = sqlite3.Row
|
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
|
||||||
self._conn.execute("PRAGMA journal_mode=WAL;")
|
self._conn.row_factory = sqlite3.Row
|
||||||
self._conn.executescript(_CREATE_TABLES_SQL)
|
self._conn.execute("PRAGMA journal_mode=WAL;")
|
||||||
self._migrate_pipeline_logs_columns()
|
self._conn.executescript(_CREATE_TABLES_SQL)
|
||||||
self._conn.commit()
|
self._migrate_pipeline_logs_columns()
|
||||||
|
self._conn.commit()
|
||||||
|
|
||||||
def _migrate_pipeline_logs_columns(self) -> None:
|
def _migrate_pipeline_logs_columns(self) -> None:
|
||||||
"""给旧库的 pipeline_logs_v2 补 namespace/session/event time 列。"""
|
"""给旧库的 pipeline_logs_v2 补 namespace/session/event time 列。"""
|
||||||
try:
|
with self._lock:
|
||||||
cols = {
|
try:
|
||||||
row["name"]
|
cols = {
|
||||||
for row in self._conn.execute("PRAGMA table_info(pipeline_logs_v2)")
|
row["name"]
|
||||||
}
|
for row in self._conn.execute("PRAGMA table_info(pipeline_logs_v2)")
|
||||||
_text_cols = ("namespace", "session_id")
|
}
|
||||||
for col in _text_cols:
|
_text_cols = ("namespace", "session_id")
|
||||||
if col not in cols:
|
for col in _text_cols:
|
||||||
|
if col not in cols:
|
||||||
|
self._conn.execute(
|
||||||
|
f"ALTER TABLE pipeline_logs_v2 ADD COLUMN {col} TEXT NOT NULL DEFAULT ''"
|
||||||
|
)
|
||||||
|
if "event_at_ms" not in cols:
|
||||||
self._conn.execute(
|
self._conn.execute(
|
||||||
f"ALTER TABLE pipeline_logs_v2 ADD COLUMN {col} TEXT NOT NULL DEFAULT ''"
|
"ALTER TABLE pipeline_logs_v2 ADD COLUMN event_at_ms INTEGER NOT NULL DEFAULT 0"
|
||||||
)
|
)
|
||||||
if "event_at_ms" not in cols:
|
except Exception as e:
|
||||||
self._conn.execute(
|
logger.debug(f"[sqlite] pipeline_logs_v2 column migration skipped: {e}")
|
||||||
"ALTER TABLE pipeline_logs_v2 ADD COLUMN event_at_ms INTEGER NOT NULL DEFAULT 0"
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.debug(f"[sqlite] pipeline_logs_v2 column migration skipped: {e}")
|
|
||||||
|
|
||||||
async def close(self) -> None:
|
async def close(self) -> None:
|
||||||
if self._conn is not None:
|
if self._conn is not None:
|
||||||
@@ -180,9 +185,10 @@ class SqliteCache(CacheBase):
|
|||||||
logger.info("SqliteCache closed")
|
logger.info("SqliteCache closed")
|
||||||
|
|
||||||
def _close_sync(self) -> None:
|
def _close_sync(self) -> None:
|
||||||
if self._conn:
|
with self._lock:
|
||||||
self._conn.close()
|
if self._conn:
|
||||||
self._conn = None
|
self._conn.close()
|
||||||
|
self._conn = None
|
||||||
|
|
||||||
# ================================================================
|
# ================================================================
|
||||||
# 统计
|
# 统计
|
||||||
@@ -255,39 +261,40 @@ class SqliteCache(CacheBase):
|
|||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
def _query():
|
def _query():
|
||||||
conditions = []
|
with self._lock:
|
||||||
params = []
|
conditions = []
|
||||||
if request_id:
|
params = []
|
||||||
conditions.append("request_id = ?")
|
if request_id:
|
||||||
params.append(request_id)
|
conditions.append("request_id = ?")
|
||||||
if memory_id:
|
params.append(request_id)
|
||||||
conditions.append("memory_id = ?")
|
if memory_id:
|
||||||
params.append(memory_id)
|
conditions.append("memory_id = ?")
|
||||||
if user_id:
|
params.append(memory_id)
|
||||||
conditions.append("user_id = ?")
|
if user_id:
|
||||||
params.append(user_id)
|
conditions.append("user_id = ?")
|
||||||
|
params.append(user_id)
|
||||||
|
|
||||||
if not conditions:
|
if not conditions:
|
||||||
return []
|
return []
|
||||||
|
|
||||||
where = " AND ".join(conditions)
|
where = " AND ".join(conditions)
|
||||||
rows = self._conn.execute(
|
rows = self._conn.execute(
|
||||||
f"SELECT * FROM memory_operations_v2 WHERE {where} "
|
f"SELECT * FROM memory_operations_v2 WHERE {where} "
|
||||||
f"ORDER BY id DESC LIMIT ?",
|
f"ORDER BY id DESC LIMIT ?",
|
||||||
params + [limit],
|
params + [limit],
|
||||||
).fetchall()
|
).fetchall()
|
||||||
result = []
|
result = []
|
||||||
for r in rows:
|
for r in rows:
|
||||||
row = dict(r)
|
row = dict(r)
|
||||||
# 反序列化 supersedes JSON 字符串 → list
|
# 反序列化 supersedes JSON 字符串 → list
|
||||||
import json as _json
|
import json as _json
|
||||||
raw_sup = row.get("supersedes", "")
|
raw_sup = row.get("supersedes", "")
|
||||||
try:
|
try:
|
||||||
row["supersedes"] = _json.loads(raw_sup) if raw_sup else []
|
row["supersedes"] = _json.loads(raw_sup) if raw_sup else []
|
||||||
except Exception:
|
except Exception:
|
||||||
row["supersedes"] = []
|
row["supersedes"] = []
|
||||||
result.append(row)
|
result.append(row)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
return await _run_in_sqlite_pool(_query)
|
return await _run_in_sqlite_pool(_query)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@@ -369,37 +376,38 @@ class SqliteCache(CacheBase):
|
|||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
def _query():
|
def _query():
|
||||||
conditions = []
|
with self._lock:
|
||||||
params = []
|
conditions = []
|
||||||
if request_id:
|
params = []
|
||||||
conditions.append("request_id = ?")
|
if request_id:
|
||||||
params.append(request_id)
|
conditions.append("request_id = ?")
|
||||||
if user_id:
|
params.append(request_id)
|
||||||
conditions.append("user_id = ?")
|
if user_id:
|
||||||
params.append(user_id)
|
conditions.append("user_id = ?")
|
||||||
if step:
|
params.append(user_id)
|
||||||
conditions.append("step = ?")
|
if step:
|
||||||
params.append(step)
|
conditions.append("step = ?")
|
||||||
|
params.append(step)
|
||||||
|
|
||||||
if not conditions:
|
if not conditions:
|
||||||
return []
|
return []
|
||||||
|
|
||||||
where = " AND ".join(conditions)
|
where = " AND ".join(conditions)
|
||||||
rows = self._conn.execute(
|
rows = self._conn.execute(
|
||||||
f"SELECT * FROM pipeline_logs_v2 WHERE {where} "
|
f"SELECT * FROM pipeline_logs_v2 WHERE {where} "
|
||||||
f"ORDER BY id ASC LIMIT ?",
|
f"ORDER BY id ASC LIMIT ?",
|
||||||
params + [limit],
|
params + [limit],
|
||||||
).fetchall()
|
).fetchall()
|
||||||
out = [dict(r) for r in rows]
|
out = [dict(r) for r in rows]
|
||||||
# 派生可读的 event_at(ISO ms)供查询接口直接用;event_at_ms 为 0 时留空
|
# 派生可读的 event_at(ISO ms)供查询接口直接用;event_at_ms 为 0 时留空
|
||||||
from datetime import datetime as _dt
|
from datetime import datetime as _dt
|
||||||
for _r in out:
|
for _r in out:
|
||||||
_ms = _r.get("event_at_ms") or 0
|
_ms = _r.get("event_at_ms") or 0
|
||||||
_r["event_at"] = (
|
_r["event_at"] = (
|
||||||
_dt.fromtimestamp(_ms / 1000.0).isoformat(timespec="milliseconds")
|
_dt.fromtimestamp(_ms / 1000.0).isoformat(timespec="milliseconds")
|
||||||
if _ms else ""
|
if _ms else ""
|
||||||
)
|
)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
return await _run_in_sqlite_pool(_query)
|
return await _run_in_sqlite_pool(_query)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@@ -414,26 +422,27 @@ class SqliteCache(CacheBase):
|
|||||||
"""存储一个分钟桶的增量指标数据(UPSERT: 同一分钟多次写入会合并)"""
|
"""存储一个分钟桶的增量指标数据(UPSERT: 同一分钟多次写入会合并)"""
|
||||||
try:
|
try:
|
||||||
def _store():
|
def _store():
|
||||||
now = datetime.now().isoformat()
|
with self._lock:
|
||||||
# UPSERT: 如果同一分钟已存在,合并数据
|
now = datetime.now().isoformat()
|
||||||
existing = self._conn.execute(
|
# UPSERT: 如果同一分钟已存在,合并数据
|
||||||
"SELECT data FROM system_metrics WHERE minute_ts = ?",
|
existing = self._conn.execute(
|
||||||
(minute_ts,),
|
"SELECT data FROM system_metrics WHERE minute_ts = ?",
|
||||||
).fetchone()
|
(minute_ts,),
|
||||||
if existing:
|
).fetchone()
|
||||||
import json as _json
|
if existing:
|
||||||
old_data = _json.loads(existing["data"])
|
import json as _json
|
||||||
merged = self._merge_metric_buckets(old_data, data)
|
old_data = _json.loads(existing["data"])
|
||||||
self._conn.execute(
|
merged = self._merge_metric_buckets(old_data, data)
|
||||||
"UPDATE system_metrics SET data = ?, created_at = ? WHERE minute_ts = ?",
|
self._conn.execute(
|
||||||
(_json.dumps(merged, default=str), now, minute_ts),
|
"UPDATE system_metrics SET data = ?, created_at = ? WHERE minute_ts = ?",
|
||||||
)
|
(_json.dumps(merged, default=str), now, minute_ts),
|
||||||
else:
|
)
|
||||||
self._conn.execute(
|
else:
|
||||||
"INSERT INTO system_metrics (minute_ts, data, created_at) VALUES (?, ?, ?)",
|
self._conn.execute(
|
||||||
(minute_ts, json.dumps(data, default=str), now),
|
"INSERT INTO system_metrics (minute_ts, data, created_at) VALUES (?, ?, ?)",
|
||||||
)
|
(minute_ts, json.dumps(data, default=str), now),
|
||||||
self._conn.commit()
|
)
|
||||||
|
self._conn.commit()
|
||||||
await _run_in_sqlite_pool(_store)
|
await _run_in_sqlite_pool(_store)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning(f"store_metrics_minute failed: {e}")
|
logger.warning(f"store_metrics_minute failed: {e}")
|
||||||
@@ -442,17 +451,18 @@ class SqliteCache(CacheBase):
|
|||||||
"""读取指定时间范围内的分钟指标数据"""
|
"""读取指定时间范围内的分钟指标数据"""
|
||||||
try:
|
try:
|
||||||
def _load():
|
def _load():
|
||||||
rows = self._conn.execute(
|
with self._lock:
|
||||||
"SELECT data FROM system_metrics WHERE minute_ts >= ? AND minute_ts <= ? ORDER BY minute_ts",
|
rows = self._conn.execute(
|
||||||
(start_ts, end_ts),
|
"SELECT data FROM system_metrics WHERE minute_ts >= ? AND minute_ts <= ? ORDER BY minute_ts",
|
||||||
).fetchall()
|
(start_ts, end_ts),
|
||||||
results = []
|
).fetchall()
|
||||||
for row in rows:
|
results = []
|
||||||
try:
|
for row in rows:
|
||||||
results.append(json.loads(row["data"]))
|
try:
|
||||||
except (json.JSONDecodeError, TypeError):
|
results.append(json.loads(row["data"]))
|
||||||
pass
|
except (json.JSONDecodeError, TypeError):
|
||||||
return results
|
pass
|
||||||
|
return results
|
||||||
return await _run_in_sqlite_pool(_load)
|
return await _run_in_sqlite_pool(_load)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning(f"load_metrics_range failed: {e}")
|
logger.warning(f"load_metrics_range failed: {e}")
|
||||||
@@ -462,11 +472,12 @@ class SqliteCache(CacheBase):
|
|||||||
"""删除指定时间之前的 metrics 数据"""
|
"""删除指定时间之前的 metrics 数据"""
|
||||||
try:
|
try:
|
||||||
def _cleanup():
|
def _cleanup():
|
||||||
self._conn.execute(
|
with self._lock:
|
||||||
"DELETE FROM system_metrics WHERE minute_ts < ?",
|
self._conn.execute(
|
||||||
(before_ts,),
|
"DELETE FROM system_metrics WHERE minute_ts < ?",
|
||||||
)
|
(before_ts,),
|
||||||
self._conn.commit()
|
)
|
||||||
|
self._conn.commit()
|
||||||
await _run_in_sqlite_pool(_cleanup)
|
await _run_in_sqlite_pool(_cleanup)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning(f"cleanup_old_metrics failed: {e}")
|
logger.warning(f"cleanup_old_metrics failed: {e}")
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import json
|
|||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import sqlite3
|
import sqlite3
|
||||||
|
import threading
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from typing import Dict, List, Any, Optional
|
from typing import Dict, List, Any, Optional
|
||||||
|
|
||||||
@@ -89,6 +90,10 @@ class HistoryStore:
|
|||||||
"""
|
"""
|
||||||
self._db_path = config.history.db_path
|
self._db_path = config.history.db_path
|
||||||
self._conn: Optional[sqlite3.Connection] = None
|
self._conn: Optional[sqlite3.Connection] = None
|
||||||
|
# sqlite3.Connection.close() must never race with an execute() on the
|
||||||
|
# same connection. The lock also serializes operations on this shared
|
||||||
|
# connection; separate HistoryStore instances remain independent.
|
||||||
|
self._conn_lock = threading.RLock()
|
||||||
|
|
||||||
async def initialize(self) -> None:
|
async def initialize(self) -> None:
|
||||||
"""创建表和索引(如果不存在)"""
|
"""创建表和索引(如果不存在)"""
|
||||||
@@ -97,20 +102,21 @@ class HistoryStore:
|
|||||||
|
|
||||||
def _init_sync(self) -> None:
|
def _init_sync(self) -> None:
|
||||||
"""同步初始化:创建目录、连接数据库、建表建索引"""
|
"""同步初始化:创建目录、连接数据库、建表建索引"""
|
||||||
# 确保目录存在
|
with self._conn_lock:
|
||||||
db_dir = os.path.dirname(self._db_path)
|
# 确保目录存在
|
||||||
if db_dir:
|
db_dir = os.path.dirname(self._db_path)
|
||||||
os.makedirs(db_dir, exist_ok=True)
|
if db_dir:
|
||||||
|
os.makedirs(db_dir, exist_ok=True)
|
||||||
|
|
||||||
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
|
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
|
||||||
self._conn.row_factory = sqlite3.Row
|
self._conn.row_factory = sqlite3.Row
|
||||||
# 开启 WAL 模式,提高并发读写性能
|
# 开启 WAL 模式,提高并发读写性能
|
||||||
self._conn.execute("PRAGMA journal_mode=WAL;")
|
self._conn.execute("PRAGMA journal_mode=WAL;")
|
||||||
|
|
||||||
self._conn.execute(_CREATE_TABLE_SQL)
|
self._conn.execute(_CREATE_TABLE_SQL)
|
||||||
for idx_sql in _CREATE_INDEXES_SQL:
|
for idx_sql in _CREATE_INDEXES_SQL:
|
||||||
self._conn.execute(idx_sql)
|
self._conn.execute(idx_sql)
|
||||||
self._conn.commit()
|
self._conn.commit()
|
||||||
|
|
||||||
# ================================================================
|
# ================================================================
|
||||||
# 记录操作
|
# 记录操作
|
||||||
@@ -218,13 +224,14 @@ class HistoryStore:
|
|||||||
return await _run_in_sqlite_pool(self._get_history_sync, memory_id)
|
return await _run_in_sqlite_pool(self._get_history_sync, memory_id)
|
||||||
|
|
||||||
def _get_history_sync(self, memory_id: str) -> List[Dict[str, Any]]:
|
def _get_history_sync(self, memory_id: str) -> List[Dict[str, Any]]:
|
||||||
if self._conn is None:
|
with self._conn_lock:
|
||||||
return []
|
if self._conn is None:
|
||||||
cursor = self._conn.execute(
|
return []
|
||||||
"SELECT * FROM memory_history WHERE memory_id = ? ORDER BY created_at ASC",
|
cursor = self._conn.execute(
|
||||||
(memory_id,),
|
"SELECT * FROM memory_history WHERE memory_id = ? ORDER BY created_at ASC",
|
||||||
)
|
(memory_id,),
|
||||||
return [dict(row) for row in cursor.fetchall()]
|
)
|
||||||
|
return [dict(row) for row in cursor.fetchall()]
|
||||||
|
|
||||||
async def get_recent(
|
async def get_recent(
|
||||||
self,
|
self,
|
||||||
@@ -244,14 +251,15 @@ class HistoryStore:
|
|||||||
return await _run_in_sqlite_pool(self._get_recent_sync, isolation_key, limit)
|
return await _run_in_sqlite_pool(self._get_recent_sync, isolation_key, limit)
|
||||||
|
|
||||||
def _get_recent_sync(self, isolation_key: str, limit: int) -> List[Dict[str, Any]]:
|
def _get_recent_sync(self, isolation_key: str, limit: int) -> List[Dict[str, Any]]:
|
||||||
if self._conn is None:
|
with self._conn_lock:
|
||||||
return []
|
if self._conn is None:
|
||||||
cursor = self._conn.execute(
|
return []
|
||||||
"SELECT * FROM memory_history WHERE isolation_key = ? "
|
cursor = self._conn.execute(
|
||||||
"ORDER BY created_at DESC LIMIT ?",
|
"SELECT * FROM memory_history WHERE isolation_key = ? "
|
||||||
(isolation_key, limit),
|
"ORDER BY created_at DESC LIMIT ?",
|
||||||
)
|
(isolation_key, limit),
|
||||||
return [dict(row) for row in cursor.fetchall()]
|
)
|
||||||
|
return [dict(row) for row in cursor.fetchall()]
|
||||||
|
|
||||||
async def count(self, isolation_key: str = "") -> int:
|
async def count(self, isolation_key: str = "") -> int:
|
||||||
"""
|
"""
|
||||||
@@ -266,16 +274,17 @@ class HistoryStore:
|
|||||||
return await _run_in_sqlite_pool(self._count_sync, isolation_key)
|
return await _run_in_sqlite_pool(self._count_sync, isolation_key)
|
||||||
|
|
||||||
def _count_sync(self, isolation_key: str) -> int:
|
def _count_sync(self, isolation_key: str) -> int:
|
||||||
if self._conn is None:
|
with self._conn_lock:
|
||||||
return 0
|
if self._conn is None:
|
||||||
if isolation_key:
|
return 0
|
||||||
cursor = self._conn.execute(
|
if isolation_key:
|
||||||
"SELECT COUNT(*) FROM memory_history WHERE isolation_key = ?",
|
cursor = self._conn.execute(
|
||||||
(isolation_key,),
|
"SELECT COUNT(*) FROM memory_history WHERE isolation_key = ?",
|
||||||
)
|
(isolation_key,),
|
||||||
else:
|
)
|
||||||
cursor = self._conn.execute("SELECT COUNT(*) FROM memory_history")
|
else:
|
||||||
return cursor.fetchone()[0]
|
cursor = self._conn.execute("SELECT COUNT(*) FROM memory_history")
|
||||||
|
return cursor.fetchone()[0]
|
||||||
|
|
||||||
async def close(self) -> None:
|
async def close(self) -> None:
|
||||||
"""关闭数据库连接"""
|
"""关闭数据库连接"""
|
||||||
@@ -284,9 +293,10 @@ class HistoryStore:
|
|||||||
logger.info("[HistoryStore] closed")
|
logger.info("[HistoryStore] closed")
|
||||||
|
|
||||||
def _close_sync(self) -> None:
|
def _close_sync(self) -> None:
|
||||||
if self._conn:
|
with self._conn_lock:
|
||||||
self._conn.close()
|
if self._conn:
|
||||||
self._conn = None
|
self._conn.close()
|
||||||
|
self._conn = None
|
||||||
|
|
||||||
# ================================================================
|
# ================================================================
|
||||||
# 内部
|
# 内部
|
||||||
@@ -339,33 +349,34 @@ class HistoryStore:
|
|||||||
role: str = "",
|
role: str = "",
|
||||||
extra: Optional[Dict[str, Any]] = None,
|
extra: Optional[Dict[str, Any]] = None,
|
||||||
) -> int:
|
) -> int:
|
||||||
if self._conn is None:
|
with self._conn_lock:
|
||||||
return -1
|
if self._conn is None:
|
||||||
now = _now_iso()
|
return -1
|
||||||
extra_json = json.dumps(extra or {}, ensure_ascii=False)
|
now = _now_iso()
|
||||||
cursor = self._conn.execute(
|
extra_json = json.dumps(extra or {}, ensure_ascii=False)
|
||||||
"""INSERT INTO memory_history
|
cursor = self._conn.execute(
|
||||||
(memory_id, isolation_key, event,
|
"""INSERT INTO memory_history
|
||||||
old_memory, new_memory, old_status, new_status,
|
(memory_id, isolation_key, event,
|
||||||
change_reason, layer, actor_id, role, extra,
|
old_memory, new_memory, old_status, new_status,
|
||||||
created_at, updated_at)
|
change_reason, layer, actor_id, role, extra,
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
created_at, updated_at)
|
||||||
(
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||||
memory_id,
|
(
|
||||||
isolation_key,
|
memory_id,
|
||||||
event,
|
isolation_key,
|
||||||
old_memory,
|
event,
|
||||||
new_memory,
|
old_memory,
|
||||||
old_status,
|
new_memory,
|
||||||
new_status,
|
old_status,
|
||||||
change_reason,
|
new_status,
|
||||||
layer,
|
change_reason,
|
||||||
actor_id,
|
layer,
|
||||||
role,
|
actor_id,
|
||||||
extra_json,
|
role,
|
||||||
now,
|
extra_json,
|
||||||
now,
|
now,
|
||||||
),
|
now,
|
||||||
)
|
),
|
||||||
self._conn.commit()
|
)
|
||||||
return cursor.lastrowid
|
self._conn.commit()
|
||||||
|
return cursor.lastrowid
|
||||||
@@ -2793,6 +2793,12 @@ def create_app(
|
|||||||
# D75: work queued for the previous model continues with this one.
|
# D75: work queued for the previous model continues with this one.
|
||||||
# Only tasks that never reached the model are re-bound.
|
# Only tasks that never reached the model are re-bound.
|
||||||
await asyncio.to_thread(app.state.memory_task_store.rebind_unstarted_profile, profile)
|
await asyncio.to_thread(app.state.memory_task_store.rebind_unstarted_profile, profile)
|
||||||
|
# A missing profile is an event-driven wait, not a provider outage.
|
||||||
|
# Release those durable rows only now that activation has changed the
|
||||||
|
# condition; the ordinary drain poll will claim them without a restart.
|
||||||
|
await asyncio.to_thread(
|
||||||
|
app.state.memory_write_queue.readmit_model_profile_waiters
|
||||||
|
)
|
||||||
# Not awaited: the catch-up reads every conversation ever imported, and
|
# Not awaited: the catch-up reads every conversation ever imported, and
|
||||||
# the answer to "which profile is active" does not depend on it.
|
# the answer to "which profile is active" does not depend on it.
|
||||||
_start_backfill(app)
|
_start_backfill(app)
|
||||||
@@ -5062,10 +5068,11 @@ def _source_writes_on(app: FastAPI, source_id: str) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
def _sedimentation_claim_enabled(app: FastAPI, source_id: str) -> bool:
|
def _sedimentation_claim_enabled(app: FastAPI, source_id: str) -> bool:
|
||||||
"""Claim queued model work only while both global and source gates allow it."""
|
"""Claim queued model work only when policy, source and model all admit it."""
|
||||||
|
|
||||||
return (
|
return (
|
||||||
_config_store(app).sedimentation_enabled()
|
_config_store(app).sedimentation_enabled()
|
||||||
|
and _model_profile_store(app).active_identity() is not None
|
||||||
and _history_import_gate_released(app)
|
and _history_import_gate_released(app)
|
||||||
and _source_writes_on(app, source_id)
|
and _source_writes_on(app, source_id)
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -282,11 +282,24 @@ class PollingLiveSyncSupervisor:
|
|||||||
and job.status
|
and job.status
|
||||||
in {ImportStatus.PENDING, ImportStatus.RUNNING, ImportStatus.INTERRUPTED}
|
in {ImportStatus.PENDING, ImportStatus.RUNNING, ImportStatus.INTERRUPTED}
|
||||||
]
|
]
|
||||||
|
bounded_history_incomplete = any(
|
||||||
|
job.source_id == source_id
|
||||||
|
and job.operation == "history"
|
||||||
|
and job.history_after is not None
|
||||||
|
and job.status is not ImportStatus.COMPLETED
|
||||||
|
for job in source_jobs
|
||||||
|
)
|
||||||
# Establish the realtime EOF independently even while history owns
|
# Establish the realtime EOF independently even while history owns
|
||||||
# this source. Otherwise the P1 job starts from the P2 checkpoint
|
# this source. Otherwise the P1 job starts from the P2 checkpoint
|
||||||
# (or from zero) and becomes a second full-history scan.
|
# (or from zero) and becomes a second full-history scan.
|
||||||
adapter = self.import_supervisor.adapters[source_id]
|
adapter = self.import_supervisor.adapters[source_id]
|
||||||
await self._live_cursors(source_id, adapter)
|
live_cursors = await self._live_cursors(source_id, adapter)
|
||||||
|
# A bounded onboarding history is the authority for everything up
|
||||||
|
# to its EOF. An adapter without a native live baseline has no safe
|
||||||
|
# realtime cursor until that history finishes; starting a sync from
|
||||||
|
# zero here would bypass `history_after` and import the old corpus.
|
||||||
|
if bounded_history_incomplete and not live_cursors:
|
||||||
|
return None
|
||||||
# Only one scan of a source may be in flight: they share the
|
# Only one scan of a source may be in flight: they share the
|
||||||
# cursor, and two readers means every file read twice. With a
|
# cursor, and two readers means every file read twice. With a
|
||||||
# catch-up still holding the source there is no free moment to ask
|
# catch-up still holding the source there is no free moment to ask
|
||||||
|
|||||||
@@ -1404,11 +1404,12 @@ class DurableWriteQueue:
|
|||||||
) -> int:
|
) -> int:
|
||||||
"""Re-admit `waiting_for_model` jobs whose wait has elapsed.
|
"""Re-admit `waiting_for_model` jobs whose wait has elapsed.
|
||||||
|
|
||||||
A model outage clears on its own, so this state is retried on a timer
|
A model outage clears on its own, so transient waiters retry on a timer
|
||||||
rather than waiting for a grant event like `waiting_for_permission`.
|
rather than waiting for a grant event like `waiting_for_permission`.
|
||||||
Keyed on `updated_at` so repeated calls while a job is still parked do
|
`model_profile_not_active` is excluded: only explicit activation can
|
||||||
not reset its own clock; the attempt consumed by the parking claim is
|
change that condition. Keyed on `updated_at` so repeated calls while a
|
||||||
returned, the same way `readmit_parked` does.
|
job is still parked do not reset its own clock; the attempt consumed by
|
||||||
|
the parking claim is returned, the same way `readmit_parked` does.
|
||||||
"""
|
"""
|
||||||
now = now or datetime.now(UTC)
|
now = now or datetime.now(UTC)
|
||||||
threshold = (now - after).isoformat()
|
threshold = (now - after).isoformat()
|
||||||
@@ -1417,8 +1418,33 @@ class DurableWriteQueue:
|
|||||||
"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
|
"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
|
||||||
"lease_expires_at=NULL, attempts=CASE WHEN attempts>0 "
|
"lease_expires_at=NULL, attempts=CASE WHEN attempts>0 "
|
||||||
"THEN attempts-1 ELSE 0 END, last_error=NULL, updated_at=? "
|
"THEN attempts-1 ELSE 0 END, last_error=NULL, updated_at=? "
|
||||||
"WHERE state=? AND updated_at <= ? AND hy_invoked=0",
|
"WHERE state=? AND updated_at <= ? AND hy_invoked=0 "
|
||||||
(PENDING, now.isoformat(), WAITING_FOR_MODEL, threshold),
|
"AND (last_error IS NULL OR last_error<>?)",
|
||||||
|
(
|
||||||
|
PENDING,
|
||||||
|
now.isoformat(),
|
||||||
|
WAITING_FOR_MODEL,
|
||||||
|
threshold,
|
||||||
|
"model_profile_not_active",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return changed.rowcount
|
||||||
|
|
||||||
|
def readmit_model_profile_waiters(self) -> int:
|
||||||
|
"""Re-admit no-profile work only after model activation changes the condition."""
|
||||||
|
now = datetime.now(UTC).isoformat()
|
||||||
|
with main_database(self.database_path).write() as connection, connection:
|
||||||
|
changed = connection.execute(
|
||||||
|
"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
|
||||||
|
"lease_expires_at=NULL, attempts=CASE WHEN attempts>0 "
|
||||||
|
"THEN attempts-1 ELSE 0 END, last_error=NULL, updated_at=? "
|
||||||
|
"WHERE state=? AND last_error=? AND hy_invoked=0",
|
||||||
|
(
|
||||||
|
PENDING,
|
||||||
|
now,
|
||||||
|
WAITING_FOR_MODEL,
|
||||||
|
"model_profile_not_active",
|
||||||
|
),
|
||||||
)
|
)
|
||||||
return changed.rowcount
|
return changed.rowcount
|
||||||
|
|
||||||
@@ -2071,7 +2097,12 @@ async def execute_job(
|
|||||||
# Never `logger.exception` here: it prints the exception's message,
|
# Never `logger.exception` here: it prints the exception's message,
|
||||||
# which may be a provider response or the user's own text. Log the
|
# which may be a provider response or the user's own text. Log the
|
||||||
# already-sanitised code instead, matching `hy-failure-classification`.
|
# already-sanitised code instead, matching `hy-failure-classification`.
|
||||||
logger.error("durable_write_job_failed job_id=%s error_code=%s", job_id, code)
|
if code == "model_profile_not_active":
|
||||||
|
logger.debug("durable_write_job_parked job_id=%s error_code=%s", job_id, code)
|
||||||
|
elif state in PARKED_STATES:
|
||||||
|
logger.info("durable_write_job_parked job_id=%s error_code=%s", job_id, code)
|
||||||
|
else:
|
||||||
|
logger.error("durable_write_job_failed job_id=%s error_code=%s", job_id, code)
|
||||||
return
|
return
|
||||||
await in_the_background(queue.settle, job_id, state=COMPLETED, owner=owner)
|
await in_the_background(queue.settle, job_id, state=COMPLETED, owner=owner)
|
||||||
|
|
||||||
|
|||||||
@@ -94,6 +94,67 @@ class SettledFixtureAdapter(IncrementalFixtureAdapter):
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class BoundedOnboardingAdapter:
|
||||||
|
"""No native baseline: bounded history must establish the live cursor."""
|
||||||
|
|
||||||
|
descriptor = IncrementalFixtureAdapter.descriptor
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.include_new = False
|
||||||
|
|
||||||
|
async def detect(self) -> bool:
|
||||||
|
return True
|
||||||
|
|
||||||
|
async def scan(self, options: ScanOptions):
|
||||||
|
current = options.cursors.get("e" * 64)
|
||||||
|
checkpoint = current.checkpoint if current is not None else None
|
||||||
|
if checkpoint not in {"byte:10", "byte:20"}:
|
||||||
|
yield ScannedEvidence(
|
||||||
|
evidence=EvidenceInput(
|
||||||
|
source_id="fixture",
|
||||||
|
conversation_id="old-conversation",
|
||||||
|
message_id="old-message",
|
||||||
|
role="user",
|
||||||
|
written_by=WrittenBy.PERSON,
|
||||||
|
created_at=datetime(2020, 1, 1, tzinfo=UTC),
|
||||||
|
verbatim="old history",
|
||||||
|
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||||
|
),
|
||||||
|
cursor=SourceCursor(
|
||||||
|
file_key="e" * 64,
|
||||||
|
file_identity="bounded",
|
||||||
|
checkpoint="byte:10",
|
||||||
|
checkpoint_anchor="f" * 64,
|
||||||
|
observed_size=20,
|
||||||
|
observed_modified_ns=1,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
checkpoint = "byte:10"
|
||||||
|
if self.include_new and checkpoint != "byte:20":
|
||||||
|
yield ScannedEvidence(
|
||||||
|
evidence=EvidenceInput(
|
||||||
|
source_id="fixture",
|
||||||
|
conversation_id="new-conversation",
|
||||||
|
message_id="new-message",
|
||||||
|
role="user",
|
||||||
|
written_by=WrittenBy.PERSON,
|
||||||
|
created_at=datetime(2026, 10, 18, tzinfo=UTC),
|
||||||
|
verbatim="new append",
|
||||||
|
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||||
|
),
|
||||||
|
cursor=SourceCursor(
|
||||||
|
file_key="e" * 64,
|
||||||
|
file_identity="bounded",
|
||||||
|
checkpoint="byte:20",
|
||||||
|
checkpoint_anchor="1" * 64,
|
||||||
|
observed_size=20,
|
||||||
|
observed_modified_ns=2,
|
||||||
|
read_to_end=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
yield ScanProgressEvent(phase=ScanPhase.DONE)
|
||||||
|
|
||||||
|
|
||||||
class HoldingHistoryAdapter:
|
class HoldingHistoryAdapter:
|
||||||
"""First scan blocks until cancelled; later scans yield one new record."""
|
"""First scan blocks until cancelled; later scans yield one new record."""
|
||||||
|
|
||||||
@@ -328,6 +389,56 @@ def test_live_sync_does_not_resume_a_history_import_the_user_paused(tmp_path) ->
|
|||||||
asyncio.run(scenario())
|
asyncio.run(scenario())
|
||||||
|
|
||||||
|
|
||||||
|
def test_bounded_onboarding_history_establishes_cursor_before_first_sync(tmp_path) -> None:
|
||||||
|
async def scenario() -> None:
|
||||||
|
database = tmp_path / "one-memory.sqlite3"
|
||||||
|
evidence = EvidenceStore(database)
|
||||||
|
jobs = ImportJobStore(database)
|
||||||
|
adapter = BoundedOnboardingAdapter()
|
||||||
|
imports = ImportSupervisor(
|
||||||
|
adapters={"fixture": adapter},
|
||||||
|
evidence_store=evidence,
|
||||||
|
job_store=jobs,
|
||||||
|
)
|
||||||
|
live_sync = PollingLiveSyncSupervisor(
|
||||||
|
source_ids=("fixture",),
|
||||||
|
import_supervisor=imports,
|
||||||
|
job_store=jobs,
|
||||||
|
)
|
||||||
|
history = imports.create_job("fixture", operation="history")
|
||||||
|
with sqlite3.connect(database) as connection, connection:
|
||||||
|
connection.execute(
|
||||||
|
"UPDATE import_jobs SET history_after=? WHERE job_id=?",
|
||||||
|
(datetime(2026, 10, 15, tzinfo=UTC).isoformat(), history.job_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
# With no native baseline, sync must wait instead of scanning from zero
|
||||||
|
# and bypassing the onboarding history window.
|
||||||
|
assert await live_sync.trigger("fixture") is None
|
||||||
|
assert not [job for job in jobs.list_jobs() if job.operation == "sync"]
|
||||||
|
|
||||||
|
imports.enqueue(history.job_id)
|
||||||
|
completed = await imports.wait(history.job_id)
|
||||||
|
assert completed.status is ImportStatus.COMPLETED
|
||||||
|
assert completed.stored == 0
|
||||||
|
assert completed.skipped == 1
|
||||||
|
assert evidence.count() == 0
|
||||||
|
|
||||||
|
# Once bounded history has advanced the cursor to EOF, normal realtime
|
||||||
|
# appends are admitted without retaining the three-day cutoff forever.
|
||||||
|
adapter.include_new = True
|
||||||
|
synced = await live_sync.trigger("fixture")
|
||||||
|
assert synced is not None
|
||||||
|
assert synced.status is ImportStatus.COMPLETED
|
||||||
|
assert synced.stored == 1
|
||||||
|
records = evidence.list_extractable()
|
||||||
|
assert [record.verbatim for record in records] == ["new append"]
|
||||||
|
await live_sync.shutdown()
|
||||||
|
await imports.shutdown()
|
||||||
|
|
||||||
|
asyncio.run(scenario())
|
||||||
|
|
||||||
|
|
||||||
def test_a_sync_left_running_after_its_worker_died_is_started_again(tmp_path) -> None:
|
def test_a_sync_left_running_after_its_worker_died_is_started_again(tmp_path) -> None:
|
||||||
"""A running row with no live worker used to be returned as 'already syncing'.
|
"""A running row with no live worker used to be returned as 'already syncing'.
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ from __future__ import annotations
|
|||||||
import asyncio
|
import asyncio
|
||||||
import contextlib
|
import contextlib
|
||||||
import json
|
import json
|
||||||
|
import logging
|
||||||
import sqlite3
|
import sqlite3
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
@@ -1369,24 +1370,30 @@ def test_a_machine_error_code_is_preserved(tmp_path) -> None:
|
|||||||
assert safe_error_code(ValueError("some sentence with spaces")) == "ValueError"
|
assert safe_error_code(ValueError("some sentence with spaces")) == "ValueError"
|
||||||
|
|
||||||
|
|
||||||
def test_an_inactive_profile_parks_for_model_before_any_send(tmp_path) -> None:
|
def test_an_inactive_profile_parks_without_error_logging(tmp_path, caplog) -> None:
|
||||||
"""A pre-bridge model-not-ready code parks and keeps the intent."""
|
"""A pre-bridge model-not-ready code parks and keeps the intent."""
|
||||||
path = tmp_path / "one-memory.sqlite3"
|
path = tmp_path / "one-memory.sqlite3"
|
||||||
queue = DurableWriteQueue(path)
|
queue = DurableWriteQueue(path)
|
||||||
record = queue.submit(_request())
|
record = queue.submit(_request())
|
||||||
_claim_through_drain(queue, record.job_id)
|
_claim_through_drain(queue, record.job_id)
|
||||||
|
|
||||||
asyncio.run(
|
with caplog.at_level(
|
||||||
execute_job(
|
logging.DEBUG,
|
||||||
record.job_id,
|
logger="one_memory.memory_server.durable_queue",
|
||||||
queue=queue,
|
):
|
||||||
supervisor=_Recorder(error=RuntimeError("model_profile_not_active")),
|
asyncio.run(
|
||||||
evidence=_EvidenceStore({"e1"}),
|
execute_job(
|
||||||
|
record.job_id,
|
||||||
|
queue=queue,
|
||||||
|
supervisor=_Recorder(error=RuntimeError("model_profile_not_active")),
|
||||||
|
evidence=_EvidenceStore({"e1"}),
|
||||||
|
)
|
||||||
)
|
)
|
||||||
)
|
|
||||||
|
|
||||||
assert _state(path, record.job_id) == WAITING_FOR_MODEL
|
assert _state(path, record.job_id) == WAITING_FOR_MODEL
|
||||||
assert _rows(path)[0]["hy_invoked"] == 0
|
assert _rows(path)[0]["hy_invoked"] == 0
|
||||||
|
assert "durable_write_job_failed" not in caplog.text
|
||||||
|
assert "durable_write_job_parked" in caplog.text
|
||||||
|
|
||||||
|
|
||||||
def test_an_authorization_stop_cancels_instead_of_failing(tmp_path) -> None:
|
def test_an_authorization_stop_cancels_instead_of_failing(tmp_path) -> None:
|
||||||
@@ -1546,7 +1553,7 @@ def test_parking_does_not_burn_the_retry_budget(tmp_path) -> None:
|
|||||||
assert queue.get("idem-1").state == PENDING
|
assert queue.get("idem-1").state == PENDING
|
||||||
|
|
||||||
|
|
||||||
def test_a_model_waiter_is_readmitted_once_its_wait_elapses(tmp_path) -> None:
|
def test_a_transient_model_waiter_is_readmitted_once_its_wait_elapses(tmp_path) -> None:
|
||||||
"""A model outage clears on its own; nothing re-admits it otherwise.
|
"""A model outage clears on its own; nothing re-admits it otherwise.
|
||||||
|
|
||||||
`waiting_for_permission` needs a grant event, but `waiting_for_model` is
|
`waiting_for_permission` needs a grant event, but `waiting_for_model` is
|
||||||
@@ -1564,7 +1571,7 @@ def test_a_model_waiter_is_readmitted_once_its_wait_elapses(tmp_path) -> None:
|
|||||||
queue.settle(
|
queue.settle(
|
||||||
record.job_id,
|
record.job_id,
|
||||||
state=WAITING_FOR_MODEL,
|
state=WAITING_FOR_MODEL,
|
||||||
error_code="model_profile_not_active",
|
error_code="model_secret_unavailable",
|
||||||
hy_invoked=False,
|
hy_invoked=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -1581,8 +1588,8 @@ def test_a_model_waiter_is_readmitted_once_its_wait_elapses(tmp_path) -> None:
|
|||||||
assert _rows(path)[0]["attempts"] == 0
|
assert _rows(path)[0]["attempts"] == 0
|
||||||
|
|
||||||
|
|
||||||
def test_readmitting_a_model_waiter_clears_its_stale_detail(tmp_path) -> None:
|
def test_no_profile_waiter_is_readmitted_only_after_activation(tmp_path) -> None:
|
||||||
"""The timer readmission path must clear the old blocking reason too."""
|
"""No timer can create a model profile; activation owns this transition."""
|
||||||
path = tmp_path / "one-memory.sqlite3"
|
path = tmp_path / "one-memory.sqlite3"
|
||||||
queue = DurableWriteQueue(path)
|
queue = DurableWriteQueue(path)
|
||||||
record = queue.submit(_request())
|
record = queue.submit(_request())
|
||||||
@@ -1594,13 +1601,50 @@ def test_readmitting_a_model_waiter_clears_its_stale_detail(tmp_path) -> None:
|
|||||||
)
|
)
|
||||||
now = datetime.now(UTC)
|
now = datetime.now(UTC)
|
||||||
|
|
||||||
assert queue.readmit_model_waiters(now=now + timedelta(minutes=2)) == 1
|
assert queue.readmit_model_waiters(now=now + timedelta(minutes=2)) == 0
|
||||||
|
parked = queue.get("idem-1")
|
||||||
|
assert parked.state == WAITING_FOR_MODEL
|
||||||
|
assert parked.detail == "model_profile_not_active"
|
||||||
|
|
||||||
|
assert queue.readmit_model_profile_waiters() == 1
|
||||||
readmitted = queue.get("idem-1")
|
readmitted = queue.get("idem-1")
|
||||||
assert readmitted.state == PENDING
|
assert readmitted.state == PENDING
|
||||||
assert readmitted.detail == ""
|
assert readmitted.detail == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_profile_backlog_stays_parked_across_maintenance_cycles(tmp_path) -> None:
|
||||||
|
path = tmp_path / "one-memory.sqlite3"
|
||||||
|
queue = DurableWriteQueue(path)
|
||||||
|
for index in range(100):
|
||||||
|
record = queue.submit(_request(key=f"no-profile-{index}"))
|
||||||
|
queue.settle(
|
||||||
|
record.job_id,
|
||||||
|
state=WAITING_FOR_MODEL,
|
||||||
|
error_code="model_profile_not_active",
|
||||||
|
hy_invoked=False,
|
||||||
|
)
|
||||||
|
with sqlite3.connect(path) as connection, connection:
|
||||||
|
connection.execute(
|
||||||
|
"UPDATE memory_write_jobs SET attempts=1,"
|
||||||
|
"updated_at='2000-01-01T00:00:00+00:00'"
|
||||||
|
)
|
||||||
|
|
||||||
|
first = datetime.now(UTC)
|
||||||
|
assert queue.readmit_model_waiters(now=first, after=timedelta(0)) == 0
|
||||||
|
assert (
|
||||||
|
queue.readmit_model_waiters(
|
||||||
|
now=first + timedelta(minutes=2),
|
||||||
|
after=timedelta(0),
|
||||||
|
)
|
||||||
|
== 0
|
||||||
|
)
|
||||||
|
with sqlite3.connect(path) as connection:
|
||||||
|
assert connection.execute(
|
||||||
|
"SELECT state,attempts,COUNT(*) FROM memory_write_jobs "
|
||||||
|
"GROUP BY state,attempts"
|
||||||
|
).fetchall() == [(WAITING_FOR_MODEL, 1, 100)]
|
||||||
|
|
||||||
|
|
||||||
def test_readmit_parked_cannot_resurrect_a_terminal_row(tmp_path) -> None:
|
def test_readmit_parked_cannot_resurrect_a_terminal_row(tmp_path) -> None:
|
||||||
"""Only parked states may be re-admitted, never a finished write.
|
"""Only parked states may be re-admitted, never a finished write.
|
||||||
|
|
||||||
|
|||||||
@@ -259,12 +259,17 @@ def test_activation_survives_a_corpus_already_imported(tmp_path) -> None:
|
|||||||
),
|
),
|
||||||
secrets=secrets,
|
secrets=secrets,
|
||||||
)
|
)
|
||||||
|
readmissions: list[bool] = []
|
||||||
|
app.state.memory_write_queue.readmit_model_profile_waiters = (
|
||||||
|
lambda: readmissions.append(True) or 0
|
||||||
|
)
|
||||||
|
|
||||||
activated = await client.post(f"/api/v1/model-profiles/{profile_id}/activate")
|
activated = await client.post(f"/api/v1/model-profiles/{profile_id}/activate")
|
||||||
|
|
||||||
assert activated.status_code == 200
|
assert activated.status_code == 200
|
||||||
assert activated.json()["is_active"] is True
|
assert activated.json()["is_active"] is True
|
||||||
assert API_KEY not in activated.text
|
assert API_KEY not in activated.text
|
||||||
|
assert readmissions == [True]
|
||||||
# Started, not waited for: the walk reads every conversation.
|
# Started, not waited for: the walk reads every conversation.
|
||||||
assert not app.state.memory_backfill.done()
|
assert not app.state.memory_backfill.done()
|
||||||
|
|
||||||
|
|||||||
@@ -111,6 +111,9 @@ def test_the_manual_source_is_admitted_by_grant_without_a_toggle(tmp_path) -> No
|
|||||||
app = SimpleNamespace(
|
app = SimpleNamespace(
|
||||||
state=SimpleNamespace(
|
state=SimpleNamespace(
|
||||||
config_store=config,
|
config_store=config,
|
||||||
|
model_profile_store=SimpleNamespace(
|
||||||
|
active_identity=lambda: ("profile", 1)
|
||||||
|
),
|
||||||
source_grant_store=SimpleNamespace(
|
source_grant_store=SimpleNamespace(
|
||||||
receipt=lambda **_kwargs: SimpleNamespace(revoked_at=None)
|
receipt=lambda **_kwargs: SimpleNamespace(revoked_at=None)
|
||||||
),
|
),
|
||||||
@@ -125,6 +128,21 @@ def test_the_manual_source_is_admitted_by_grant_without_a_toggle(tmp_path) -> No
|
|||||||
assert _sedimentation_claim_enabled(app, "one-memory-manual") is False
|
assert _sedimentation_claim_enabled(app, "one-memory-manual") is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_claim_gate_waits_without_polling_when_no_model_is_active(tmp_path) -> None:
|
||||||
|
config = ConfigStore(tmp_path)
|
||||||
|
app = SimpleNamespace(
|
||||||
|
state=SimpleNamespace(
|
||||||
|
config_store=config,
|
||||||
|
model_profile_store=SimpleNamespace(active_identity=lambda: None),
|
||||||
|
source_grant_store=SimpleNamespace(
|
||||||
|
receipt=lambda **_kwargs: SimpleNamespace(revoked_at=None)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert _sedimentation_claim_enabled(app, "one-memory-manual") is False
|
||||||
|
|
||||||
|
|
||||||
def test_first_import_gate_waits_for_every_selected_history_job() -> None:
|
def test_first_import_gate_waits_for_every_selected_history_job() -> None:
|
||||||
state = SimpleNamespace(
|
state = SimpleNamespace(
|
||||||
status="importing",
|
status="importing",
|
||||||
|
|||||||
@@ -174,20 +174,20 @@ it("continues to step3 while sample GET is pending without calling completion AP
|
|||||||
expect(screen.queryByText(/^已生成.*记忆/)).toBeNull();
|
expect(screen.queryByText(/^已生成.*记忆/)).toBeNull();
|
||||||
});
|
});
|
||||||
it.each([
|
it.each([
|
||||||
"pending",
|
["pending", "已扫描 100 条,正在筛选"],
|
||||||
"paused",
|
["paused", "已扫描 100 条,尚未保存"],
|
||||||
"interrupted",
|
["interrupted", "已扫描 100 条,尚未保存"],
|
||||||
"failed",
|
["failed", "已扫描 100 条,尚未保存"],
|
||||||
"cancelled",
|
["cancelled", "已扫描 100 条,尚未保存"],
|
||||||
"completed",
|
["completed", "已扫描 100 条,本次没有可保存的对话"],
|
||||||
] as const)(
|
] as const)(
|
||||||
"shows %s without mutating jobs or fetching missing content",
|
"shows %s without mutating jobs or fetching missing content",
|
||||||
(status) => {
|
(status, expected) => {
|
||||||
setup({
|
setup({
|
||||||
jobs: [{ ...job, status, stored: 0 }],
|
jobs: [{ ...job, status, stored: 0 }],
|
||||||
sources: [{ ...source, imported_messages: 0 }],
|
sources: [{ ...source, imported_messages: 0 }],
|
||||||
});
|
});
|
||||||
expect(screen.getByText("已扫描 100 条,正在保存")).toBeTruthy();
|
expect(screen.getByText(expected)).toBeTruthy();
|
||||||
expect(api.receipt).not.toHaveBeenCalled();
|
expect(api.receipt).not.toHaveBeenCalled();
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -38,6 +38,71 @@ it("uses the selected job's message count, never the source total or fake percen
|
|||||||
expect(container.textContent).not.toContain("%");
|
expect(container.textContent).not.toContain("%");
|
||||||
expect(container.textContent).not.toContain("持续导入中");
|
expect(container.textContent).not.toContain("持续导入中");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("does not describe scanned-but-skipped history as saved", () => {
|
||||||
|
const { container } = render(
|
||||||
|
<R6ImportProgress
|
||||||
|
sourceIds={["cursor"]}
|
||||||
|
sources={[source]}
|
||||||
|
jobs={[
|
||||||
|
{
|
||||||
|
...job,
|
||||||
|
status: "completed",
|
||||||
|
discovered: 12_375,
|
||||||
|
stored: 0,
|
||||||
|
skipped: 12_375,
|
||||||
|
history_after: "2026-10-15T00:00:00Z",
|
||||||
|
},
|
||||||
|
]}
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
expect(
|
||||||
|
screen.getByText("已扫描 12,375 条,本次没有符合时间范围的对话"),
|
||||||
|
).toBeTruthy();
|
||||||
|
expect(screen.getByText("导入完成")).toBeTruthy();
|
||||||
|
expect(container.textContent).not.toContain("正在保存");
|
||||||
|
expect(container.textContent).not.toContain("已保存 12,375");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("calls an active bounded scan filtering rather than saving", () => {
|
||||||
|
render(
|
||||||
|
<R6ImportProgress
|
||||||
|
sourceIds={["cursor"]}
|
||||||
|
sources={[source]}
|
||||||
|
jobs={[
|
||||||
|
{
|
||||||
|
...job,
|
||||||
|
discovered: 3_522,
|
||||||
|
stored: 0,
|
||||||
|
history_after: "2026-10-15T00:00:00Z",
|
||||||
|
},
|
||||||
|
]}
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
expect(screen.getByText("已扫描 3,522 条,正在筛选最近 3 天")).toBeTruthy();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports only stored rows as saved after a completed partial import", () => {
|
||||||
|
const { container } = render(
|
||||||
|
<R6ImportProgress
|
||||||
|
sourceIds={["cursor"]}
|
||||||
|
sources={[source]}
|
||||||
|
jobs={[
|
||||||
|
{
|
||||||
|
...job,
|
||||||
|
status: "completed",
|
||||||
|
discovered: 25_177,
|
||||||
|
stored: 20_826,
|
||||||
|
skipped: 4_351,
|
||||||
|
},
|
||||||
|
]}
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
expect(screen.getByText("已保存 20,826 条消息")).toBeTruthy();
|
||||||
|
expect(screen.getByText("导入完成")).toBeTruthy();
|
||||||
|
expect(container.textContent).not.toContain("25,177");
|
||||||
|
});
|
||||||
|
|
||||||
it.each(["paused", "interrupted", "failed", "completed", "cancelled"] as const)(
|
it.each(["paused", "interrupted", "failed", "completed", "cancelled"] as const)(
|
||||||
"does not animate %s as active work",
|
"does not animate %s as active work",
|
||||||
(status) => {
|
(status) => {
|
||||||
|
|||||||
@@ -3,6 +3,21 @@ import { useEffect, useLayoutEffect, useRef } from "react";
|
|||||||
import { AppMark } from "./app-mark";
|
import { AppMark } from "./app-mark";
|
||||||
import "./r6-import-progress.css";
|
import "./r6-import-progress.css";
|
||||||
|
|
||||||
|
function progressText(job: ImportJob): string | null {
|
||||||
|
if (job.stored > 0) return `已保存 ${job.stored.toLocaleString()} 条消息`;
|
||||||
|
if (job.discovered <= 0) return null;
|
||||||
|
const scanned = `已扫描 ${job.discovered.toLocaleString()} 条`;
|
||||||
|
if (job.status === "completed")
|
||||||
|
return job.history_after
|
||||||
|
? `${scanned},本次没有符合时间范围的对话`
|
||||||
|
: `${scanned},本次没有可保存的对话`;
|
||||||
|
if (job.status === "pending" || job.status === "running")
|
||||||
|
return job.history_after
|
||||||
|
? `${scanned},正在筛选最近 3 天`
|
||||||
|
: `${scanned},正在筛选`;
|
||||||
|
return `${scanned},尚未保存`;
|
||||||
|
}
|
||||||
|
|
||||||
export function R6ImportProgress({
|
export function R6ImportProgress({
|
||||||
sourceIds,
|
sourceIds,
|
||||||
sources,
|
sources,
|
||||||
@@ -86,8 +101,9 @@ export function R6ImportProgress({
|
|||||||
interrupted: "已中断",
|
interrupted: "已中断",
|
||||||
failed: "导入遇到问题",
|
failed: "导入遇到问题",
|
||||||
cancelled: "已取消",
|
cancelled: "已取消",
|
||||||
completed: "已保存",
|
completed: "导入完成",
|
||||||
};
|
};
|
||||||
|
const progress = job ? progressText(job) : null;
|
||||||
return (
|
return (
|
||||||
<article
|
<article
|
||||||
className="r6-import-progress-card"
|
className="r6-import-progress-card"
|
||||||
@@ -102,10 +118,8 @@ export function R6ImportProgress({
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<h2>{source?.display_name ?? id}</h2>
|
<h2>{source?.display_name ?? id}</h2>
|
||||||
{job && job.stored > 0 ? (
|
{progress ? (
|
||||||
<p>{`已保存 ${job.stored.toLocaleString()} 条消息`}</p>
|
<p>{progress}</p>
|
||||||
) : job && job.discovered > 0 ? (
|
|
||||||
<p>{`已扫描 ${job.discovered.toLocaleString()} 条,正在保存`}</p>
|
|
||||||
) : noImportedHistory ? (
|
) : noImportedHistory ? (
|
||||||
<p>没有可保存的历史对话</p>
|
<p>没有可保存的历史对话</p>
|
||||||
) : null}
|
) : null}
|
||||||
|
|||||||
@@ -12,8 +12,9 @@ L1_RAW 对外读路径不可见:闸门焊在向量库唯一查询入口。
|
|||||||
hybrid_search_native / list_by_user 四个查询方法的共同必经处,
|
hybrid_search_native / list_by_user 四个查询方法的共同必经处,
|
||||||
上层无法绕开,新增接口自动受保护。
|
上层无法绕开,新增接口自动受保护。
|
||||||
|
|
||||||
按 ID 直取走主键查询、不经过 filter,故在 `client.async_get` 处单独挡
|
按 ID 直取走主键查询、不经过 filter,故在 `client.async_get` 处默认挡住。
|
||||||
(GET /v1/memories/{id} 与 /v1/memories/batch 的共同必经处)。
|
Python SDK 调用方可显式传 `include_raw=True` 读取;HTTP 接口不传该参数,
|
||||||
|
继续保持原有隐藏行为。
|
||||||
|
|
||||||
内部管线需读原始对话时显式 allow_raw_layer=True 放行:
|
内部管线需读原始对话时显式 allow_raw_layer=True 放行:
|
||||||
- `writer._get_recent_dialogue_history`:提取前拼多轮对话历史喂 LLM
|
- `writer._get_recent_dialogue_history`:提取前拼多轮对话历史喂 LLM
|
||||||
@@ -124,28 +125,63 @@ class TestListNoLongerFiltersInProcess:
|
|||||||
"list 又出现进程内过滤 raw 的写法,应交由 vdb 侧 filter 完成"
|
"list 又出现进程内过滤 raw 的写法,应交由 vdb 侧 filter 完成"
|
||||||
|
|
||||||
|
|
||||||
class TestGetByIdPathBlocksRaw:
|
class TestGetByIdRawOptIn:
|
||||||
"""按 ID 直取:走主键查询不经 filter,须在 async_get 处挡住。"""
|
"""按 ID 直取默认隐藏 raw,只有 SDK 显式选择时才返回。"""
|
||||||
|
|
||||||
def test_async_get_drops_raw_node(self):
|
@staticmethod
|
||||||
|
def _client(node):
|
||||||
|
from hy_memory.client import HyMemoryClient
|
||||||
|
|
||||||
|
class Store:
|
||||||
|
async def get_by_id(self, memory_id):
|
||||||
|
return node if memory_id == node.node_id else None
|
||||||
|
|
||||||
|
async def no_op_log(**kwargs):
|
||||||
|
return None
|
||||||
|
|
||||||
|
client = HyMemoryClient.__new__(HyMemoryClient)
|
||||||
|
client._vector_store = Store()
|
||||||
|
client._graph_store = None
|
||||||
|
client._backend_identity = lambda: {}
|
||||||
|
client._maybe_store_op_log = no_op_log
|
||||||
|
return client
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _raw_node():
|
||||||
|
return MemoryNode(
|
||||||
|
node_id="raw-1",
|
||||||
|
user_id="u1",
|
||||||
|
agent_id="a1",
|
||||||
|
session_id="s1",
|
||||||
|
content="原始对话",
|
||||||
|
layer=MemoryLayer.L1_RAW,
|
||||||
|
status=MemoryStatus.ACTIVE,
|
||||||
|
custom={"meta": [{"source": "test"}]},
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_default_still_hides_raw(self):
|
||||||
|
from hy_memory.client import HyMemoryClient
|
||||||
|
|
||||||
|
client = self._client(self._raw_node())
|
||||||
|
raw_get = HyMemoryClient.async_get.__wrapped__
|
||||||
|
assert run(raw_get(client, "raw-1")) is None
|
||||||
|
|
||||||
|
def test_explicit_include_raw_returns_lite_memory(self):
|
||||||
|
from hy_memory.client import HyMemoryClient
|
||||||
|
|
||||||
|
client = self._client(self._raw_node())
|
||||||
|
raw_get = HyMemoryClient.async_get.__wrapped__
|
||||||
|
result = run(raw_get(client, "raw-1", include_raw=True))
|
||||||
|
|
||||||
|
assert result["memory_id"] == "raw-1"
|
||||||
|
assert result["content"] == "原始对话"
|
||||||
|
assert result["layer"] == MemoryLayer.L1_RAW.value
|
||||||
|
assert result["meta"] == [{"source": "test"}]
|
||||||
|
|
||||||
|
def test_public_get_exposes_keyword_only_opt_in(self):
|
||||||
import inspect
|
import inspect
|
||||||
from hy_memory.client import HyMemoryClient
|
from hy_memory.client import HyMemoryClient
|
||||||
src = inspect.getsource(HyMemoryClient.async_get)
|
|
||||||
assert "MemoryLayer.L1_RAW" in src, \
|
|
||||||
"async_get 未挡 raw,batch 与按 ID 单取会漏出原始对话"
|
|
||||||
|
|
||||||
def test_block_happens_before_return(self):
|
param = inspect.signature(HyMemoryClient.get).parameters["include_raw"]
|
||||||
"""挡的位置必须在两处 return 之前,否则等于没挡。"""
|
assert param.default is False
|
||||||
import inspect
|
assert param.kind is inspect.Parameter.KEYWORD_ONLY
|
||||||
from hy_memory.client import HyMemoryClient
|
|
||||||
lines = inspect.getsource(HyMemoryClient.async_get).splitlines()
|
|
||||||
block_at = next(
|
|
||||||
i for i, l in enumerate(lines)
|
|
||||||
if "MemoryLayer.L1_RAW" in l and "vdb_node" in l
|
|
||||||
)
|
|
||||||
first_return_at = next(
|
|
||||||
i for i, l in enumerate(lines)
|
|
||||||
if l.strip().startswith("return {")
|
|
||||||
)
|
|
||||||
assert block_at < first_return_at, \
|
|
||||||
f"挡 raw 在第 {block_at} 行、首个 return 在第 {first_return_at} 行,顺序错了"
|
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
import asyncio
|
||||||
|
import threading
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from hy_memory.config import MemoryConfig
|
||||||
|
from hy_memory.data.cache_sqlite import SqliteCache
|
||||||
|
from hy_memory.data.history_store import HistoryStore
|
||||||
|
|
||||||
|
|
||||||
|
class _BlockingCursor:
|
||||||
|
def fetchone(self):
|
||||||
|
return (1,)
|
||||||
|
|
||||||
|
def fetchall(self):
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
class _BlockingConnection:
|
||||||
|
"""Hold execute() open so a concurrent close() can be observed."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.entered = threading.Event()
|
||||||
|
self.release = threading.Event()
|
||||||
|
self.closed = threading.Event()
|
||||||
|
|
||||||
|
def execute(self, *_args, **_kwargs):
|
||||||
|
self.entered.set()
|
||||||
|
if not self.release.wait(timeout=2):
|
||||||
|
raise TimeoutError("test did not release the blocking query")
|
||||||
|
if self.closed.is_set():
|
||||||
|
raise AssertionError("connection was closed while execute() was active")
|
||||||
|
return _BlockingCursor()
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
self.closed.set()
|
||||||
|
|
||||||
|
|
||||||
|
def _config(tmp_path) -> MemoryConfig:
|
||||||
|
config = MemoryConfig()
|
||||||
|
config.history.db_path = str(tmp_path / "history.db")
|
||||||
|
config.cache.db_path = str(tmp_path / "cache.db")
|
||||||
|
return config
|
||||||
|
|
||||||
|
|
||||||
|
async def _wait_for(event: threading.Event) -> None:
|
||||||
|
assert await asyncio.wait_for(asyncio.to_thread(event.wait, 1), timeout=2)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_history_close_waits_for_active_query(tmp_path):
|
||||||
|
store = HistoryStore(_config(tmp_path))
|
||||||
|
connection = _BlockingConnection()
|
||||||
|
store._conn = connection
|
||||||
|
|
||||||
|
query = asyncio.create_task(store.count())
|
||||||
|
await _wait_for(connection.entered)
|
||||||
|
closing = asyncio.create_task(store.close())
|
||||||
|
await asyncio.sleep(0.05)
|
||||||
|
|
||||||
|
assert not connection.closed.is_set()
|
||||||
|
connection.release.set()
|
||||||
|
assert await query == 1
|
||||||
|
await closing
|
||||||
|
assert connection.closed.is_set()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_cache_close_waits_for_active_query(tmp_path):
|
||||||
|
store = SqliteCache(_config(tmp_path))
|
||||||
|
connection = _BlockingConnection()
|
||||||
|
store._conn = connection
|
||||||
|
|
||||||
|
query = asyncio.create_task(store.load_metrics_range("0", "9"))
|
||||||
|
await _wait_for(connection.entered)
|
||||||
|
closing = asyncio.create_task(store.close())
|
||||||
|
await asyncio.sleep(0.05)
|
||||||
|
|
||||||
|
assert not connection.closed.is_set()
|
||||||
|
connection.release.set()
|
||||||
|
assert await query == []
|
||||||
|
await closing
|
||||||
|
assert connection.closed.is_set()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_history_concurrent_reads_and_writes_are_serialized(tmp_path):
|
||||||
|
store = HistoryStore(_config(tmp_path))
|
||||||
|
await store.initialize()
|
||||||
|
try:
|
||||||
|
inserted = await asyncio.gather(
|
||||||
|
*(
|
||||||
|
store.record_add(
|
||||||
|
memory_id=f"memory-{index}",
|
||||||
|
content="content",
|
||||||
|
layer="l2_fact",
|
||||||
|
isolation_key="user",
|
||||||
|
)
|
||||||
|
for index in range(200)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert all(row_id > 0 for row_id in inserted)
|
||||||
|
assert len(set(inserted)) == 200
|
||||||
|
|
||||||
|
counts = await asyncio.gather(*(store.count() for _ in range(100)))
|
||||||
|
assert counts == [200] * 100
|
||||||
|
finally:
|
||||||
|
await store.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_cache_concurrent_pipeline_log_reads_and_writes_are_serialized(tmp_path):
|
||||||
|
store = SqliteCache(_config(tmp_path))
|
||||||
|
await store.initialize()
|
||||||
|
try:
|
||||||
|
written = await asyncio.gather(
|
||||||
|
*(
|
||||||
|
store.store_pipeline_log(
|
||||||
|
request_id="request",
|
||||||
|
user_id="user",
|
||||||
|
agent_id="agent",
|
||||||
|
step="EXTRACT",
|
||||||
|
prompt=f"prompt-{index}",
|
||||||
|
response="response",
|
||||||
|
)
|
||||||
|
for index in range(200)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert written == [True] * 200
|
||||||
|
|
||||||
|
pages = await asyncio.gather(
|
||||||
|
*(store.get_pipeline_logs(request_id="request", limit=200) for _ in range(50))
|
||||||
|
)
|
||||||
|
assert all(len(page) == 200 for page in pages)
|
||||||
|
finally:
|
||||||
|
await store.close()
|
||||||
Reference in new issue
Block a user