Compare commits

...
Author SHA1 Message Date
kristinhe 0fd3211f9c fix(memhub): stop rejecting a source list Core can grant
The applications list was empty on every launch while every endpoint
answered 200 (issue #139). `useSources()` is cache-only, so the list is
whatever `getSources()` committed, and its transport guard spelled the root
kinds by hand — only `default` and `environment_override`. Core has a third,
`user_selected`, which is what a source gets when the person picks its root.
One such row failed the guard for the whole list, and the mount-time read is a
single `Promise.all`, so `commitSources` never ran and
`sourceSnapshot.confirmedAt` was never written.

The kinds are now listed once, in the contract, and both the web guard and the
CLI validator read that list instead of repeating it. A parity test pins Core's
`RootCandidateKind` to it, because a union is erased at build time and nothing
else would notice the next divergence.

An unrecognised kind is still rejected. A source Core grants a root to is not.
2026-10-24 08:15:29 +08:00
kristinhe1111 6b0bd86679 Merge pull request #156 from Tencent-Hunyuan/fix/build-17-test-index
docs(memhub): regenerate the core test index for build 17
2026-10-23 20:22:05 +08:00
kristinhe1111 91250d7c87 fix(desktop): keep an unknown startup failure code in the dialog
Build 17 routed every startup error through the friendly-text map, so three
desktop contracts that assert the dialog body (page_load_failed,
runtime_broker_unavailable, host_permission_store_unavailable) started failing.

Only codes we wrote copy for become sentences now; anything else keeps the code
it was thrown with, which is what support reads.
2026-10-23 20:02:05 +08:00
kristinhe1111 3c89a535fb docs(memhub): regenerate the core test index for build 17
The build-17 test file was added without regenerating TEST_INDEX.md, so the
public structure gate failed on main. Generated artifact only.
2026-10-23 19:49:44 +08:00
kristinhe1111 1151878265 Merge pull request #155 from Tencent-Hunyuan/release/memhub-build-17
fix(memhub): build 17 — migration fixes carry through to first launch
2026-10-23 19:34:32 +08:00
kristinhe e82c4c939e fix(memhub): name a blocked cutover without its counts
The cutover gate reports sdk_data_plane_cutover_work_pending plus a JSON body of
table counts. That string went straight into the document the startup dialog
reads, so the person saw braces and table names.

Collapse it to sdk_migration_work_pending and let the dialog say the data is
already upgraded but earlier memory work is still finishing. Any other code that
is not a plain identifier becomes sdk_migration_cutover_failed, so no free text
reaches the dialog.
2026-10-23 19:33:31 +08:00
kristinhe ca4a66349b fix(memhub): explain the wait and the failure on first launch
Three things a person meets before the app can open:

- The waiting window had no drag region element, so a frameless window could not
  be moved at all; the CSS alone never applied to anything.
- The page said only that the service was starting. It now says a first launch
  or a memory upgrade can take a few minutes and the app opens by itself, in all
  three interface languages.
- A migration failure surfaced as a raw code in one path and a generic sentence
  in the other. Every migration code now maps to a sentence that says what
  happened, what happens to their memories, and what to do next; the dispatch
  path reuses the same mapping instead of printing error.message.

Build number 17 carries the diagnostics fix and the merged SDK migration fix.
2026-10-23 19:19:06 +08:00
kristinhe 82044d6172 fix(memhub): name the migration failure instead of one opaque code
A full disk, an unconfigured model profile and a Coding classification gap all
reached the person as sdk_migration_backfill_failed, so the startup dialog could
only ever say the upgrade failed.

Classify the exception chain before the code is written: ENOSPC/EDQUOT becomes
sdk_migration_disk_space_insufficient, SDK productivity errors become
sdk_migration_coding_inventory_failed, the legacy working_fact import failure
keeps its own name, and existing sdk_data_plane_/sdk_migration_ codes pass
through. Anything unrecognized keeps its message out of the document.
2026-10-23 19:18:56 +08:00
kristinhe 7f23b74c36 fix(memhub): separate local diagnostics from safe exports 2026-10-23 19:03:50 +08:00
kristinhe 617e2d0ba6 fix(memhub): keep diagnostics from blocking startup 2026-10-23 19:03:50 +08:00
feitianxiang 083b837f32 Merge pull request #154 from Tencent-Hunyuan/fix/sdk-migration-turns-and-coding
fix(memhub): harden SDK migration inventory and fencing
2026-10-23 17:37:15 +08:00
alvinfeiandCursor 1dc7e957c0 fix(memhub): harden SDK migration inventory
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-23 17:34:59 +08:00
feitianxiang 3b44d8381c Merge pull request #153 from Tencent-Hunyuan/fix/sdk-migration-turns-and-coding
fix(memhub): repair legacy SDK authority migration
2026-10-23 17:27:39 +08:00
alvinfeiandCursor 790d6d3d5e fix(memhub): repair SDK authority migration
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-23 17:26:40 +08:00
feitianxiang 4c311b2e93 Merge pull request #149 from Tencent-Hunyuan/fix/memhub-1.0.5-release-doc-test
test(memhub): update 1.0.5 release doc expectations
2026-10-22 21:31:04 +08:00
alvinfeiandCursor 801a2af36b test(memhub): update release documentation expectations
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 21:30:31 +08:00
feitianxiang 0a8e43e570 Merge pull request #148 from Tencent-Hunyuan/release/memhub-1.0.5
release(memhub): bump to 1.0.5 and pin hy-memory 2.0.5
2026-10-22 21:22:40 +08:00
alvinfeiandCursor f5a38c38b3 release(memhub): pin hy-memory 2.0.5
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 21:20:40 +08:00
feitianxiang d08e070816 Merge pull request #147 from Tencent-Hunyuan/release/hy-memory-2.0.5
release: hy-memory 2.0.5
2026-10-22 20:58:49 +08:00
alvinfeiandCursor 29bd9146a4 release: hy-memory 2.0.5
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 20:52:21 +08:00
26c33ee40f refactor(sdk): internalize legacy data migration (#146)
* refactor(sdk): keep legacy migration out of client APIs

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(migration): stage SDK data before cutover

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 20:49:23 +08:00
83 changed files with 4770 additions and 1818 deletions

No files matched your search

+89 -61
View File
@@ -2,6 +2,78 @@
Hy Memory and MemHub are versioned and released independently.
## Unreleased
- Fixed legacy-to-SDK startup migration to adopt native Coding Experience and
Working Fact IDs instead of importing them as Chat layers. Raw migration now
rebuilds bounded user/assistant turns from the latest three days, excludes
tools, and enforces the same rolling retention after cutover.
- Paused host capture writes with a live Desktop migration fence; Cursor
History catch-up resumes from its existing checkpoint after cutover.
- Removed MemHub's private `hy_memory_bridge` implementation. MemHub now calls
public `HyMemoryClient` methods at runtime; prompt/reconcile/response
monkey-patches and private VDB/Coding runtime access were deleted. Kuzu Graph
projection/admin and stopped-writer App migration are the documented product
exceptions.
- Added a MemHub-managed native Hermes MemoryProvider with automatic prefetch,
persistent per-session memory-ID deduplication, authorized Broker reads, and
recall-context stripping to prevent memory recursion.
- Added an extensible MemHub interface-localization layer with Simplified
Chinese, English, and Spanish catalogs shared by Web and Electron-native
surfaces. Interface language remains independent from memory content and
model processing language.
- Fixed governed-memory revisions and repeated evidence links failing when the
visibility dirty-marker trigger encounters the same memory twice. Startup
now replaces the affected legacy trigger without changing schema versions
or governed and visibility data.
- Kept post-write authority verification failures available for read-only
reconciliation while preventing automatic or generic confirmed retries from
invoking the model again. Automatic reconciliation skips those sticky rows
before applying its batch limit, and linked durable jobs exclusively own
retry scheduling for their memory tasks.
- Removed MemHub's task-scoped HyPermission store wrappers and global
model-content consent switch. The trusted embedded Hy Memory SDK now uses
its native local stores directly.
- Existing installations ignore the retired config flag, readmit previously
parked pre-invocation work, and drop only the obsolete
`hy_node_permissions` proof table. Evidence, governed memories, model
profiles, SDK stores, and source/host grants are preserved.
- Fixed the applications list going permanently blank when a source's root was
chosen by the person rather than detected. The web transport guard listed
the root kinds by hand and rejected the whole list over the `user_selected`
kind Core already grants; the kinds now come from the shared contract, so the
two cannot drift again, and a parity test pins Core's enumeration to it.
## Hy Memory 2.0.5 — 2026-10-22
- Published
[`hy-memory==2.0.5`](https://pypi.org/project/hy-memory/2.0.5/).
- Extended `get()` across Chat, Coding Experience, Working Fact, and raw IDs;
the returned `layer`, `kind`, and `variant` identify the actual record.
- Added explicit raw retrieval through the existing `search()` and
`list_memories()` contracts, including user, agent, session, and source
filters.
- Added public embedding-service injection and corrected extraction language
detection for structured OpenAI-style messages.
- Added lifecycle status filters, durable idempotent write outcomes, and
quiesced local data-plane snapshot/restore.
- Kept legacy migration mechanics out of the public Client API; separate
migration-only raw/import methods from development previews were removed.
- Physical delete now reports partial VDB/Graph cleanup as failure.
- Coding search no longer injects the version-evolution diff block by default;
opt in with `MEMORY_CODING_CHAIN_BLOCK_ENABLED=true`.
- Serialized SDK SQLite cache/history connection lifecycle and tightened
Coding reconciliation to its agent/session/workspace authority.
- Clarified persistent caller context through `meta`, while `metadata` remains
legacy processing configuration.
## Hy Memory 2.0.5.dev4 — 2026-10-21
- Development preview used for SDK-authority migration validation; not the
final `2.0.5` release.
- Published as
[`hy-memory==2.0.5.dev4`](https://pypi.org/project/hy-memory/2.0.5.dev4/).
## Hy Memory 2.0.5.dev3 — 2026-10-21
- Development preview; not the final `2.0.5` release.
@@ -28,6 +100,12 @@ Hy Memory and MemHub are versioned and released independently.
APIs, offline local data-plane snapshot/restore, and fail-closed physical
delete receipts.
## Hy Memory 2.0.4 — 2026-10-17
- Published
[`hy-memory==2.0.4`](https://pypi.org/project/hy-memory/2.0.4/) as a
maintenance release.
## Hy Memory 2.0.3 — 2026-10-17
- Added an optional runtime HTTP transport factory so embedded products can
@@ -38,66 +116,6 @@ Hy Memory and MemHub are versioned and released independently.
unchanged memory extraction, reconciliation, storage, and Coding/Working
behavior.
## Unreleased
- Removed MemHub's private `hy_memory_bridge` implementation. MemHub now calls
public `HyMemoryClient` methods through explicit runtime, write, migration,
and read modules; prompt/reconcile/response monkey-patches and private
VDB/Coding access were deleted. Kuzu Graph projection/admin remains the one
documented product exception.
- Extended public SDK `get()` to resolve Chat, Coding Experience, and Working
Fact IDs, including owner/session fields and explicit ambiguity handling;
added constructor-time public embedding-service injection.
- Added a MemHub-managed native Hermes MemoryProvider with automatic prefetch,
persistent per-session memory-ID deduplication, authorized Broker reads, and
recall-context stripping to prevent memory recursion.
- Added an extensible MemHub interface-localization layer with Simplified
Chinese, English, and Spanish catalogs shared by Web and Electron-native
surfaces. Interface language remains independent from memory content and
model processing language.
- `search(..., layers=["l1_raw"])` now explicitly searches SDK raw records in
pro/ultra mode and keeps existing user/agent/session/source filters; default
search continues to hide raw.
- Added `import_memory_exact` for idempotent migration of an existing semantic
memory with its canonical ID, lifecycle and supersession links, without
extraction or reconciliation.
- Added SDK-authoritative lifecycle listing with explicit status filters,
idempotent durable write outcomes, exact raw import/get/list APIs, and
quiesced local data-plane snapshot/restore contracts for the MemHub
authority migration.
- Physical delete now reports incomplete VDB/Graph cleanup as a partial failure
instead of claiming success after only one store was changed.
- Coding search no longer injects the version-evolution diff block
(`chain_block`) by default. Enable with
`MEMORY_CODING_CHAIN_BLOCK_ENABLED=true` or
`CodingConfig.chain_block_enabled=True` (write-side observe unchanged).
- Fixed governed-memory revisions and repeated evidence links failing when the
visibility dirty-marker trigger encounters the same memory twice. Startup
now replaces the affected legacy trigger without changing schema versions
or governed and visibility data.
- Serialized each SDK SQLite store's shared connection lifecycle so concurrent
history/cache operations cannot race with `close()` and crash the process.
- Scoped Coding reconciliation targets to the current HY authority,
conversation, and workspace, preventing near-duplicate sessions from
mutating an experience or working fact owned by another evidence context.
- Kept post-write authority verification failures available for read-only
reconciliation while preventing automatic or generic confirmed retries from
invoking the model again. Automatic reconciliation skips those sticky rows
before applying its batch limit, and linked durable jobs exclusively own
retry scheduling for their memory tasks.
- Clarified that `add(metadata=...)` carries legacy SDK processing options,
while persistent caller business context belongs in `meta`.
- Added an opt-in `get(..., include_raw=True)` path for reading `L1_RAW`
memories, including memories written in lite mode. The default remains
hidden for backward compatibility.
- Removed MemHub's task-scoped HyPermission store wrappers and global
model-content consent switch. The trusted embedded Hy Memory SDK now uses
its native local stores directly.
- Existing installations ignore the retired config flag, readmit previously
parked pre-invocation work, and drop only the obsolete
`hy_node_permissions` proof table. Evidence, governed memories, model
profiles, SDK stores, and source/host grants are preserved.
## Hy Memory 2.0.2 — 2026-10-17
- Published
@@ -112,7 +130,17 @@ Hy Memory and MemHub are versioned and released independently.
Because PyPI release metadata is immutable, the next SDK patch release must
carry the updated license metadata; `2.0.1` retains its published metadata.
## MemHub 1.0.3 — Unreleased
## MemHub 1.0.5 — Unreleased
- Updated the source version to `1.0.5` (build 14).
- Pinned the public SDK to the stable `hy-memory==2.0.5` release.
## MemHub 1.0.4 — 2026-10-18
- Updated the source version to `1.0.4` (build 13).
- Published the `v1.0.4` source and release metadata.
## MemHub 1.0.3 — 2026-10-17
- Updated the source version to `1.0.3` (build 7).
- Pinned the public SDK to `hy-memory==2.0.3`.
+7 -6
View File
@@ -273,8 +273,8 @@ catalog 也可以在没有基础 L0 时形成只读的合成画像。
```python
memory = client.get("memory_id")
# lite 模式写入的是 L1_RAW;按 ID 读取时必须显式选择返回原文。
raw_memory = client.get("memory_id", include_raw=True)
# raw ID 返回 L1_RAW,fact ID 返回对应事实层。
raw_memory = client.get("raw_id")
result = client.update("memory_id", "更新后的内容")
result = client.delete("memory_id")
result = client.delete_all(user_id="user_001")
@@ -284,7 +284,7 @@ page = client.list_memories(
limit=100,
offset=0,
order="desc",
layers=None,
layers=["l1_raw"], # 显式列出 raw;不传仍沿用默认语义层列表
with_evolution=False,
)
@@ -294,9 +294,10 @@ result = client.clone_user("user_001", "user_002", agent_id=None)
对应异步方法为 `async_get()`、`async_update()`、`async_delete()`、
`async_delete_all()`、`async_list_memories()`、`async_clone_user()`。
`get()` / `async_get()` 默认继续隐藏 `L1_RAW`,因此不会改变现有调用方的原文访问
边界。只有显式传入 `include_raw=True` 才返回该层;这也是 lite 模式按 ID 读取
写入内容及其 `meta` 的方式。MemHub 的 HTTP 读取接口不开放此参数。
`get()` / `async_get()` 是统一的 ID 精确读取:返回结果的 `layer`、`kind` 和
`variant` 表明实际类型。授权由调用 SDK 前的产品边界完成,不由 SDK 根据 layer
猜测。`list_memories()` 默认仍不包含 `L1_RAW`;显式传
`layers=["l1_raw"]` 时按同一列表合同返回 raw。
普通 `list_memories()` 返回 `vdb` 桶,并在 Graph 有结果时附带 `graph`;传入
`workspace_id` 时返回生产力记忆的 `scene="productivity"` 和 `experiences`。
+63 -440
View File
@@ -1886,7 +1886,6 @@ class HyMemoryClient:
self,
memory_id: str,
*,
include_raw: bool = False,
kind: Optional[MemoryKind] = None,
variant: Optional[MemoryVariant] = None,
) -> Optional[Dict[str, Any]]:
@@ -1895,8 +1894,6 @@ class HyMemoryClient:
Args:
memory_id: 记忆 ID
include_raw: 是否允许返回 L1_RAW 原始记忆。默认 False,保持原有隐藏行为;
lite 模式的记忆均为 L1_RAW,需显式设为 True 才能按 ID 读取。
kind: 可选 namespace("chat" / "experience" / "fact")。
variant: 可选生产力变体("working" / "coding")。
@@ -1906,7 +1903,6 @@ class HyMemoryClient:
return self._loop_thread.run(
self.async_get(
memory_id,
include_raw=include_raw,
kind=kind,
variant=variant,
)
@@ -1978,6 +1974,9 @@ class HyMemoryClient:
branch: Optional[str] = None,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
layers: Optional[List[Any]] = None,
statuses: Optional[List[Any]] = None,
with_evolution: bool = False,
@@ -2000,6 +1999,9 @@ class HyMemoryClient:
user_ids: 可选,多用户批量列举。非空时优先于 user_id,返回并集
(分页在合并排序后统一施加)。
agent_ids: 可选,多 agent 批量列举。非空时优先于 agent_id。
session_ids: 可选,只返回指定 session。
include_sources: 可选,只返回指定来源。
exclude_sources: 可选,排除指定来源;与 include_sources 冲突时排除优先。
layers: 可选,只返回指定层级(字符串或 MemoryLayer)。不传时返回
L1_RAW 以外的全部层级,与改动前一致。
statuses: 可选,只返回指定生命周期(字符串或 MemoryStatus)。不传时
@@ -2015,6 +2017,9 @@ class HyMemoryClient:
self.async_list_memories(
user_id=user_id, agent_id=agent_id,
user_ids=user_ids, agent_ids=agent_ids,
session_ids=session_ids,
include_sources=include_sources,
exclude_sources=exclude_sources,
limit=limit, offset=offset, order=order,
workspace_id=workspace_id, branch=branch,
layers=layers, statuses=statuses, with_evolution=with_evolution,
@@ -2975,106 +2980,6 @@ class HyMemoryClient:
include_runtime_diag=True,
)
def import_raw(
self,
content: str,
*,
user_id: str,
agent_id: str = "default_agent",
session_id: str = "default_session",
source: str = "default",
memory_at: Optional[datetime] = None,
metadata: Optional[Dict[str, Any]] = None,
raw_id: Optional[str] = None,
request_id: Optional[str] = None,
) -> Dict[str, Any]:
"""Persist one exact raw record without extraction or reconciliation."""
return self._loop_thread.run(
self.async_import_raw(
content,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
memory_at=memory_at,
metadata=metadata,
raw_id=raw_id,
request_id=request_id,
),
include_runtime_diag=True,
)
def import_memory_exact(
self,
content: str,
*,
memory_id: str,
user_id: str,
agent_id: str = "default_agent",
session_id: str = "default_session",
source: str = "default",
layer: Any = "l2_fact",
status: Any = "active",
owner: Optional[str] = None,
tags: Optional[List[str]] = None,
supersedes: Optional[List[str]] = None,
superseded_by: Optional[List[str]] = None,
is_latest: bool = True,
memory_at: Optional[datetime] = None,
metadata: Optional[Dict[str, Any]] = None,
request_id: Optional[str] = None,
) -> Dict[str, Any]:
"""Import one existing semantic memory without extraction/reconcile."""
return self._loop_thread.run(
self.async_import_memory_exact(
content,
memory_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=layer,
status=status,
owner=owner,
tags=tags,
supersedes=supersedes,
superseded_by=superseded_by,
is_latest=is_latest,
memory_at=memory_at,
metadata=metadata,
request_id=request_id,
),
include_runtime_diag=True,
)
def get_raw(self, raw_id: str) -> Optional[Dict[str, Any]]:
"""Read one SDK raw record through the explicit raw API."""
return self._loop_thread.run(self.async_get_raw(raw_id))
def list_raw(
self,
*,
user_id: str,
agent_id: Optional[str] = None,
session_id: Optional[str] = None,
source: Optional[str] = None,
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> Dict[str, Any]:
"""List raw records for an authorized SDK owner/scope."""
return self._loop_thread.run(
self.async_list_raw(
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
limit=limit,
offset=offset,
order=order,
)
)
@_ensure_internal_loop
async def async_add_extracted(
self,
@@ -3227,316 +3132,6 @@ class HyMemoryClient:
}
@_ensure_internal_loop
async def async_import_raw(
self,
content: str,
*,
user_id: str,
agent_id: str = "default_agent",
session_id: str = "default_session",
source: str = "default",
memory_at: Optional[datetime] = None,
metadata: Optional[Dict[str, Any]] = None,
raw_id: Optional[str] = None,
request_id: Optional[str] = None,
) -> Dict[str, Any]:
"""Persist exact source content as L1_RAW without running a model."""
from .utils.source_filter import normalize_source
if not isinstance(content, str) or not content:
raise ValueError("content is required")
if not user_id:
raise ValueError("user_id is required")
request_id = request_id or str(uuid.uuid4())
replay = await self._claim_write_outcome(
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
operation="import_raw",
)
if replay is not None:
return replay
try:
result = await self._async_import_raw_impl(
content,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=normalize_source(source),
memory_at=memory_at,
metadata=metadata,
raw_id=raw_id,
request_id=request_id,
)
except BaseException:
await self._mark_write_outcome_unknown(request_id)
raise
await self._finish_write_outcome(request_id, "succeeded", result)
return result
async def _async_import_raw_impl(
self,
content: str,
*,
user_id: str,
agent_id: str,
session_id: str,
source: str,
memory_at: Optional[datetime],
metadata: Optional[Dict[str, Any]],
raw_id: Optional[str],
request_id: str,
) -> Dict[str, Any]:
from .models.memory import MemoryLayer, MemoryNode, MemoryStatus, SourceType
node_id = raw_id or str(uuid.uuid4())
existing = await self._vector_store.get_by_id(node_id)
if existing is not None:
identity = (
existing.layer == MemoryLayer.L1_RAW,
existing.content == content,
existing.user_id == user_id,
existing.agent_id == agent_id,
existing.session_id == session_id,
existing.source == source,
)
if not all(identity):
raise ValueError("raw_id is already bound to different source content")
return {
"success": True,
"raw_id": node_id,
"request_id": request_id,
"replayed": True,
}
custom = dict(metadata or {})
custom["request_id"] = request_id
custom["input_kind"] = "raw_exact_import"
embedding = await self._embed_service.embed(content)
node = MemoryNode(
node_id=node_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=MemoryLayer.L1_RAW,
content=content,
source_type=SourceType.EXPLICIT,
status=MemoryStatus.ACTIVE,
is_latest=True,
embedding=embedding,
memory_at=memory_at,
custom=custom,
)
stored_id = await self._vector_store.upsert(node)
if stored_id != node_id:
raise RuntimeError("raw import changed the requested identity")
return {"success": True, "raw_id": node_id, "request_id": request_id}
@_ensure_internal_loop
async def async_import_memory_exact(
self,
content: str,
*,
memory_id: str,
user_id: str,
agent_id: str = "default_agent",
session_id: str = "default_session",
source: str = "default",
layer: Any = "l2_fact",
status: Any = "active",
owner: Optional[str] = None,
tags: Optional[List[str]] = None,
supersedes: Optional[List[str]] = None,
superseded_by: Optional[List[str]] = None,
is_latest: bool = True,
memory_at: Optional[datetime] = None,
metadata: Optional[Dict[str, Any]] = None,
request_id: Optional[str] = None,
) -> Dict[str, Any]:
"""Import a canonical memory node exactly, with no model invocation."""
from .models.memory import MemoryLayer, MemoryStatus
from .utils.source_filter import normalize_source
if not content or not memory_id or not user_id:
raise ValueError("content, memory_id and user_id are required")
normalized_layer = (
layer if isinstance(layer, MemoryLayer) else MemoryLayer.from_string(str(layer))
)
if normalized_layer == MemoryLayer.L1_RAW:
raise ValueError("use import_raw for L1_RAW")
normalized_status = (
status if isinstance(status, MemoryStatus) else MemoryStatus(str(status).lower())
)
request_id = request_id or str(uuid.uuid4())
replay = await self._claim_write_outcome(
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
operation="import_memory_exact",
)
if replay is not None:
return replay
try:
result = await self._async_import_memory_exact_impl(
content,
memory_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=normalize_source(source),
layer=normalized_layer,
status=normalized_status,
owner=owner,
tags=tags or [],
supersedes=supersedes or [],
superseded_by=superseded_by or [],
is_latest=is_latest,
memory_at=memory_at,
metadata=metadata,
request_id=request_id,
)
except BaseException:
await self._mark_write_outcome_unknown(request_id)
raise
await self._finish_write_outcome(request_id, "succeeded", result)
return result
async def _async_import_memory_exact_impl(
self,
content: str,
*,
memory_id: str,
user_id: str,
agent_id: str,
session_id: str,
source: str,
layer,
status,
owner: Optional[str],
tags: List[str],
supersedes: List[str],
superseded_by: List[str],
is_latest: bool,
memory_at: Optional[datetime],
metadata: Optional[Dict[str, Any]],
request_id: str,
) -> Dict[str, Any]:
from .models.memory import MemoryNode, SourceType
existing = await self._vector_store.get_by_id(memory_id)
if existing is not None:
identity = (
existing.content == content,
existing.user_id == user_id,
existing.agent_id == agent_id,
existing.session_id == session_id,
existing.source == source,
existing.layer == layer,
existing.status == status,
existing.owner == owner,
list(existing.tags or []) == tags,
list(existing.supersedes or []) == supersedes,
list(existing.superseded_by or []) == superseded_by,
existing.is_latest == is_latest,
)
if not all(identity):
raise ValueError("memory_id is already bound to different memory content")
return {
"success": True,
"memory_id": memory_id,
"memory_ids": [memory_id],
"request_id": request_id,
"replayed": True,
}
custom = dict(metadata or {})
custom["request_id"] = request_id
custom["input_kind"] = "memory_exact_import"
embedding = await self._embed_service.embed(content)
node = MemoryNode(
node_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=layer,
content=content,
source_type=SourceType.EXPLICIT,
status=status,
owner=owner,
tags=list(tags),
supersedes=list(supersedes),
superseded_by=list(superseded_by),
is_latest=is_latest,
embedding=embedding,
memory_at=memory_at,
custom=custom,
)
stored_id = await self._vector_store.upsert(node)
if stored_id != memory_id:
raise RuntimeError("exact memory import changed the requested identity")
return {
"success": True,
"memory_id": memory_id,
"memory_ids": [memory_id],
"request_id": request_id,
}
@_ensure_internal_loop
async def async_get_raw(self, raw_id: str) -> Optional[Dict[str, Any]]:
from .models.memory import MemoryLayer
node = await self._vector_store.get_by_id(raw_id)
if node is None or node.layer != MemoryLayer.L1_RAW:
return None
return self._raw_node_to_public(node)
@_ensure_internal_loop
async def async_list_raw(
self,
*,
user_id: str,
agent_id: Optional[str] = None,
session_id: Optional[str] = None,
source: Optional[str] = None,
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> Dict[str, Any]:
from .models.memory import MemoryLayer, MemoryStatus
if not user_id:
raise ValueError("user_id is required")
if limit < 1 or offset < 0:
raise ValueError("limit must be positive and offset must be non-negative")
nodes = await self._vector_store.list_by_user(
user_id=user_id,
agent_id=agent_id,
status_filter=[MemoryStatus.ACTIVE],
layers=[MemoryLayer.L1_RAW],
include_sources=[source] if source else None,
limit=100_000,
only_latest=False,
with_vectors=False,
allow_raw_layer=True,
)
if session_id:
nodes = [node for node in nodes if node.session_id == session_id]
nodes = self._sort_memory_nodes(nodes, order=order)
total = len(nodes)
return {
"raw": [
self._raw_node_to_public(node)
for node in nodes[offset: offset + limit]
],
"total": total,
"limit": limit,
"offset": offset,
}
async def _search_explicit_raw_layer(
self,
query: str,
@@ -4500,7 +4095,6 @@ class HyMemoryClient:
self,
memory_id: str,
*,
include_raw: bool = False,
kind: Optional[MemoryKind] = None,
variant: Optional[MemoryVariant] = None,
) -> Optional[Dict[str, Any]]:
@@ -4513,15 +4107,12 @@ class HyMemoryClient:
Args:
memory_id: 记忆 ID
include_raw: 是否允许返回 L1_RAW 原始记忆,默认 False。
kind: 可选 namespace("chat" / "experience" / "fact")。
variant: 可选生产力变体("working" / "coding")。
Returns:
统一 memory dict,或 None。
"""
from .models.memory import MemoryLayer
if kind not in (None, "chat", "experience", "fact"):
raise ValueError(
"kind must be one of: chat, experience, fact"
@@ -4583,26 +4174,6 @@ class HyMemoryClient:
vdb_node = by_name.get("chat:vdb")
graph_node = by_name.get("chat:graph")
# L1_RAW 默认不对外可见:按 ID 直取走主键查询、绕过向量库的 layer
# filter,故在此统一挡掉。SDK 调用方可通过 include_raw=True 显式读取;
# HTTP 接口不传该参数,继续保持隐藏。内部写链路不经过此方法。
#
# 置于 gather 之后、埋点之前:命中 raw 仍照常落 GET 埋点(保留可观测性,
# 便于发现有调用方拿 raw id 试探),只是不把内容返回给调用方。
# 对调用方的表现与"该 id 不存在"完全一致,不额外暴露"存在但被挡"的信息。
if (
not include_raw
and vdb_node is not None
and vdb_node.layer == MemoryLayer.L1_RAW
):
vdb_node = None
if (
not include_raw
and graph_node is not None
and getattr(graph_node, "layer", None) == MemoryLayer.L1_RAW
):
graph_node = None
matches = []
chat_node = vdb_node if vdb_node is not None else graph_node
if chat_node is not None:
@@ -4682,7 +4253,8 @@ class HyMemoryClient:
"confidence": vdb_node.confidence,
"custom": vdb_node.custom or {},
"meta": list((vdb_node.custom or {}).get("meta") or []),
"source": "vdb",
"source": vdb_node.source,
"retrieval_source": "vdb",
"request_id": request_id,
"elapsed_ms": elapsed,
}
@@ -4709,7 +4281,8 @@ class HyMemoryClient:
"owner": getattr(graph_node, "owner", None),
"confidence": graph_node.confidence,
"custom": getattr(graph_node, "custom", {}) or {},
"source": "graph",
"source": getattr(graph_node, "source", "") or "",
"retrieval_source": "graph",
"request_id": request_id,
"elapsed_ms": elapsed,
}
@@ -5282,6 +4855,7 @@ class HyMemoryClient:
"user_id": node.user_id,
"agent_id": node.agent_id,
"session_id": getattr(node, "session_id", None),
"source": getattr(node, "source", None),
"speculate": getattr(node, "speculate", None),
# 不透出 source_raw_memory_id:见 async_search 同处说明。
"tags": node.tags or [],
@@ -5353,6 +4927,9 @@ class HyMemoryClient:
order: str,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
layers: Optional[List] = None,
statuses: Optional[List] = None,
with_evolution: bool = False,
@@ -5378,6 +4955,9 @@ class HyMemoryClient:
status_filter=effective_statuses,
layers=layers,
with_vectors=False,
allow_raw_layer=bool(
layers is not None and MemoryLayer.L1_RAW in layers
),
)
# L0 是 user-level,不应被 agent 过滤。默认列表或显式包含 L0 时,额外按
@@ -5399,6 +4979,22 @@ class HyMemoryClient:
])
all_nodes.extend(hit["node"] for hit in l0_hits if hit.get("node") is not None)
if session_ids:
wanted_sessions = set(session_ids)
all_nodes = [
node for node in all_nodes if node.session_id in wanted_sessions
]
if include_sources:
wanted_sources = set(include_sources)
all_nodes = [
node for node in all_nodes if node.source in wanted_sources
]
if exclude_sources:
blocked_sources = set(exclude_sources)
all_nodes = [
node for node in all_nodes if node.source not in blocked_sources
]
all_nodes = self._sort_memory_nodes(all_nodes, order=order)
total = len(all_nodes)
page_nodes = all_nodes[offset: offset + limit]
@@ -5462,6 +5058,9 @@ class HyMemoryClient:
order: str,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
layers: Optional[List] = None,
statuses: Optional[List] = None,
) -> Optional[Dict[str, Any]]:
@@ -5509,6 +5108,21 @@ class HyMemoryClient:
node for node in graph_nodes
if getattr(node, "status", None) in effective_statuses
]
if session_ids:
wanted_sessions = set(session_ids)
graph_nodes = [
node for node in graph_nodes if node.session_id in wanted_sessions
]
if include_sources:
wanted_sources = set(include_sources)
graph_nodes = [
node for node in graph_nodes if node.source in wanted_sources
]
if exclude_sources:
blocked_sources = set(exclude_sources)
graph_nodes = [
node for node in graph_nodes if node.source not in blocked_sources
]
graph_nodes = self._sort_memory_nodes(graph_nodes, order=order)
total = len(graph_nodes)
@@ -5620,6 +5234,9 @@ class HyMemoryClient:
branch: Optional[str] = None,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
layers: Optional[List[Any]] = None,
statuses: Optional[List[Any]] = None,
with_evolution: bool = False,
@@ -5696,6 +5313,9 @@ class HyMemoryClient:
vdb = await self._list_vdb_bucket(
user_id=user_id, agent_id=agent_id,
user_ids=user_ids, agent_ids=agent_ids,
session_ids=session_ids,
include_sources=include_sources,
exclude_sources=exclude_sources,
limit=limit, offset=offset, order=order,
layers=effective_layers, statuses=effective_statuses,
with_evolution=with_evolution,
@@ -5703,6 +5323,9 @@ class HyMemoryClient:
graph = await self._list_graph_bucket(
user_id=user_id, agent_id=agent_id,
user_ids=user_ids, agent_ids=agent_ids,
session_ids=session_ids,
include_sources=include_sources,
exclude_sources=exclude_sources,
limit=limit, offset=offset, order=order,
layers=effective_layers, statuses=effective_statuses,
)
+5 -1
View File
@@ -9,8 +9,12 @@
## Architecture Principles
1. Memory 读写与存储 backend **只经 SDK**(config / env)。
2. MemHub 对 memory 的原子操作 **只经 SDK**;缺接口先补 SDK。
2. MemHub 运行时对 memory 的原子操作 **只经 SDK 公共 Client**;缺正常业务
接口先补 SDK。Runtime writers 停止后的 App 启动迁移可直接操作底层存储,
但不得把迁移专用方法加入公共 Client。
3. MemHub **不存 memory 实体**;**可以存 `memory_id`** 做组织/回执。原文进入 **SDK raw**。
4. SDK raw 只保留滚动三天;一条 raw 对应一个完整 turn,只含真人 user 与
assistant,tool 不持久化。提取后的高层 memory 不受该窗口影响。
生产入口通过共享 Runtime 租约取得公开 `HyMemoryClient`;MemHub 只在调用前执行
模型配置、密钥和 harness/model 门禁,不维护第二套 memory 正文权威。
+5 -5
View File
@@ -18,13 +18,13 @@ If you run into a problem, please open an Issue with the version, platform,repro
## 发行与平台合同
源码当前固定使用公开 PyPI 开发预览 `hy-memory==2.0.5.dev3`。MemHub 源码版本为 `1.0.4`
(build 13),GitHub 仓库为
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.5`。MemHub 源码版本为 `1.0.5`
(build 14),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
`v1.0.4` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
`v1.0.5` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
@@ -45,7 +45,7 @@ Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命
## 仓库结构
- `apps/core`:Python / FastAPI Kernel,负责授权、模型、队列、SDK bridge 与 API。
- `apps/core`:Python / FastAPI Kernel,负责授权、模型、队列、SDK Runtime 边界与 API。
- `apps/cli`:安装、控制和诊断共享 Runtime 的 `one-memory` CLI。
- `apps/web`:React / TypeScript 管理界面。
- `apps/desktop`:Electron 桌面壳与图形安装入口。
@@ -117,7 +117,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
## Hy Memory
SDK 依赖固定为公开 PyPI 开发预览 `hy-memory==2.0.5.dev3`。SDK 配置、权限边界及 search
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.5`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+3 -3
View File
@@ -3,9 +3,9 @@
"dataLicense": "CC0-1.0",
"SPDXID": "SPDXRef-DOCUMENT",
"name": "one-memory",
"documentNamespace": "https://one-memory.local/spdx/2026-10-22T08:09:31Z",
"documentNamespace": "https://one-memory.local/spdx/2026-10-22T13:18:16Z",
"creationInfo": {
"created": "2026-10-22T08:09:31Z",
"created": "2026-10-22T13:18:16Z",
"creators": [
"Tool: one-memory-generate-sbom"
]
@@ -2352,7 +2352,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.5.dev3",
"versionInfo": "2.0.5",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+2 -2
View File
@@ -2,14 +2,14 @@
MemHub itself is licensed under Apache-2.0 (see `LICENSE`).
It bundles the runtime dependencies listed below.
Generated from the installed dependency tree on 2026-10-22T08:09:31Z
Generated from the installed dependency tree on 2026-10-22T13:18:16Z
by `python3 scripts/generate_sbom.py`. Do not edit by hand.
Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.5.dev3
## hy-memory 2.0.5
License: Apache-2.0
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/cli",
"version": "1.0.4",
"version": "1.0.5",
"description": "MemHub bootstrap CLI for installing and managing the local-first shared memory runtime",
"license": "Apache-2.0",
"repository": {
+42 -1
View File
@@ -1,6 +1,7 @@
import { mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { inflateRawSync } from "node:zlib";
import { DiagnosticLogger } from "@one-memory/runtime-control";
import { afterEach, expect, it, vi } from "vitest";
@@ -29,6 +30,27 @@ async function fixture() {
return dataDir;
}
function readZipEntry(archive: Buffer, expected: string): string {
let offset = 0;
while (archive.readUInt32LE(offset) === 0x04034b50) {
const method = archive.readUInt16LE(offset + 8);
const compressedSize = archive.readUInt32LE(offset + 18);
const nameLength = archive.readUInt16LE(offset + 26);
const extraLength = archive.readUInt16LE(offset + 28);
const nameStart = offset + 30;
const dataStart = nameStart + nameLength + extraLength;
const name = archive.subarray(nameStart, nameStart + nameLength).toString();
const compressed = archive.subarray(dataStart, dataStart + compressedSize);
if (name === expected) {
if (method === 0) return compressed.toString();
if (method === 8) return inflateRawSync(compressed).toString();
throw new Error("unsupported zip compression");
}
offset = dataStart + compressedSize;
}
throw new Error(`missing ${expected}`);
}
it("prints matching diagnostic events as NDJSON without starting Runtime", async () => {
const dataDir = await fixture();
let stdout = "";
@@ -74,6 +96,25 @@ it("exports one support zip to the requested downloads directory", async () => {
});
expect(receipt.path).toMatch(/MemHub-日志-3800-\d{8}-\d{4}\.zip$/);
expect(await readdir(downloads)).toEqual([path.basename(receipt.path)]);
const archive = await readFile(receipt.path);
expect(JSON.parse(readZipEntry(archive, "summary.json")).commit).toBeNull();
});
it("fails closed for an explicitly supplied invalid commit", async () => {
const dataDir = await fixture();
const downloads = await mkdtemp(path.join(os.tmpdir(), "om-cli-downloads-"));
roots.push(downloads);
vi.spyOn(process.stdout, "write").mockImplementation(() => true);
await expect(
run(["logs", "export", "--json"], {
ONE_MEMORY_DATA_DIR: dataDir,
ONE_MEMORY_EXPORT_DIR: downloads,
ONE_MEMORY_BUILD_NUMBER: "3800",
ONE_MEMORY_COMMIT: "unknown",
}),
).rejects.toThrow("diagnostic_commit_invalid");
expect(await readdir(downloads)).toEqual([]);
});
it("never emits a secret from a malformed manually written event", async () => {
+11 -5
View File
@@ -1048,15 +1048,21 @@ async function runCommand(
entry: "cli",
}));
const target = path.join(downloads, diagnosticZipFilename(build));
const environmentCommit = environment.ONE_MEMORY_COMMIT?.trim();
const embeddedCommit =
typeof __ONE_MEMORY_COMMIT__ === "string"
? __ONE_MEMORY_COMMIT__.trim()
: "";
const commit =
environmentCommit ||
(embeddedCommit && embeddedCommit !== "unknown"
? embeddedCommit
: undefined);
const receipt = await exportDiagnosticZip({
dataDir: config.dataDir,
target,
productVersion: build,
commit:
environment.ONE_MEMORY_COMMIT?.trim() ||
(typeof __ONE_MEMORY_COMMIT__ === "string"
? __ONE_MEMORY_COMMIT__
: "unknown"),
...(commit === undefined ? {} : { commit }),
coreSha256: await sha256File(config.coreCommand).catch(() => undefined),
home: os.homedir(),
crashReportsDir: path.join(
+15 -18
View File
@@ -1,15 +1,17 @@
import type {
GovernedMemory,
HomeSummary,
ImportIssue,
ImportJob,
LiveSyncHealth,
MemoryDetail,
MemoryEvidenceExcerpt,
MemoryLibraryPage,
MemoryProcessingSummary,
ModelProfile,
SourceResponse,
import {
CANDIDATE_KINDS,
type CandidateKind,
type GovernedMemory,
type HomeSummary,
type ImportIssue,
type ImportJob,
type LiveSyncHealth,
type MemoryDetail,
type MemoryEvidenceExcerpt,
type MemoryLibraryPage,
type MemoryProcessingSummary,
type ModelProfile,
type SourceResponse,
} from "@one-memory/contracts";
/** The only memory states a response may carry, and the only ones a filter may ask for. */
export const MEMORY_STATUSES = ["confirmed", "deleted", "superseded"] as const;
@@ -74,11 +76,6 @@ const SYNC_COMPLETENESS = [
"additions_only",
"one_time_snapshot",
] as const;
const CANDIDATE_KINDS = [
"default",
"environment_override",
"user_selected",
] as const;
const LOCATION_STATES = [
"found",
"not_found",
@@ -100,7 +97,7 @@ export interface ValidatedMemoryTask {
export interface SourceConnectionCandidate {
source_id: string;
display_name: string;
candidate_kind: (typeof CANDIDATE_KINDS)[number];
candidate_kind: CandidateKind;
candidate_root_id: string;
candidate_revision?: number;
requires_override_confirmation: boolean;
+31 -1
View File
@@ -1,4 +1,4 @@
import { mkdtemp, rm } from "node:fs/promises";
import { mkdtemp, rm, symlink } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { DiagnosticLogger } from "@one-memory/runtime-control";
@@ -41,11 +41,15 @@ it("redacts unsafe details and absolute stack frames from the read surface", ()
session_id: "11111111-1111-4111-8111-111111111111",
details: {
count: 1,
reason: "/Users/me/private-query.txt",
status: "sk-live-secret-token",
source_id: "credential.private",
prompt: "secret-token-123",
authorization: "Bearer secret-token-123",
} as unknown as Record<string, string | number | boolean | null>,
safe_stack: [
"packages/runtime-control/src/example.ts:42:7",
"packages/runtime/sk-live-secret-token.ts:1:1",
"/Users/me/private.py:1",
"C:\\Users\\me\\private.py:1",
],
@@ -56,6 +60,8 @@ it("redacts unsafe details and absolute stack frames from the read surface", ()
"packages/runtime-control/src/example.ts:42:7",
]);
expect(JSON.stringify(redacted)).not.toContain("secret-token-123");
expect(JSON.stringify(redacted)).not.toContain("sk-live-secret-token");
expect(JSON.stringify(redacted)).not.toContain("credential.private");
expect(JSON.stringify(redacted)).not.toContain("/Users/me");
expect(JSON.stringify(redacted)).not.toContain("C:\\\\Users");
});
@@ -132,3 +138,27 @@ it("follows a newly appended event without stopping Runtime", async () => {
});
expect(seen).toEqual(["service.ready", "core.failed"]);
});
it("fails follow mode when the diagnostic directory is unsafe", async () => {
const dataDir = await mkdtemp(
path.join(os.tmpdir(), "om-cli-follow-unsafe-"),
);
const external = await mkdtemp(
path.join(os.tmpdir(), "om-cli-follow-external-"),
);
roots.push(dataDir, external);
await symlink(external, path.join(dataDir, "logs"));
const query = parseLogQuery(["--follow"]);
expect(query).not.toBeNull();
if (!query) throw new Error("query_missing");
await expect(
followLogs({
dataDir,
query,
signal: new AbortController().signal,
maxEvents: 1,
emit: () => undefined,
wait: async () => undefined,
}),
).rejects.toThrow("log_directory_unreadable");
});
+6 -102
View File
@@ -3,6 +3,7 @@ import path from "node:path";
import type { DiagnosticEventV1 } from "@one-memory/runtime-control";
import {
diagnosticLogFiles,
projectDiagnosticEventForSafeOutput,
readDiagnosticEvents,
} from "@one-memory/runtime-control";
@@ -24,54 +25,6 @@ export interface LogQuery {
const ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/;
const NAME = /^[a-z][a-z0-9]*(?:[._-][a-z0-9]+){0,15}$/;
const SAFE_DETAIL_KEYS = new Set([
"action",
"attempt",
"count",
"method",
"mode",
"phase",
"platform",
"reason",
"result_count",
"source_id",
"spawned",
"status",
"version",
"parent_session_id",
"invalid_lines",
"truncated",
"runtime_state",
"core_state",
"broker_state",
"instance_id",
"open_file_count",
"memory_bytes",
"thread_count",
"database_error_count",
"file_count",
"sqlite_file_count",
"byte_file_count",
"total_bytes",
"copied_bytes",
"bytes_per_second",
"stored_count",
"revised_count",
"skipped_count",
"error_count",
"pending_count",
"prepared_count",
"queue_depth",
"in_flight_count",
"call_count",
"created_count",
"updated_count",
"superseded_count",
"deleted_count",
"operation_kind",
"file_kind",
]);
export function parseLogQuery(arguments_: readonly string[]): LogQuery | null {
if (arguments_.filter((item) => item === "--json").length > 1) return null;
const values: Partial<LogQuery> = { errorsOnly: false, follow: false };
@@ -237,63 +190,13 @@ export function renderLogText(
return `${lines.join("\n")}\n`;
}
/** Defence in depth for the CLI read surface. `DiagnosticLogger` already
* allow-lists keys, but a future writer, a manually edited log, or a malformed
* line must not turn `memhub logs` into an exfiltration path. */
/** Defence in depth for the CLI read surface. Local writers accept arbitrary
* bounded scalar keys, so every outward read surface must reuse the same
* field-and-value projection as the support-bundle exporter. */
export function redactDiagnosticEvent(
event: DiagnosticEventV1,
): DiagnosticEventV1 {
const redacted: DiagnosticEventV1 = {
schema_version: 1,
timestamp: event.timestamp,
level: event.level,
service: event.service,
component: event.component,
event: event.event,
outcome: event.outcome,
session_id: event.session_id,
...(event.trace_id === undefined ? {} : { trace_id: event.trace_id }),
...(event.incident_id === undefined
? {}
: { incident_id: event.incident_id }),
...(event.instance_id === undefined
? {}
: { instance_id: event.instance_id }),
...(event.operation_id === undefined
? {}
: { operation_id: event.operation_id }),
...(event.request_id === undefined ? {} : { request_id: event.request_id }),
...(event.job_id === undefined ? {} : { job_id: event.job_id }),
...(event.task_id === undefined ? {} : { task_id: event.task_id }),
...(event.duration_ms === undefined
? {}
: { duration_ms: event.duration_ms }),
...(event.state_before === undefined
? {}
: { state_before: event.state_before }),
...(event.state_after === undefined
? {}
: { state_after: event.state_after }),
...(event.error_code === undefined ? {} : { error_code: event.error_code }),
...(event.error_type === undefined ? {} : { error_type: event.error_type }),
...(event.safe_stack === undefined
? {}
: {
safe_stack: event.safe_stack.filter(
(frame) =>
!frame.startsWith("/") &&
!/^[A-Za-z]:\\/.test(frame) &&
!frame.includes(".."),
),
}),
};
const details = Object.fromEntries(
Object.entries(event.details ?? {}).filter(([key]) =>
SAFE_DETAIL_KEYS.has(key),
),
);
if (Object.keys(details).length > 0) redacted.details = details;
return redacted;
return projectDiagnosticEventForSafeOutput(event);
}
export async function followLogs(options: {
@@ -315,6 +218,7 @@ export async function followLogs(options: {
...options.query,
tail: 5_000,
});
if (report.failure) throw new Error(report.failure);
for (const event of report.events) {
const identity = JSON.stringify(event);
if (seen.has(identity)) continue;
+2 -2
View File
@@ -4,7 +4,7 @@ build-backend = "uv_build"
[project]
name = "one-memory"
version = "1.0.4"
version = "1.0.5"
description = "MemHub local-first memory core"
readme = "README.md"
requires-python = ">=3.12"
@@ -12,7 +12,7 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.5.dev3",
"hy-memory==2.0.5",
# Retrieval's derived vector index directly consumes these locked versions.
"chromadb==1.5.9",
"numpy==2.5.2",
+30 -41
View File
@@ -998,9 +998,7 @@ def create_app(
app.state.recall_target = GovernedRecallTarget(
app.state.memory_governance_store,
filter_memories=app.state.source_read_access.memories,
semantic_search=lambda query: _sdk_memory_runtime(
app
).semantic_recall(query),
semantic_search=lambda query: _sdk_memory_runtime(app).semantic_recall(query),
# Read per request rather than cached: an import that lands
# while the app is open should make its workspace reachable
# without a restart. Affordable only because the read is served
@@ -1102,9 +1100,7 @@ def create_app(
authorize_generation=lambda profile, question, grounding: (
_authorize_answer_generation(app, profile, question, grounding)
),
semantic_search=lambda query: _sdk_memory_runtime(app).semantic_recall(
query
),
semantic_search=lambda query: _sdk_memory_runtime(app).semantic_recall(query),
resolve_workspace=lambda workspace_path: resolve_workspace_scopes(
workspace_path,
evidence_store.known_workspaces(user_id="local"),
@@ -1128,21 +1124,21 @@ def create_app(
profiles=app.state.model_profile_store,
secrets=app.state.secret_store,
data_dir=runtime_settings.data_dir,
transport_factory=lambda profile: lambda: AuditedModelTransport(
app.state.model_request_audit_store,
profile_id=profile.profile_id,
profile_revision=profile.revision,
provider_kind=str(profile.provider_kind),
model=profile.model,
inner=(
BudgetedModelTransport(model_request_budget)
if model_request_budget is not None
else None
),
transport_factory=lambda profile: (
lambda: AuditedModelTransport(
app.state.model_request_audit_store,
profile_id=profile.profile_id,
profile_revision=profile.revision,
provider_kind=str(profile.provider_kind),
model=profile.model,
inner=(
BudgetedModelTransport(model_request_budget)
if model_request_budget is not None
else None
),
)
),
sedimentation_enabled=lambda: _config_store(
app
).sedimentation_enabled(),
sedimentation_enabled=lambda: _config_store(app).sedimentation_enabled(),
)
app.state.sdk_memory_runtime = SdkMemoryRuntime(
coordinator=app.state.sdk_write_coordinator,
@@ -1151,6 +1147,8 @@ def create_app(
extraction_settings=app.state.extraction_settings_store,
data_dir=runtime_settings.data_dir,
)
if app.state.memory_authority is MemoryAuthority.SDK:
app.state.sdk_memory_runtime.start_raw_retention()
# C2 owns the single production budget (handle 5s + reserve 250ms,
# total 5.25s, lease 10s, renew 1s, claim 250ms). The composition
# root only consumes it: no local formula, and the 250ms claim
@@ -1165,10 +1163,8 @@ def create_app(
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
)
for job in unknown:
receipt = (
await app.state.sdk_write_coordinator.inquire_write_outcome(
job.request_id
)
receipt = await app.state.sdk_write_coordinator.inquire_write_outcome(
job.request_id
)
if receipt is None:
continue
@@ -1608,9 +1604,7 @@ def create_app(
)
if getattr(
app.state, "memory_authority", MemoryAuthority.LEGACY
) is MemoryAuthority.SDK and request.url.path.startswith(
legacy_paths
):
) is MemoryAuthority.SDK and request.url.path.startswith(legacy_paths):
return JSONResponse(
status_code=409,
content={"detail": "sdk_memory_authority_required"},
@@ -1732,7 +1726,7 @@ def create_app(
trace_id=trace_id,
instance_id=runtime_settings.instance_id,
duration_ms=elapsed_ms(started),
details={**diagnostic_details, "status": str(response.status_code)},
details={**diagnostic_details, "status_code": response.status_code},
)
return response
except Exception as error:
@@ -3240,13 +3234,14 @@ def create_app(
try:
client = await _sdk_client_handle(app).client()
value = await asyncio.to_thread(
client.list_raw,
client.list_memories,
user_id="local",
source=source,
layers=["l1_raw"],
include_sources=[source] if source else None,
offset=offset,
limit=limit,
)
page = HySdkRawPage.from_sdk(value, limit=limit, offset=offset)
page = HySdkRawPage.from_memory_list(value, limit=limit, offset=offset)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_raw_list_failed") from None
return SdkRawPage(items=page.raw, total=page.total, offset=offset, limit=limit)
@@ -3780,9 +3775,7 @@ def create_app(
all_memories, projected, nodes, edges = await asyncio.to_thread(read_governed_part)
try:
hy_graph = (
None
if revoked
else await _sdk_memory_runtime(app).graph_snapshot(limit=limit)
None if revoked else await _sdk_memory_runtime(app).graph_snapshot(limit=limit)
)
except HySdkError:
hy_graph = None
@@ -3891,9 +3884,7 @@ def create_app(
active_queue, in_flight_count = supervisor.active_depths()
planned, current = supervisor.manual_batch_progress()
sdk_counts = (
supervisor.status_counts()
if app.state.memory_authority is MemoryAuthority.SDK
else {}
supervisor.status_counts() if app.state.memory_authority is MemoryAuthority.SDK else {}
)
# `running` counts rows the store still marks running, which includes
# work a worker has already given up on. Reporting that as "what is
@@ -3914,9 +3905,7 @@ def create_app(
),
recent_completions=depths.recent_completions,
running=sdk_counts.get("running", running_rows),
stranded_running=(
0 if sdk_counts else max(0, running_rows - in_flight_count)
),
stranded_running=(0 if sdk_counts else max(0, running_rows - in_flight_count)),
unexecuted_policy_failures=store.unexecuted_policy_failures(),
waiting_for_model=sdk_counts.get(
"waiting_for_model",
@@ -5283,7 +5272,7 @@ def _report_visibility_failure(phase: str, error: BaseException) -> None:
event="memory_visibility.background_failed",
outcome="failed",
error_type=type(error).__name__,
details={"phase": phase, "reason": str(error)[:200]},
details={"phase": phase},
)
+160 -21
View File
@@ -31,13 +31,60 @@ def _end_by(seconds: str | None) -> None:
if __name__ == "__main__" and sys.argv[1:2] == ["capture"]:
_end_by(os.environ.get("ONE_MEMORY_CAPTURE_DEADLINE_S"))
import errno # noqa: E402
import json # noqa: E402
from collections.abc import Sequence # noqa: E402
import re # noqa: E402
from collections.abc import Iterator, Sequence # noqa: E402
from typing import TextIO # noqa: E402
from one_memory.config import load_settings # noqa: E402
from one_memory.runtime_client import RuntimeStatus, inspect_runtime # noqa: E402
# A single `sdk_migration_backfill_failed` told the person nothing: running out
# of disk, a model that was never configured, and a Coding/Working-Fact gap are
# three different next steps, and the startup dialog can only act on the name it
# is given. These are the names that reach it.
_SAFE_MIGRATION_CODE = re.compile(r"^[a-z][a-z0-9_]{0,95}$")
_OUT_OF_SPACE = frozenset({errno.ENOSPC, getattr(errno, "EDQUOT", errno.ENOSPC)})
def _exception_chain(error: BaseException) -> Iterator[BaseException]:
seen: set[int] = set()
current: BaseException | None = error
while current is not None and id(current) not in seen:
seen.add(id(current))
yield current
current = current.__cause__ or current.__context__
def migration_failure_code(
error: BaseException, *, fallback: str = "sdk_migration_backfill_failed"
) -> str:
"""Name a migration failure; never guess a code from free text.
Errs toward the caller's generic code: an exception this does not recognize
keeps its message out of the document Desktop reads.
"""
chain = list(_exception_chain(error))
for item in chain:
if isinstance(item, OSError) and item.errno in _OUT_OF_SPACE:
return "sdk_migration_disk_space_insufficient"
for item in chain:
code = getattr(item, "code", "")
if not isinstance(code, str) or not _SAFE_MIGRATION_CODE.match(code):
continue
if code.startswith(("sdk_data_plane_", "sdk_migration_")):
return code
if code.startswith("hy_sdk_productivity_"):
# The Coding inventory could not be read or classified; that is a
# data-shape problem the person cannot fix by retrying.
return "sdk_migration_coding_inventory_failed"
if code == "hy_sdk_memory_import_failed" and any(
"working_fact" in str(part) for part in chain
):
return "sdk_migration_working_fact_conflict"
return fallback
def main() -> None:
if sys.argv[1:] == ["--keychain-read"]:
@@ -345,9 +392,16 @@ def migration_bundle(options: Sequence[str], stream: TextIO) -> int:
document = {
"schema_version": 1,
"state": "failed",
"code": error.code,
"code": migration_failure_code(error, fallback=error.code),
}
_write_restore(stream, as_json, document, f"迁移备份失败:{error.code}")
_write_restore(stream, as_json, document, f"迁移备份失败:{document['code']}")
return 1
except OSError as error:
# A full disk while copying is the most common first-launch failure and
# the only one the person fixes by deleting something.
code = migration_failure_code(error, fallback="sdk_migration_bundle_copy_failed")
document = {"schema_version": 1, "state": "failed", "code": code}
_write_restore(stream, as_json, document, f"迁移备份失败:{code}")
return 1
document = {
"schema_version": 1,
@@ -361,6 +415,21 @@ def migration_bundle(options: Sequence[str], stream: TextIO) -> int:
return 0
def cutover_failure_code(error: BaseException) -> str:
"""The cutover's own codes, minus the counts it appends to one of them.
`sdk_data_plane_cutover_work_pending` carries a JSON body of table names and
counts. That belongs in the diagnostic event, not in the single string the
startup dialog reads, so it collapses to one name here.
"""
code = getattr(error, "code", "")
if isinstance(code, str) and code.startswith("sdk_data_plane_cutover_work_pending"):
return "sdk_migration_work_pending"
if isinstance(code, str) and _SAFE_MIGRATION_CODE.match(code):
return code
return "sdk_migration_cutover_failed"
def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
"""Backfill legacy memory/raw into SDK while keeping legacy authority."""
migration_id: str | None = None
@@ -431,7 +500,12 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
MigrationBundleRejected,
inspect_sdk_migration_bundle,
)
from one_memory.sdk_migration_runner import BackfillResult, run_sdk_data_plane_backfill
from one_memory.sdk_migration_runner import (
BackfillResult,
prepare_sdk_migration_staging,
run_sdk_data_plane_backfill,
sdk_migration_staging,
)
from one_memory.secrets import KeyringSecretStore, SecretStoreUnavailableError
database = settings.data_dir / "one-memory.sqlite3"
@@ -456,14 +530,15 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
except SecretStoreUnavailableError:
code = "secure_storage_unavailable"
except Exception as error:
message = str(error)
code = (
str(error)
if str(error)
message
if message
in {
"model_profile_not_active",
"model_secret_unavailable",
}
else "sdk_migration_preflight_failed"
else migration_failure_code(error, fallback="sdk_migration_preflight_failed")
)
else:
namespace_fingerprint = hashlib.sha256(HY_MEMORY_NAMESPACE.encode()).hexdigest()
@@ -482,13 +557,27 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
separators=(",", ":"),
)
backend_fingerprint = hashlib.sha256(backend_material.encode()).hexdigest()
staging = sdk_migration_staging(
data_dir=settings.data_dir,
migration_id=migration_id,
namespace=HY_MEMORY_NAMESPACE,
)
try:
staging_data_dir = prepare_sdk_migration_staging(
paths=staging,
database_path=database,
)
except Exception as error:
code = migration_failure_code(error, fallback="sdk_migration_staging_failed")
staging_data_dir = None
async def run_backfill() -> BackfillResult:
assert staging_data_dir is not None
lease = await asyncio.to_thread(
acquire_client,
profile,
api_key=secret,
data_dir=settings.data_dir,
data_dir=staging_data_dir,
)
try:
return await run_sdk_data_plane_backfill(
@@ -506,10 +595,14 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
await asyncio.to_thread(lease.close)
try:
result = asyncio.run(run_backfill())
except Exception:
code = "sdk_migration_backfill_failed"
result = asyncio.run(run_backfill()) if staging_data_dir is not None else None
except Exception as error:
code = migration_failure_code(error)
else:
if result is None:
document = {"schema_version": 1, "state": "failed", "code": code}
_write_restore(stream, as_json, document, f"SDK backfill 失败:{code}")
return 1
document = {
"schema_version": 1,
"state": result.state,
@@ -585,21 +678,40 @@ def sdk_cutover(options: Sequence[str], stream: TextIO) -> int:
_write_restore(stream, as_json, document, "预览通过;加 --yes 后提交 SDK authority。")
return 4
from one_memory.hy_runtime_config import HY_MEMORY_VERSION
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE, HY_MEMORY_VERSION
from one_memory.sdk_data_plane_migration import DataPlaneMigrationError
from one_memory.sdk_migration_bundle import MigrationBundleRejected
from one_memory.sdk_migration_runner import commit_reconciled_cutover
from one_memory.sdk_migration_runner import (
activate_sdk_migration_staging,
commit_reconciled_cutover,
resume_legacy_cleanup,
sdk_migration_staging,
)
try:
marker = commit_reconciled_cutover(
database_path = settings.data_dir / "one-memory.sqlite3"
staging = sdk_migration_staging(
data_dir=settings.data_dir,
database_path=settings.data_dir / "one-memory.sqlite3",
migration_id=migration_id,
bundle_id=bundle_id,
sdk_version=HY_MEMORY_VERSION,
namespace=HY_MEMORY_NAMESPACE,
)
except (DataPlaneMigrationError, MigrationBundleRejected) as error:
code = getattr(error, "code", str(error))
marker = activate_sdk_migration_staging(
paths=staging,
database_path=database_path,
commit=lambda: commit_reconciled_cutover(
data_dir=settings.data_dir,
database_path=database_path,
migration_id=migration_id,
bundle_id=bundle_id,
sdk_version=HY_MEMORY_VERSION,
),
)
resume_legacy_cleanup(
data_dir=settings.data_dir,
database_path=database_path,
)
except (DataPlaneMigrationError, MigrationBundleRejected, OSError) as error:
code = cutover_failure_code(error)
document = {
"schema_version": 1,
"state": "blocked",
@@ -640,10 +752,12 @@ def sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
import sqlite3
from one_memory.database_access import main_database
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
from one_memory.sdk_data_plane_migration import (
MemoryAuthority,
SdkDataPlaneMigrationStore,
)
from one_memory.sdk_migration_runner import sdk_migration_staging
from one_memory.target_schema import legacy_tables_present
migration_store = SdkDataPlaneMigrationStore(database_path)
@@ -653,10 +767,18 @@ def sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
generation = hashlib.sha256(f"{stat.st_size}:{stat.st_mtime_ns}".encode()).hexdigest()[:16]
if authority is MemoryAuthority.SDK:
legacy_tables = legacy_tables_present(database_path)
staging_cleanup = False
if marker is not None:
staging = sdk_migration_staging(
data_dir=settings.data_dir,
migration_id=marker.migration_id,
namespace=HY_MEMORY_NAMESPACE,
)
staging_cleanup = staging.protective_root.exists() or staging.staging_home.exists()
document = {
"schema_version": 1,
"authority": "sdk",
"state": "cleanup_required" if legacy_tables else "complete",
"state": ("cleanup_required" if legacy_tables or staging_cleanup else "complete"),
"migration_id": marker.migration_id if marker else None,
"generation": generation,
"legacy_tables": list(legacy_tables),
@@ -681,11 +803,28 @@ def sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
else 0
),
}
active_migration = (
database.execute(
"SELECT migration_id,backup_bundle_id "
"FROM sdk_data_plane_migrations "
"WHERE state IN ('running','blocked','ready') "
"ORDER BY created_at DESC LIMIT 1"
).fetchone()
if "sdk_data_plane_migrations" in tables
else None
)
document = {
"schema_version": 1,
"authority": "legacy",
"state": "required" if any(counts.values()) else "empty",
"migration_id": None,
"migration_id": (
str(active_migration["migration_id"]) if active_migration is not None else None
),
"bundle_id": (
str(active_migration["backup_bundle_id"])
if active_migration is not None
else None
),
"counts": counts,
"generation": generation,
}
+217 -122
View File
@@ -20,8 +20,10 @@ from __future__ import annotations
import atexit
import json
import logging
import math
import os
import re
import stat
import sys
import threading
import time
@@ -72,80 +74,30 @@ class _Refused(OSError):
"""The path is not one we are willing to write to, and never will be."""
DiagnosticLevel = Literal["info", "warn", "error"]
DiagnosticLevel = Literal["debug", "info", "warn", "error"]
_IDENTIFIER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$")
_EVENT = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+){0,15}$")
_DETAIL_KEYS = frozenset(
{
"action",
"attempt",
"count",
"method",
"mode",
"phase",
"platform",
"reason",
"result_count",
"source_id",
"spawned",
"status",
"version",
"parent_session_id",
"invalid_lines",
"truncated",
"open_file_count",
"memory_bytes",
"thread_count",
"database_error_count",
"route_path",
"sqlite_errorcode",
"sqlite_errorname",
"file_count",
"sqlite_file_count",
"byte_file_count",
"total_bytes",
"copied_bytes",
"bytes_per_second",
"stored_count",
"revised_count",
"skipped_count",
"error_count",
"pending_count",
"prepared_count",
"queue_depth",
"in_flight_count",
"call_count",
"created_count",
"updated_count",
"superseded_count",
"deleted_count",
"operation_kind",
"index_complete",
"lexical_hits",
"semantic_hits",
"generation",
"timed_out",
"failure_site",
"error_class",
"file_kind",
"file_size",
"file_mtime_ns",
"file_inode",
"cursor_new",
"cursor_old",
"cursor_read_to_end",
"cursor_rewind_branch",
"summed_ms",
"model_ms",
"accounted_ms",
"unaccounted_ms",
"conversation_hash",
# The sedimentation switch: a bounded boolean, the same shape as
# `spawned` and `index_complete`. Left out of this list it did not
# drop a log line, it turned the switch's own request into a 500.
"enabled",
}
_DIAGNOSTIC_LOG_NAME = re.compile(
r"^diagnostic-(runtime|core|host-bridge|desktop|cli)\.jsonl(?:\.\d+)?$"
)
_DETAIL_KEY = re.compile(r"^[A-Za-z][A-Za-z0-9_.-]{0,63}$")
_CODE_FRAME = re.compile(
r"^(?:(?:apps|packages|scripts)/(?:[A-Za-z0-9_-]+/)*[A-Za-z0-9_.-]+:\d+:\d+"
r"|one_memory/(?:[A-Za-z0-9_-]+/)*[A-Za-z0-9_.-]+:\d+:[A-Za-z_][A-Za-z0-9_]*"
r"|[A-Za-z0-9_][A-Za-z0-9_.-]*\.py:\d+:[A-Za-z_][A-Za-z0-9_]*)$"
)
_SHORT_CREDENTIAL_TOKEN = re.compile(
r"(?<![A-Za-z0-9])([A-Za-z0-9]{24,47})(?:[_-])?(?![A-Za-z0-9])"
)
_REFUSED_DETAIL_KEYS = frozenset({"__proto__", "prototype", "constructor"})
_MAX_DETAIL_KEYS = 32
_MAX_SAFE_INTEGER = (1 << 53) - 1
_EXPORT_SECRET_MARKER = re.compile(
r"(?:^|[._:-])(?:authorization|credential|password|prompt|query|secret|token)(?:$|[._:-])",
re.IGNORECASE,
)
_TRACE_STRING_DETAIL_KEYS = frozenset({"phase", "mode", "status", "reason", "failure_site"})
_TRACE_NUMBER_DETAIL_KEYS = frozenset({"count"})
_trace_id: ContextVar[str | None] = ContextVar("one_memory_trace_id", default=None)
_active: DiagnosticLog | None = None
_database_errors: Counter[tuple[str, str]] = Counter()
@@ -264,7 +216,12 @@ class DiagnosticLog:
}
value.update({key: _identifier(item) for key, item in identifiers.items() if item})
if duration_ms is not None:
if not isinstance(duration_ms, int) or duration_ms < 0:
if (
not isinstance(duration_ms, int)
or isinstance(duration_ms, bool)
or duration_ms < 0
or duration_ms > _MAX_SAFE_INTEGER
):
raise ValueError("diagnostic_duration_invalid")
value["duration_ms"] = duration_ms
for field_name, item in {
@@ -276,7 +233,9 @@ class DiagnosticLog:
if item is not None:
value[field_name] = _identifier(item)
if safe_stack:
value["safe_stack"] = [_safe_stack_line(line) for line in safe_stack[:256]]
if len(safe_stack) > 16:
raise ValueError("diagnostic_stack_invalid")
value["safe_stack"] = [_safe_stack_line(line) for line in safe_stack]
if details:
value["details"] = _details(details)
self._submit(_serialize(value))
@@ -289,6 +248,8 @@ class DiagnosticLog:
event: str,
outcome: str,
) -> dict[str, object]:
if level not in ("debug", "info", "warn", "error"):
raise ValueError("diagnostic_level_invalid")
return {
"schema_version": 1,
"timestamp": datetime.now(UTC).isoformat().replace("+00:00", "Z"),
@@ -651,24 +612,17 @@ def _drain_diagnostics_at_exit() -> None:
def diagnostic_event(**values: object) -> None:
active = _active
if active is None:
return
if values.get("level") == "debug" and not logging.getLogger().isEnabledFor(logging.DEBUG):
return
try:
active = _active
if active is None:
return
if values.get("level") == "debug" and not logging.getLogger().isEnabledFor(logging.DEBUG):
return
active.write(**values) # type: ignore[arg-type]
except OSError:
# Diagnostics must not become a new product outage when a disk is full
# or the log directory is temporarily unavailable.
return
except ValueError:
# Same promise, same reason, for a rejected payload instead of a failed
# write. An unlisted `details` key once turned every external recall
# into a 500 and then killed the process, and the sedimentation switch
# into a permanent failure. The payload itself is still wrong and is
# still caught in CI by tests/test_diagnostic_payload_keys.py; what
# must not happen is the caller paying for it.
except Exception:
# Diagnostics are best-effort: validation, serialization, logging, and
# filesystem failures must never become a product outage. BaseException
# deliberately remains visible so process-control signals still work.
return
@@ -691,46 +645,95 @@ def read_trace_events(
if (request_id is None) == (trace_id is None):
raise ValueError("exactly_one_trace_identity_required")
key, expected = ("request_id", request_id) if request_id is not None else ("trace_id", trace_id)
if expected is None or not _IDENTIFIER.fullmatch(expected):
if expected is None or not _export_identifier(expected):
raise ValueError("invalid_trace_identity")
selected: list[dict[str, object]] = []
allowed_details = {"phase", "mode", "status", "reason", "failure_site", "count"}
files = sorted(
(data_dir / "logs").glob("diagnostic-*.jsonl*"),
key=lambda path: path.stat().st_mtime,
)
for path in files:
if path.is_symlink() or not path.is_file():
directory = data_dir / "logs"
try:
directory_info = directory.lstat()
except FileNotFoundError:
return ()
if stat.S_ISLNK(directory_info.st_mode) or not stat.S_ISDIR(directory_info.st_mode):
raise ValueError("diagnostic_directory_unsafe")
files: list[tuple[Path, float]] = []
for path in directory.iterdir():
if not _DIAGNOSTIC_LOG_NAME.fullmatch(path.name):
continue
for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
try:
info = path.lstat()
except OSError:
continue
if stat.S_ISREG(info.st_mode) and not stat.S_ISLNK(info.st_mode):
files.append((path, info.st_mtime))
for path, _mtime in sorted(files, key=lambda item: item[1]):
for line in _read_regular_file_no_follow(path).splitlines():
try:
event = json.loads(line)
except (TypeError, json.JSONDecodeError):
continue
if not isinstance(event, dict) or event.get(key) != expected:
continue
details = event.get("details")
projected = {
name: event[name]
for name in (
"timestamp",
"component",
"event",
"outcome",
"duration_ms",
"error_code",
"error_type",
)
if name in event
timestamp = event.get("timestamp")
if not isinstance(timestamp, str):
continue
try:
datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
except ValueError:
continue
component = event.get("component")
event_name = event.get("event")
outcome = event.get("outcome")
projected: dict[str, object] = {
"timestamp": timestamp,
"component": (
component
if isinstance(component, str) and _export_identifier(component)
else "diagnostic.redacted"
),
"event": (
event_name
if isinstance(event_name, str) and _export_identifier(event_name)
else "diagnostic.redacted"
),
"outcome": (
outcome
if isinstance(outcome, str) and _export_identifier(outcome)
else "redacted"
),
}
duration_ms = event.get("duration_ms")
if (
isinstance(duration_ms, int)
and not isinstance(duration_ms, bool)
and 0 <= duration_ms <= _MAX_SAFE_INTEGER
):
projected["duration_ms"] = duration_ms
for name in ("error_code", "error_type"):
value = event.get(name)
if isinstance(value, str) and _export_identifier(value):
projected[name] = value
details = event.get("details")
if isinstance(details, dict):
projected["details"] = {
name: value for name, value in details.items() if name in allowed_details
}
safe_details = _trace_details(details)
if safe_details:
projected["details"] = safe_details
selected.append(projected)
return tuple(selected[-max(1, min(limit, 500)) :])
def _read_regular_file_no_follow(path: Path) -> str:
descriptor = os.open(path, os.O_RDONLY | _O_NOFOLLOW)
try:
if not stat.S_ISREG(os.fstat(descriptor).st_mode):
raise ValueError("diagnostic_file_unsafe")
with os.fdopen(descriptor, encoding="utf-8", errors="replace") as handle:
descriptor = -1
return handle.read()
finally:
if descriptor >= 0:
os.close(descriptor)
def current_trace_id() -> str | None:
return _trace_id.get()
@@ -837,8 +840,77 @@ def diagnostic_exception(error: BaseException) -> dict[str, object]:
}
def _contains_credential_material(value: str) -> bool:
return bool(
re.search(
r"(?:^|[^A-Za-z0-9])(?:sk-|sk_|omhg_|ghp_|github_pat_)[A-Za-z0-9._-]+",
value,
re.IGNORECASE,
)
or re.search(r"(?:^|[^A-Za-z0-9])xox[baprs]-[A-Za-z0-9-]+", value, re.IGNORECASE)
or re.search(r"(?:^|[^A-Za-z0-9])(?:AKIA|ASIA)[A-Z0-9]{16}(?![A-Za-z0-9])", value)
or re.search(r"(?:^|[^A-Za-z0-9])AKID[A-Za-z0-9]{12,}", value, re.IGNORECASE)
or re.search(r"(?:^|[^A-Za-z0-9])AIza[A-Za-z0-9_-]{20,}", value)
or re.search(r"(?:^|[^A-Za-z0-9])GOCSPX-[A-Za-z0-9_-]+", value)
or any(
_is_conservative_short_credential(match.group(1))
for match in _SHORT_CREDENTIAL_TOKEN.finditer(value)
)
or re.search(
r"\b[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b",
value,
)
or any(
len(candidate) >= 48
and any(character.isalpha() for character in candidate)
and any(character.isdigit() for character in candidate)
for candidate in re.findall(r"\b[A-Za-z0-9._-]{48,}\b", value)
)
)
def _is_conservative_short_credential(value: str) -> bool:
return bool(
re.search(r"[A-Z]", value)
and re.search(r"[a-z]", value)
and re.search(r"\d", value)
and not re.fullmatch(r"[A-Fa-f0-9]+", value)
)
def _export_identifier(value: str) -> bool:
return bool(
_IDENTIFIER.fullmatch(value)
and not _EXPORT_SECRET_MARKER.search(value)
and not _contains_credential_material(value)
)
def _trace_details(values: Mapping[object, object]) -> dict[str, str | int | float]:
result: dict[str, str | int | float] = {}
for key, value in values.items():
if (
key in _TRACE_STRING_DETAIL_KEYS
and isinstance(value, str)
and _export_identifier(value)
):
result[str(key)] = value
elif key in _TRACE_NUMBER_DETAIL_KEYS and isinstance(value, (int, float)):
if isinstance(value, bool):
continue
safe_integer = isinstance(value, int) and abs(value) <= _MAX_SAFE_INTEGER
safe_float = (
isinstance(value, float)
and math.isfinite(value)
and (not value.is_integer() or abs(value) <= _MAX_SAFE_INTEGER)
)
if safe_integer or safe_float:
result[str(key)] = value
return result
def _serialize(value: Mapping[str, object]) -> str:
return json.dumps(value, ensure_ascii=True, separators=(",", ":")) + "\n"
return json.dumps(value, ensure_ascii=True, allow_nan=False, separators=(",", ":")) + "\n"
def _uuid(value: str | None) -> str | None:
@@ -864,24 +936,47 @@ def _event_name(value: str) -> str:
def _safe_stack_line(value: str) -> str:
if not value or len(value) > 256 or "\n" in value or "\r" in value:
if (
not value
or len(value) > 256
or any(character in value for character in "\0\r\n")
or _contains_credential_material(value)
):
raise ValueError("diagnostic_stack_invalid")
if Path(value.split(":", 1)[0]).is_absolute() or "file://" in value:
raise ValueError("diagnostic_stack_invalid")
return value
if _CODE_FRAME.fullmatch(value):
return value
if value.startswith("thread|"):
frames = value.removeprefix("thread|").split(" > ")
if frames and all(_CODE_FRAME.fullmatch(frame) for frame in frames):
return value
raise ValueError("diagnostic_stack_invalid")
def _details(
values: Mapping[str, str | int | float | bool | None],
) -> dict[str, str | int | float | bool | None]:
if len(values) > _MAX_DETAIL_KEYS:
raise ValueError("diagnostic_detail_count_invalid")
result: dict[str, str | int | float | bool | None] = {}
for key, value in values.items():
if key not in _DETAIL_KEYS:
if (
not isinstance(key, str)
or not _DETAIL_KEY.fullmatch(key)
or key in _REFUSED_DETAIL_KEYS
):
raise ValueError("diagnostic_detail_key_invalid")
if isinstance(value, str):
if not value or len(value) > 256 or "\n" in value or "\r" in value:
if not value or len(value) > 256 or any(character in value for character in "\0\r\n"):
raise ValueError("diagnostic_detail_value_invalid")
elif value is not None and not isinstance(value, (int, float, bool)):
elif isinstance(value, float):
if not math.isfinite(value) or (value.is_integer() and abs(value) > _MAX_SAFE_INTEGER):
raise ValueError("diagnostic_detail_value_invalid")
elif isinstance(value, bool) or value is None:
pass
elif isinstance(value, int):
if abs(value) > _MAX_SAFE_INTEGER:
raise ValueError("diagnostic_detail_value_invalid")
else:
raise ValueError("diagnostic_detail_value_invalid")
result[key] = value
return result
@@ -3,8 +3,8 @@
Every site that catches `Exception` broadly must call `report_exception`
before (or instead of) continuing, so a failure never looks like success
and a stack is always on disk. Content (conversation text, keys, SQL
parameters) never reaches the log: only exception type, a truncated
message, and file:line:function frames.
parameters, and exception messages) never reaches the diagnostic log: only
exception type, a repeat count, and file:line:function frames.
"""
from __future__ import annotations
@@ -67,7 +67,7 @@ def report_exception(
event=event,
outcome="failed",
**payload,
details={"reason": _safe_message(error), "count": _RECENT[key]},
details={"count": _RECENT[key]},
)
if full:
_logger.error(
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.5.dev3"
HY_MEMORY_VERSION = "2.0.5"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -1,4 +1,10 @@
"""Migration-only access to the HY SDK's public inventory and import APIs."""
"""Privileged offline access to the HY SDK data plane during App migration.
Runtime product code must use public ``HyMemoryClient`` methods. This module
is the deliberately isolated exception: App startup has stopped all writers,
so the one-time legacy migration may access SDK stores directly without
turning migration mechanics into public Client APIs.
"""
from __future__ import annotations
@@ -6,7 +12,7 @@ import asyncio
from collections.abc import Callable, Mapping
from datetime import datetime
from pathlib import Path
from typing import Protocol
from typing import Any, Protocol
from hy_memory import ( # type: ignore[import-untyped]
DataPlaneRestore,
@@ -14,6 +20,13 @@ from hy_memory import ( # type: ignore[import-untyped]
restore_local_data_plane,
snapshot_local_data_plane,
)
from hy_memory.models.memory import ( # type: ignore[import-untyped]
MemoryLayer,
MemoryNode,
MemoryStatus,
SourceType,
)
from hy_memory.utils.source_filter import normalize_source # type: ignore[import-untyped]
from one_memory.hy_sdk_types import (
HySdkMemoryPage,
@@ -25,6 +38,11 @@ from one_memory.hy_sdk_types import (
class MigrationClient(Protocol):
_loop_thread: Any
_vector_store: Any
_embed_service: Any
_graph_store: Any
def list_memories(
self,
*,
@@ -36,32 +54,30 @@ class MigrationClient(Protocol):
order: str = "desc",
) -> Mapping[str, object]: ...
def import_memory_exact(self, content: str, **kwargs: object) -> Mapping[str, object]: ...
def list_raw(
self,
*,
user_id: str,
agent_id: str | None = None,
session_id: str | None = None,
source: str | None = None,
limit: int = 100,
offset: int = 0,
order: str = "desc",
) -> Mapping[str, object]: ...
def import_raw(self, content: str, **kwargs: object) -> Mapping[str, object]: ...
def delete(self, memory_id: str) -> Mapping[str, object]: ...
def get_write_outcome(self, request_id: str) -> Mapping[str, object] | None: ...
def get(
self,
memory_id: str,
*,
kind: str | None = None,
variant: str | None = None,
) -> Mapping[str, object] | None: ...
async def _coding_store_for_exact_get(self, variant: str) -> Any: ...
class HySdkMigration:
"""Typed async adapter over the SDK's exact migration primitives."""
"""Typed offline adapter over one initialized SDK data plane."""
def __init__(self, client: MigrationClient) -> None:
self.client = client
self._loop_thread = client._loop_thread
self._vector_store = client._vector_store
self._embed_service = client._embed_service
self._graph_store = client._graph_store
async def list_memories(
self,
@@ -85,7 +101,139 @@ class HySdkMigration:
)
return _memory_page(raw, limit=limit, offset=offset)
async def import_memory_exact(
async def list_productivity_memories(self) -> tuple[dict[str, object], ...]:
raw = await _sdk_call(
"hy_sdk_productivity_inventory_failed",
self._loop_thread.run,
self._list_productivity_memories(),
)
return tuple(raw)
async def _list_productivity_memories(self) -> list[dict[str, object]]:
records: list[dict[str, object]] = []
for variant in ("working", "coding"):
store = await self.client._coding_store_for_exact_get(variant)
if store is None:
continue
with store._lock:
experiences = store._conn.execute(
"SELECT experience_id,user_id,agent_id,solution "
"FROM coding_memory_meta ORDER BY experience_id"
).fetchall()
facts = store._conn.execute(
"SELECT fact_id,user_id,agent_id,statement,status "
"FROM working_fact ORDER BY fact_id"
).fetchall()
records.extend(
{
"memory_id": str(row["experience_id"]),
"content": str(row["solution"]),
"status": "active",
"user_id": str(row["user_id"]),
"agent_id": str(row["agent_id"]),
"kind": "experience",
"variant": variant,
}
for row in experiences
)
records.extend(
{
"memory_id": str(row["fact_id"]),
"content": str(row["statement"]),
"status": str(row["status"]),
"user_id": str(row["user_id"]),
"agent_id": str(row["agent_id"]),
"kind": "fact",
"variant": variant,
}
for row in facts
)
return records
async def read_productivity_memory(
self,
memory_id: str,
*,
kind: str,
variant: str | None,
) -> tuple[HySdkMemoryRecord, str] | None:
raw = await _sdk_call(
"hy_sdk_productivity_get_failed",
self.client.get,
memory_id,
kind=kind,
variant=variant,
)
if raw is None:
return None
if not isinstance(raw, Mapping) or raw.get("variant") not in {
"working",
"coding",
}:
raise HySdkMigrationError("hy_sdk_productivity_variant_invalid")
return HySdkMemoryRecord.from_sdk(raw), str(raw["variant"])
async def delete_productivity_memory(
self,
memory_id: str,
*,
kind: str,
variant: str | None,
user_id: str,
) -> dict[str, object]:
raw = await _sdk_call(
"hy_sdk_productivity_delete_failed",
self._loop_thread.run,
self._delete_productivity_memory(
memory_id,
kind=kind,
variant=variant,
user_id=user_id,
),
)
return _successful_result(raw, "hy_sdk_productivity_delete_failed")
async def _delete_productivity_memory(
self,
memory_id: str,
*,
kind: str,
variant: str | None,
user_id: str,
) -> dict[str, object]:
if kind not in {"experience", "fact"}:
raise ValueError("productivity kind must be experience or fact")
matches: list[tuple[Any, str]] = []
for candidate in ((variant,) if variant is not None else ("working", "coding")):
store = await self.client._coding_store_for_exact_get(candidate)
if store is None:
continue
existing = (
await store.get_by_id(memory_id, user_id=user_id)
if kind == "experience"
else await store.get_fact(memory_id, user_id=user_id)
)
if existing is not None:
matches.append((store, candidate))
if len(matches) > 1:
raise ValueError("productivity memory id is ambiguous across variants")
if not matches:
return {"success": True, "deleted_count": 0}
store, _actual_variant = matches[0]
if kind == "experience":
deleted = await store.delete(memory_id, user_id=user_id)
elif kind == "fact":
await store._fact_vec.delete(memory_id)
with store._lock:
store._conn.execute(
"DELETE FROM working_fact WHERE fact_id=? AND user_id=?",
(memory_id, user_id),
)
store._conn.commit()
deleted = True
return {"success": True, "deleted_count": int(deleted)}
async def write_legacy_memory(
self,
*,
content: str,
@@ -107,27 +255,119 @@ class HySdkMigration:
) -> dict[str, object]:
raw = await _sdk_call(
"hy_sdk_memory_import_failed",
self.client.import_memory_exact,
content,
memory_id=memory_id,
self._loop_thread.run,
self._write_legacy_memory(
content=content,
memory_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=layer,
status=status,
owner=owner,
tags=tags,
supersedes=supersedes,
superseded_by=superseded_by,
is_latest=is_latest,
memory_at=memory_at,
metadata=metadata,
request_id=request_id,
),
)
return _successful_result(raw, "hy_sdk_memory_import_failed")
async def _write_legacy_memory(
self,
*,
content: str,
memory_id: str,
user_id: str,
agent_id: str,
session_id: str,
source: str,
layer: str,
status: str,
owner: str | None,
tags: tuple[str, ...],
supersedes: tuple[str, ...],
superseded_by: tuple[str, ...],
is_latest: bool,
memory_at: datetime | None,
metadata: Mapping[str, object],
request_id: str,
) -> dict[str, object]:
if not content or not memory_id or not user_id:
raise ValueError("content, memory_id and user_id are required")
normalized_layer = MemoryLayer.from_string(layer)
if normalized_layer is MemoryLayer.L1_RAW:
raise ValueError("raw content must use the raw migration lane")
normalized_status = MemoryStatus(status.lower())
normalized_source = normalize_source(source)
existing = await self._vector_store.get_by_id(memory_id)
if existing is not None:
identity = (
existing.content == content,
existing.user_id == user_id,
existing.agent_id == agent_id,
existing.session_id == session_id,
existing.source == normalized_source,
existing.layer == normalized_layer,
existing.status == normalized_status,
existing.owner == owner,
list(existing.tags or []) == list(tags),
list(existing.supersedes or []) == list(supersedes),
list(existing.superseded_by or []) == list(superseded_by),
existing.is_latest == is_latest,
)
if not all(identity):
raise ValueError("memory_id is already bound to different memory content")
await self._project_graph(existing)
return {
"success": True,
"memory_id": memory_id,
"memory_ids": [memory_id],
"request_id": request_id,
"replayed": True,
}
custom = dict(metadata)
custom["migration_request_id"] = request_id
embedding = await self._embed_service.embed(content)
node = MemoryNode(
node_id=memory_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
layer=layer,
status=status,
source=normalized_source,
layer=normalized_layer,
content=content,
source_type=SourceType.EXPLICIT,
status=normalized_status,
owner=owner,
tags=list(tags),
supersedes=list(supersedes),
superseded_by=list(superseded_by),
is_latest=is_latest,
embedding=embedding,
memory_at=memory_at,
metadata=dict(metadata),
request_id=request_id,
custom=custom,
)
return _successful_result(raw, "hy_sdk_memory_import_failed")
stored_id = await self._vector_store.upsert(node)
if stored_id != memory_id:
raise RuntimeError("exact migration changed the requested memory identity")
await self._project_graph(node)
return {
"success": True,
"memory_id": memory_id,
"memory_ids": [memory_id],
"request_id": request_id,
}
async def list_raw(
async def _project_graph(self, node: MemoryNode) -> None:
if self._graph_store is not None:
await self._graph_store.upsert_memory_node(node)
async def read_raw_inventory(
self,
*,
user_id: str,
@@ -140,18 +380,57 @@ class HySdkMigration:
) -> HySdkRawPage:
raw = await _sdk_call(
"hy_sdk_raw_list_failed",
self.client.list_raw,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
limit=limit,
offset=offset,
order=order,
self._loop_thread.run,
self._read_raw_inventory(
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
limit=limit,
offset=offset,
order=order,
),
)
return _raw_page(raw, limit=limit, offset=offset)
async def import_raw(
async def _read_raw_inventory(
self,
*,
user_id: str,
agent_id: str | None,
session_id: str | None,
source: str | None,
limit: int,
offset: int,
order: str,
) -> dict[str, object]:
if not user_id:
raise ValueError("user_id is required")
nodes = await self._vector_store.list_by_user(
user_id=user_id,
agent_id=agent_id,
status_filter=list(MemoryStatus),
layers=[MemoryLayer.L1_RAW],
include_sources=[source] if source else None,
limit=100_000,
only_latest=False,
with_vectors=False,
allow_raw_layer=True,
)
if session_id:
nodes = [node for node in nodes if node.session_id == session_id]
nodes.sort(
key=_node_timestamp,
reverse=order.lower() != "asc",
)
return {
"raw": [_raw_node(node) for node in nodes[offset : offset + limit]],
"total": len(nodes),
"limit": limit,
"offset": offset,
}
async def write_legacy_raw(
self,
*,
content: str,
@@ -166,18 +445,92 @@ class HySdkMigration:
) -> dict[str, object]:
raw = await _sdk_call(
"hy_sdk_raw_import_failed",
self.client.import_raw,
content,
raw_id=raw_id,
self._loop_thread.run,
self._write_legacy_raw(
content=content,
raw_id=raw_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
memory_at=memory_at,
metadata=metadata,
request_id=request_id,
),
)
return _successful_result(raw, "hy_sdk_raw_import_failed")
async def delete_raw(self, raw_id: str) -> dict[str, object]:
raw = await _sdk_call(
"hy_sdk_raw_delete_failed",
self._loop_thread.run,
self._delete_raw(raw_id),
)
return _successful_result(raw, "hy_sdk_raw_delete_failed")
async def _delete_raw(self, raw_id: str) -> dict[str, object]:
existing = await self._vector_store.get_by_id(raw_id)
if existing is None or existing.layer is not MemoryLayer.L1_RAW:
return {"success": True, "deleted_count": 0}
deleted = await self._vector_store.delete(raw_id)
return {"success": True, "deleted_count": int(bool(deleted))}
async def _write_legacy_raw(
self,
*,
content: str,
raw_id: str,
user_id: str,
agent_id: str,
session_id: str,
source: str,
memory_at: datetime | None,
metadata: Mapping[str, object],
request_id: str,
) -> dict[str, object]:
if not content or not raw_id or not user_id:
raise ValueError("content, raw_id and user_id are required")
normalized_source = normalize_source(source)
existing = await self._vector_store.get_by_id(raw_id)
if existing is not None:
identity = (
existing.layer is MemoryLayer.L1_RAW,
existing.content == content,
existing.user_id == user_id,
existing.agent_id == agent_id,
existing.session_id == session_id,
existing.source == normalized_source,
)
if not all(identity):
raise ValueError("raw_id is already bound to different source content")
return {
"success": True,
"raw_id": raw_id,
"request_id": request_id,
"replayed": True,
}
custom = dict(metadata)
custom["migration_request_id"] = request_id
embedding = await self._embed_service.embed(content)
node = MemoryNode(
node_id=raw_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
source=source,
source=normalized_source,
layer=MemoryLayer.L1_RAW,
content=content,
source_type=SourceType.EXPLICIT,
status=MemoryStatus.ACTIVE,
is_latest=True,
embedding=embedding,
memory_at=memory_at,
metadata=dict(metadata),
request_id=request_id,
custom=custom,
)
return _successful_result(raw, "hy_sdk_raw_import_failed")
stored_id = await self._vector_store.upsert(node)
if stored_id != raw_id:
raise RuntimeError("exact migration changed the requested raw identity")
return {"success": True, "raw_id": raw_id, "request_id": request_id}
async def delete(self, memory_id: str) -> dict[str, object]:
raw = await _sdk_call(
@@ -338,6 +691,28 @@ def _raw_page(raw: object, *, limit: int, offset: int) -> HySdkRawPage:
)
def _node_timestamp(node: Any) -> float:
value = getattr(node, "memory_at", None) or getattr(node, "gmt_created", None)
return value.timestamp() if isinstance(value, datetime) else 0.0
def _raw_node(node: Any) -> dict[str, object]:
def timestamp(value: object) -> int | None:
return int(value.timestamp()) if isinstance(value, datetime) else None
return {
"raw_id": node.node_id,
"content": node.content,
"user_id": node.user_id,
"agent_id": node.agent_id,
"session_id": node.session_id,
"source": node.source,
"memory_at": timestamp(node.memory_at),
"gmt_created": timestamp(node.gmt_created),
"custom": node.custom or {},
}
def _raw_record(raw: object) -> HySdkRawRecord:
if not isinstance(raw, Mapping):
raise HySdkMigrationError("hy_sdk_raw_record_shape_invalid")
@@ -22,7 +22,7 @@ class HySdkRuntimeError(HySdkError):
class HySdkMigrationError(HySdkError):
"""A public SDK migration operation failed or returned an invalid shape."""
"""The privileged offline SDK data-plane migration failed."""
class HySdkReadError(HySdkError):
@@ -200,11 +200,21 @@ class HySdkRawPage(BaseModel):
offset: int = Field(ge=0)
@classmethod
def from_sdk(cls, raw: object, *, limit: int, offset: int) -> HySdkRawPage:
def from_memory_list(
cls,
raw: object,
*,
limit: int,
offset: int,
) -> HySdkRawPage:
"""Parse ``list_memories(layers=['l1_raw'])`` into the App raw DTO."""
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
rows = raw.get("raw")
total = raw.get("total")
bucket = raw.get("vdb")
if not isinstance(bucket, Mapping):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
rows = bucket.get("memories")
total = bucket.get("total")
if (
not isinstance(rows, list)
or not isinstance(total, int)
@@ -212,8 +222,13 @@ class HySdkRawPage(BaseModel):
or total < 0
):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
converted = []
for row in rows:
if not isinstance(row, Mapping) or row.get("layer") != "l1_raw":
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
converted.append({**row, "raw_id": row.get("memory_id")})
return cls(
raw=tuple(HySdkRawRecord.from_sdk(row) for row in rows),
raw=tuple(HySdkRawRecord.from_sdk(row) for row in converted),
total=total,
limit=limit,
offset=offset,
@@ -557,12 +557,10 @@ class PollingLiveSyncSupervisor:
)
def _note_internal_error(self, source_id: str, error: Exception) -> None:
# Type alone used to be all this said. It is not enough to act on:
# `OperationalError` covers a locked database, a missing table and a
# corrupt page alike, and without the message and the frames two dozen
# of these in a log could only be guessed at. The stack is the
# redacted, `one_memory`-relative one, so it names the call site
# without printing anyone's paths or query text.
# The ordinary local log keeps the exception message for debugging.
# The structured diagnostic event deliberately keeps only the stable
# error code, source id, type, and relative stack: exception messages
# can contain paths, query text, or credentials and are exportable data.
logging.getLogger(__name__).warning(
"live_sync_internal_error %s: %s", type(error).__name__, error
)
@@ -572,10 +570,7 @@ class PollingLiveSyncSupervisor:
event="live_sync.internal_error",
outcome="failed",
error_code="live_sync_internal_error",
# `reason` is the listed key that can carry the message; see
# `diagnostic_log._DETAIL_KEYS`. Truncated because a detail value
# longer than 256 characters is rejected.
details={"source_id": source_id, "reason": str(error)[:200]},
details={"source_id": source_id},
**diagnostic_exception(error),
)
if isinstance(error, sqlite3.Error):
@@ -867,10 +867,7 @@ class ImportRunner:
outcome="failed",
job_id=job_id,
error_code="import_heartbeat_failed",
# `reason` is the listed key for this; see
# `diagnostic_log._DETAIL_KEYS`. Truncated: a detail value
# over 256 characters is rejected.
details={"reason": str(error)[:200]},
details={"phase": "heartbeat"},
**diagnostic_exception(error),
)
@@ -1,6 +1,7 @@
"""Write visible hook text into Evidence. Tool output stays out."""
import json
import os
import time
from datetime import UTC, datetime
from enum import StrEnum
@@ -74,6 +75,7 @@ _FILE_FALLBACK = {
}
_FALLBACK_EVENTS = frozenset({"Stop", "stop", "SubagentStop", "StopFailure"})
_TEXT_BLOCKS = frozenset({"text"})
SDK_MIGRATION_CAPTURE_FENCE = ".sdk-migration-capture-fence.json"
def database_path() -> Path:
@@ -88,17 +90,42 @@ def capture_stdio(source_id: str, stdin: TextIO, output: TextIO) -> int:
del output
if source_id not in CAPTURE_SOURCES:
return 2
database = database_path()
if sdk_migration_capture_fenced(database.parent):
return 0
try:
payload = json.loads(stdin.read() or "")
except json.JSONDecodeError:
return 0
try:
capture_hook(source_id, payload, database_path())
capture_hook(source_id, payload, database)
except Exception:
return 0
return 0
def sdk_migration_capture_fenced(data_dir: Path) -> bool:
"""Ignore stale fences; only a live Desktop migration pauses hook writes."""
marker = data_dir / SDK_MIGRATION_CAPTURE_FENCE
try:
document = json.loads(marker.read_text(encoding="utf-8"))
pid = int(document["pid"])
if pid < 1:
raise ValueError("invalid migration fence pid")
os.kill(pid, 0)
return True
except FileNotFoundError:
return False
except ProcessLookupError:
marker.unlink(missing_ok=True)
return False
except PermissionError:
return True
except (KeyError, TypeError, ValueError, json.JSONDecodeError, OSError):
marker.unlink(missing_ok=True)
return False
def capture_hook(source_id: str, payload: object, database: Path) -> bool:
if source_id not in CAPTURE_SOURCES or not isinstance(payload, dict):
return False
@@ -14,6 +14,7 @@ stuck in it.
"""
import logging
import re
import sys
import threading
import time
@@ -23,6 +24,7 @@ from types import FrameType
logger = logging.getLogger("one_memory.loop_watchdog")
DEFAULT_STALL_SECONDS = 2.0
_SAFE_FALLBACK_FILE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_.-]*\.py$")
class LoopWatchdog:
@@ -120,22 +122,28 @@ def _safe_stack_for_main_thread() -> tuple[str, ...]:
def _safe_stacks_for_all_threads() -> tuple[str, ...]:
"""Return one content-free, bounded stack summary for every live thread."""
names = {thread.ident: thread.name for thread in threading.enumerate()}
"""Return bounded stack summaries without recording thread names."""
lines: list[str] = []
for ident, frame in sorted(sys._current_frames().items()):
for _ident, frame in sorted(sys._current_frames().items()):
stack: list[str] = []
# Most specific frame first so the 256-character privacy bound keeps
# the likely culprit rather than only the thread bootstrap frames.
# Most specific frame first; append only complete frames so truncation
# can never turn a valid location into free text.
for entry in reversed(traceback.extract_stack(frame)[-8:]):
parts = entry.filename.replace("\\", "/").split("/")
try:
root = parts.index("one_memory")
relative = "/".join(parts[root:])
except ValueError:
relative = parts[-1]
fallback = parts[-1]
relative = fallback if _SAFE_FALLBACK_FILE.fullmatch(fallback) else "unknown.py"
name = entry.name if entry.name.isidentifier() else "call"
stack.append(f"{relative}:{entry.lineno}:{name}")
thread_name = names.get(ident, "unknown").replace("\n", " ").replace("\r", " ")
lines.append(f"thread={thread_name[:48]}|{' > '.join(stack)}"[:256])
candidate = f"{relative}:{entry.lineno}:{name}"
separator = "" if not stack else " > "
if len("thread|") + len(" > ".join(stack)) + len(separator) + len(candidate) > 256:
break
stack.append(candidate)
if stack:
lines.append(f"thread|{' > '.join(stack)}")
if len(lines) == 16:
break
return tuple(lines)
@@ -3,6 +3,8 @@
from __future__ import annotations
import asyncio
import logging
from contextlib import suppress
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
@@ -25,6 +27,18 @@ from one_memory.memory_server.durable_queue import COMPLETED, DurableWriteQueue
from one_memory.sdk_write_processor import SdkWriteCoordinator
from one_memory.semantic_deadline import SemanticMemoryHit
logger = logging.getLogger(__name__)
RAW_RETENTION_DAYS = 3
RAW_RETENTION_CHECK_SECONDS = 6 * 60 * 60
_RAW_STATUSES = (
"active",
"superseded",
"shadow",
"negated",
"conflicted",
"archived",
)
class SdkMemoryRuntime:
"""Product services that remain after legacy task/governance writes retire."""
@@ -43,6 +57,68 @@ class SdkMemoryRuntime:
self.write_queue = write_queue
self.extraction_settings = extraction_settings
self.data_dir = data_dir
self._raw_retention_task: asyncio.Task[None] | None = None
def start_raw_retention(self) -> None:
if self._raw_retention_task is None or self._raw_retention_task.done():
self._raw_retention_task = asyncio.create_task(
self._raw_retention_loop(),
name="one-memory-sdk-raw-retention",
)
async def prune_expired_raw(self, *, now: datetime | None = None) -> int:
cutoff = (now or datetime.now(UTC)) - timedelta(days=RAW_RETENTION_DAYS)
cutoff_timestamp = int(cutoff.timestamp())
client = await self.read_handle.client()
deleted = 0
while True:
page = await asyncio.to_thread(
client.list_memories,
user_id="local",
layers=["l1_raw"],
statuses=list(_RAW_STATUSES),
order="asc",
limit=500,
offset=0,
)
bucket = page.get("vdb") if isinstance(page, dict) else None
rows = bucket.get("memories") if isinstance(bucket, dict) else None
if not isinstance(rows, list) or not rows:
return deleted
expired = [
row
for row in rows
if isinstance(row, dict)
and isinstance(row.get("memory_id"), str)
and int(row.get("memory_at") or row.get("gmt_created") or 0)
< cutoff_timestamp
]
if not expired:
return deleted
removed_page = 0
for row in expired:
result = await asyncio.to_thread(client.delete, row["memory_id"])
if isinstance(result, dict) and result.get("success") is True:
count = int(result.get("deleted_count") or 0)
deleted += count
removed_page += count
if removed_page == 0:
return deleted
async def _raw_retention_loop(self) -> None:
while True:
await asyncio.sleep(RAW_RETENTION_CHECK_SECONDS)
try:
removed = await self.prune_expired_raw()
if removed:
logger.info("sdk_raw_retention_deleted count=%d", removed)
except asyncio.CancelledError:
raise
except Exception as error:
logger.warning(
"sdk_raw_retention_failed error_type=%s",
type(error).__name__,
)
async def semantic_recall(
self,
@@ -279,7 +355,12 @@ class SdkMemoryRuntime:
return None
async def shutdown(self) -> None:
return None
if self._raw_retention_task is None:
return
self._raw_retention_task.cancel()
with suppress(asyncio.CancelledError):
await self._raw_retention_task
self._raw_retention_task = None
def _agent_ids(raw: object) -> tuple[str, ...]:
@@ -7,12 +7,13 @@ import json
import sqlite3
from collections.abc import Awaitable, Mapping
from dataclasses import dataclass
from datetime import datetime
from datetime import UTC, datetime
from pathlib import Path
from typing import Protocol
from one_memory.database_access import main_database
from one_memory.hy_sdk_types import HySdkMemoryPage, HySdkRawPage
from one_memory.hy_sdk_types import HySdkMemoryPage, HySdkMemoryRecord, HySdkRawPage
from one_memory.input_sanitization import sanitize_model_input
from one_memory.sdk_data_plane_migration import SdkDataPlaneMigrationStore
@@ -35,6 +36,8 @@ class LegacyMemoryInventory:
tags: tuple[str, ...]
supersedes: tuple[str, ...]
superseded_by: tuple[str, ...]
productivity_kind: str | None = None
productivity_variant: str | None = None
@dataclass(frozen=True)
@@ -44,6 +47,8 @@ class SdkMemoryInventory:
agent_id: str
content_hash: str
status: str
productivity_kind: str | None = None
productivity_variant: str | None = None
@dataclass(frozen=True)
@@ -92,6 +97,17 @@ _KIND_LAYER = {
"schema": "l6_schema",
"intention": "l7_intention",
}
_PRODUCTIVITY_LAYER = {
"coding_experience": ("experience", "coding"),
"working_experience": ("experience", "working"),
"coding_fact": ("fact", "coding"),
# Historical receipts called every productivity fact "working_fact",
# including facts stored beside a coding-variant experience. Resolve the
# actual variant from the SDK exact lookup instead of trusting this label.
"working_fact": ("fact", None),
}
_RAW_TURN_MAX_CHARACTERS = 60_000
_RAW_USER_MAX_CHARACTERS = 30_000
SDK_LIFECYCLE_STATUSES = (
"active",
@@ -115,7 +131,7 @@ class SdkMigrationClient(Protocol):
order: str = "desc",
) -> Awaitable[HySdkMemoryPage]: ...
def import_memory_exact(
def write_legacy_memory(
self,
*,
content: str,
@@ -133,7 +149,7 @@ class SdkMigrationClient(Protocol):
request_id: str,
) -> Awaitable[dict[str, object]]: ...
def list_raw(
def read_raw_inventory(
self,
*,
user_id: str,
@@ -145,7 +161,7 @@ class SdkMigrationClient(Protocol):
order: str = "desc",
) -> Awaitable[HySdkRawPage]: ...
def import_raw(
def write_legacy_raw(
self,
*,
content: str,
@@ -161,6 +177,29 @@ class SdkMigrationClient(Protocol):
def delete(self, memory_id: str) -> Awaitable[dict[str, object]]: ...
def read_productivity_memory(
self,
memory_id: str,
*,
kind: str,
variant: str | None,
) -> Awaitable[tuple[HySdkMemoryRecord, str] | None]: ...
def list_productivity_memories(
self,
) -> Awaitable[tuple[dict[str, object], ...]]: ...
def delete_productivity_memory(
self,
memory_id: str,
*,
kind: str,
variant: str | None,
user_id: str,
) -> Awaitable[dict[str, object]]: ...
def delete_raw(self, raw_id: str) -> Awaitable[dict[str, object]]: ...
def read_legacy_memory_inventory(database_path: Path) -> tuple[LegacyMemoryInventory, ...]:
"""Read legacy memory intent without constructing a Governance writer."""
@@ -204,6 +243,7 @@ def read_legacy_memory_inventory(database_path: Path) -> tuple[LegacyMemoryInven
tags = tuple(value for value in tags_value if isinstance(value, str))
kind = str(row["kind"])
status = str(row["status"])
productivity = _PRODUCTIVITY_LAYER.get(mapped_layer or "")
records.append(
LegacyMemoryInventory(
legacy_memory_id=legacy_id,
@@ -225,6 +265,8 @@ def read_legacy_memory_inventory(database_path: Path) -> tuple[LegacyMemoryInven
tags=tags,
supersedes=tuple(supersedes.get(legacy_id, ())),
superseded_by=tuple(superseded_by.get(legacy_id, ())),
productivity_kind=productivity[0] if productivity else None,
productivity_variant=productivity[1] if productivity else None,
)
)
return tuple(records)
@@ -239,43 +281,116 @@ def _parse_datetime(value: object) -> datetime | None:
return None
def read_legacy_raw_inventory(database_path: Path) -> tuple[LegacyRawInventory, ...]:
def read_legacy_raw_inventory(
database_path: Path,
*,
cutoff: datetime,
) -> tuple[LegacyRawInventory, ...]:
"""Rebuild recent legacy messages into one user/assistant raw per turn."""
if cutoff.tzinfo is None:
cutoff = cutoff.replace(tzinfo=UTC)
with main_database(database_path).read(row_factory=sqlite3.Row) as connection:
rows = connection.execute(
"""
SELECT evidence_id,user_id,source_id,workspace,project,
conversation_id,message_id,source_ordinal,content_kind,
role,written_by,created_at,imported_at,verbatim,content_hash,revision
FROM evidence ORDER BY COALESCE(ingest_seq,rowid),evidence_id
FROM evidence
WHERE content_kind='raw_conversation'
AND (
(role='user' AND written_by='person')
OR role='assistant'
)
ORDER BY source_id,conversation_id,workspace,
COALESCE(created_at,imported_at),
COALESCE(ingest_seq,rowid),evidence_id
"""
).fetchall()
return tuple(
LegacyRawInventory(
evidence_id=str(row["evidence_id"]),
user_id=str(row["user_id"]),
agent_id="one-memory",
session_id=str(row["conversation_id"]),
source=str(row["source_id"]),
content=str(row["verbatim"]),
content_hash=str(row["content_hash"]),
memory_at=_parse_datetime(row["created_at"] or row["imported_at"]),
metadata={
key: row[key]
for key in (
"workspace",
"project",
"message_id",
"source_ordinal",
"content_kind",
"role",
"written_by",
"revision",
)
if row[key] is not None
},
turns: list[LegacyRawInventory] = []
current_key: tuple[str, str, str, str] | None = None
current: list[sqlite3.Row] = []
def finish() -> None:
if not current or str(current[0]["role"]) != "user":
return
assistants = [row for row in current[1:] if str(row["role"]) == "assistant"]
if not assistants:
return
rows_for_turn = [current[0], *assistants]
completed_at = _parse_datetime(
assistants[-1]["created_at"] or assistants[-1]["imported_at"]
)
for row in rows
)
if completed_at is not None and completed_at.tzinfo is None:
completed_at = completed_at.replace(tzinfo=UTC)
if completed_at is None or completed_at.astimezone(UTC) < cutoff.astimezone(UTC):
return
evidence_ids = [str(row["evidence_id"]) for row in rows_for_turn]
source = str(current[0]["source_id"])
session_id = str(current[0]["conversation_id"])
digest = hashlib.sha256(
"\0".join((source, session_id, *evidence_ids)).encode()
).hexdigest()
raw_user_text = str(rows_for_turn[0]["verbatim"] or "")
user_text = sanitize_model_input(raw_user_text)[:_RAW_USER_MAX_CHARACTERS]
content_parts = [f"[user]: {user_text}"]
truncated = len(raw_user_text) > len(user_text)
for row in assistants:
sanitized = sanitize_model_input(str(row["verbatim"] or ""))
text = sanitized
prefix = "[assistant]: "
remaining = _RAW_TURN_MAX_CHARACTERS - len("\n".join(content_parts)) - 1
if remaining <= len(prefix):
truncated = True
break
available = remaining - len(prefix)
if len(sanitized) > available:
text = text[:available]
truncated = True
content_parts.append(f"{prefix}{text}")
if len(text) < len(sanitized):
break
content = "\n".join(content_parts)
if not content:
return
turns.append(
LegacyRawInventory(
evidence_id=f"turn:{digest}",
user_id=str(current[0]["user_id"]),
agent_id="all-apps",
session_id=session_id,
source=source,
content=content,
content_hash=content_hash(content),
memory_at=_parse_datetime(
current[0]["created_at"] or current[0]["imported_at"]
),
metadata={
"evidence_ids": evidence_ids,
"workspace": current[0]["workspace"],
"project": current[0]["project"],
"input_kind": "legacy_turn",
"truncated": truncated,
},
)
)
for row in rows:
key = (
str(row["source_id"]),
str(row["conversation_id"]),
str(row["user_id"]),
str(row["workspace"] or ""),
)
if key != current_key:
finish()
current_key, current = key, []
if str(row["role"]) == "user":
finish()
current = [row]
elif current:
current.append(row)
finish()
return tuple(turns)
def classify_memory_inventory(
@@ -283,13 +398,39 @@ def classify_memory_inventory(
sdk: tuple[SdkMemoryInventory, ...],
) -> tuple[MemoryMigrationDecision, ...]:
"""Classify every legacy and SDK memory; no item may silently disappear."""
sdk_by_id = {item.memory_id: item for item in sdk}
claimed_sdk_ids: set[str] = set()
sdk_by_id: dict[str, list[SdkMemoryInventory]] = {}
for sdk_item in sdk:
sdk_by_id.setdefault(sdk_item.memory_id, []).append(sdk_item)
claimed_sdk_items: set[tuple[str, str | None, str | None]] = set()
decisions: list[MemoryMigrationDecision] = []
for item in legacy:
current = sdk_by_id.get(item.canonical_memory_id)
candidates = [
candidate
for candidate in sdk_by_id.get(item.canonical_memory_id, ())
if (
candidate.productivity_kind == item.productivity_kind
and (
item.productivity_kind is None
or item.productivity_variant is None
or candidate.productivity_variant == item.productivity_variant
)
)
]
if len(candidates) > 1:
claimed_sdk_items.update(_sdk_inventory_key(candidate) for candidate in candidates)
decisions.append(
MemoryMigrationDecision(
item_key=f"memory:{item.legacy_memory_id}",
action="quarantine",
legacy_memory_id=item.legacy_memory_id,
canonical_memory_id=item.canonical_memory_id,
reason="sdk_memory_id_ambiguous",
)
)
continue
current = candidates[0] if candidates else None
if current is not None:
claimed_sdk_ids.add(current.memory_id)
claimed_sdk_items.add(_sdk_inventory_key(current))
if item.status == "deleted":
decisions.append(
MemoryMigrationDecision(
@@ -313,6 +454,17 @@ def classify_memory_inventory(
)
continue
if current is None:
if item.productivity_kind is not None:
decisions.append(
MemoryMigrationDecision(
item_key=f"memory:{item.legacy_memory_id}",
action="quarantine",
legacy_memory_id=item.legacy_memory_id,
canonical_memory_id=item.canonical_memory_id,
reason="mapped_productivity_memory_missing",
)
)
continue
decisions.append(
MemoryMigrationDecision(
item_key=f"memory:{item.legacy_memory_id}",
@@ -323,7 +475,19 @@ def classify_memory_inventory(
)
)
continue
if (
if item.productivity_kind is not None and (
current.user_id == item.user_id
and current.productivity_kind == item.productivity_kind
and (
item.productivity_variant is None
or current.productivity_variant == item.productivity_variant
)
):
action = (
"matched" if item.legacy_memory_id == item.canonical_memory_id else "record_alias"
)
reason = "existing_productivity_identity"
elif (
current.content_hash == item.content_hash
and current.status == item.status
and current.user_id == item.user_id
@@ -346,10 +510,10 @@ def classify_memory_inventory(
)
for sdk_item in sdk:
if sdk_item.memory_id not in claimed_sdk_ids:
if _sdk_inventory_key(sdk_item) not in claimed_sdk_items:
decisions.append(
MemoryMigrationDecision(
item_key=f"sdk:{sdk_item.memory_id}",
item_key=_sdk_item_key(sdk_item),
action="adopt",
legacy_memory_id=None,
canonical_memory_id=sdk_item.memory_id,
@@ -359,13 +523,33 @@ def classify_memory_inventory(
return tuple(decisions)
def _sdk_inventory_key(
item: SdkMemoryInventory,
) -> tuple[str, str | None, str | None]:
return (
item.memory_id,
item.productivity_kind,
item.productivity_variant,
)
def _sdk_item_key(item: SdkMemoryInventory) -> str:
namespace = (
f"{item.productivity_variant}:{item.productivity_kind}"
if item.productivity_kind is not None
else "chat"
)
return f"sdk:{namespace}:{item.memory_id}"
async def read_sdk_memory_inventory(
client: SdkMigrationClient,
*,
legacy: tuple[LegacyMemoryInventory, ...] = (),
user_id: str = "local",
page_size: int = 500,
) -> tuple[SdkMemoryInventory, ...]:
records: dict[str, SdkMemoryInventory] = {}
records: list[SdkMemoryInventory] = []
offset = 0
while True:
page = await client.list_memories(
@@ -376,18 +560,45 @@ async def read_sdk_memory_inventory(
order="asc",
)
for memory in page.memories:
records[memory.memory_id] = SdkMemoryInventory(
memory_id=memory.memory_id,
user_id=memory.user_id,
agent_id=memory.agent_id,
content_hash=content_hash(memory.content),
status=memory.status,
records.append(
SdkMemoryInventory(
memory_id=memory.memory_id,
user_id=memory.user_id,
agent_id=memory.agent_id,
content_hash=content_hash(memory.content),
status=memory.status,
)
)
consumed = len(page.memories)
offset += consumed
if consumed == 0 or offset >= page.total:
break
return tuple(records[key] for key in sorted(records))
del legacy
for memory in await client.list_productivity_memories():
try:
records.append(
SdkMemoryInventory(
memory_id=str(memory["memory_id"]),
user_id=str(memory["user_id"]),
agent_id=str(memory["agent_id"]),
content_hash=content_hash(str(memory["content"])),
status=str(memory["status"]),
productivity_kind=str(memory["kind"]),
productivity_variant=str(memory["variant"]),
)
)
except KeyError as error:
raise RuntimeError("invalid productivity inventory row") from error
return tuple(
sorted(
records,
key=lambda item: (
item.memory_id,
item.productivity_kind or "",
item.productivity_variant or "",
),
)
)
def record_memory_migration_plan(
@@ -399,12 +610,12 @@ def record_memory_migration_plan(
) -> tuple[MemoryMigrationDecision, ...]:
decisions = classify_memory_inventory(legacy, sdk)
legacy_by_id = {item.legacy_memory_id: item for item in legacy}
sdk_by_id = {item.memory_id: item for item in sdk}
sdk_by_item_key = {_sdk_item_key(item): item for item in sdk}
for decision in decisions:
source_hash = (
legacy_by_id[decision.legacy_memory_id].content_hash
if decision.legacy_memory_id is not None
else sdk_by_id[decision.canonical_memory_id].content_hash
else sdk_by_item_key[decision.item_key].content_hash
)
operation_key = hashlib.sha256(
f"{migration_id}:{decision.item_key}:{decision.action}:{source_hash}".encode()
@@ -457,7 +668,15 @@ async def execute_memory_migration_plan(
continue
if decision.action in {"delete", "verified_deleted"}:
if decision.action == "delete":
await client.delete(decision.canonical_memory_id)
if item is not None and item.productivity_kind is not None:
await client.delete_productivity_memory(
decision.canonical_memory_id,
kind=item.productivity_kind,
variant=item.productivity_variant,
user_id=item.user_id,
)
else:
await client.delete(decision.canonical_memory_id)
store.settle_item(
migration_id=migration_id,
item_key=decision.item_key,
@@ -472,7 +691,7 @@ async def execute_memory_migration_plan(
request_id = hashlib.sha256(
f"{migration_id}:{decision.item_key}:import_exact".encode()
).hexdigest()
await client.import_memory_exact(
await client.write_legacy_memory(
content=item.content,
memory_id=item.canonical_memory_id,
user_id=item.user_id,
@@ -514,7 +733,7 @@ async def read_sdk_raw_inventory(
records: dict[str, SdkRawInventory] = {}
offset = 0
while True:
page = await client.list_raw(
page = await client.read_raw_inventory(
user_id=user_id,
limit=page_size,
offset=offset,
@@ -595,10 +814,10 @@ def classify_raw_inventory(
decisions.append(
RawMigrationDecision(
item_key=f"sdk-raw:{raw.raw_id}",
action="adopt",
action="delete",
evidence_id=None,
raw_id=raw.raw_id,
reason="sdk_only",
reason="outside_canonical_three_day_turns",
)
)
return tuple(decisions)
@@ -644,11 +863,12 @@ async def execute_raw_migration_plan(
) -> None:
legacy_by_id = {item.evidence_id: item for item in legacy}
for decision in decisions:
if decision.action == "adopt":
if decision.action == "delete":
await client.delete_raw(decision.raw_id)
store.settle_item(
migration_id=migration_id,
item_key=decision.item_key,
state="adopted",
state="deleted",
target_memory_id=decision.raw_id,
)
continue
@@ -657,9 +877,9 @@ async def execute_raw_migration_plan(
raise RuntimeError(f"missing legacy raw inventory for {decision.item_key}")
if decision.action == "import_exact":
request_id = hashlib.sha256(
f"{migration_id}:{decision.item_key}:import_raw".encode()
f"{migration_id}:{decision.item_key}:write_legacy_raw".encode()
).hexdigest()
await client.import_raw(
await client.write_legacy_raw(
content=item.content,
raw_id=decision.raw_id,
user_id=item.user_id,
@@ -677,8 +897,15 @@ async def execute_raw_migration_plan(
target_memory_id=decision.raw_id,
target_hash=item.content_hash,
)
store.record_raw_alias(
migration_id=migration_id,
legacy_evidence_id=item.evidence_id,
canonical_raw_id=decision.raw_id,
)
evidence_ids = item.metadata.get("evidence_ids")
for evidence_id in (
evidence_ids
if isinstance(evidence_ids, list)
else [item.evidence_id]
):
if isinstance(evidence_id, str) and evidence_id:
store.record_raw_alias(
migration_id=migration_id,
legacy_evidence_id=evidence_id,
canonical_raw_id=decision.raw_id,
)
@@ -4,8 +4,12 @@ from __future__ import annotations
import hashlib
import json
import shutil
import sqlite3
from collections.abc import Callable
from contextlib import suppress
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from pathlib import Path
from one_memory.database_access import main_database
@@ -47,6 +51,139 @@ class BackfillResult:
raw_items: int
@dataclass(frozen=True)
class SdkMigrationStaging:
migration_id: str
staging_home: Path
live_root: Path
staging_root: Path
protective_root: Path
state_file: Path
def sdk_migration_staging(
*,
data_dir: Path,
migration_id: str,
namespace: str,
) -> SdkMigrationStaging:
"""Return same-filesystem paths without trusting the migration ID as a path."""
token = hashlib.sha256(migration_id.encode()).hexdigest()[:16]
home = data_dir.resolve()
staging_home = home / ".sdk-migration" / token
live_root = home / "hy-memory" / namespace
return SdkMigrationStaging(
migration_id=migration_id,
staging_home=staging_home,
live_root=live_root,
staging_root=staging_home / "hy-memory" / namespace,
protective_root=live_root.parent / f".{namespace}.pre-migration-{token}",
state_file=staging_home / "swap-state.json",
)
def _write_staging_state(paths: SdkMigrationStaging, phase: str) -> None:
paths.staging_home.mkdir(parents=True, exist_ok=True)
temporary = paths.state_file.with_suffix(".tmp")
temporary.write_text(
json.dumps(
{"schema_version": 1, "migration_id": paths.migration_id, "phase": phase},
sort_keys=True,
),
encoding="utf-8",
)
temporary.replace(paths.state_file)
def recover_sdk_migration_swap(
*,
paths: SdkMigrationStaging,
database_path: Path,
) -> None:
"""Recover an interrupted directory swap before opening any SDK store."""
if not paths.protective_root.exists():
return
authority = SdkDataPlaneMigrationStore(database_path).authority()
if authority.value == "sdk":
if not paths.live_root.is_dir():
raise DataPlaneMigrationError("sdk_migration_committed_store_missing")
shutil.rmtree(paths.protective_root)
shutil.rmtree(paths.staging_home, ignore_errors=True)
return
if paths.live_root.exists():
if paths.staging_root.exists():
raise DataPlaneMigrationError("sdk_migration_swap_state_ambiguous")
paths.staging_root.parent.mkdir(parents=True, exist_ok=True)
paths.live_root.replace(paths.staging_root)
paths.protective_root.replace(paths.live_root)
_write_staging_state(paths, "recovered")
def prepare_sdk_migration_staging(
*,
paths: SdkMigrationStaging,
database_path: Path,
) -> Path:
"""Build a private copy; incomplete retries restart from the unchanged live root."""
recover_sdk_migration_swap(paths=paths, database_path=database_path)
migration_store = SdkDataPlaneMigrationStore(database_path)
try:
progress = migration_store.progress(paths.migration_id)
reuse_ready = progress.state == "ready" and progress.phase == "reconciled"
except Exception:
reuse_ready = False
if paths.staging_root.exists():
if paths.staging_root.is_symlink() or not paths.staging_root.is_dir():
raise DataPlaneMigrationError("sdk_migration_staging_invalid")
if reuse_ready:
return paths.staging_home
shutil.rmtree(paths.staging_root)
paths.live_root.mkdir(parents=True, exist_ok=True)
from one_memory.hy_sdk_migration import snapshot_data_plane
snapshot_data_plane(
source_root=paths.live_root,
target_root=paths.staging_root,
writers_stopped=True,
)
_write_staging_state(paths, "prepared")
return paths.staging_home
def activate_sdk_migration_staging(
*,
paths: SdkMigrationStaging,
database_path: Path,
commit: Callable[[], CutoverMarker],
) -> CutoverMarker:
"""Atomically expose staged SDK files and roll them back if cutover fails."""
recover_sdk_migration_swap(paths=paths, database_path=database_path)
if not paths.staging_root.is_dir():
raise DataPlaneMigrationError("sdk_migration_staging_missing")
paths.live_root.mkdir(parents=True, exist_ok=True)
_write_staging_state(paths, "activating")
paths.live_root.replace(paths.protective_root)
try:
paths.staging_root.replace(paths.live_root)
_write_staging_state(paths, "staging_live")
marker = commit()
except BaseException:
if paths.live_root.exists():
paths.staging_root.parent.mkdir(parents=True, exist_ok=True)
paths.live_root.replace(paths.staging_root)
if paths.protective_root.exists():
paths.protective_root.replace(paths.live_root)
_write_staging_state(paths, "rolled_back")
raise
with suppress(OSError):
_write_staging_state(paths, "committed")
# Authority is already committed; leftover paths are detected by
# sdk-migration-status and removed by sdk-cleanup on the next startup.
shutil.rmtree(paths.protective_root, ignore_errors=True)
shutil.rmtree(paths.staging_home, ignore_errors=True)
return marker
def prepare_legacy_write_queue_for_cutover(database_path: Path) -> tuple[int, int]:
"""Freeze old jobs and move unstarted task rows into the surviving queue."""
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
@@ -217,7 +354,6 @@ def commit_reconciled_cutover(
reconciliation_digest=digest,
)
inspect_sdk_migration_bundle(data_dir=data_dir, bundle_id=marker.backup_bundle_id)
drop_legacy_tables(database_path)
return marker
@@ -232,6 +368,16 @@ def resume_legacy_cleanup(*, data_dir: Path, database_path: Path) -> tuple[str,
data_dir=data_dir,
bundle_id=marker.backup_bundle_id,
)
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
recover_sdk_migration_swap(
paths=sdk_migration_staging(
data_dir=data_dir,
migration_id=marker.migration_id,
namespace=HY_MEMORY_NAMESPACE,
),
database_path=database_path,
)
return drop_legacy_tables(database_path)
@@ -261,14 +407,27 @@ async def run_sdk_data_plane_backfill(
store.set_phase(migration_id, phase="inventory", state="running")
legacy_memories = read_legacy_memory_inventory(database_path)
sdk_memories = await read_sdk_memory_inventory(client)
sdk_memories = await read_sdk_memory_inventory(client, legacy=legacy_memories)
memory_decisions = record_memory_migration_plan(
store,
migration_id=migration_id,
legacy=legacy_memories,
sdk=sdk_memories,
)
legacy_raw = read_legacy_raw_inventory(database_path)
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
started = database.execute(
"SELECT created_at FROM sdk_data_plane_migrations WHERE migration_id=?",
(migration_id,),
).fetchone()
if started is None:
raise DataPlaneMigrationError("sdk_data_plane_migration_start_missing")
started_at = datetime.fromisoformat(str(started["created_at"]))
if started_at.tzinfo is None:
started_at = started_at.replace(tzinfo=UTC)
legacy_raw = read_legacy_raw_inventory(
database_path,
cutoff=started_at.astimezone(UTC) - timedelta(days=3),
)
sdk_raw = await read_sdk_raw_inventory(client)
raw_decisions = record_raw_migration_plan(
store,
+3 -1
View File
@@ -54,6 +54,7 @@
| `test_claude_adoption_probe.py` | `contract` | The external-use verifier must distinguish a read from merely mentioning it. |
| `test_cli_commands.py` | `contract` | 覆盖 status reports the verified runtime as json、doctor fails when runtime identity cannot be verified、serve and legacy empty command share the same entry 等 4 个场景。 |
| `test_cli_lifecycle.py` | `integration` | 覆盖 cli starts and stops without residual runtime 场景。 |
| `test_cli_migration_error_codes.py` | `unit` | The startup dialog can only act on the code it is handed. |
| `test_cli_restore.py` | `contract` | 覆盖 restore requires a source、restore refuses while the runtime is running、restore previews without confirmation 等 4 个场景。 |
| `test_codex_header_provenance.py` | `contract` | 覆盖 header workspace versions and native ids reach file origin、new adapter rereads header after a real checkpoint、machine and unknown session sources do not establish human or agent sender 等 14 个场景。 |
| `test_connection_lifetime.py` | `integration` | Every SQLite connection a request opens must be closed when the request ends. |
@@ -286,6 +287,7 @@
| `test_source_authorization_state.py` | `unit` | Why a source cannot be read, named separately from whether it can. |
| `test_source_candidate_api.py` | `integration` | Candidate location changes must not implicitly grant or import anything. |
| `test_source_candidate_connections.py` | `contract` | 覆盖 empty candidate revisions preserve default fingerprint、persisted candidate requires exact revision、connection resolves candidate in transaction and prior replay never regrants 等 7 个场景。 |
| `test_source_candidate_kind_parity.py` | `contract` | 来源根目录类型在三处必须同时存在:python 枚举、TS 契约、前端运行时校验。 |
| `test_source_candidates.py` | `contract` | 覆盖 preview is metadata only and cancel changes nothing、select persists only candidate and restarts with revision、compare and swap and aba cannot reuse old preview 等 9 个场景。 |
| `test_source_connection_intents.py` | `integration` | No user data: connection retries must retain the original intent and job. |
| `test_source_discovery.py` | `contract` | 覆盖 discovery is sorted bounded and suffix scoped、discovery does not follow directory or file symlinks 场景。 |
@@ -327,4 +329,4 @@
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
共 322 个测试文件。
共 324 个测试文件。
@@ -14,13 +14,14 @@ from pathlib import Path
import pytest
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
EXPECTED_TEST_FILE_COUNT = 322
EXPECTED_TEST_FILES_SHA256 = "d242783adb4015ebdbb5c099f3b5aa88a99c828ca6fddc8ca8d6c74c56e48945"
EXPECTED_TEST_FILE_COUNT = 324
EXPECTED_TEST_FILES_SHA256 = "3fcc7a3336de56aa8fa6b0dddd6101441a837095c89dd5f31d2a1b66ec7e3764"
# Pure transformations, validation models, parity checks and state machines.
# This list is intentionally small: uncertain files default to contract.
UNIT_FILES = frozenset(
{
"test_cli_migration_error_codes.py",
"test_context_v2_contract_parity.py",
"test_daemon_contract.py",
"test_diagnostic_payload_keys.py",
@@ -107,6 +107,12 @@ SDK_PRIVATE_ACCESS_EXCEPTIONS = {
f"{PACKAGE_LABEL}/hy_graph_projection.py": frozenset(
{"_graph_store", "_loop_thread", "_run"}
),
# Startup migration runs while Runtime writers are stopped. It is the one
# non-runtime path allowed to manipulate SDK storage directly so that
# migration-only mechanics never become public HyMemoryClient methods.
f"{PACKAGE_LABEL}/hy_sdk_migration.py": frozenset(
{"_embed_service", "_graph_store", "_loop_thread", "_vector_store"}
),
}
@@ -241,7 +247,7 @@ def test_no_new_module_writes_legacy_memory_authority() -> None:
def test_no_sdk_private_implementation_access() -> None:
"""No debt allowlist: MemHub calls public client methods only."""
"""Runtime uses public APIs; only Graph and stopped-writer migration are privileged."""
offenders: list[str] = []
for path in _all_module_files():
tree = ast.parse(path.read_text(encoding="utf-8"))
@@ -266,16 +272,18 @@ def test_no_sdk_private_implementation_access() -> None:
if unexpected:
offenders.append(f"{relative}: {sorted(unexpected)}")
assert not offenders, (
"SDK private implementation access is forbidden; add a public "
f"HyMemoryClient API instead: {offenders}"
"SDK private implementation access is forbidden outside the isolated "
f"Graph and offline-migration boundaries: {offenders}"
)
def test_sdk_modules_import_only_the_public_package_root() -> None:
"""Implementation submodules are not a public product contract."""
"""Runtime imports only public SDK symbols; offline migration is privileged."""
offenders: list[str] = []
for path in _all_module_files():
relative = _relative(path)
if relative == f"{PACKAGE_LABEL}/hy_sdk_migration.py":
continue
tree = ast.parse(path.read_text(encoding="utf-8"))
for node in ast.walk(tree):
if isinstance(node, ast.Import) and any(
@@ -0,0 +1,131 @@
"""The startup dialog can only act on the code it is handed.
`sdk_migration_backfill_failed` covered a full disk, a missing model profile and
a Coding classification gap alike, so every one of them reached the person as
the same unactionable sentence. These tests pin the names that replace it.
"""
from __future__ import annotations
import errno
from pathlib import Path
import pytest
from one_memory.cli import cutover_failure_code, migration_failure_code
class _Coded(RuntimeError):
def __init__(self, code: str) -> None:
super().__init__(code)
self.code = code
def test_running_out_of_disk_is_named_however_deep_it_is_wrapped() -> None:
error = OSError(errno.ENOSPC, "No space left on device")
wrapped: BaseException = error
for _ in range(3):
try:
raise RuntimeError("wrapper") from wrapped
except RuntimeError as caught:
wrapped = caught
assert migration_failure_code(wrapped) == "sdk_migration_disk_space_insufficient"
# The explicit context chain counts too: `raise ... from None` discards the
# cause but Python still keeps the original as __context__.
try:
try:
raise OSError(errno.ENOSPC, "No space left on device")
except OSError:
raise RuntimeError("sdk_snapshot_failed") from None
except RuntimeError as caught:
assert migration_failure_code(caught) == "sdk_migration_disk_space_insufficient"
def test_quota_exhaustion_counts_as_no_space(tmp_path: Path) -> None:
quota = getattr(errno, "EDQUOT", errno.ENOSPC)
assert (
migration_failure_code(OSError(quota, "Disk quota exceeded"))
== "sdk_migration_disk_space_insufficient"
)
def test_a_coding_inventory_gap_is_not_reported_as_a_generic_failure() -> None:
assert (
migration_failure_code(_Coded("hy_sdk_productivity_variant_invalid"))
== "sdk_migration_coding_inventory_failed"
)
def test_the_legacy_working_fact_failure_keeps_its_own_name() -> None:
error = _Coded("hy_sdk_memory_import_failed")
error.__cause__ = ValueError("Invalid memory layer: working_fact")
assert migration_failure_code(error) == "sdk_migration_working_fact_conflict"
def test_migration_state_codes_pass_through_because_they_are_already_actionable() -> None:
assert (
migration_failure_code(_Coded("sdk_data_plane_migration_item_conflict"))
== "sdk_data_plane_migration_item_conflict"
)
assert (
migration_failure_code(_Coded("sdk_migration_staging_invalid"))
== "sdk_migration_staging_invalid"
)
def test_unknown_failures_keep_their_message_out_of_the_code() -> None:
error = RuntimeError("/Users/someone/private/path blew up")
assert migration_failure_code(error) == "sdk_migration_backfill_failed"
assert (
migration_failure_code(error, fallback="sdk_migration_preflight_failed")
== "sdk_migration_preflight_failed"
)
def test_a_non_code_attribute_never_becomes_a_code() -> None:
class Odd(RuntimeError):
code = 17
assert migration_failure_code(Odd("x")) == "sdk_migration_backfill_failed"
def test_the_chain_survives_a_cycle() -> None:
first = RuntimeError("first")
second = RuntimeError("second")
first.__cause__ = second
second.__cause__ = first
assert migration_failure_code(first) == "sdk_migration_backfill_failed"
@pytest.mark.parametrize(
"code",
[
"sdk_migration_disk_space_insufficient",
"sdk_migration_working_fact_conflict",
"sdk_migration_coding_inventory_failed",
"sdk_migration_work_pending",
],
)
def test_the_codes_the_desktop_translates_stay_in_sync(code: str) -> None:
"""Desktop maps these exact names to sentences; renaming one blinds it."""
desktop = Path(__file__).resolve().parents[2] / "desktop" / "src" / "stable-runtime.ts"
assert desktop.is_file(), desktop
assert f'"{code}"' in desktop.read_text(encoding="utf-8")
def test_cutover_work_counts_do_not_become_the_dialog_text() -> None:
"""The blocker code carries a JSON body; the person gets one name."""
blocked = _Coded('sdk_data_plane_cutover_work_pending:{"memory_tasks":591}')
assert cutover_failure_code(blocked) == "sdk_migration_work_pending"
assert (
cutover_failure_code(_Coded("sdk_data_plane_cutover_missing"))
== "sdk_data_plane_cutover_missing"
)
assert (
cutover_failure_code(RuntimeError("/Users/someone/private"))
== "sdk_migration_cutover_failed"
)
+59 -3
View File
@@ -1,10 +1,15 @@
import asyncio
import io
import json
import sqlite3
from types import SimpleNamespace
from one_memory import cli
from one_memory.config import Settings
from one_memory.memory_server.migration_runner import SqliteMigrationStore, migrate
from one_memory.memory_server.migrations import MIGRATIONS
from one_memory.runtime_client import RuntimeStatus
from one_memory.sdk_data_plane_migration import SdkDataPlaneMigrationStore
def test_restore_requires_a_source(tmp_path, monkeypatch) -> None:
@@ -302,7 +307,12 @@ def test_sdk_migrate_runs_offline_backfill_without_cutover(tmp_path, monkeypatch
pass
class Lease:
client = object()
client = SimpleNamespace(
_loop_thread=None,
_vector_store=None,
_embed_service=None,
_graph_store=None,
)
def close(self):
return None
@@ -358,10 +368,18 @@ def test_sdk_cutover_is_offline_confirmed_and_returns_marker(tmp_path, monkeypat
lambda **_kwargs: SimpleNamespace(
migration_id="migration-1",
backup_bundle_id="bundle-1",
sdk_version="2.0.5.dev3",
sdk_version="2.0.5",
completed_at="2026-10-21T00:00:00Z",
),
)
monkeypatch.setattr(
"one_memory.sdk_migration_runner.activate_sdk_migration_staging",
lambda *, commit, **_kwargs: commit(),
)
monkeypatch.setattr(
"one_memory.sdk_migration_runner.resume_legacy_cleanup",
lambda **_kwargs: (),
)
output = io.StringIO()
code = cli.run(
@@ -380,7 +398,7 @@ def test_sdk_cutover_is_offline_confirmed_and_returns_marker(tmp_path, monkeypat
assert code == 0
document = output.getvalue().replace(" ", "")
assert '"state":"cutover"' in document
assert '"sdk_version":"2.0.5.dev3"' in document
assert '"sdk_version":"2.0.5"' in document
def test_sdk_cutover_never_runs_while_runtime_is_live(tmp_path, monkeypatch) -> None:
@@ -423,3 +441,41 @@ def test_sdk_migration_status_distinguishes_empty_and_legacy_data(tmp_path, monk
assert cli.run(["sdk-migration-status", "--json"], output=required) == 0
assert '"state": "required"' in required.getvalue()
assert '"raw": 1' in required.getvalue()
def test_sdk_migration_status_exposes_a_resumable_active_identity(tmp_path, monkeypatch) -> None:
monkeypatch.setattr(
cli,
"load_settings",
lambda: Settings(_env_file=None, data_dir=tmp_path),
)
database = tmp_path / "one-memory.sqlite3"
asyncio.run(migrate(SqliteMigrationStore(database), MIGRATIONS))
with sqlite3.connect(database) as connection:
connection.execute(
"INSERT INTO evidence ("
"evidence_id,idempotency_key,user_id,source_id,conversation_id,"
"content_kind,role,written_by,verbatim,content_hash,revision,"
"imported_at,updated_at"
") VALUES ('e1','k','local','cursor','s','raw_conversation',"
"'user','person','hello','hash',1,'now','now')"
)
SdkDataPlaneMigrationStore(database).start(
migration_id="sdk-authority-v2-0123456789abcdef",
source_runtime_build="17",
target_sdk_version="2.0.5",
source_schema_version=MIGRATIONS[-1].version,
namespace_fingerprint="namespace",
backend_fingerprint="backend",
backup_bundle_id=(
"sdk-authority-v2-0123456789abcdef-pre-cutover"
),
)
output = io.StringIO()
assert cli.run(["sdk-migration-status", "--json"], output=output) == 0
document = json.loads(output.getvalue())
assert document["migration_id"] == "sdk-authority-v2-0123456789abcdef"
assert document["bundle_id"] == (
"sdk-authority-v2-0123456789abcdef-pre-cutover"
)
+310 -18
View File
@@ -18,11 +18,13 @@ from one_memory.diagnostic_log import (
diagnostic_exception,
diagnostics_enabled,
diagnostics_requested,
read_trace_events,
record_database_error,
reset_diagnostics,
reset_trace_id,
resource_snapshot,
)
from one_memory.exception_reporting import report_exception
def test_debug_events_are_collected_only_at_debug_level(tmp_path, monkeypatch) -> None:
@@ -105,7 +107,31 @@ def test_structured_log_is_private_correlated_and_owner_only(tmp_path):
assert log.path.stat().st_mode & 0o777 == 0o600
def test_arbitrary_detail_and_absolute_stack_are_rejected(tmp_path):
def test_arbitrary_valid_detail_keys_and_scalars_are_recorded(tmp_path):
log = DiagnosticLog(tmp_path)
details = {
"Custom.metric-v1": "ready",
"noneValue": None,
"boolValue": False,
"intValue": 7,
"floatValue": 1.5,
}
log.write(
level="info",
component="core.lifecycle",
event="core.started",
outcome="succeeded",
details=details,
)
assert log.flush()
assert json.loads(log.path.read_text())["details"] == details
@pytest.mark.parametrize(
"key",
["__proto__", "prototype", "constructor", "_private", "9startsWithDigit", "a" * 65],
)
def test_dangerous_or_malformed_detail_keys_are_rejected(tmp_path, key):
log = DiagnosticLog(tmp_path)
with pytest.raises(ValueError, match="diagnostic_detail_key_invalid"):
log.write(
@@ -113,8 +139,76 @@ def test_arbitrary_detail_and_absolute_stack_are_rejected(tmp_path):
component="hy.bridge",
event="hy.failed",
outcome="failed",
details={"prompt": "private"},
details={key: "value"},
)
@pytest.mark.parametrize("value", [{}, [], object()])
def test_complex_detail_values_are_rejected(tmp_path, value):
log = DiagnosticLog(tmp_path)
with pytest.raises(ValueError, match="diagnostic_detail_value_invalid"):
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
details={"payload": value},
)
@pytest.mark.parametrize("value", [float("nan"), float("inf"), float("-inf")])
def test_non_finite_detail_values_are_rejected(tmp_path, value):
log = DiagnosticLog(tmp_path)
with pytest.raises(ValueError, match="diagnostic_detail_value_invalid"):
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
details={"metric": value},
)
@pytest.mark.parametrize("value", [2**53, -(2**53), float(2**53), -float(2**53)])
def test_numeric_detail_values_outside_json_safe_range_are_rejected(tmp_path, value):
log = DiagnosticLog(tmp_path)
with pytest.raises(ValueError, match="diagnostic_detail_value_invalid"):
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
details={"count": value},
)
@pytest.mark.parametrize("value", ["", "nul\0byte", "carriage\rreturn", "line\nfeed", "x" * 257])
def test_invalid_detail_strings_are_rejected(tmp_path, value):
log = DiagnosticLog(tmp_path)
with pytest.raises(ValueError, match="diagnostic_detail_value_invalid"):
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
details={"message": value},
)
def test_more_than_32_detail_keys_are_rejected(tmp_path):
log = DiagnosticLog(tmp_path)
with pytest.raises(ValueError, match="diagnostic_detail_count_invalid"):
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
details={f"key{index}": index for index in range(33)},
)
def test_absolute_or_credential_bearing_stack_is_rejected(tmp_path):
log = DiagnosticLog(tmp_path)
with pytest.raises(ValueError, match="diagnostic_stack_invalid"):
log.write(
level="error",
@@ -123,16 +217,90 @@ def test_arbitrary_detail_and_absolute_stack_are_rejected(tmp_path):
outcome="failed",
safe_stack=("/Users/me/private.py:1",),
)
with pytest.raises(ValueError, match="diagnostic_stack_invalid"):
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
safe_stack=("one_memory/sk-live-secret-token.py:1:call",),
)
def test_a_rejected_payload_does_not_break_the_caller(tmp_path, monkeypatch) -> None:
"""Rejection protects the log; it must not become the caller's outage.
def test_safe_stack_uses_strict_shared_grammar_and_sixteen_line_limit(tmp_path):
log = DiagnosticLog(tmp_path)
for line in (
"arbitrary stack text",
"thread=worker|threading.py:12:wait",
"FILE:///Users/me/private.py:1:call",
"one_memory/worker.py:9:call /Users/private",
"one_memory/GOCSPX-privateCredential.py:9:call",
"packages/ASIA1234567890ABCDEF/file.ts:1:1",
"packages/Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8/file.ts:1:1",
):
with pytest.raises(ValueError, match="diagnostic_stack_invalid"):
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
safe_stack=(line,),
)
Two of our own nodes shipped with unlisted `details` keys. Both sat on a
request path, so the rejection surfaced as a 500 to the person using the
app — including the sedimentation switch, whose every request failed
because logging the toggle raised before the endpoint could answer.
"""
accepted = (
"apps/desktop/src/main.ts:80:1",
"one_memory/diagnostic_log.py:895:_safe_stack_line",
"threading.py:1045:_bootstrap_inner",
"thread|threading.py:1045:_bootstrap_inner > one_memory/worker.py:9:call",
)
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
safe_stack=accepted,
)
with pytest.raises(ValueError, match="diagnostic_stack_invalid"):
log.write(
level="error",
component="hy.bridge",
event="hy.failed",
outcome="failed",
safe_stack=tuple(f"one_memory/worker.py:{index + 1}:call" for index in range(17)),
)
assert log.flush()
assert json.loads(log.path.read_text())["safe_stack"] == list(accepted)
def test_invalid_level_is_strict_but_diagnostic_event_remains_fail_open(
tmp_path, monkeypatch
) -> None:
log = DiagnosticLog(tmp_path)
with pytest.raises(ValueError, match="diagnostic_level_invalid"):
log.write(
level="fatal", # type: ignore[arg-type]
component="core.lifecycle",
event="core.failed",
outcome="failed",
)
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
receipt = configure_diagnostics(tmp_path / "fail-open")
try:
diagnostic_event(
level="fatal",
component="core.lifecycle",
event="core.failed",
outcome="failed",
)
finally:
reset_diagnostics()
assert not receipt.path.exists()
def test_a_rejected_payload_does_not_break_the_caller_or_later_events(
tmp_path, monkeypatch
) -> None:
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
receipt = configure_diagnostics(tmp_path)
try:
@@ -141,20 +309,122 @@ def test_a_rejected_payload_does_not_break_the_caller(tmp_path, monkeypatch) ->
component="memory.sedimentation",
event="memory_sedimentation.toggled",
outcome="succeeded",
details={"prompt": "private"},
details={"payload": ["not", "scalar"]},
)
diagnostic_event(
level="info",
component="memory.sedimentation",
event="memory_sedimentation.toggled",
outcome="succeeded",
details={"custom.metric": 1},
)
finally:
reset_diagnostics()
# Rejected, so nothing is written — but the call returned.
assert not receipt.path.exists() or receipt.path.read_text() == ""
events = [json.loads(line) for line in receipt.path.read_text().splitlines()]
assert [event["details"] for event in events] == [{"custom.metric": 1}]
def test_trace_projection_does_not_expose_arbitrary_local_detail_keys(tmp_path):
log = DiagnosticLog(tmp_path)
log.write(
level="info",
component="core.api",
event="memory.request.completed",
outcome="succeeded",
request_id="request-1",
details={
"phase": "recall",
"reason": "/Users/private/query.txt",
"status": "sk-live-secret-token",
"custom.private": "local-only",
},
)
assert log.flush()
events = read_trace_events(tmp_path, request_id="request-1")
assert events[0]["details"] == {"phase": "recall"}
def test_trace_projection_rejects_symlinked_directory_and_redacts_top_level_secrets(
tmp_path,
) -> None:
request_id = "request-1"
logs = tmp_path / "logs"
logs.mkdir()
event = {
"timestamp": "2026-10-23T01:00:00Z",
"request_id": request_id,
"component": "secret.token",
"event": "credential.failed",
"outcome": "Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8",
"error_code": "GOCSPX-clientSecret_A1",
"error_type": "ASIA1234567890ABCDEF",
"details": {
"phase": "recall",
"reason": "x_ASIA1234567890ABCDEF",
"mode": "x_GOCSPX-clientSecret_A1",
"status": "Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8",
"failure_site": "AIzaSyExampleCredential1234567890",
"count": 1,
},
}
(logs / "diagnostic-core.jsonl").write_text(json.dumps(event) + "\n")
projected = read_trace_events(tmp_path, request_id=request_id)
assert projected == (
{
"timestamp": "2026-10-23T01:00:00Z",
"component": "diagnostic.redacted",
"event": "diagnostic.redacted",
"outcome": "redacted",
"details": {"phase": "recall", "count": 1},
},
)
external = tmp_path / "external"
external.mkdir()
(logs / "diagnostic-core.jsonl").unlink()
logs.rmdir()
logs.symlink_to(external, target_is_directory=True)
with pytest.raises(ValueError, match="diagnostic_directory_unsafe"):
read_trace_events(tmp_path, request_id=request_id)
def test_credential_detection_is_conservative_and_matches_export_rules() -> None:
for value in (
"ASIA1234567890ABCDEF",
"GOCSPX-clientSecret_A1",
"Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8",
"Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8_",
):
assert diagnostic_log._contains_credential_material(value)
for value in (
"11111111-1111-4111-8111-111111111111",
"abcdef1234567890abcdef1234567890abcdef12",
"cursor",
):
assert not diagnostic_log._contains_credential_material(value)
def test_trace_reader_never_follows_a_diagnostic_file_symlink(tmp_path) -> None:
logs = tmp_path / "logs"
logs.mkdir()
outside = tmp_path / "outside.jsonl"
outside.write_text(
json.dumps(
{
"timestamp": "2026-10-23T01:00:00Z",
"request_id": "request-1",
"component": "core.api",
"event": "memory.request.completed",
"outcome": "succeeded",
}
)
+ "\n"
)
(logs / "diagnostic-core.jsonl").symlink_to(outside)
assert read_trace_events(tmp_path, request_id="request-1") == ()
def test_the_sedimentation_toggle_payload_is_recorded(tmp_path, monkeypatch) -> None:
"""`enabled` is a bounded boolean and has to be a legal detail key.
Omitting it from the allow-list is what made the switch's own endpoint
answer 500 while still saving the new value.
"""
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
receipt = configure_diagnostics(tmp_path)
try:
@@ -181,7 +451,7 @@ def test_exception_diagnostic_keeps_code_location_not_message():
component="hy.bridge",
event="hy.failed",
outcome="failed",
details={"prompt": secret},
details={"prompt": f"{secret}\n"},
)
except ValueError as error:
safe = diagnostic_exception(error)
@@ -191,6 +461,28 @@ def test_exception_diagnostic_keeps_code_location_not_message():
assert str(__file__) not in json.dumps(safe)
def test_report_exception_never_writes_the_exception_message_to_diagnostics(
tmp_path, monkeypatch
) -> None:
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
receipt = configure_diagnostics(tmp_path)
secret = "sk-live-private-provider-response"
try:
report_exception(
"core.test",
"core.test.failed",
RuntimeError(f"provider failed with {secret} at /Users/private/query.txt"),
)
finally:
reset_diagnostics()
payload = receipt.path.read_text()
assert secret not in payload
assert "/Users/private" not in payload
event = json.loads(payload)
assert event["details"] == {"count": 1}
assert event["error_type"] == "RuntimeError"
def test_old_logs_are_removed_but_symlinks_are_never_followed(tmp_path):
log = DiagnosticLog(tmp_path)
log.path.parent.mkdir(parents=True)
@@ -449,7 +449,7 @@ def test_invalid_input_is_still_rejected_on_the_caller_side(tmp_path):
component="hy.bridge",
event="hy.failed",
outcome="failed",
details={"prompt": "private"},
details={"invalid key": "private"},
)
log.close()
@@ -1,22 +1,57 @@
"""An instrument must never be able to break the thing it measures.
`diagnostic_event` validates its payload: an unlisted `details` key, a missing
`level`, or a detail value over 256 characters all raise. Two of our own nodes
shipped with unlisted keys, and because they sat on a request path and on the
loop heartbeat, they turned every external recall into a 500 and then killed
the process. Both were caught only by running the built runtime, not by tests.
`diagnostic_event` validates its payload: a malformed `details` key, a missing
`level`, or an unsafe detail value is rejected. A diagnostic payload must never
be able to break the request path or loop heartbeat it is intended to measure.
So this checks the payloads at their source, by walking every literal call.
"""
import ast
import re
from pathlib import Path
import pytest
from one_memory.diagnostic_log import _DETAIL_KEYS
ROOT = Path(__file__).resolve().parent.parent / "src" / "one_memory"
DETAIL_KEY = re.compile(r"^[A-Za-z][A-Za-z0-9_.-]{0,63}$")
REFUSED_DETAIL_KEYS = {"__proto__", "prototype", "constructor"}
EXPORTED_STRING_DETAIL_KEYS = {
"action",
"method",
"mode",
"phase",
"platform",
"reason",
"source_id",
"status",
"version",
"parent_session_id",
"runtime_state",
"core_state",
"broker_state",
"instance_id",
"operation_kind",
"file_kind",
"last_probe",
"signal",
}
PRIVATE_DETAIL_KEYS = {
"api_key",
"authorization",
"content",
"credential",
"memory",
"memory_text",
"message",
"messages",
"password",
"prompt",
"query",
"raw_text",
"secret",
"token",
}
#: Every module, not a hand-picked list. The two payloads that caused real
#: outages were in files nobody thought to add here, so the only shape of this
@@ -33,6 +68,19 @@ def _literal_calls(path: Path):
yield node
def _contains_free_text_conversion(node: ast.AST) -> bool:
for item in ast.walk(node):
if (
isinstance(item, ast.Call)
and isinstance(item.func, ast.Name)
and item.func.id in {"str", "repr", "_safe_message"}
):
return True
if isinstance(item, ast.Attribute) and item.attr in {"args", "message"}:
return True
return False
@pytest.mark.parametrize("relative", SOURCES)
def test_every_diagnostic_payload_is_valid(relative: str) -> None:
path = ROOT / relative
@@ -45,7 +93,22 @@ def test_every_diagnostic_payload_is_valid(relative: str) -> None:
for keyword in node.keywords:
if keyword.arg != "details" or not isinstance(keyword.value, ast.Dict):
continue
for key in keyword.value.keys:
if isinstance(key, ast.Constant) and key.value not in _DETAIL_KEYS:
problems.append(f"{relative}:{node.lineno}: unlisted details key {key.value!r}")
for key, expression in zip(keyword.value.keys, keyword.value.values, strict=True):
if not isinstance(key, ast.Constant):
continue
value = key.value
if (
not isinstance(value, str)
or not DETAIL_KEY.fullmatch(value)
or value in REFUSED_DETAIL_KEYS
):
problems.append(f"{relative}:{node.lineno}: invalid details key {value!r}")
elif value.lower() in PRIVATE_DETAIL_KEYS:
problems.append(f"{relative}:{node.lineno}: private details key {value!r}")
elif value in EXPORTED_STRING_DETAIL_KEYS and _contains_free_text_conversion(
expression
):
problems.append(
f"{relative}:{node.lineno}: exported detail {value!r} uses free text"
)
assert not problems, "\n".join(problems)
+30 -4
View File
@@ -2,6 +2,7 @@
import asyncio
import json
import os
import sqlite3
from contextlib import closing
from io import StringIO
@@ -20,6 +21,7 @@ from one_memory.memory_server.durable_queue import (
DurableWriteQueue,
execute_job,
)
from one_memory.sdk_write_processor import SdkWriteReceipt
def _grant(tmp_path, source_id: str, *, sync: bool) -> EvidenceStore:
@@ -233,14 +235,14 @@ def test_hook_job_completes_after_file_identity_adoption(tmp_path) -> None:
).fetchone()
class IdentityCheckingSupervisor:
async def process_records(
async def process_sdk_job(
self,
records,
*,
request_id,
before_hy_invocation,
source_admitted,
):
assert source_admitted is True
assert request_id
with closing(sqlite3.connect(database)) as connection:
connection.execute("BEGIN")
for record in records:
@@ -249,7 +251,7 @@ def test_hook_job_completes_after_file_identity_adoption(tmp_path) -> None:
assert ingest_journal.matches_current(connection, current)
connection.rollback()
await before_hy_invocation()
return ()
return SdkWriteReceipt()
drain = DurableWriteDrain(queue, execute=lambda _job: None)
assert drain.claim(limit=1) == (job_id,)
@@ -560,6 +562,30 @@ def test_capture_failure_stays_silent(tmp_path, monkeypatch) -> None:
assert output.getvalue() == ""
def test_capture_is_a_silent_noop_during_sdk_migration(tmp_path, monkeypatch) -> None:
database = tmp_path / "one-memory.sqlite3"
marker = tmp_path / ".sdk-migration-capture-fence.json"
marker.write_text(json.dumps({"schema_version": 1, "pid": os.getpid()}))
monkeypatch.setattr("one_memory.live_capture.database_path", lambda: database)
monkeypatch.setattr(
"sys.stdin",
StringIO(
json.dumps(
{
"hook_event_name": "beforeSubmitPrompt",
"conversation_id": "session",
"prompt": "not persisted during migration",
}
)
),
)
output = StringIO()
assert cli.run(["capture", "cursor"], output=output) == 0
assert output.getvalue() == ""
assert not database.exists()
def test_codex_harness_preamble_is_not_a_user_message(tmp_path) -> None:
database = tmp_path / "one-memory.sqlite3"
_grant(tmp_path, "codex", sync=True)
+4 -2
View File
@@ -102,6 +102,8 @@ def test_stall_captures_a_safe_stack_for_every_thread() -> None:
assert watchdog.stall_stacks
captured = watchdog.stall_stacks[0]
assert any(line.startswith("thread=MainThread|") for line in captured)
assert any(line.startswith("thread=source-reader-test|") for line in captured)
assert captured
assert len(captured) <= 16
assert all(line.startswith("thread|") for line in captured)
assert all("source-reader-test" not in line and "MainThread" not in line for line in captured)
assert all("/Users/" not in line and "\n" not in line for line in captured)
@@ -2,10 +2,16 @@ from __future__ import annotations
import asyncio
import json
import sqlite3
import threading
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from hy_memory.models.memory import MemoryLayer, MemoryStatus
from one_memory.database_access import close_main_databases, main_database
from one_memory.hy_sdk_migration import HySdkMigration
from one_memory.hy_sdk_types import (
HySdkMemoryPage,
HySdkMemoryRecord,
@@ -21,6 +27,7 @@ from one_memory.sdk_data_plane_migration import (
MemoryAuthority,
SdkDataPlaneMigrationStore,
)
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
from one_memory.sdk_migration_bundle import create_sdk_migration_bundle
from one_memory.sdk_migration_inventory import (
LegacyMemoryInventory,
@@ -40,10 +47,15 @@ from one_memory.sdk_migration_inventory import (
record_raw_migration_plan,
)
from one_memory.sdk_migration_runner import (
activate_sdk_migration_staging,
commit_reconciled_cutover,
prepare_legacy_write_queue_for_cutover,
prepare_sdk_migration_staging,
recover_sdk_migration_swap,
resume_legacy_cleanup,
run_sdk_data_plane_backfill,
sdk_cutover_blockers,
sdk_migration_staging,
)
from one_memory.target_schema import legacy_tables_present
@@ -212,9 +224,11 @@ def test_startup_cutover_freezes_pending_job_input_and_preserves_the_queue(store
database_path=store.database_path,
migration_id="m1",
bundle_id="bundle-1",
sdk_version="2.0.5.dev3",
sdk_version="2.0.5",
)
assert marker.migration_id == "m1"
assert legacy_tables_present(store.database_path)
resume_legacy_cleanup(data_dir=data_dir, database_path=store.database_path)
assert legacy_tables_present(store.database_path) == ()
assert store.authority() is MemoryAuthority.SDK
preserved = DurableWriteQueue(store.database_path, sdk_authority=True)
@@ -357,6 +371,12 @@ def _legacy(
status="active",
layer="l2_fact",
):
productivity = {
"coding_experience": ("experience", "coding"),
"working_experience": ("experience", "working"),
"coding_fact": ("fact", "coding"),
"working_fact": ("fact", None),
}.get(layer)
return LegacyMemoryInventory(
legacy_memory_id=legacy_id,
canonical_memory_id=canonical_id or legacy_id,
@@ -371,16 +391,27 @@ def _legacy(
tags=(),
supersedes=(),
superseded_by=(),
productivity_kind=productivity[0] if productivity else None,
productivity_variant=productivity[1] if productivity else None,
)
def _sdk(memory_id, *, content="body", status="active"):
def _sdk(
memory_id,
*,
content="body",
status="active",
productivity_kind=None,
productivity_variant=None,
):
return SdkMemoryInventory(
memory_id=memory_id,
user_id="local",
agent_id="one-memory",
content_hash=content_hash(content),
status=status,
productivity_kind=productivity_kind,
productivity_variant=productivity_variant,
)
@@ -411,7 +442,7 @@ def test_inventory_classification_covers_every_legacy_and_sdk_state():
assert by_key["memory:delete-existing"].action == "delete"
assert by_key["memory:divergent"].action == "quarantine"
assert by_key["memory:unknown-layer"].reason == "legacy_layer_unknown"
assert by_key["sdk:sdk-only"].action == "adopt"
assert by_key["sdk:chat:sdk-only"].action == "adopt"
assert len(decisions) == 8
@@ -421,7 +452,10 @@ class _MigrationClient:
self.imported = []
self.imported_raw = []
self.deleted = []
self.deleted_productivity = []
self.deleted_raw = []
self.raw = ()
self.productivity = {}
async def list_memories(self, *, limit, offset, **_kwargs):
page = self.memories[offset : offset + limit]
@@ -432,7 +466,7 @@ class _MigrationClient:
offset=offset,
)
async def import_memory_exact(self, **kwargs):
async def write_legacy_memory(self, **kwargs):
self.imported.append(kwargs)
return {"success": True, "memory_id": kwargs["memory_id"]}
@@ -440,7 +474,35 @@ class _MigrationClient:
self.deleted.append(memory_id)
return {"success": True, "deleted_count": 1}
async def list_raw(self, *, limit, offset, **_kwargs):
async def read_productivity_memory(self, memory_id, *, kind, variant):
if variant is not None:
memory = self.productivity.get((memory_id, kind, variant))
return None if memory is None else (memory, variant)
for candidate in ("working", "coding"):
memory = self.productivity.get((memory_id, kind, candidate))
if memory is not None:
return memory, candidate
return None
async def list_productivity_memories(self):
return tuple(
{
"memory_id": memory.memory_id,
"content": memory.content,
"status": memory.status or "active",
"user_id": memory.user_id,
"agent_id": memory.agent_id,
"kind": kind,
"variant": variant,
}
for (_memory_id, kind, variant), memory in self.productivity.items()
)
async def delete_productivity_memory(self, memory_id, *, kind, variant, user_id):
self.deleted_productivity.append((memory_id, kind, variant, user_id))
return {"success": True, "deleted_count": 1}
async def read_raw_inventory(self, *, limit, offset, **_kwargs):
rows = self.raw[offset : offset + limit]
return HySdkRawPage(
raw=rows,
@@ -449,10 +511,14 @@ class _MigrationClient:
offset=offset,
)
async def import_raw(self, **kwargs):
async def write_legacy_raw(self, **kwargs):
self.imported_raw.append(kwargs)
return {"success": True, "raw_id": kwargs["raw_id"]}
async def delete_raw(self, raw_id):
self.deleted_raw.append(raw_id)
return {"success": True, "deleted_count": 1}
async def get_sdk_memory(self, memory_id):
return next(
(memory for memory in self.memories if memory.memory_id == memory_id),
@@ -491,6 +557,84 @@ def test_sdk_inventory_is_paged_without_dropping_lifecycle_records():
]
def test_sdk_inventory_adopts_mapped_productivity_ids_and_deletes_user_removed_fact(store):
active = _legacy("exp", layer="coding_experience")
deleted = _legacy("fact", layer="working_fact", status="deleted")
client = _MigrationClient()
client.productivity = {
("exp", "experience", "coding"): HySdkMemoryRecord(
memory_id="exp",
content="sdk experience",
layer="experience",
status="active",
is_latest=True,
user_id="local",
agent_id="scope",
),
("fact", "fact", "coding"): HySdkMemoryRecord(
memory_id="fact",
content="sdk fact",
layer="fact",
status="active",
is_latest=True,
user_id="local",
agent_id="scope",
),
("sdk-only", "experience", "working"): HySdkMemoryRecord(
memory_id="sdk-only",
content="sdk only",
layer="experience",
status="active",
is_latest=True,
user_id="local",
agent_id="another-scope",
),
}
sdk = asyncio.run(read_sdk_memory_inventory(client, legacy=(active, deleted)))
_start(store)
decisions = record_memory_migration_plan(
store,
migration_id="m1",
legacy=(active, deleted),
sdk=sdk,
)
by_key = {decision.item_key: decision for decision in decisions}
assert by_key["memory:exp"].action == "matched"
assert by_key["memory:fact"].action == "delete"
assert by_key["sdk:working:experience:sdk-only"].action == "adopt"
asyncio.run(
execute_memory_migration_plan(
client,
store,
migration_id="m1",
legacy=(active, deleted),
decisions=decisions,
)
)
assert client.imported == []
assert client.deleted_productivity == [("fact", "fact", None, "local")]
def test_productivity_id_in_both_variants_is_quarantined():
legacy = (_legacy("fact", layer="working_fact"),)
sdk = (
_sdk(
"fact",
productivity_kind="fact",
productivity_variant="working",
),
_sdk(
"fact",
productivity_kind="fact",
productivity_variant="coding",
),
)
(decision, *_sdk_only) = classify_memory_inventory(legacy, sdk)
assert decision.action == "quarantine"
assert decision.reason == "sdk_memory_id_ambiguous"
def test_recorded_plan_executes_import_delete_alias_and_adoption(store):
_start(store)
legacy = (
@@ -529,32 +673,218 @@ def test_recorded_plan_executes_import_delete_alias_and_adoption(store):
assert progress.counts == {"adopted": 1, "deleted": 2, "verified": 3}
def test_legacy_raw_inventory_preserves_exact_source_metadata(store):
def test_legacy_raw_inventory_builds_recent_user_assistant_turns_without_tools(store):
database = store.database_path
with main_database(database).write() as connection, connection:
connection.execute(
connection.executemany(
"""
INSERT INTO evidence (
evidence_id,idempotency_key,user_id,source_id,workspace,project,
conversation_id,message_id,source_ordinal,content_kind,role,
written_by,created_at,verbatim,content_hash,revision,
imported_at,updated_at,ingest_seq
) VALUES (
'e1','key','local','cursor','/repo','project','session','message',
'7','raw_conversation','user','person','2026-10-01T00:00:00+00:00',
'verbatim',?,2,'2026-10-02T00:00:00+00:00',
'2026-10-02T00:00:00+00:00',1
)
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
""",
(content_hash("verbatim"),),
(
(
"u1",
"k1",
"local",
"cursor",
"/repo",
"project",
"session",
"m1",
"1",
"raw_conversation",
"user",
"person",
"2026-10-22T00:00:00+00:00",
"question",
content_hash("question"),
1,
"2026-10-22T00:00:00+00:00",
"2026-10-22T00:00:00+00:00",
1,
),
(
"t1",
"k2",
"local",
"cursor",
"/repo",
"project",
"session",
"m2",
"2",
"raw_conversation",
"tool",
"tool",
"2026-10-22T00:00:01+00:00",
"secret tool output",
content_hash("secret tool output"),
1,
"2026-10-22T00:00:01+00:00",
"2026-10-22T00:00:01+00:00",
2,
),
(
"a1",
"k3",
"local",
"cursor",
"/repo",
"project",
"session",
"m3",
"3",
"raw_conversation",
"assistant",
"agent",
"2026-10-22T00:00:02+00:00",
"answer",
content_hash("answer"),
1,
"2026-10-22T00:00:02+00:00",
"2026-10-22T00:00:02+00:00",
3,
),
(
"old-u",
"k4",
"local",
"cursor",
"/repo",
"project",
"old-session",
"m4",
"4",
"raw_conversation",
"user",
"person",
"2026-10-01T00:00:00+00:00",
"old question",
content_hash("old question"),
1,
"2026-10-01T00:00:00+00:00",
"2026-10-01T00:00:00+00:00",
4,
),
(
"old-a",
"k5",
"local",
"cursor",
"/repo",
"project",
"old-session",
"m5",
"5",
"raw_conversation",
"assistant",
"agent",
"2026-10-01T00:00:01+00:00",
"old answer",
content_hash("old answer"),
1,
"2026-10-01T00:00:01+00:00",
"2026-10-01T00:00:01+00:00",
5,
),
),
)
(raw,) = read_legacy_raw_inventory(database)
assert raw.evidence_id == "e1"
(raw,) = read_legacy_raw_inventory(
database,
cutoff=datetime.fromisoformat("2026-10-20T00:00:00+00:00"),
)
assert raw.evidence_id.startswith("turn:")
assert raw.session_id == "session"
assert raw.source == "cursor"
assert raw.metadata["role"] == "user"
assert raw.metadata["revision"] == 2
assert raw.memory_at.isoformat() == "2026-10-01T00:00:00+00:00"
assert raw.agent_id == "all-apps"
assert raw.content == "[user]: question\n[assistant]: answer"
assert raw.metadata["evidence_ids"] == ["u1", "a1"]
assert "tool" not in raw.content
assert raw.memory_at.isoformat() == "2026-10-22T00:00:00+00:00"
def test_raw_turn_cutoff_uses_completion_time_and_keeps_workspaces_separate(store):
database = store.database_path
rows = (
("u0", "k0", "shared", "w0", "user", "person", "before", "2026-10-19T23:59:00+00:00", 1),
("a0", "k1", "shared", "w0", "assistant", "agent", "after", "2026-10-20T00:01:00+00:00", 2),
("u1", "k2", "same-session", "w1", "user", "person", "one", "2026-10-21T00:00:00+00:00", 3),
(
"a1",
"k3",
"same-session",
"w1",
"assistant",
"agent",
"first",
"2026-10-21T00:01:00+00:00",
4,
),
("u2", "k4", "same-session", "w2", "user", "person", "two", "2026-10-21T00:02:00+00:00", 5),
(
"a2",
"k5",
"same-session",
"w2",
"assistant",
"agent",
"second",
"2026-10-21T00:03:00+00:00",
6,
),
)
with main_database(database).write() as connection, connection:
connection.executemany(
"""
INSERT INTO evidence (
evidence_id,idempotency_key,user_id,source_id,conversation_id,
workspace,content_kind,role,written_by,verbatim,content_hash,
revision,created_at,imported_at,updated_at,ingest_seq
) VALUES (?,?,'local','cursor',?,?,'raw_conversation',?,?,?,? ,1,?,?,?,?)
""",
tuple(
(
evidence_id,
key,
session,
workspace,
role,
written_by,
text,
content_hash(text),
timestamp,
timestamp,
timestamp,
sequence,
)
for (
evidence_id,
key,
session,
workspace,
role,
written_by,
text,
timestamp,
sequence,
) in rows
),
)
turns = read_legacy_raw_inventory(
database,
cutoff=datetime.fromisoformat("2026-10-20T00:00:00+00:00"),
)
assert {turn.content for turn in turns} == {
"[user]: before\n[assistant]: after",
"[user]: one\n[assistant]: first",
"[user]: two\n[assistant]: second",
}
assert {turn.metadata["workspace"] for turn in turns} == {"w0", "w1", "w2"}
def _legacy_raw(evidence_id, *, content="verbatim"):
@@ -582,7 +912,7 @@ def _sdk_raw(raw_id, *, content="verbatim"):
)
def test_raw_inventory_matches_exact_scope_imports_missing_and_adopts_sdk_only(store):
def test_raw_inventory_matches_turns_imports_missing_and_deletes_noncanonical_sdk_raw(store):
legacy = (_legacy_raw("e1"), _legacy_raw("e2", content="missing"))
sdk = (_sdk_raw("raw-existing"), _sdk_raw("raw-sdk-only", content="sdk-only"))
decisions = classify_raw_inventory(legacy, sdk)
@@ -591,7 +921,7 @@ def test_raw_inventory_matches_exact_scope_imports_missing_and_adopts_sdk_only(s
assert by_key["raw:e1"].raw_id == "raw-existing"
assert by_key["raw:e2"].action == "import_exact"
assert by_key["raw:e2"].raw_id == "evidence:e2"
assert by_key["sdk-raw:raw-sdk-only"].action == "adopt"
assert by_key["sdk-raw:raw-sdk-only"].action == "delete"
_start(store)
recorded = record_raw_migration_plan(
@@ -611,7 +941,8 @@ def test_raw_inventory_matches_exact_scope_imports_missing_and_adopts_sdk_only(s
)
)
assert [item["raw_id"] for item in client.imported_raw] == ["evidence:e2"]
assert store.progress("m1").counts == {"adopted": 1, "verified": 2}
assert client.deleted_raw == ["raw-sdk-only"]
assert store.progress("m1").counts == {"deleted": 1, "verified": 2}
def test_sdk_raw_inventory_pages_all_records():
@@ -666,6 +997,20 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
""",
(content_hash("verbatim"),),
)
connection.execute(
"""
INSERT INTO evidence (
evidence_id,idempotency_key,user_id,source_id,conversation_id,
content_kind,role,written_by,verbatim,content_hash,revision,
imported_at,updated_at,ingest_seq
) VALUES (
'a1','assistant-key','local','cursor','session','raw_conversation',
'assistant','agent','reply',?,1,
'2026-10-21T00:00:01+00:00','2026-10-21T00:00:01+00:00',2
)
""",
(content_hash("reply"),),
)
client = _MigrationClient()
result = asyncio.run(
run_sdk_data_plane_backfill(
@@ -674,7 +1019,7 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
migration_id="migration",
bundle_id="bundle",
source_runtime_build="v1.0.4",
target_sdk_version="2.0.5.dev3",
target_sdk_version="2.0.5",
source_schema_version=55,
namespace_fingerprint="namespace",
backend_fingerprint="backend",
@@ -685,7 +1030,9 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
assert result.phase == "reconciled"
assert result.counts == {"verified": 2}
assert [item["memory_id"] for item in client.imported] == ["memory-1"]
assert [item["raw_id"] for item in client.imported_raw] == ["evidence:e1"]
assert len(client.imported_raw) == 1
assert client.imported_raw[0]["content"] == "[user]: verbatim\n[assistant]: reply"
migrated_raw_id = client.imported_raw[0]["raw_id"]
assert store.authority() is MemoryAuthority.LEGACY
# Simulate a process dying after both SDK side effects but before startup
# could finish. The next inventory observes them as matched rather than
@@ -694,13 +1041,13 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
client.memories = (_record("memory-1", content="preference"),)
client.raw = (
HySdkRawRecord(
raw_id="evidence:e1",
content="verbatim",
raw_id=migrated_raw_id,
content="[user]: verbatim\n[assistant]: reply",
user_id="local",
agent_id="one-memory",
agent_id="all-apps",
session_id="session",
source="cursor",
custom={"role": "user", "written_by": "person"},
custom={"input_kind": "legacy_turn"},
),
)
resumed = asyncio.run(
@@ -710,7 +1057,7 @@ def test_offline_backfill_reaches_ready_without_committing_cutover(store):
migration_id="migration",
bundle_id="bundle",
source_runtime_build="v1.0.4",
target_sdk_version="2.0.5.dev3",
target_sdk_version="2.0.5",
source_schema_version=55,
namespace_fingerprint="namespace",
backend_fingerprint="backend",
@@ -743,7 +1090,7 @@ def test_offline_backfill_blocks_cutover_when_an_item_is_quarantined(store):
migration_id="migration",
bundle_id="bundle",
source_runtime_build="v1.0.4",
target_sdk_version="2.0.5.dev3",
target_sdk_version="2.0.5",
source_schema_version=55,
namespace_fingerprint="namespace",
backend_fingerprint="backend",
@@ -754,3 +1101,330 @@ def test_offline_backfill_blocks_cutover_when_an_item_is_quarantined(store):
assert result.phase == "review_required"
assert result.counts == {"quarantine": 1}
assert store.authority() is MemoryAuthority.LEGACY
class _MigrationLoop:
def run(self, coroutine):
return asyncio.run(coroutine)
class _MigrationVectorStore:
def __init__(self):
self.nodes = {}
async def get_by_id(self, memory_id):
return self.nodes.get(memory_id)
async def upsert(self, node):
self.nodes[node.node_id] = node
return node.node_id
async def list_by_user(self, *, user_id, layers, **_kwargs):
return [
node for node in self.nodes.values() if node.user_id == user_id and node.layer in layers
]
class _MigrationEmbedder:
async def embed(self, _content):
return [0.1, 0.2]
class _MigrationGraphStore:
def __init__(self):
self.nodes = {}
async def upsert_memory_node(self, node):
self.nodes[node.node_id] = node
return node.node_id
class _MigrationFactVector:
def __init__(self):
self.deleted = []
async def delete(self, memory_id):
self.deleted.append(memory_id)
return True
class _MigrationCodingStore:
def __init__(self):
self._conn = sqlite3.connect(":memory:", check_same_thread=False)
self._conn.row_factory = sqlite3.Row
self._conn.execute(
"CREATE TABLE coding_memory_meta ("
"experience_id TEXT PRIMARY KEY,user_id TEXT,agent_id TEXT,solution TEXT)"
)
self._conn.execute(
"CREATE TABLE working_fact ("
"fact_id TEXT PRIMARY KEY,user_id TEXT,agent_id TEXT,"
"statement TEXT,status TEXT)"
)
self._conn.execute(
"INSERT INTO working_fact VALUES ('fact-1','local','scope','fact body','active')"
)
self._conn.commit()
self._lock = threading.Lock()
self._fact_vec = _MigrationFactVector()
async def get_fact(self, memory_id, *, user_id):
row = self._conn.execute(
"SELECT 1 FROM working_fact WHERE fact_id=? AND user_id=?",
(memory_id, user_id),
).fetchone()
return object() if row is not None else None
class _PrivilegedMigrationClient:
def __init__(self):
self._loop_thread = _MigrationLoop()
self._vector_store = _MigrationVectorStore()
self._embed_service = _MigrationEmbedder()
self._graph_store = _MigrationGraphStore()
self._coding = _MigrationCodingStore()
async def _coding_store_for_exact_get(self, variant):
return self._coding if variant == "coding" else None
def list_memories(self, **_kwargs):
return {"vdb": {"memories": [], "total": 0}, "request_id": "list"}
def delete(self, _memory_id):
return {"success": True, "deleted_count": 1}
def get_write_outcome(self, _request_id):
return None
def test_offline_migration_writes_sdk_stores_without_public_client_import_methods():
client = _PrivilegedMigrationClient()
migration = HySdkMigration(client)
raw_result = asyncio.run(
migration.write_legacy_raw(
content="verbatim",
raw_id="raw-1",
user_id="local",
agent_id="one-memory",
session_id="session",
source="cursor",
memory_at=None,
metadata={"role": "user"},
request_id="migrate-raw-1",
)
)
memory_result = asyncio.run(
migration.write_legacy_memory(
content="prefers tea",
memory_id="memory-1",
user_id="local",
agent_id="one-memory",
session_id="session",
source="legacy-governance",
layer="l2_fact",
status="active",
owner="user",
tags=("preference",),
supersedes=(),
superseded_by=(),
is_latest=True,
memory_at=None,
metadata={},
request_id="migrate-memory-1",
)
)
raw_page = asyncio.run(migration.read_raw_inventory(user_id="local"))
productivity = asyncio.run(migration.list_productivity_memories())
deleted = asyncio.run(
migration.delete_productivity_memory(
"fact-1",
kind="fact",
variant=None,
user_id="local",
)
)
assert raw_result["raw_id"] == "raw-1"
assert memory_result["memory_id"] == "memory-1"
assert [item.raw_id for item in raw_page.raw] == ["raw-1"]
assert productivity == (
{
"memory_id": "fact-1",
"content": "fact body",
"status": "active",
"user_id": "local",
"agent_id": "scope",
"kind": "fact",
"variant": "coding",
},
)
assert client._vector_store.nodes["raw-1"].layer is MemoryLayer.L1_RAW
assert client._vector_store.nodes["memory-1"].status is MemoryStatus.ACTIVE
assert client._graph_store.nodes["memory-1"].content == "prefers tea"
assert deleted["deleted_count"] == 1
assert client._coding._fact_vec.deleted == ["fact-1"]
assert client._coding._conn.execute("SELECT COUNT(*) FROM working_fact").fetchone()[0] == 0
assert not hasattr(client, "import_raw")
assert not hasattr(client, "import_memory_exact")
def test_raw_http_dto_uses_the_unified_memory_list_shape():
page = HySdkRawPage.from_memory_list(
{
"vdb": {
"memories": [
{
"memory_id": "raw-1",
"content": "verbatim",
"layer": "l1_raw",
"status": "active",
"user_id": "local",
"agent_id": "one-memory",
"session_id": "session",
"source": "cursor",
"custom": {"role": "user"},
}
],
"total": 1,
"limit": 100,
"offset": 0,
}
},
limit=100,
offset=0,
)
assert page.raw[0].raw_id == "raw-1"
assert page.raw[0].custom == {"role": "user"}
def test_sdk_raw_retention_keeps_only_the_latest_three_days():
now = datetime(2026, 10, 23, tzinfo=UTC)
class Client:
def __init__(self):
self.rows = [
{
"memory_id": "old",
"memory_at": int((now - timedelta(days=4)).timestamp()),
},
{
"memory_id": "recent",
"memory_at": int((now - timedelta(days=2)).timestamp()),
},
]
def list_memories(self, **_kwargs):
return {
"vdb": {
"memories": list(self.rows),
"total": len(self.rows),
"limit": 500,
"offset": 0,
}
}
def delete(self, memory_id):
before = len(self.rows)
self.rows = [row for row in self.rows if row["memory_id"] != memory_id]
return {
"success": True,
"deleted_count": before - len(self.rows),
}
client = Client()
class Handle:
async def client(self):
return client
runtime = SdkMemoryRuntime(
coordinator=None,
read_handle=Handle(),
write_queue=None,
extraction_settings=None,
data_dir=Path("."),
)
assert asyncio.run(runtime.prune_expired_raw(now=now)) == 1
assert [row["memory_id"] for row in client.rows] == ["recent"]
def _staging_fixture(tmp_path, store):
data_dir = tmp_path / "home"
live = data_dir / "hy-memory" / "governed-v2"
live.mkdir(parents=True)
(live / "sdk-only.txt").write_text("old", encoding="utf-8")
paths = sdk_migration_staging(
data_dir=data_dir,
migration_id="migration-1",
namespace="governed-v2",
)
prepare_sdk_migration_staging(
paths=paths,
database_path=store.database_path,
)
return paths
def test_staging_preserves_sdk_only_data_and_swaps_after_commit(tmp_path, store):
paths = _staging_fixture(tmp_path, store)
staged = paths.staging_root / "sdk-only.txt"
assert staged.read_text(encoding="utf-8") == "old"
staged.write_text("migrated", encoding="utf-8")
marker = object()
result = activate_sdk_migration_staging(
paths=paths,
database_path=store.database_path,
commit=lambda: marker,
)
assert result is marker
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
assert not paths.protective_root.exists()
assert not paths.staging_home.exists()
def test_incomplete_staging_retry_restarts_from_unchanged_live_root(tmp_path, store):
paths = _staging_fixture(tmp_path, store)
(paths.staging_root / "sdk-only.txt").write_text("partial", encoding="utf-8")
prepare_sdk_migration_staging(
paths=paths,
database_path=store.database_path,
)
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
def test_failed_cutover_restores_live_store_and_keeps_staging(tmp_path, store):
paths = _staging_fixture(tmp_path, store)
(paths.staging_root / "sdk-only.txt").write_text("migrated", encoding="utf-8")
def fail():
raise DataPlaneMigrationError("cutover_failed")
with pytest.raises(DataPlaneMigrationError, match="cutover_failed"):
activate_sdk_migration_staging(
paths=paths,
database_path=store.database_path,
commit=fail,
)
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
def test_restart_recovers_swap_when_authority_was_not_committed(tmp_path, store):
paths = _staging_fixture(tmp_path, store)
(paths.staging_root / "sdk-only.txt").write_text("migrated", encoding="utf-8")
paths.live_root.replace(paths.protective_root)
paths.staging_root.replace(paths.live_root)
recover_sdk_migration_swap(
paths=paths,
database_path=store.database_path,
)
assert (paths.live_root / "sdk-only.txt").read_text(encoding="utf-8") == "old"
assert (paths.staging_root / "sdk-only.txt").read_text(encoding="utf-8") == "migrated"
@@ -91,7 +91,7 @@ def test_bundle_contains_management_sdk_config_and_verified_manifest(tmp_path):
assert manifest["bundle_id"] == "migration-1"
assert manifest["sdk_namespace"] == "governed-v2"
assert manifest["runtime_build"] == "build-13"
assert manifest["snapshot_sdk_version"] == "2.0.5.dev3"
assert manifest["snapshot_sdk_version"] == "2.0.5"
paths = {item["path"]: item for item in manifest["files"]}
assert {
"management/one-memory.sqlite3",
@@ -0,0 +1,38 @@
"""来源根目录类型在三处必须同时存在:python 枚举、TS 契约、前端运行时校验。
起因(issue #139):「我的应用」永久空白,所有接口 200,但前端 sources 恒为空。
根因是 web 侧 `isSourceResponse` 手写了一份类型清单,只列了 default 与
environment_override;Core 后来加了 user_selected(用户自己选的目录,
`sources/connect` 带 source_roots 时就是它)。一行这样的记录就会让整张列表
被拒,而承载它的 prefetch 不允许抛错上报 —— 于是「应用列表暂时不可读取」,
没有任何一处会红。
TS 联合类型漏一个成员不会让构建失败,这里补上那道守卫;前端改成从契约取清单
之后,剩下的风险只有 python 与 TS 之间再漂移。
"""
import re
from pathlib import Path
from one_memory.imports.sources.registry import RootCandidateKind
REPO_ROOT = Path(__file__).resolve().parents[3]
CONTRACT = REPO_ROOT / "packages" / "contracts" / "src" / "index.ts"
def _typescript_kinds() -> set[str]:
source = CONTRACT.read_text(encoding="utf-8")
match = re.search(
r"export const CANDIDATE_KINDS = \[(.*?)\] as const;", source, re.DOTALL
)
assert match, f"没在 {CONTRACT} 里找到 CANDIDATE_KINDS"
return set(re.findall(r'"([a-z_]+)"', match.group(1)))
def test_python_and_the_contract_name_the_same_kinds() -> None:
python_kinds = {kind.value for kind in RootCandidateKind}
assert python_kinds == _typescript_kinds(), (
"来源根目录类型在 python 与 TS 契约之间不一致:"
f"python={sorted(python_kinds)} ts={sorted(_typescript_kinds())}。"
f"请同步 {CONTRACT.relative_to(REPO_ROOT)} 的 CANDIDATE_KINDS。"
)
+5 -5
View File
@@ -991,7 +991,7 @@ wheels = [
[[package]]
name = "hy-memory"
version = "2.0.5.dev3"
version = "2.0.5"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "chromadb" },
@@ -1008,9 +1008,9 @@ dependencies = [
{ name = "scikit-learn" },
{ name = "tiktoken" },
]
sdist = { url = "https://files.pythonhosted.org/packages/cb/83/67c261bc379eb45f1cc460cf55eb057867d485695c2799bd861827dbcedb/hy_memory-2.0.5.dev3.tar.gz", hash = "sha256:8ae8c2b2d9736b304d26654a5c2903160f430ec5065206ae724d5cb841820d75", size = 836317, upload-time = "2026-10-21T09:59:27.311Z" }
sdist = { url = "https://files.pythonhosted.org/packages/94/56/7a279122bb56b067c390b748b8a83b6a90bb2a8f237c1d3989b6317681cd/hy_memory-2.0.5.tar.gz", hash = "sha256:bfb1e9e9815377477702327fa318331d555b8256a67b5bad8d70d6a682bd9a4f", size = 839060, upload-time = "2026-10-22T12:59:30.502Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b6/5f/30a88b0278c070c63d11f0c25a360e1cfe130d90a53a5eece9359c4ca7ec/hy_memory-2.0.5.dev3-py3-none-any.whl", hash = "sha256:621193ac3a527c0b8736d66f17c4cce4956dc4a882d9dfc724b9df4d34fe816e", size = 900779, upload-time = "2026-10-21T09:59:24.716Z" },
{ url = "https://files.pythonhosted.org/packages/c5/97/090ca18b6a13d0216d2df607cb67732ff7a8eba2d0359b9ad95283b2cbdc/hy_memory-2.0.5-py3-none-any.whl", hash = "sha256:c24649d188eaa03d3b4e4e4ec20e818da884170b5f775f3cf23ecfa6b60523d6", size = 902923, upload-time = "2026-10-22T12:59:28.018Z" },
]
[[package]]
@@ -1801,7 +1801,7 @@ wheels = [
[[package]]
name = "one-memory"
version = "1.0.4"
version = "1.0.5"
source = { editable = "." }
dependencies = [
{ name = "aiomysql" },
@@ -1837,7 +1837,7 @@ requires-dist = [
{ name = "fastapi", specifier = ">=0.141.1" },
{ name = "filelock", specifier = ">=3.32.4" },
{ name = "httpx", specifier = ">=0.28.1" },
{ name = "hy-memory", specifier = "==2.0.5.dev3" },
{ name = "hy-memory", specifier = "==2.0.5" },
{ name = "keyring", specifier = ">=25.7.0" },
{ name = "numpy", specifier = "==2.5.2" },
{ name = "onnxruntime", specifier = "==1.23.2" },
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/desktop",
"version": "1.0.4",
"version": "1.0.5",
"private": true,
"main": "dist/main.js",
"scripts": {
@@ -94,12 +94,14 @@ it("returns only diagnostic events written during this desktop launch", async ()
await mkdir(logs, { recursive: true });
const target = path.join(logs, "diagnostic-core.jsonl");
const old = {
schema_version: 1,
timestamp: "2026-10-14T09:00:00.000Z",
level: "info",
service: "core",
component: "core.startup",
event: "core.old",
outcome: "succeeded",
session_id: "11111111-1111-4111-8111-111111111111",
};
await writeFile(target, `${JSON.stringify(old)}\n`);
const cursor = await captureDebugLogCursor(dataDir);
@@ -109,6 +111,11 @@ it("returns only diagnostic events written during this desktop launch", async ()
timestamp: "2026-10-14T10:01:00.000Z",
event: "core.current",
level: "warn",
details: {
count: 1,
"custom.private": "/Users/private/query.txt",
credential_hint: "ghp_private_token",
},
};
await appendFile(target, `${JSON.stringify(current)}\nnot-json\n`);
@@ -118,8 +125,11 @@ it("returns only diagnostic events written during this desktop launch", async ()
expect(snapshot.entries[0]).toMatchObject({
event: "core.current",
level: "warn",
details: { count: 1 },
source: "diagnostic",
});
expect(JSON.stringify(snapshot)).not.toContain("/Users/private");
expect(JSON.stringify(snapshot)).not.toContain("ghp_private_token");
});
it("tracks diagnostic JSONL across rotation without rereading the old prefix", async () => {
@@ -130,12 +140,14 @@ it("tracks diagnostic JSONL across rotation without rereading the old prefix", a
const target = path.join(logs, "diagnostic-core.jsonl");
const event = (name: string, timestamp: string) =>
JSON.stringify({
schema_version: 1,
timestamp,
level: "info",
service: "core",
component: "core.test",
event: name,
outcome: "succeeded",
session_id: "22222222-2222-4222-8222-222222222222",
});
await writeFile(target, `${event("before", "2026-10-14T09:00:00.000Z")}\n`);
const cursor = await captureDebugLogCursor(dataDir);
+51 -51
View File
@@ -1,10 +1,13 @@
import { lstat, open, readdir } from "node:fs/promises";
import path from "node:path";
import {
type DiagnosticEventV1,
type DiagnosticLogger,
diagnosticEvent,
diagnosticZipFilename,
exportDiagnosticZip,
inspectBackgroundHealth,
projectDiagnosticEventForSafeOutput,
recordDiagnosticMarker,
} from "@one-memory/runtime-control";
import type { BrowserWindow, IpcMainInvokeEvent } from "electron";
@@ -370,42 +373,58 @@ function debugLogEntry(
const timestamp = typeof value.timestamp === "string" ? value.timestamp : "";
const timestampMs = Date.parse(timestamp);
if (!Number.isFinite(timestampMs) || timestampMs < startedAt) return null;
const level = value.level;
const service = value.service;
const component = value.component;
const event = value.event;
const outcome = value.outcome;
if (
!["info", "warn", "error"].includes(String(level)) ||
!["runtime", "core", "host-bridge", "desktop", "cli"].includes(
String(service),
) ||
typeof component !== "string" ||
typeof event !== "string" ||
typeof outcome !== "string"
) {
let projected: DiagnosticEventV1;
try {
projected = projectDiagnosticEventForSafeOutput(
diagnosticEvent({
timestamp,
level: value.level as never,
service: value.service as never,
component: value.component as never,
event: value.event as never,
outcome: value.outcome as never,
session_id: value.session_id as never,
...(typeof value.trace_id === "string"
? { trace_id: value.trace_id }
: {}),
...(typeof value.duration_ms === "number"
? { duration_ms: value.duration_ms }
: {}),
...(typeof value.error_code === "string"
? { error_code: value.error_code }
: {}),
...(typeof value.error_type === "string"
? { error_type: value.error_type }
: {}),
...(value.details && typeof value.details === "object"
? { details: value.details as never }
: {}),
}),
);
} catch {
return null;
}
const details = safeDebugDetails(value.details);
return {
id: `${filename}:${lineNumber}:${timestamp}`,
timestamp,
level: level as DebugLogEntry["level"],
service: service as DebugLogEntry["service"],
component,
event,
outcome,
...(typeof value.duration_ms === "number"
? { duration_ms: value.duration_ms }
: {}),
...(typeof value.error_code === "string"
? { error_code: value.error_code }
: {}),
...(typeof value.error_type === "string"
? { error_type: value.error_type }
: {}),
...(typeof value.trace_id === "string" ? { trace_id: value.trace_id } : {}),
...(details === undefined ? {} : { details }),
timestamp: projected.timestamp,
level: projected.level,
service: projected.service,
component: projected.component,
event: projected.event,
outcome: projected.outcome,
...(projected.duration_ms === undefined
? {}
: { duration_ms: projected.duration_ms }),
...(projected.error_code === undefined
? {}
: { error_code: projected.error_code }),
...(projected.error_type === undefined
? {}
: { error_type: projected.error_type }),
...(projected.trace_id === undefined
? {}
: { trace_id: projected.trace_id }),
...(projected.details === undefined ? {} : { details: projected.details }),
source: "diagnostic",
};
}
@@ -523,25 +542,6 @@ function logLineTimestamp(
return new Date(Math.max(startedAt, fallback)).toISOString();
}
function safeDebugDetails(
value: unknown,
): Record<string, string | number | boolean | null> | undefined {
if (!value || typeof value !== "object" || Array.isArray(value))
return undefined;
const details: Record<string, string | number | boolean | null> = {};
for (const [key, item] of Object.entries(value as Record<string, unknown>)) {
if (
item === null ||
typeof item === "string" ||
typeof item === "number" ||
typeof item === "boolean"
) {
details[key] = item;
}
}
return Object.keys(details).length ? details : undefined;
}
async function pipelineLogFiles(root: string): Promise<PipelineLogCursor[]> {
const rootInfo = await lstat(root).catch(() => null);
if (!rootInfo?.isDirectory() || rootInfo.isSymbolicLink()) return [];
+8 -3
View File
@@ -113,10 +113,14 @@ import {
installServiceWithDisabledFallback,
isInstalledApplicationsApp,
stableRuntimeErrorCode,
stableRuntimeFailureDetail,
startStableRuntimeRepair,
} from "./stable-runtime";
import { STARTUP_SCREEN_URL, startupScreenHtml } from "./startup-screen";
import { runStartupSdkMigration } from "./startup-sdk-migration";
import {
requireCompletedStartupSdkMigration,
runStartupSdkMigration,
} from "./startup-sdk-migration";
import { resolveTrialProfile } from "./trial-profile";
import { createWindowOptions } from "./window-options";
@@ -1095,8 +1099,9 @@ async function initialize(isQuitting: () => boolean): Promise<void> {
? "blocked"
: "skipped",
...(migration.code === undefined ? {} : { error_code: migration.code }),
details: { state: migration.state },
details: { runtime_state: migration.state },
});
requireCompletedStartupSdkMigration(migration);
};
try {
@@ -1667,7 +1672,7 @@ if (desktopVersion !== null) {
languagePreferences?.read().locale ?? "zh-CN",
"nativeErrorTitle",
),
error instanceof Error ? error.message : "desktop_unavailable",
stableRuntimeFailureDetail(error),
);
},
});
@@ -16,6 +16,7 @@ import {
isDisabledServiceInstallError,
isInstalledApplicationsApp,
stableRuntimeErrorCode,
stableRuntimeFailureDetail,
stableRuntimeUserMessage,
startStableRuntimeRepair,
} from "./stable-runtime";
@@ -274,4 +275,46 @@ describe("startup install errors", () => {
"MemHub 启动失败",
);
});
it("says why a data upgrade failed, and what the person can do", () => {
const noSpace = stableRuntimeUserMessage(
new Error("sdk_migration_disk_space_insufficient"),
);
// The only failure here the person fixes themselves: name the action.
expect(noSpace).toContain("磁盘空间不足");
expect(noSpace).toContain("重新启动");
expect(noSpace).toContain("不会丢失");
expect(noSpace).not.toContain("sdk_migration_disk_space_insufficient");
const noModel = stableRuntimeUserMessage(
new Error("model_profile_not_active"),
);
expect(noModel).toContain("模型");
expect(noModel).not.toContain("model_profile_not_active");
const coding = stableRuntimeUserMessage(
new Error("sdk_migration_working_fact_conflict"),
);
expect(coding).toContain("Coding");
expect(coding).toContain("不会丢失");
});
});
describe("startup dialog body", () => {
it("keeps a code we have no copy for, so support can see which failure fired", () => {
expect(stableRuntimeFailureDetail(new Error("page_load_failed"))).toBe(
"page_load_failed",
);
expect(
stableRuntimeFailureDetail(new Error("runtime_broker_unavailable")),
).toBe("runtime_broker_unavailable");
});
it("shows the sentence we wrote for a migration failure", () => {
const detail = stableRuntimeFailureDetail(
new Error("sdk_migration_disk_space_insufficient"),
);
expect(detail).toContain("磁盘空间不足");
expect(detail).not.toContain("sdk_migration_disk_space_insufficient");
});
});
+58
View File
@@ -126,6 +126,23 @@ const SAFE_STABLE_RUNTIME_ERROR_CODES = new Set([
"installed_version_conflict",
"distribution_downgrade_refused",
"install_current_unmanaged",
// Migration runs before the app window exists, so these are the only names
// a first launch can be given. They stay codes here and become sentences in
// STABLE_RUNTIME_USER_MESSAGES: a person reading the dialog never sees them.
"model_profile_not_active",
"model_secret_unavailable",
"sdk_migration_preflight_failed",
"sdk_migration_disk_space_insufficient",
"sdk_migration_staging_failed",
"sdk_migration_backfill_failed",
"sdk_migration_working_fact_conflict",
"sdk_migration_coding_inventory_failed",
"sdk_migration_work_pending",
"sdk_migration_cutover_failed",
"sdk_migration_bundle_copy_failed",
"sdk_migration_response_invalid",
"sdk_migration_status_unavailable",
"sdk_migration_incomplete",
]);
export function stableRuntimeErrorCode(error: unknown): string {
@@ -191,6 +208,34 @@ export const STABLE_RUNTIME_USER_MESSAGES: Record<string, string> = {
"已安装的运行环境比这个安装包更新,无法用旧安装包启动。请安装最新版本的 MemHub;你的记忆数据不会受到影响。",
install_current_unmanaged:
"运行环境的安装目录被改动过,无法安全接管。请重新安装 MemHub,你的记忆数据不会受到影响。",
model_profile_not_active:
"还没有配置可用的模型,无法升级记忆数据格式。请在旧版本中完成模型配置后重启;你的记忆数据不会受到影响。",
model_secret_unavailable:
"模型密钥读不到(可能被系统钥匙串拒绝)。请允许 MemHub 访问钥匙串后重新启动;你的记忆数据不会受到影响。",
sdk_migration_disk_space_insufficient:
"磁盘空间不足,升级记忆数据格式时中断了。请至少腾出与记忆数据同等大小的可用空间(可在「关于本机 → 储存空间」查看),然后重新启动 MemHub;升级会自动继续,原有记忆数据不会丢失。",
sdk_migration_staging_failed:
"升级准备工作没有完成,通常是磁盘空间不足或文件被占用。请清理磁盘空间后重新启动;原有记忆数据不会丢失。",
sdk_migration_bundle_copy_failed:
"升级前备份没有写完,通常是磁盘空间不足。请清理磁盘空间后重新启动;原有记忆数据不会丢失。",
sdk_migration_working_fact_conflict:
"旧版数据里有一条 Coding 记忆的记录类型在升级中无法安全对应。为避免写错你的记忆,升级已停止。请把这个提示反馈给我们;原有记忆数据不会丢失。",
sdk_migration_coding_inventory_failed:
"读取旧版 Coding 记忆时发现无法识别的结构,升级已停止。请把这个提示反馈给我们;原有记忆数据不会丢失。",
sdk_migration_backfill_failed:
"升级记忆数据格式时失败,已停止且没有切换数据源。请重新启动重试;如果仍然失败,请把这个提示反馈给我们,原有记忆数据不会丢失。",
sdk_migration_work_pending:
"记忆数据已经升级完成,但还有之前没处理完的记忆任务,所以暂时不能切换数据源。请保持 MemHub 打开几分钟,等这些任务处理完再重新启动;原有记忆数据不会丢失。",
sdk_migration_cutover_failed:
"切换数据源时失败,原有记忆数据仍然完整可用。请重新启动重试;如果仍然失败,请把这个提示反馈给我们。",
sdk_migration_preflight_failed:
"升级前的检查没有通过,升级已停止。请重新启动重试;如果仍然失败,请把这个提示反馈给我们,原有记忆数据不会丢失。",
sdk_migration_status_unavailable:
"读不到升级状态,无法判断是否需要升级。请重新启动 MemHub;如果仍然失败,请重新安装,你的记忆数据不会受到影响。",
sdk_migration_response_invalid:
"升级过程返回了无法识别的内容,已停止。原有记忆数据不会丢失;请重新启动重试,如果仍然失败,请把这个提示反馈给我们。",
sdk_migration_incomplete:
"记忆数据格式升级没有完成,MemHub 暂时无法打开。请重新启动重试;如果仍然失败,请把这个提示反馈给我们,原有记忆数据不会丢失。",
desktop_start_failed:
"MemHub 启动失败。请重新启动;如果仍然打不开,请重新安装,你的记忆数据不会受到影响。",
};
@@ -203,3 +248,16 @@ export function stableRuntimeUserMessage(error: unknown): string {
""
);
}
/** What the startup dialog body says.
*
* A code we wrote copy for becomes a sentence. Every other code is shown as it
* is: this dialog is the last thing a person sees before the app closes, and an
* unexpected failure is more useful as the code support can look up than as a
* reassuring sentence that erases which failure actually fired. */
export function stableRuntimeFailureDetail(error: unknown): string {
if (!(error instanceof Error)) {
return STABLE_RUNTIME_USER_MESSAGES.desktop_start_failed ?? "";
}
return STABLE_RUNTIME_USER_MESSAGES[error.message] ?? error.message;
}
@@ -21,6 +21,21 @@ describe("startup screen", () => {
);
expect(spanish).not.toMatch(/[一-鿿]/);
});
it("says an upgrade may take a while instead of leaving a silent wait", () => {
expect(startupScreenHtml()).toContain(
"首次启动或升级记忆数据可能需要几分钟,完成后会自动进入应用。",
);
expect(startupScreenHtml("en")).toContain(
"The first launch or a memory-data upgrade can take a few minutes.",
);
});
it("keeps a draggable strip so the waiting window can be moved", () => {
const html = startupScreenHtml();
// CSS alone does not move a frameless window: the region has to be a real
// element in the document, and it has to be the top strip.
expect(html).toContain('<div class="window-drag-region"');
expect(html).toContain("-webkit-app-region: drag");
});
it("is served from the app origin, not from the web bundle", () => {
expect(STARTUP_SCREEN_URL).toBe("one-memory://app/__startup");
});
+22
View File
@@ -37,6 +37,19 @@ export function startupScreenHtml(locale: GuiLocale = "zh-CN"): string {
"Helvetica Neue", Arial, sans-serif;
-webkit-font-smoothing: antialiased;
}
/* A frameless window still needs somewhere to grab; without this the
whole wait is undraggable. */
.window-drag-region {
position: absolute;
z-index: 1;
top: 0;
right: 0;
left: 0;
height: 58px;
-webkit-app-region: drag;
-webkit-user-select: none;
user-select: none;
}
@media (prefers-color-scheme: dark) {
body {
background: #15181d;
@@ -61,6 +74,13 @@ export function startupScreenHtml(locale: GuiLocale = "zh-CN"): string {
font-size: 13px;
opacity: 0.62;
}
p.hint {
max-width: 320px;
font-size: 12px;
line-height: 1.55;
text-align: center;
opacity: 0.45;
}
.bar {
position: relative;
width: 168px;
@@ -101,9 +121,11 @@ export function startupScreenHtml(locale: GuiLocale = "zh-CN"): string {
</style>
</head>
<body>
<div class="window-drag-region" aria-hidden="true"></div>
<main>
<h1>MemHub</h1>
<p>${translate(locale, "startupPreparing")}</p>
<p class="hint">${translate(locale, "startupUpgradeHint")}</p>
<div class="bar"><span></span></div>
</main>
</body>
@@ -1,10 +1,38 @@
import { access, mkdtemp, readFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { expect, it, vi } from "vitest";
import { runStartupSdkMigration } from "./startup-sdk-migration";
import {
requireCompletedStartupSdkMigration,
runStartupSdkMigration,
withSdkMigrationCaptureFence,
} from "./startup-sdk-migration";
const core = { command: "/core/one-memory", args: [] };
const captureFence = async <T>(
_dataDir: string,
action: () => Promise<T>,
): Promise<T> => action();
it("fences hook capture only while startup migration is running", async () => {
const dataDir = await mkdtemp(
path.join(os.tmpdir(), "memhub-sdk-migration-"),
);
const marker = path.join(dataDir, ".sdk-migration-capture-fence.json");
await withSdkMigrationCaptureFence(dataDir, async () => {
const document = JSON.parse(await readFile(marker, "utf8")) as {
pid: number;
};
expect(document.pid).toBe(process.pid);
});
await expect(access(marker)).rejects.toThrow();
});
it("runs bundle backfill and cutover inside the startup maintenance fence", async () => {
const calls: string[][] = [];
const ordering: string[] = [];
const execute = vi.fn(async (_core, args: string[]) => {
calls.push(args);
const command = args[0];
@@ -30,7 +58,19 @@ it("runs bundle backfill and cutover inside the startup maintenance fence", asyn
action: () => Promise<T>,
): Promise<T> => {
fenceCalls.push(dataDir);
return action();
ordering.push("runtime-fence:start");
const result = await action();
ordering.push("runtime-fence:end");
return result;
};
const orderedCaptureFence = async <T>(
_dataDir: string,
action: () => Promise<T>,
): Promise<T> => {
ordering.push("capture-fence:start");
const result = await action();
ordering.push("capture-fence:end");
return result;
};
const result = await runStartupSdkMigration({
@@ -39,10 +79,17 @@ it("runs bundle backfill and cutover inside the startup maintenance fence", asyn
runtimeBuild: "build-13",
execute,
fence,
captureFence: orderedCaptureFence,
});
expect(result).toEqual({ state: "cutover" });
expect(fenceCalls).toEqual(["/data"]);
expect(ordering).toEqual([
"capture-fence:start",
"runtime-fence:start",
"runtime-fence:end",
"capture-fence:end",
]);
expect(calls.map((args) => args[0])).toEqual([
"sdk-migration-status",
"migration-bundle",
@@ -70,6 +117,7 @@ it("does not stop Runtime when authority is already SDK or the home is empty", a
runtimeBuild: "build-13",
execute: vi.fn(async () => ({ exitCode: 0, document })),
fence,
captureFence,
});
expect(result).toEqual({ state: "not_required" });
expect(fenced).toBe(false);
@@ -83,12 +131,16 @@ it("reports an unreadable migration status as deferred", async () => {
runtimeBuild: "build-13",
execute: vi.fn(async () => ({ exitCode: 1, document: {} })),
fence: async (_dataDir, action) => action(),
captureFence,
});
expect(result).toEqual({
state: "deferred",
code: "sdk_migration_status_unavailable",
});
expect(() => requireCompletedStartupSdkMigration(result)).toThrow(
"sdk_migration_status_unavailable",
);
});
it("resumes legacy table cleanup under the maintenance fence", async () => {
@@ -110,6 +162,7 @@ it("resumes legacy table cleanup under the maintenance fence", async () => {
: { exitCode: 0, document: { state: "clean" } };
}),
fence: async (_dataDir, action) => action(),
captureFence,
});
expect(result).toEqual({ state: "cutover" });
@@ -140,6 +193,7 @@ it("keeps legacy authority when backfill is blocked", async () => {
runtimeBuild: "build-13",
execute,
fence: async (_dataDir, action) => action(),
captureFence,
});
expect(result).toEqual({ state: "deferred", code: "quarantine" });
@@ -147,3 +201,42 @@ it("keeps legacy authority when backfill is blocked", async () => {
"sdk-cutover",
);
});
it("reuses one v2 migration identity after a blocked startup", async () => {
const calls: string[][] = [];
const migrationId = "sdk-authority-v2-0123456789abcdef";
const bundleId = `${migrationId}-pre-cutover`;
const execute = vi.fn(async (_core, args: string[]) => {
calls.push(args);
return args[0] === "sdk-migration-status"
? {
exitCode: 0,
document: {
authority: "legacy",
state: "required",
generation: "ffffffffffffffff",
migration_id: migrationId,
bundle_id: bundleId,
},
}
: args[0] === "sdk-migrate"
? { exitCode: 4, document: { state: "blocked", code: "quarantine" } }
: { exitCode: 0, document: { state: "reused" } };
});
await runStartupSdkMigration({
dataDir: "/data",
core,
runtimeBuild: "build-17",
execute,
fence: async (_dataDir, action) => action(),
captureFence,
});
expect(calls.find((args) => args[0] === "migration-bundle")).toContain(
bundleId,
);
expect(calls.find((args) => args[0] === "sdk-migrate")).toContain(
migrationId,
);
});
+109 -62
View File
@@ -1,14 +1,25 @@
import { spawn } from "node:child_process";
import { mkdir, rename, rm, writeFile } from "node:fs/promises";
import path from "node:path";
import { withRuntimeMaintenanceFence } from "@one-memory/runtime-control";
const MIGRATION_PREFIX = "sdk-authority-v1";
const MIGRATION_PREFIX = "sdk-authority-v2";
const MAX_OUTPUT_BYTES = 1024 * 1024;
const CAPTURE_FENCE = ".sdk-migration-capture-fence.json";
export interface StartupMigrationResult {
state: "not_required" | "cutover" | "deferred";
code?: string;
}
export function requireCompletedStartupSdkMigration(
result: StartupMigrationResult,
): void {
if (result.state === "deferred") {
throw new Error(result.code ?? "sdk_migration_incomplete");
}
}
interface CommandResult {
exitCode: number;
document: Record<string, unknown>;
@@ -31,6 +42,7 @@ interface StartupMigrationOptions {
dataDir: string,
) => Promise<CommandResult>;
fence?: <T>(dataDir: string, action: () => Promise<T>) => Promise<T>;
captureFence?: <T>(dataDir: string, action: () => Promise<T>) => Promise<T>;
}
export async function runStartupSdkMigration(
@@ -38,6 +50,7 @@ export async function runStartupSdkMigration(
): Promise<StartupMigrationResult> {
const execute = options.execute ?? executeCoreCommand;
const fence = options.fence ?? withRuntimeMaintenanceFence;
const captureFence = options.captureFence ?? withSdkMigrationCaptureFence;
const status = await execute(
options.core,
["sdk-migration-status", "--json"],
@@ -50,16 +63,18 @@ export async function runStartupSdkMigration(
status.document.authority === "sdk" &&
status.document.state === "cleanup_required"
) {
return fence(options.dataDir, async () => {
const cleanup = await execute(
options.core,
["sdk-cleanup", "--yes", "--json"],
options.dataDir,
);
return cleanup.exitCode === 0 && cleanup.document.state === "clean"
? { state: "cutover" }
: deferred(cleanup);
});
return captureFence(options.dataDir, () =>
fence(options.dataDir, async () => {
const cleanup = await execute(
options.core,
["sdk-cleanup", "--yes", "--json"],
options.dataDir,
);
return cleanup.exitCode === 0 && cleanup.document.state === "clean"
? { state: "cutover" }
: deferred(cleanup);
}),
);
}
if (
status.document.authority === "sdk" ||
@@ -71,57 +86,89 @@ export async function runStartupSdkMigration(
if (typeof generation !== "string" || !/^[a-f0-9]{16}$/.test(generation)) {
return { state: "deferred", code: "sdk_migration_generation_invalid" };
}
const migrationId = `${MIGRATION_PREFIX}-${generation}`;
const bundleId = `${migrationId}-pre-cutover`;
return fence(options.dataDir, async () => {
const bundle = await execute(
options.core,
[
"migration-bundle",
"--bundle-id",
bundleId,
"--runtime-build",
options.runtimeBuild,
"--yes",
"--json",
],
options.dataDir,
);
if (bundle.exitCode !== 0) return deferred(bundle);
const backfill = await execute(
options.core,
[
"sdk-migrate",
"--migration-id",
migrationId,
"--bundle-id",
bundleId,
"--yes",
"--json",
],
options.dataDir,
);
if (backfill.exitCode !== 0 || backfill.document.state !== "ready") {
return deferred(backfill);
}
const cutover = await execute(
options.core,
[
"sdk-cutover",
"--migration-id",
migrationId,
"--bundle-id",
bundleId,
"--yes",
"--json",
],
options.dataDir,
);
if (cutover.exitCode !== 0 || cutover.document.state !== "cutover") {
return deferred(cutover);
}
return { state: "cutover" };
});
const previousMigrationId = status.document.migration_id;
const previousBundleId = status.document.bundle_id;
const canResume =
typeof previousMigrationId === "string" &&
previousMigrationId.startsWith(`${MIGRATION_PREFIX}-`) &&
typeof previousBundleId === "string" &&
previousBundleId === `${previousMigrationId}-pre-cutover`;
const migrationId = canResume
? previousMigrationId
: `${MIGRATION_PREFIX}-${generation}`;
const bundleId = canResume ? previousBundleId : `${migrationId}-pre-cutover`;
return captureFence(options.dataDir, () =>
fence(options.dataDir, async () => {
const bundle = await execute(
options.core,
[
"migration-bundle",
"--bundle-id",
bundleId,
"--runtime-build",
options.runtimeBuild,
"--yes",
"--json",
],
options.dataDir,
);
if (bundle.exitCode !== 0) return deferred(bundle);
const backfill = await execute(
options.core,
[
"sdk-migrate",
"--migration-id",
migrationId,
"--bundle-id",
bundleId,
"--yes",
"--json",
],
options.dataDir,
);
if (backfill.exitCode !== 0 || backfill.document.state !== "ready") {
return deferred(backfill);
}
const cutover = await execute(
options.core,
[
"sdk-cutover",
"--migration-id",
migrationId,
"--bundle-id",
bundleId,
"--yes",
"--json",
],
options.dataDir,
);
if (cutover.exitCode !== 0 || cutover.document.state !== "cutover") {
return deferred(cutover);
}
return { state: "cutover" };
}),
);
}
export async function withSdkMigrationCaptureFence<T>(
dataDir: string,
action: () => Promise<T>,
): Promise<T> {
await mkdir(dataDir, { recursive: true, mode: 0o700 });
const marker = path.join(dataDir, CAPTURE_FENCE);
const temporary = `${marker}.${process.pid}.tmp`;
await writeFile(
temporary,
`${JSON.stringify({ schema_version: 1, pid: process.pid })}\n`,
{ encoding: "utf8", mode: 0o600 },
);
await rename(temporary, marker);
try {
return await action();
} finally {
await rm(marker, { force: true });
await rm(temporary, { force: true });
}
}
function deferred(result: CommandResult): StartupMigrationResult {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/site",
"version": "1.0.4",
"version": "1.0.5",
"private": true,
"type": "module",
"scripts": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/web",
"version": "1.0.4",
"version": "1.0.5",
"private": true,
"type": "module",
"scripts": {
@@ -227,6 +227,27 @@ describe("source list transport validation", () => {
await expect(getSources()).resolves.toEqual([source]);
});
// Every kind Core can grant, not just the ones this guard happened to know.
// A source whose root the person picked themselves is `user_selected`; the
// hand-written list here skipped it, so one such row rejected the whole
// list and every app disappeared while all endpoints answered 200.
it.each(["default", "environment_override", "user_selected"])(
"accepts the %s candidate kind",
async (candidate_kind) => {
fake.get.mockResolvedValue({ data: [{ ...source, candidate_kind }] });
await expect(getSources()).resolves.toEqual([
{ ...source, candidate_kind },
]);
},
);
it("rejects a candidate kind Core cannot grant", async () => {
fake.get.mockResolvedValue({
data: [{ ...source, candidate_kind: "invented" }],
});
await expect(getSources()).rejects.toThrow("invalid source list");
});
it.each([
["negative messages", { imported_messages: -1 }],
["NaN messages", { imported_messages: Number.NaN }],
+8 -2
View File
@@ -6,6 +6,8 @@ import type {
MatterRun,
} from "@one-memory/contracts";
import {
CANDIDATE_KINDS,
type CandidateKind,
CORE_SERVICE,
type ConnectSourcesResponse,
type ConversationBrowserPage,
@@ -1544,8 +1546,12 @@ export function isSourceResponse(value: unknown): value is SourceResponse {
typeof candidate.candidate_path === "string" &&
measuredCount(candidate.imported_messages) &&
measuredCount(candidate.imported_conversations) &&
(candidate.candidate_kind === "default" ||
candidate.candidate_kind === "environment_override") &&
// Taken from the contract rather than spelled out here: this list used to
// be written by hand, Core gained `user_selected`, and one row of that kind
// failed the guard for the whole list. Nothing renders, every endpoint
// answers 200, and the prefetch that carried the rejection cannot report
// it (issue #139).
CANDIDATE_KINDS.includes(candidate.candidate_kind as CandidateKind) &&
// Guarded rather than defaulted: an unrecognised value must not quietly
// read as "additions_only" and tell the user we track less than we do,
// nor as "full_dynamic" and promise more.
+8 -2
View File
@@ -129,12 +129,18 @@ directly for low-latency Cypher projection and store-width administration.
This exception does not extend to VDB, Coding, prompt, reconcile, or lifecycle.
- Storage backends:仅 SDK config/env(Chroma/FAISS/Qdrant、Kuzu/Neo4j、cache/history sqlite|mysql、coding sqlite 等)。
- Queue:pending `memory_write_jobs` 只表示尚未调用 SDK 的待处理请求,因此没有 SDK raw/memory ID。worker 单次调用 SDK `add()`;SDK 内部完成 raw 写入与 extraction/reconciliation,成功后 MemHub 仅将返回的 raw ID / memory IDs 记作 job outcome。`import_raw()` 只用于历史数据的 exact migration,不拆分正常写路径。
- Queue:pending `memory_write_jobs` 只表示尚未调用 SDK 的待处理请求,因此没有 SDK raw/memory ID。worker 单次调用 SDK `add()`;SDK 内部完成 raw 写入与 extraction/reconciliation,成功后 MemHub 仅将返回的 raw ID / memory IDs 记作 job outcome。旧用户启动迁移在 Runtime writers 停止期间直接操作 SDK 存储,不向公开 `HyMemoryClient` 增加迁移专用方法,也不拆分正常写路径。
- Raw retention:SDK L1 raw 只保留最近三天;每个完整 turn 只保存真人 user 与 assistant,tool 不进入 raw。高层 Chat/Coding/Working memory 不受该窗口影响。
- Recall:Host Bridge / Core 先做宿主安装凭据与 tool grant,再调 **SDK search/get**。当前产品没有逐 memory 的 A-harness→B-harness 可见性开关,默认允许跨 harness 搜索;未来若增加来源限制,直接透传 SDK `source` / `include_sources` / `exclude_sources`,不在 MemHub 建 memory ACL。
- Context wire:cutover marker 提交前只使用 legacy Context v1;提交后 Host Bridge 只使用 SDK-native Context v2。两种 schema 原样校验、原样返回,不把 SDK item 补造成 v1 的 `revision`、`evidence_ids`、`paths` 或 `adoption`,已选择的 v2 路径失败时也不回退 legacy 权威。
- Ask wire:marker 前保留 Governance Ask v1;marker 后 Web / Host Bridge 只调用 SDK Ask v2。v2 的引用只含 SDK 原生 `memory_id/content/layer/status/source/gmt_created`,生成过程中重复 `get` 校验同一 SDK 快照,不补造 revision、Evidence 或 scopes,SDK 失败不回退 v1。
- Matter:未形成已发布产品合同,不迁移占位数据;所有 `matter*` 表在升级清理时物理删除,fresh install 不创建,Runtime 不注册 Matter API。
- 旧用户迁移:由 App 在可见的启动 loading 阶段统一执行;此时 Core 尚未对外提供产品读写。启动流程创建/校验完整 bundle、续跑 ledger、完成 SDK backfill 与 reconciliation,满足全部门禁后才提交 marker 并进入主界面。普通用户不需要另跑 CLI,也不按旧 App 内 SDK 版本分叉迁移。
- SDK backfill 只写同文件系统的私有 staging 副本;验证后先原子切换 SDK
root,再提交 authority marker。切换或 marker 提交失败必须恢复原 live root;
App 不得在 migration `deferred` 状态继续启动 Runtime。
- 迁移期间宿主 capture hook 由 Desktop 活跃 PID marker 静默挡住,不再追加
Evidence;cutover 后先从宿主 History checkpoint 补齐遗漏 turn,再恢复 live sync。
- Matter 若未来重新开发,需以新的 SDK-native 合同重新设计,不继承当前占位 schema。
Pinned SDK 版本见发布锁文件 / `MEMORY.md`(随发布更新)。
@@ -279,7 +285,7 @@ Support is declared per immutable GitHub Release manifest:
- Linux x64/arm64 Runtime on native GHA runners.
- macOS arm64/x64 App/Runtime: Developer ID, notarization, native acceptance.
- Windows is not supported.
- `hy-memory==2.0.5.dev3` is the current development-preview SDK dependency; the release
- `hy-memory==2.0.5` is the current stable SDK dependency; the release
lock and [`../MEMORY.md`](../MEMORY.md) are authoritative when the pin changes.
See [`INSTALLATION.md`](INSTALLATION.md).
+1 -1
View File
@@ -1,6 +1,6 @@
# MemHub 安装与发行
MemHub `v1.0.4`(build 13)的发行目标如下:
MemHub `v1.0.5`(build 14)的发行目标如下:
| 平台 | 发行物 | 构建与验收 |
| --- | --- | --- |
+9 -5
View File
@@ -6,10 +6,13 @@
2. 记忆沉淀有没有持续运行、产出了多少条;
3. 如果停住,停在导入、模型调用还是最终写入。
默认 `info` 下的结构化诊断日志不包含会话正文、模型输入输出或 API Key。`debug`
会额外启用 HY pipeline trace,其中可能包含完整 prompt/response 和获授权的对话正文。
排查时优先通过 App 的“导出日志”提供文件,不要直接发送整个 `~/.one-memory` 数据目录;
debug 期间产生的 pipeline 日志在分享前必须由开发人员确认并脱敏。
默认 `info` 下的官方结构化事件不应写入会话正文、模型输入输出或 API Key;但本地原始日志允许
新增受限标量标签,仍应按私人数据处理,不能直接对外发送。`debug` 会额外启用 HY pipeline
trace,其中可能包含完整 prompt/response 和获授权的对话正文。
排查时只通过 App 的“导出日志”提供文件:导出过程会按安全白名单重新生成诊断事件,不会打包
原始日志。不要直接发送 `diagnostic-*.jsonl`、整个 `~/.one-memory` 数据目录或 debug pipeline
日志;后者如确需分享,必须由开发人员另行确认并脱敏。
## 1. Log level 设什么
@@ -39,7 +42,8 @@ debug 可能把完整模型 prompt/response 写入本机调试日志,不得长
## 2. 日志在哪里
App 内推荐直接使用“运行诊断 → 导出日志”。开发人员远程指导时,优先使用
`memhub logs`,它会再次过滤不安全字段。需要现场查看原始结构化日志时,Core 日志默认位于:
`memhub logs`,它会再次过滤不安全字段。原始结构化日志只用于本机现场查看,不得作为附件发送。
Core 日志默认位于:
```text
~/.one-memory/logs/diagnostic-core.jsonl
+24 -8
View File
@@ -15,9 +15,17 @@ Desktop -> Runtime -> Core -> HY -> Host 的实际执行链。日志不是用户
可选关联字段为 `trace_id`、`incident_id`、`instance_id`、`operation_id`、`request_id`、
`job_id`、`task_id`、`duration_ms`、状态、稳定错误码、错误类型和清理后的仓库相对栈帧。
`details` 只接受代码声明的标量白名单。禁止记录对话原文、查询、回答、模型输入输出、
请求体、响应体、token、API Key、secret ref、完整用户路径或异常 message/cause。未知字段和
不合规标识拒绝写入。日志写入失败不能阻断产品主链。
`details` 的本地写入合同只约束结构,不维护业务字段白名单:最多 32 个 key;key 必须是
1–64 字符的字母、数字、点、下划线或连字符并以字母开头;value 只接受 `null`、布尔值、
跨 Python/JavaScript 可无损表示的有限数字(整数限于 ±(2^53-1))或不超过 256 字符且不含
控制换行的短字符串,不接受对象和数组。这样新增诊断标签
不需要先修改中央白名单。诊断日志的校验、序列化、排队或落盘发生任何异常时都只能丢弃该条
日志,不能阻断产品主链。
本地写入放宽不代表允许官方代码记录用户内容。对话原文、查询、回答、模型输入输出、请求体、
响应体、token、API Key、secret ref、完整用户路径和异常 message/cause 仍禁止进入诊断事件,
由类型、静态检查和隐私测试共同防止误用;确需保存完整用户输入的调试能力必须使用独立的本地
trace/session 文件,不得混入可导出的诊断日志。
## 关联规则
@@ -34,9 +42,16 @@ Desktop -> Runtime -> Core -> HY -> Host 的实际执行链。日志不是用户
host-bridge、desktop 和 cli。单文件 4 MiB,保留 3 份轮转;额外清理 14 天以前的文件并将
总量限制在 128 MiB。POSIX 目录为 0700、文件为 0600。
读取器只枚举上述固定文件名的 regular file,不跟随符号链接,不接受任意路径。损坏行跳过
并计数,不让一行损坏遮蔽其余证据。诊断导出是单个脱敏 JSON,包含版本、平台、架构、
非秘密状态和最多 5000 条已验证事件,不包含数据库或配置文件。
读取器只枚举上述固定文件名的 regular file,不跟随符号链接,不接受任意路径。日常本地查询
遇到损坏行时跳过并计数,不让一行损坏遮蔽其余证据。
导出是独立的安全边界:它重新读取并校验全部候选事件,只把明确列入导出白名单的字段投影为
新的 `diagnostics/events.jsonl`;details 不只检查 key,还按字段约束 value 类型,并拒绝自由文本、
路径和凭证形态的字符串。绝不复制原始 `diagnostic-*.jsonl`、普通运行日志或崩溃报告。
导出包还可以包含固定 schema 的系统摘要和问题时间点,但不接受调用方传入的任意对象展开。
任何事件、marker、安全投影或日志目录读取失败时,导出必须 fail-closed:删除临时文件、不发布
诊断包,更不能回退为打包原始日志。发布使用已设好权限的临时文件原子创建最终目标,已存在的
目标不得覆盖,发布后也不存在再 chmod 失败留下伪成功文件的窗口。
## 覆盖与节流
@@ -56,7 +71,8 @@ host-bridge、desktop 和 cli。单文件 4 MiB,保留 3 份轮转;额外清
## 验收
隐私测试必须先植入一条可检测的秘密作为正向对照,再证明真实链路日志和导出均不含该秘密。
至少用隔离数据制造 Core 启动失败、HY 失败和宿主连接失败,并仅凭日志还原失败组件、时间、
隐私测试必须先向本地日志植入带未知 key 的可检测秘密作为正向对照,再证明本地读取仍保留该
证据、导出安全投影必定剔除它;另需证明损坏行会阻止导出且不会留下伪成功文件。至少用隔离
数据制造 Core 启动失败、HY 失败和宿主连接失败,并仅凭允许导出的字段还原失败组件、时间、
关联 ID、稳定错误码和代码位置。自动测试通过只能标记 verified,正式安装 App 的真实体验仍需
单独验收。
@@ -19,6 +19,9 @@ Before the cutover marker is committed:
- SDK stores are inspected and backfilled, but are not merged into live search;
- new imports, writes, queue drains, corrections, deletes and session delivery
are paused.
- Host capture commands detect the live Desktop migration fence and return
silently without writing Evidence. After cutover, History catch-up runs from
the existing source cursor before ordinary live sync.
After the cutover marker is committed:
@@ -49,12 +52,9 @@ The migration preflight records and validates:
- embedding model identity and dimensions;
- source grants, model profiles, config, queue leases, and pending operations.
The migration fails closed when the installed SDK lacks:
- complete lifecycle/raw inventory;
- exact raw and exact memory import;
- durable request outcome lookup;
- consistent snapshot/export and restore.
The migration fails closed when it cannot inspect every SDK memory namespace
(Chat, Coding Experience, Working Fact), write a missing ordinary memory, or
create and restore a consistent data-plane snapshot.
MemHub tags `v1.0.0` through `v1.0.4` all use the `governed-v2` namespace and
the same SDK SQL table families (`memory_operations_v2`, `pipeline_logs_v2`,
@@ -71,10 +71,10 @@ because an earlier App wrote it.
While Core and Runtime writers are stopped, build four inventories:
- `E`: Evidence rows and revisions;
- `E`: completed user/assistant turns from the rolling three-day legacy window;
- `G`: current Governance rows, statuses, supersession links, and latest content;
- `M`: task-output mappings from legacy `memory_id` to SDK `hy_memory_id`;
- `S`: complete SDK inventory, including raw and non-active lifecycle records.
- `S`: complete SDK inventory, including Coding/Working and non-active records.
Every item must end in exactly one class:
@@ -85,7 +85,7 @@ Every item must end in exactly one class:
| Governance-only | current legacy memory has no SDK entity | exact import without extraction |
| SDK-only | SDK entity has no legacy mirror | retain and adopt |
| divergent | both exist but content, owner or lifecycle differs | reconcile explicit user intent or quarantine |
| raw-only | Evidence has no SDK raw receipt | exact raw import |
| raw turn missing | canonical recent turn has no matching SDK raw | write one bounded L1 raw turn |
| unknown outcome | a task may have reached SDK without a confirmed result | SDK outcome inquiry; never blind retry |
An unclassified item blocks cutover.
@@ -102,26 +102,30 @@ The current product has no MemHub revision history or Undo.
- SDK supersession remains an SDK lifecycle relationship. Existing SDK chains
are retained. Governance links are used only to repair a missing SDK chain
during exact import.
- Mapped Coding Experience and Working Fact IDs are resolved in their native
SDK stores and adopted without conversion to Chat layers. A mapped ID missing
from its native store blocks cutover rather than fabricating another type.
- Pending correction/delete records are settled before ordinary backfill.
`update(memory_id, content)` modifies the same SDK ID. There is no product CAS,
revision number, or restore operation.
## 5. Evidence to SDK raw
## 5. Legacy Evidence to bounded SDK raw
Evidence must not be replayed through ordinary `add()`: doing so would extract
and reconcile again, changing semantic memories.
Evidence rows are message-level legacy storage, not SDK raw chunks. Migration
uses them only to rebuild completed turns from the three days preceding the
stable migration start time:
The SDK exact-raw import preserves:
- one human `user` opener plus its following `assistant` records becomes one
deterministic L1 raw node;
- tool rows and agent-authored user envelopes are excluded;
- a turn is bounded to 60,000 characters and records whether it was truncated;
- existing SDK raw outside this canonical set is removed in staging;
- high-level Chat/Coding/Working memories are retained independently of raw.
- source, conversation/session, workspace, role and author;
- original event time and stable source ordering;
- verbatim content and source identity;
- a deterministic migration key.
The operation is idempotent and returns an SDK raw ID. Evidence revisions and
repairs are preserved in the migration bundle for audit; only the current
accepted raw record becomes the normal product read.
No extraction or reconciliation is rerun while rebuilding raw. After cutover,
the normal Cursor History catch-up scans forward from its source checkpoint and
recovers turns skipped while capture was fenced.
## 6. Full migration bundle
@@ -163,6 +167,13 @@ Before cleanup, every surviving reference is rewritten to its canonical SDK
ID. Alias, item-ledger, and detailed cutover tables are then dropped; the
authority singleton and verified bundle remain.
Backfill never mutates the live SDK root. Startup first copies the complete
local SDK data plane, including SDK-only records, to a same-filesystem staging
home. Exact legacy writes and reconciliation run there. Cutover renames the
live root to a protective path, renames staging into place, and only then
commits SDK authority. A failed commit restores the previous root; restart
recovery resolves any interrupted rename before opening SDK stores.
## 8. Queue and unknown outcomes
Queue rows and leases remain MemHub management state.
@@ -172,8 +183,11 @@ Queue rows and leases remain MemHub management state.
- Its worker calls SDK `add()` once. The SDK owns raw persistence plus
extraction/reconciliation; a successful job stores only the returned raw ID,
canonical memory IDs and terminal outcome.
- Exact `import_raw()` is a historical migration primitive, not the first half
of a two-stage normal write path.
- Legacy raw and semantic backfill is an App-owned privileged offline
operation while Runtime writers are stopped. It writes SDK stores directly;
no migration-only import/get/list method is added to public
`HyMemoryClient`, and this is not the first half of a two-stage normal write
path.
- `hy_invoked=0` may be safely retried after normal lease recovery.
- `memory_write_jobs.hy_invoked=1`, `state=unknown`, or any non-terminal SDK
+2 -2
View File
@@ -1,8 +1,8 @@
{
"name": "one-memory",
"version": "1.0.4",
"version": "1.0.5",
"release": {
"build_number": 13,
"build_number": 17,
"macos_min_version": "13.0",
"app_update": "manual",
"runtime_manifest_schema": 1,
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/contracts",
"version": "1.0.4",
"version": "1.0.5",
"private": true,
"type": "module",
"types": "./src/index.ts",
+15 -2
View File
@@ -147,6 +147,19 @@ export type SourceAuthorizationState =
| "source_root_unavailable"
| "candidate_changed";
/** The kinds of root a source can be granted for — Core's `RootCandidateKind`.
*
* A value, not only a type, because the web layer has to check what Core sent
* at runtime. Issue #139: the transport guard listed the kinds by hand, Core
* added `user_selected`, and one such row rejected the entire source list.
*/
export const CANDIDATE_KINDS = [
"default",
"environment_override",
"user_selected",
] as const;
export type CandidateKind = (typeof CANDIDATE_KINDS)[number];
export interface SourceResponse {
source_id: string;
display_name: string;
@@ -157,7 +170,7 @@ export interface SourceResponse {
detected: boolean | null;
root_label: string | null;
candidate_path: string;
candidate_kind: "default" | "environment_override" | "user_selected";
candidate_kind: CandidateKind;
candidate_revision?: number;
candidate_change_allowed?: boolean;
candidate_change_reason?: string | null;
@@ -190,7 +203,7 @@ export interface SourceGrantReceipt {
source_id: string;
root_id: string;
root_label: string;
candidate_kind: "default" | "environment_override" | "user_selected";
candidate_kind: CandidateKind;
granted_at: string;
revoked_at: string | null;
revision: number;
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/host-bridge",
"version": "1.0.4",
"version": "1.0.5",
"private": true,
"type": "module",
"exports": {
@@ -192,16 +192,12 @@ describe("RecallBroker", () => {
level: "error",
component: "host.broker",
outcome: "failed",
details: expect.objectContaining({
action: "host",
code: "connection_control_unavailable",
error_type: "Error",
}),
error_code: "connection_control_unavailable",
error_type: "Error",
details: { action: "host" },
});
expect(typeof exception?.details).toBe("object");
const details = exception?.details as Record<string, unknown>;
expect(typeof details.stack_origin).toBe("string");
expect(details.stack_origin).toContain("recall-broker");
expect(exception?.details).not.toHaveProperty("message");
expect(exception?.details).not.toHaveProperty("stack_origin");
});
it.each(["recall", "ask"])(
"answers %s without waiting for a slow diagnostic write",
@@ -702,20 +702,10 @@ export class RecallBroker {
event: "connection_control.exception",
outcome: "failed",
instance_id: this.options.instanceId,
details: {
action: action ?? "invalid",
code,
error_type:
error instanceof Error ? error.constructor.name : typeof error,
message:
error instanceof Error
? error.message.slice(0, 256)
: String(error).slice(0, 256),
stack_origin:
error instanceof Error && error.stack
? (error.stack.split("\n").at(1)?.trim() ?? "").slice(0, 256)
: null,
},
error_code: code,
error_type:
error instanceof Error ? error.constructor.name : typeof error,
details: { action: action ?? "invalid" },
});
}
const fingerprint = `error:${code}`;
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/i18n",
"version": "1.0.4",
"version": "1.0.5",
"private": true,
"type": "module",
"types": "./src/index.ts",
@@ -1933,6 +1933,8 @@ const baseEnglishMessages = {
languageLoadError: "Could not load the language setting. Please retry.",
nativeErrorTitle: "MemHub could not start",
startupPreparing: "Starting and preparing the memory service…",
startupUpgradeHint:
"The first launch or a memory-data upgrade can take a few minutes. MemHub opens automatically when it is done.",
} as const;
export const englishMessages = {
@@ -52,6 +52,8 @@ const languageChineseMessages = {
languageLoadError: "无法加载语言设置,请重试。",
nativeErrorTitle: "MemHub 无法启动",
startupPreparing: "正在启动,正在准备记忆服务…",
startupUpgradeHint:
"首次启动或升级记忆数据可能需要几分钟,完成后会自动进入应用。",
} as const;
const languageSpanishMessages: Record<
@@ -71,6 +73,8 @@ const languageSpanishMessages: Record<
languageLoadError: "No se pudo cargar el idioma. Inténtalo de nuevo.",
nativeErrorTitle: "MemHub no pudo iniciarse",
startupPreparing: "Iniciando y preparando el servicio de memorias…",
startupUpgradeHint:
"El primer inicio o la actualización de tus memorias puede tardar unos minutos. La aplicación se abrirá automáticamente al terminar.",
};
type Catalog = {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/operations",
"version": "1.0.4",
"version": "1.0.5",
"private": true,
"type": "module",
"types": "./src/index.ts",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/runtime-control",
"version": "1.0.4",
"version": "1.0.5",
"private": true,
"type": "module",
"bin": {
@@ -680,7 +680,7 @@ export class CoreSupervisor {
outcome: "failed",
instance_id: this.activeInstanceId,
error_code: "core_restart_circuit_open",
details: { streak: this.restartStreak },
details: { restart_streak: this.restartStreak },
});
return;
}
@@ -765,7 +765,7 @@ export class CoreSupervisor {
outcome: "failed",
instance_id: this.activeInstanceId,
error_code: "core_restart_exhausted",
details: { attempts: budget, streak: this.restartStreak },
details: { attempts: budget, restart_streak: this.restartStreak },
});
await this.options.diagnostics?.log({
level: "error",
@@ -774,7 +774,7 @@ export class CoreSupervisor {
outcome: "failed",
instance_id: this.activeInstanceId,
error_code: "core_restart_circuit_open",
details: { streak: this.restartStreak },
details: { restart_streak: this.restartStreak },
});
}
} finally {
@@ -1,4 +1,5 @@
import {
chmod,
lstat,
mkdir,
mkdtemp,
@@ -21,8 +22,10 @@ import {
diagnosticLogFiles,
exportDiagnosticBundle,
exportDiagnosticZip,
projectDiagnosticEventForSafeOutput,
pruneDiagnosticLogs,
readDiagnosticEvents,
recordDiagnosticMarker,
writeDiagnosticEventSync,
} from "./diagnostic-log.js";
@@ -66,6 +69,30 @@ describe("diagnostic log", () => {
});
});
it("keeps the synchronous writer fail-open for invalid input", async () => {
const dataDir = await root();
expect(() =>
writeDiagnosticEventSync(
{
dataDir,
service: "desktop",
sessionId: "55555555-5555-4555-8555-555555555555",
},
{
level: "error",
component: "desktop.lifecycle",
event: "desktop.early_start_failed",
outcome: "failed",
details: { nested: ["private"] } as never,
},
),
).not.toThrow();
expect(await readDiagnosticEvents(dataDir)).toEqual({
events: [],
invalid_lines: 0,
});
});
it("writes structured, private events and reads them by correlation identity", async () => {
const dataDir = await root();
const session = "11111111-1111-4111-8111-111111111111";
@@ -89,6 +116,8 @@ describe("diagnostic log", () => {
open_file_count: 7,
memory_bytes: 4096,
thread_count: 4,
"cache.hit-rate": 0.75,
custom_label: "warm",
},
});
const result = await readDiagnosticEvents(dataDir, { traceId: trace });
@@ -101,6 +130,8 @@ describe("diagnostic log", () => {
event: "runtime.ready",
details: expect.objectContaining({
open_file_count: 7,
"cache.hit-rate": 0.75,
custom_label: "warm",
}),
});
const file = (await diagnosticLogFiles(dataDir))[0];
@@ -184,7 +215,7 @@ describe("diagnostic log", () => {
}
});
it("rejects arbitrary detail fields and absolute stack paths", () => {
it("strictly rejects complex detail values, dangerous keys and absolute stacks", () => {
const base = {
timestamp: new Date().toISOString(),
level: "error" as const,
@@ -195,11 +226,142 @@ describe("diagnostic log", () => {
session_id: "11111111-1111-4111-8111-111111111111",
};
expect(() =>
diagnosticEvent({ ...base, details: { prompt: "private" } }),
diagnosticEvent({
...base,
details: { nested: { secret: "private" } } as never,
}),
).toThrow("diagnostic_detail_value_invalid");
expect(() =>
diagnosticEvent({ ...base, details: { constructor: "private" } }),
).toThrow("diagnostic_detail_key_invalid");
expect(() =>
diagnosticEvent({ ...base, details: { "9invalid": "private" } }),
).toThrow("diagnostic_detail_key_invalid");
expect(() =>
diagnosticEvent({
...base,
details: Object.fromEntries(
Array.from({ length: 33 }, (_, index) => [`key_${index}`, index]),
),
}),
).toThrow("diagnostic_detail_count_invalid");
expect(() =>
diagnosticEvent({ ...base, details: { message: "line1\nline2" } }),
).toThrow("diagnostic_text_invalid");
expect(() =>
diagnosticEvent({ ...base, details: { metric: Number.NaN } }),
).toThrow("diagnostic_detail_value_invalid");
expect(() =>
diagnosticEvent({
...base,
details: { count: Number.MAX_SAFE_INTEGER + 1 },
}),
).toThrow("diagnostic_detail_value_invalid");
expect(() =>
diagnosticEvent({ ...base, details: { list: ["private"] } as never }),
).toThrow("diagnostic_detail_value_invalid");
expect(() =>
diagnosticEvent({ ...base, safe_stack: ["/Users/me/private.py:1"] }),
).toThrow("diagnostic_stack_invalid");
expect(() =>
diagnosticEvent({
...base,
safe_stack: ["packages/runtime/sk-live-secret-token.ts:1:1"],
}),
).toThrow("diagnostic_stack_invalid");
for (const line of [
"arbitrary stack text",
"thread=worker|threading.py:12:wait",
"FILE:///Users/me/private.py:1:call",
"one_memory/worker.py:9:call /Users/private",
"one_memory/GOCSPX-privateCredential.py:9:call",
"packages/ASIA1234567890ABCDEF/file.ts:1:1",
"packages/Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8/file.ts:1:1",
]) {
expect(() => diagnosticEvent({ ...base, safe_stack: [line] })).toThrow(
"diagnostic_stack_invalid",
);
}
expect(() =>
diagnosticEvent({
...base,
safe_stack: Array.from(
{ length: 17 },
(_, index) => `one_memory/worker.py:${index + 1}:call`,
),
}),
).toThrow("diagnostic_stack_invalid");
const accepted = diagnosticEvent({
...base,
safe_stack: [
"apps/desktop/src/main.ts:80:1",
"one_memory/diagnostic_log.py:895:_safe_stack_line",
"threading.py:1045:_bootstrap_inner",
"thread|threading.py:1045:_bootstrap_inner > one_memory/worker.py:9:call",
],
});
expect(accepted.safe_stack).toHaveLength(4);
const projected = projectDiagnosticEventForSafeOutput({
...accepted,
safe_stack: [
"free text",
"thread=worker|one_memory/worker.py:9:call",
"packages/runtime-control/src/diagnostic-log.ts:42:7",
"thread|threading.py:1045:_bootstrap_inner > one_memory/worker.py:9:call",
],
});
expect(projected.safe_stack).toEqual([
"packages/runtime-control/src/diagnostic-log.ts:42:7",
"thread|threading.py:1045:_bootstrap_inner > one_memory/worker.py:9:call",
]);
const conservative = projectDiagnosticEventForSafeOutput(
diagnosticEvent({
...base,
details: {
source_id: "cursor",
reason: "abcdef1234567890abcdef1234567890abcdef12",
status: "11111111-1111-4111-8111-111111111111",
},
}),
);
expect(conservative.details).toEqual({
source_id: "cursor",
reason: "abcdef1234567890abcdef1234567890abcdef12",
status: "11111111-1111-4111-8111-111111111111",
});
});
it("drops malformed logger events without interrupting later writes", async () => {
const dataDir = await root();
const logger = new DiagnosticLogger({
dataDir,
service: "desktop",
sessionId: "11111111-1111-4111-8111-111111111111",
});
await expect(
logger.log({
level: "info",
component: "desktop.sdk_migration",
event: "sdk_migration.startup",
outcome: "succeeded",
details: { nested: { secret: "private" } } as never,
}),
).resolves.toBeUndefined();
await logger.log({
level: "info",
component: "desktop.sdk_migration",
event: "sdk_migration.startup",
outcome: "succeeded",
details: { arbitrary_state: "cutover" },
});
const result = await readDiagnosticEvents(dataDir);
expect(result.invalid_lines).toBe(0);
expect(result.events).toHaveLength(1);
expect(result.events[0].details).toEqual({ arbitrary_state: "cutover" });
});
it("keeps code locations but strips exception messages and absolute prefixes", () => {
@@ -215,29 +377,162 @@ describe("diagnostic log", () => {
expect(JSON.stringify(safe)).not.toContain("/Users/me");
});
it("marks incidents and exports only validated events", async () => {
it("exports events through an independent detail whitelist", async () => {
const dataDir = await root();
const logger = new DiagnosticLogger({ dataDir, service: "desktop" });
const incident = await logger.markIncident({
component: "desktop.diagnostics",
status: "degraded",
});
await writeFile(
path.join(dataDir, "logs", "diagnostic-core.jsonl"),
"not-json\n",
{ mode: 0o600 },
);
await logger.log({
level: "info",
component: "desktop.diagnostics",
event: "diagnostics.checked",
outcome: "succeeded",
details: {
count: 1,
reason: "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1c2VyIn0.signaturevalue",
status: "ASIA1234567890ABCDEF",
source_id: "GOCSPX-clientSecret_A1",
mode: "Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8",
action: "AIzaSyExampleCredential1234567890",
platform: "x_ASIA1234567890ABCDEF",
phase: "x_GOCSPX-clientSecret_A1",
local_only_secret: "must-not-export",
},
});
await logger.log({
level: "error",
component: "credential.private",
event: "token.failed",
outcome: "ghp_private_token",
trace_id: "ghp_private_token",
instance_id: "ASIA1234567890ABCDEF",
request_id: "GOCSPX-clientSecret_A1",
task_id: "Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8",
error_code: "secret.token",
});
const target = path.join(await root(), "bundle.json");
const result = await exportDiagnosticBundle({
dataDir,
target,
productVersion: "1.2.3",
status: { status: "degraded" },
status: { status: "degraded", local_only_secret: "must-not-export" },
});
expect(result).toEqual({ event_count: 1, invalid_lines: 1 });
expect(result).toEqual({ event_count: 3, invalid_lines: 0 });
const payload = JSON.parse(await readFile(target, "utf8"));
expect(payload.events[0].incident_id).toBe(incident.incident_id);
expect(JSON.stringify(payload)).not.toContain(dataDir);
expect(payload.events[1].details).toEqual({ count: 1 });
expect(payload.events[2]).toMatchObject({
component: "diagnostic.redacted",
event: "diagnostic.redacted",
outcome: "redacted",
});
expect(payload.events[2].trace_id).toBeUndefined();
expect(payload.events[2].instance_id).toBeUndefined();
expect(payload.events[2].request_id).toBeUndefined();
expect(payload.events[2].task_id).toBeUndefined();
expect(payload.events[2].error_code).toBeUndefined();
expect(JSON.stringify(payload)).not.toContain("local_only_secret");
expect(JSON.stringify(payload)).not.toContain("must-not-export");
expect(JSON.stringify(payload)).not.toContain("eyJhbGciOiJIUzI1NiJ9");
expect(JSON.stringify(payload)).not.toContain("ASIA1234567890ABCDEF");
expect(JSON.stringify(payload)).not.toContain("GOCSPX-clientSecret_A1");
expect(JSON.stringify(payload)).not.toContain("x_ASIA1234567890ABCDEF");
expect(JSON.stringify(payload)).not.toContain("x_GOCSPX-clientSecret_A1");
expect(JSON.stringify(payload)).not.toContain("Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8");
expect(JSON.stringify(payload)).not.toContain(
"AIzaSyExampleCredential1234567890",
);
expect(JSON.stringify(payload)).not.toContain("ghp_private_token");
});
it("fails closed when exported build identity has a credential shape", async () => {
const dataDir = await root();
const outputDir = await root();
await expect(
exportDiagnosticBundle({
dataDir,
target: path.join(outputDir, "bundle.json"),
productVersion: "xoxb-123456-abcdef",
}),
).rejects.toThrow("diagnostic_product_version_invalid");
await expect(
exportDiagnosticZip({
dataDir,
target: path.join(outputDir, "commit.zip"),
productVersion: "1.2.3",
commit: "ghp_private_token",
}),
).rejects.toThrow("diagnostic_commit_invalid");
await expect(
exportDiagnosticZip({
dataDir,
target: path.join(outputDir, "sha.zip"),
productVersion: "1.2.3",
coreSha256: "AKIDExampleSecretId1234567890",
}),
).rejects.toThrow("diagnostic_sha256_invalid");
expect(await readdir(outputDir)).toEqual([]);
});
it("fails closed without publishing a JSON bundle when any line is invalid", async () => {
const dataDir = await root();
await mkdir(path.join(dataDir, "logs"), { recursive: true });
await writeFile(
path.join(dataDir, "logs", "diagnostic-core.jsonl"),
"not-json\n",
{ mode: 0o600 },
);
const outputDir = await root();
const target = path.join(outputDir, "bundle.json");
await expect(
exportDiagnosticBundle({
dataDir,
target,
productVersion: "1.2.3",
}),
).rejects.toThrow("diagnostic_export_invalid_lines");
expect(await readdir(outputDir)).toEqual([]);
});
it("fails closed when the diagnostic directory cannot be read", async () => {
if (process.platform === "win32") return;
const dataDir = await root();
const logs = path.join(dataDir, "logs");
await mkdir(logs, { recursive: true, mode: 0o700 });
await writeFile(path.join(logs, "diagnostic-core.jsonl"), "{}\n");
const outputDir = await root();
const target = path.join(outputDir, "bundle.json");
await chmod(logs, 0o000);
try {
await expect(
exportDiagnosticBundle({
dataDir,
target,
productVersion: "1.2.3",
}),
).rejects.toMatchObject({ code: "EACCES" });
} finally {
await chmod(logs, 0o700);
}
expect(await readdir(outputDir)).toEqual([]);
});
it("never overwrites an existing export target or leaves temporary files", async () => {
const dataDir = await root();
const outputDir = await root();
const target = path.join(outputDir, "bundle.json");
await writeFile(target, "existing-artifact", { mode: 0o600 });
await expect(
exportDiagnosticBundle({
dataDir,
target,
productVersion: "1.2.3",
}),
).rejects.toMatchObject({ code: "EEXIST" });
expect(await readFile(target, "utf8")).toBe("existing-artifact");
expect(await readdir(outputDir)).toEqual(["bundle.json"]);
});
it("preserves microsecond ordering across service log files", async () => {
@@ -280,7 +575,7 @@ describe("diagnostic log", () => {
expect(result.events[1].timestamp).toBe("2026-10-14T00:00:00.123999Z");
});
it("exports a private seven-day zip with summary, logs, crashes and markers", async () => {
it("exports only reprojected events, a strict summary and README", async () => {
const dataDir = await root();
const outputDir = await root();
const crashDir = await root();
@@ -326,39 +621,178 @@ describe("diagnostic log", () => {
markers: [
{
timestamp: "2026-10-05T01:30:00.000Z",
description: "蓝莓没出来",
description:
"蓝莓没出来 /Users/private/query.txt ASIA1234567890ABCDEF GOCSPX-clientSecret_A1 Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8 token=ghp_private_token xoxb-123456-abcdef AKIDExampleSecretId1234567890 AIzaSyExampleCredential1234567890 eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1c2VyIn0.signaturevalue",
},
],
summary: {
sources: [{ source_id: "workbuddy", imported_messages: 12 }],
memory_tasks: { unknown: 1, last_error_codes: { model_failed: 1 } },
summary_secret: conversation,
background: {
status: "exported",
runtime_state: "ready",
core_state: "healthy",
broker_state: "ready",
instance_id: "instance-1",
secret: conversation,
},
},
});
expect(result.filename).toBe("MemHub.zip");
expect(result).toEqual({ filename: "MemHub.zip", file_count: 3 });
const files = unzipSync(await readFile(target));
expect(Object.keys(files)).toEqual(
expect.arrayContaining([
"README.txt",
"summary.json",
"logs/diagnostic-core.jsonl",
"logs/core.log",
"crashes/MemHub-2026-10-05.ips",
]),
);
const summary = files["summary.json"];
expect(Object.keys(files).sort()).toEqual([
"README.txt",
"diagnostics/events.jsonl",
"summary.json",
]);
const eventFile = files["diagnostics/events.jsonl"];
const summaryFile = files["summary.json"];
const readme = files["README.txt"];
if (!summary || !readme) throw new Error("missing support bundle metadata");
expect(strFromU8(summary)).toContain("model_failed");
expect(strFromU8(readme)).toContain("蓝莓没出来");
if (!eventFile || !summaryFile || !readme)
throw new Error("missing support bundle metadata");
const exportedEvent = JSON.parse(strFromU8(eventFile).trim());
expect(exportedEvent.details).toEqual({ count: 1 });
const summary = JSON.parse(strFromU8(summaryFile));
expect(summary.background).toEqual({
status: "exported",
runtime_state: "ready",
core_state: "healthy",
broker_state: "ready",
instance_id: "instance-1",
});
expect(summary.sources).toBeUndefined();
expect(summary.summary_secret).toBeUndefined();
expect(summary.problem_markers).toEqual([
{
timestamp: "2026-10-05T01:30:00.000Z",
description: "user_marked_incident",
},
]);
expect(strFromU8(readme)).toContain("user_marked_incident");
expect(strFromU8(readme)).not.toContain("蓝莓没出来");
const all = Object.values(files)
.map((value) => strFromU8(value))
.join("\n");
expect(all).not.toContain(secret);
expect(all).not.toContain("ghp_private_token");
expect(all).not.toContain("xoxb-123456-abcdef");
expect(all).not.toContain("AKIDExampleSecretId1234567890");
expect(all).not.toContain("ASIA1234567890ABCDEF");
expect(all).not.toContain("GOCSPX-clientSecret_A1");
expect(all).not.toContain("Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8");
expect(all).not.toContain("AIzaSyExampleCredential1234567890");
expect(all).not.toContain("eyJhbGciOiJIUzI1NiJ9");
expect(all).not.toContain(conversation);
expect(all).not.toContain(home);
expect(all).not.toContain("/Users/");
expect(all).toContain("~/MemHub");
});
it("fails closed without publishing a ZIP when a diagnostic line is invalid", async () => {
const dataDir = await root();
await mkdir(path.join(dataDir, "logs"), { recursive: true });
await writeFile(
path.join(dataDir, "logs", "diagnostic-core.jsonl"),
"not-json\n",
);
const outputDir = await root();
const target = path.join(outputDir, "MemHub.zip");
await expect(
exportDiagnosticZip({
dataDir,
target,
productVersion: "3800",
}),
).rejects.toThrow("diagnostic_export_invalid_lines");
expect(await readdir(outputDir)).toEqual([]);
});
it("stores and exports only the fixed marker description", async () => {
const dataDir = await root();
const markerFile = path.join(dataDir, "logs", "problem-markers.jsonl");
await recordDiagnosticMarker(
dataDir,
{
timestamp: "2026-10-05T01:00:00.000Z",
description:
"蓝莓 /Users/private/query.txt ASIA1234567890ABCDEF GOCSPX-clientSecret_A1 Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8",
},
"/Users/private",
);
expect(await readFile(markerFile, "utf8")).toBe(
`${JSON.stringify({ timestamp: "2026-10-05T01:00:00.000Z", description: "user_marked_incident" })}\n`,
);
await writeFile(
markerFile,
`${await readFile(markerFile, "utf8")}${JSON.stringify({ timestamp: "2026-10-05T01:30:00.000Z", description: "legacy arbitrary text" })}\n`,
{ mode: 0o600 },
);
const target = path.join(await root(), "markers.zip");
await exportDiagnosticZip({
dataDir,
target,
productVersion: "3800",
now: new Date("2026-10-05T02:00:00.000Z"),
});
const files = unzipSync(await readFile(target));
const summaryFile = files["summary.json"];
if (!summaryFile) throw new Error("missing summary.json");
const payload = JSON.stringify(JSON.parse(strFromU8(summaryFile)));
expect(payload.match(/user_marked_incident/g)).toHaveLength(2);
expect(payload).not.toContain("legacy arbitrary text");
expect(payload).not.toContain("蓝莓");
expect(payload).not.toContain("/Users/");
});
it("never follows a problem marker symlink for reads or writes", async () => {
const dataDir = await root();
const logs = path.join(dataDir, "logs");
await mkdir(logs, { recursive: true });
const outside = path.join(await root(), "outside-marker");
await writeFile(outside, "private");
await symlink(outside, path.join(logs, "problem-markers.jsonl"));
await expect(
recordDiagnosticMarker(dataDir, {
timestamp: "2026-10-05T01:00:00.000Z",
description: "anything",
}),
).rejects.toThrow("diagnostic_marker_file_unsafe");
await expect(
exportDiagnosticZip({
dataDir,
target: path.join(await root(), "markers.zip"),
productVersion: "3800",
}),
).rejects.toThrow("diagnostic_marker_file_unsafe");
expect(await readFile(outside, "utf8")).toBe("private");
});
it("rejects invalid explicit and persisted markers without publishing a ZIP", async () => {
const dataDir = await root();
const logs = path.join(dataDir, "logs");
await mkdir(logs, { recursive: true });
const outputDir = await root();
const explicitTarget = path.join(outputDir, "explicit.zip");
await expect(
exportDiagnosticZip({
dataDir,
target: explicitTarget,
productVersion: "3800",
markers: [{ timestamp: "invalid", description: "marker" }],
}),
).rejects.toThrow("diagnostic_marker_invalid");
await writeFile(path.join(logs, "problem-markers.jsonl"), "not-json\n");
const persistedTarget = path.join(outputDir, "persisted.zip");
await expect(
exportDiagnosticZip({
dataDir,
target: persistedTarget,
productVersion: "3800",
}),
).rejects.toThrow("diagnostic_marker_invalid");
expect(await readdir(outputDir)).toEqual([]);
});
it("an incident retrieves its surrounding session rather than only its marker", async () => {
File diff suppressed because it is too large. Load diff
@@ -59,6 +59,7 @@ export {
diagnosticZipFilename,
exportDiagnosticBundle,
exportDiagnosticZip,
projectDiagnosticEventForSafeOutput,
pruneDiagnosticLogs,
readDiagnosticEvents,
recordDiagnosticMarker,
+1 -1
View File
@@ -36,7 +36,7 @@ MEMMY_LICENSE = (
MEMMY_PROVENANCE = MEMMY_LICENSE.with_name("PROVENANCE.md")
MEMMY_PIN = "25acd5f29fc7df73b943713abe61f1b2f9839887"
HY_MEMORY_NAME = "hy-memory"
HY_MEMORY_VERSION = "2.0.5.dev3"
HY_MEMORY_VERSION = "2.0.5"
class SbomError(RuntimeError):
+2 -2
View File
@@ -12,8 +12,8 @@ test("installation docs disclose publication and lifecycle boundaries", async ()
"utf8",
);
assert.match(readme, /`hy-memory==2\.0\.5\.dev3`/);
assert.match(readme, /`v1\.0\.4` 的发行目标是 Linux x64\/arm64 Runtime/);
assert.match(readme, /`hy-memory==2\.0\.5`/);
assert.match(readme, /`v1\.0\.5` 的发行目标是 Linux x64\/arm64 Runtime/);
assert.match(readme, /Linux 制品由 GitHub Actions/);
assert.match(readme, /macOS.*Developer ID 签名、公证/s);
assert.match(readme, /release-manifest\.json/);
+3 -3
View File
@@ -13,7 +13,7 @@ spec.loader.exec_module(module)
def test_hy_memory_notice_comes_from_the_installed_dist_info(tmp_path: Path) -> None:
dist_info = tmp_path / "hy_memory-2.0.5.dev3.dist-info"
dist_info = tmp_path / "hy_memory-2.0.5.dist-info"
licenses = dist_info / "licenses"
licenses.mkdir(parents=True)
(dist_info / "METADATA").write_text(
@@ -21,7 +21,7 @@ def test_hy_memory_notice_comes_from_the_installed_dist_info(tmp_path: Path) ->
(
"Metadata-Version: 2.4",
"Name: hy-memory",
"Version: 2.0.5.dev3",
"Version: 2.0.5",
"License-Expression: Apache-2.0",
"",
)
@@ -38,7 +38,7 @@ def test_hy_memory_notice_comes_from_the_installed_dist_info(tmp_path: Path) ->
notice = module.hy_memory_notice(dist_info)
assert notice["name"] == "hy-memory"
assert notice["version"] == "2.0.5.dev3"
assert notice["version"] == "2.0.5"
assert notice["license"] == "Apache-2.0"
assert notice["copyright"] == "Copyright (C) 2026 Tencent. All rights reserved."
assert "Licensed under the Apache License" in notice["text"]
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "hy-memory"
version = "2.0.5.dev3"
version = "2.0.5"
description = "HY Memory - Industrial-grade dual-system cognitive memory framework"
readme = "README.md"
requires-python = ">=3.9"
+11 -18
View File
@@ -12,9 +12,8 @@ L1_RAW 对外读路径不可见:闸门焊在向量库唯一查询入口。
hybrid_search_native / list_by_user 四个查询方法的共同必经处,
上层无法绕开,新增接口自动受保护。
按 ID 直取走主键查询、不经过 filter,故在 `client.async_get` 处默认挡住。
Python SDK 调用方可显式传 `include_raw=True` 读取;HTTP 接口不传该参数,
继续保持原有隐藏行为。
按 ID 直取是精确读取:raw ID 返回 L1_RAW,fact ID 返回对应事实。权限由
调用 SDK 前的产品边界检查,不由 SDK 根据 layer 猜测。
内部管线需读原始对话时显式 allow_raw_layer=True 放行:
- `writer._get_recent_dialogue_history`:提取前拼多轮对话历史喂 LLM
@@ -125,8 +124,8 @@ class TestListNoLongerFiltersInProcess:
"list 又出现进程内过滤 raw 的写法,应交由 vdb 侧 filter 完成"
class TestGetByIdRawOptIn:
"""按 ID 直取默认隐藏 raw,只有 SDK 显式选择时才返回。"""
class TestGetByIdReturnsExactLayer:
"""按 ID 直取返回命中的真实层级。"""
@staticmethod
def _client(node):
@@ -153,35 +152,29 @@ class TestGetByIdRawOptIn:
user_id="u1",
agent_id="a1",
session_id="s1",
source="cursor",
content="原始对话",
layer=MemoryLayer.L1_RAW,
status=MemoryStatus.ACTIVE,
custom={"meta": [{"source": "test"}]},
)
def test_default_still_hides_raw(self):
def test_raw_id_returns_lite_memory(self):
from hy_memory.client import HyMemoryClient
client = self._client(self._raw_node())
raw_get = HyMemoryClient.async_get.__wrapped__
assert run(raw_get(client, "raw-1")) is None
def test_explicit_include_raw_returns_lite_memory(self):
from hy_memory.client import HyMemoryClient
client = self._client(self._raw_node())
raw_get = HyMemoryClient.async_get.__wrapped__
result = run(raw_get(client, "raw-1", include_raw=True))
result = run(raw_get(client, "raw-1"))
assert result["memory_id"] == "raw-1"
assert result["content"] == "原始对话"
assert result["layer"] == MemoryLayer.L1_RAW.value
assert result["source"] == "cursor"
assert result["retrieval_source"] == "vdb"
assert result["meta"] == [{"source": "test"}]
def test_public_get_exposes_keyword_only_opt_in(self):
def test_public_get_has_no_raw_visibility_switch(self):
import inspect
from hy_memory.client import HyMemoryClient
param = inspect.signature(HyMemoryClient.get).parameters["include_raw"]
assert param.default is False
assert param.kind is inspect.Parameter.KEYWORD_ONLY
assert "include_raw" not in inspect.signature(HyMemoryClient.get).parameters
+40 -14
View File
@@ -36,13 +36,16 @@ def _node(
content="x",
layer=MemoryLayer.L2_FACT,
status=MemoryStatus.ACTIVE,
session_id="s1",
source="default",
):
import random
n = MemoryNode(
node_id=str(uuid.uuid4()),
user_id=user_id,
agent_id="a1",
session_id="s1",
session_id=session_id,
source=source,
content=content,
layer=layer,
status=status,
@@ -159,30 +162,53 @@ class TestVdbBucketLayerFilter:
))
assert {m["content"] for m in got["memories"]} == {"fact", "summary"}
def test_explicit_raw_is_blocked(self, faiss_store):
"""显式只要 L1_RAW 也拿不到:raw 对外读路径一律不可见(契约变更)。
改动前:默认排除只是缺省策略,调用方点名 L1_RAW 即以调用方为准。
改动后:向量库侧 allow_raw_layer 默认 False,排除条件随请求下发,
raw 不出库;调用方无论传什么 layers 都拿不到原始对话。
仅内部管线(writer 拼对话历史、clone 全量复制)显式放行。
"""
def test_explicit_raw_returns_raw(self, faiss_store):
"""默认列表不含 raw;调用方点名 L1_RAW 时按统一 list 合同返回。"""
self._seed(faiss_store)
got = run(_client_with(faiss_store)._list_vdb_bucket(
user_id="u1", agent_id=None, limit=100, offset=0, order="desc",
layers=[MemoryLayer.L1_RAW],
))
assert got["memories"] == []
assert got["total"] == 0
assert [m["content"] for m in got["memories"]] == ["raw"]
assert got["memories"][0]["layer"] == MemoryLayer.L1_RAW.value
assert got["total"] == 1
def test_raw_mixed_with_other_layers_only_drops_raw(self, faiss_store):
"""raw 与其他层混传时只剔除 raw,其余层照常返回。"""
def test_raw_mixed_with_other_layers_returns_both(self, faiss_store):
self._seed(faiss_store)
got = run(_client_with(faiss_store)._list_vdb_bucket(
user_id="u1", agent_id=None, limit=100, offset=0, order="desc",
layers=[MemoryLayer.L1_RAW, MemoryLayer.L2_FACT],
))
assert [m["content"] for m in got["memories"]] == ["fact"]
assert {m["content"] for m in got["memories"]} == {"raw", "fact"}
def test_raw_uses_generic_session_and_source_filters(self, faiss_store):
run(faiss_store.upsert_batch([
_node(
content="cursor raw",
layer=MemoryLayer.L1_RAW,
session_id="s1",
source="cursor",
),
_node(
content="claude raw",
layer=MemoryLayer.L1_RAW,
session_id="s2",
source="claude",
),
]))
got = run(_client_with(faiss_store)._list_vdb_bucket(
user_id="u1",
agent_id=None,
limit=100,
offset=0,
order="desc",
layers=[MemoryLayer.L1_RAW],
session_ids=["s2"],
include_sources=["claude"],
exclude_sources=["cursor"],
))
assert [item["content"] for item in got["memories"]] == ["claude raw"]
assert got["memories"][0]["source"] == "claude"
def test_status_filter_defaults_to_active_and_can_read_superseded(self, faiss_store):
run(faiss_store.upsert_batch([
+50 -232
View File
@@ -1,69 +1,22 @@
from __future__ import annotations
import inspect
from unittest.mock import AsyncMock
import pytest
from hy_memory.client import HyMemoryClient
from hy_memory.models.memory import MemoryLayer, MemoryStatus
class OutcomeCache:
def __init__(self):
self.rows = {}
async def claim_write_outcome(
self, request_id, user_id, agent_id, session_id, operation
):
if request_id in self.rows:
return self.rows[request_id]
self.rows[request_id] = {
"request_id": request_id,
"user_id": user_id,
"agent_id": agent_id,
"session_id": session_id,
"operation": operation,
"status": "running",
"result": {},
}
return None
async def finish_write_outcome(self, request_id, status, result):
self.rows[request_id].update(status=status, result=dict(result))
return True
async def get_write_outcome(self, request_id):
return self.rows.get(request_id)
from hy_memory.models.memory import (
MemoryLayer,
MemoryNode,
MemoryStatus,
SourceType,
)
class RawStore:
def __init__(self):
self.nodes = {}
self.upserts = 0
async def get_by_id(self, node_id):
return self.nodes.get(node_id)
async def upsert(self, node):
self.upserts += 1
self.nodes[node.node_id] = node
return node.node_id
async def list_by_user(
self,
*,
user_id,
agent_id=None,
include_sources=None,
**_kwargs,
):
return [
node
for node in self.nodes.values()
if node.user_id == user_id
and (agent_id is None or node.agent_id == agent_id)
and (not include_sources or node.source in include_sources)
]
def __init__(self, nodes: list[MemoryNode]):
self.nodes = nodes
async def search(
self,
@@ -77,7 +30,7 @@ class RawStore:
**_kwargs,
):
rows = []
for node in self.nodes.values():
for node in self.nodes:
if node.user_id not in user_ids:
continue
if agent_ids and node.agent_id not in agent_ids:
@@ -92,186 +45,51 @@ class RawStore:
return rows[:limit]
def _client():
def raw_node(raw_id: str, session_id: str, source: str) -> MemoryNode:
return MemoryNode(
node_id=raw_id,
user_id="u1",
agent_id="a1",
session_id=session_id,
source=source,
layer=MemoryLayer.L1_RAW,
content=f"content {raw_id}",
source_type=SourceType.EXPLICIT,
status=MemoryStatus.ACTIVE,
is_latest=True,
embedding=[0.1, 0.2],
)
def test_migration_only_methods_are_not_public_client_api():
for name in (
"get_raw",
"async_get_raw",
"list_raw",
"async_list_raw",
"import_raw",
"async_import_raw",
"import_memory_exact",
"async_import_memory_exact",
):
assert not hasattr(HyMemoryClient, name)
assert "include_raw" not in inspect.signature(HyMemoryClient.get).parameters
@pytest.mark.asyncio
async def test_existing_search_contract_can_explicitly_search_raw():
client = HyMemoryClient.__new__(HyMemoryClient)
client._cache = OutcomeCache()
client._vector_store = RawStore()
client._vector_store = RawStore(
[
raw_node("raw-1", "s1", "cursor"),
raw_node("raw-2", "s2", "claude"),
]
)
client._embed_service = AsyncMock()
client._embed_service.embed.return_value = [0.1, 0.2]
return client
@pytest.mark.asyncio
async def test_import_raw_preserves_identity_without_extraction():
client = _client()
result = await HyMemoryClient.async_import_raw.__wrapped__(
result = await HyMemoryClient._search_explicit_raw_layer.__wrapped__(
client,
"verbatim",
user_id="u1",
agent_id="a1",
session_id="s1",
source="cursor",
raw_id="raw-1",
request_id="request-1",
metadata={"role": "user", "source_order": 7},
)
assert result == {
"success": True,
"raw_id": "raw-1",
"request_id": "request-1",
}
node = client._vector_store.nodes["raw-1"]
assert node.layer == MemoryLayer.L1_RAW
assert node.content == "verbatim"
assert node.custom["role"] == "user"
assert node.custom["source_order"] == 7
assert node.custom["input_kind"] == "raw_exact_import"
@pytest.mark.asyncio
async def test_import_raw_request_replay_does_not_write_twice():
client = _client()
first = await HyMemoryClient.async_import_raw.__wrapped__(
client, "same", user_id="u1", raw_id="raw-1", request_id="request-1"
)
second = await HyMemoryClient.async_import_raw.__wrapped__(
client, "same", user_id="u1", raw_id="raw-1", request_id="request-1"
)
assert second == {**first, "replayed": True}
assert client._vector_store.upserts == 1
@pytest.mark.asyncio
async def test_existing_raw_id_must_match_exact_content_and_owner():
client = _client()
await HyMemoryClient.async_import_raw.__wrapped__(
client, "first", user_id="u1", raw_id="raw-1", request_id="request-1"
)
with pytest.raises(ValueError, match="different source content"):
await HyMemoryClient.async_import_raw.__wrapped__(
client, "changed", user_id="u1", raw_id="raw-1", request_id="request-2"
)
@pytest.mark.asyncio
async def test_raw_get_and_list_use_explicit_api():
client = _client()
for raw_id, session_id, source in (
("raw-1", "s1", "cursor"),
("raw-2", "s2", "claude"),
):
await HyMemoryClient.async_import_raw.__wrapped__(
client,
raw_id,
user_id="u1",
session_id=session_id,
source=source,
raw_id=raw_id,
request_id=f"request-{raw_id}",
)
item = await HyMemoryClient.async_get_raw.__wrapped__(client, "raw-1")
assert item["raw_id"] == "raw-1"
assert item["content"] == "raw-1"
page = await HyMemoryClient.async_list_raw.__wrapped__(
client,
user_id="u1",
source="claude",
limit=10,
offset=0,
)
assert page["total"] == 1
assert [item["raw_id"] for item in page["raw"]] == ["raw-2"]
@pytest.mark.asyncio
async def test_exact_memory_import_preserves_id_lifecycle_and_chain():
client = _client()
result = await HyMemoryClient.async_import_memory_exact.__wrapped__(
client,
"older preference",
memory_id="memory-1",
user_id="u1",
agent_id="a1",
session_id="s1",
source="cursor",
layer="l2_fact",
status="superseded",
owner="user",
tags=["preference"],
superseded_by=["memory-2"],
is_latest=False,
request_id="request-memory-1",
)
assert result["memory_ids"] == ["memory-1"]
node = client._vector_store.nodes["memory-1"]
assert node.content == "older preference"
assert node.layer == MemoryLayer.L2_FACT
assert node.status == MemoryStatus.SUPERSEDED
assert node.superseded_by == ["memory-2"]
assert node.is_latest is False
assert node.custom["input_kind"] == "memory_exact_import"
@pytest.mark.asyncio
async def test_exact_memory_import_refuses_same_id_with_different_body():
client = _client()
await HyMemoryClient.async_import_memory_exact.__wrapped__(
client,
"first",
memory_id="memory-1",
user_id="u1",
request_id="request-memory-1",
)
with pytest.raises(ValueError, match="different memory content"):
await HyMemoryClient.async_import_memory_exact.__wrapped__(
client,
"changed",
memory_id="memory-1",
user_id="u1",
request_id="request-memory-2",
)
@pytest.mark.asyncio
async def test_exact_memory_import_rejects_raw_layer():
client = _client()
with pytest.raises(ValueError, match="use import_raw"):
await HyMemoryClient.async_import_memory_exact.__wrapped__(
client,
"raw",
memory_id="memory-1",
user_id="u1",
layer="l1_raw",
request_id="request-memory-1",
)
@pytest.mark.asyncio
async def test_existing_search_contract_can_explicitly_search_raw_with_source_filter():
client = _client()
for raw_id, session_id, source in (
("raw-1", "s1", "cursor"),
("raw-2", "s2", "claude"),
):
await HyMemoryClient.async_import_raw.__wrapped__(
client,
f"content {raw_id}",
user_id="u1",
agent_id="a1",
session_id=session_id,
source=source,
raw_id=raw_id,
request_id=f"request-{raw_id}",
)
result = await client._search_explicit_raw_layer(
"content",
user_ids=["u1"],
agent_ids=["a1"],