Compare commits

..
Author SHA1 Message Date
alvinfeiandCursor 8367667264 ci: upload tagged Linux runtimes to draft releases
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-17 00:30:45 +08:00
316 changed files with 24000 additions and 18329 deletions

No files matched your search

+2 -45
View File
@@ -2,26 +2,6 @@
Hy Memory and MemHub are versioned and released independently.
## Hy Memory 2.0.3 — 2026-10-17
- Added an optional runtime HTTP transport factory so embedded products can
enforce request budgets without wrapping or restricting SDK local-store
access.
- Published
[`hy-memory==2.0.3`](https://pypi.org/project/hy-memory/2.0.3/) with
unchanged memory extraction, reconciliation, storage, and Coding/Working
behavior.
## Unreleased
- Removed MemHub's task-scoped HyPermission store wrappers and global
model-content consent switch. The trusted embedded Hy Memory SDK now uses
its native local stores directly.
- Existing installations ignore the retired config flag, readmit previously
parked pre-invocation work, and drop only the obsolete
`hy_node_permissions` proof table. Evidence, governed memories, model
profiles, SDK stores, and source/host grants are preserved.
## Hy Memory 2.0.2 — 2026-10-17
- Published
@@ -36,34 +16,11 @@ Hy Memory and MemHub are versioned and released independently.
Because PyPI release metadata is immutable, the next SDK patch release must
carry the updated license metadata; `2.0.1` retains its published metadata.
## MemHub 1.0.3 — Unreleased
## MemHub 1.0.1 — Unreleased
- Updated the source version to `1.0.3` (build 7).
- Pinned the public SDK to `hy-memory==2.0.3`.
- Removed the HyPermission store wrappers and global model-content consent
switch; the trusted embedded SDK now uses native local stores.
- Prevented uncertain writes from blocking later source work and added one
reconciled automatic retry after ten seconds.
- Added product-facing memory pipeline counters and the log troubleshooting
runbook.
- Allowed a verified packaged Desktop Runtime from a different commit to take
over an equal-build install atomically while preserving downgrade and
tamper refusal.
## MemHub 1.0.2 — 2026-10-17
- Updated the source version to `1.0.2` (build 6).
- Updated the source version to `1.0.1` (build 5).
- Pinned the public SDK to `hy-memory==2.0.2`.
- Pinned `onnxruntime==1.23.2` to retain macOS x64 packaging support.
- Fixed relaunch and upgrade failures when a prior managed Runtime exists.
- Embedded canonical build identity when an external packaging pipeline omits
release environment fields.
- Added automatic Linux Runtime upload to tagged Draft Releases.
## MemHub 1.0.1 — Superseded before release
- The candidate tag was not published as a GitHub Release; `1.0.2` replaces it
after the managed-Runtime relaunch fix.
## MemHub 1.0.0
+8 -8
View File
@@ -32,7 +32,7 @@ English | [中文](README.zh.md)
## Latest News
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.3`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.3/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.2`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.2/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** on Hugging Face along with its [technical report](https://arxiv.org/abs/2610.22196) on attributed reinforcement learning for long-term memory writing!
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **Released** **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)** (submitted to **ICLR 2027**).
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **Accepted** at **EMNLP 2026**: **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)**!
@@ -63,13 +63,13 @@ Welcome to the **Hy Memory Research Family**, covering cognitive chat memory, lo
### 1. Installation
```bash
pip install hy-memory==2.0.3 # core (includes ChromaDB and Kuzu)
pip install "hy-memory[qdrant]==2.0.3" # + Qdrant
pip install "hy-memory[faiss]==2.0.3" # + FAISS
pip install "hy-memory[graph]==2.0.3" # + Neo4j
pip install "hy-memory[mysql]==2.0.3" # + MySQL cache
pip install "hy-memory[rerank]==2.0.3" # + local multilingual cross-encoder rerank
pip install "hy-memory[all]==2.0.3" # everything
pip install hy-memory==2.0.2 # core (includes ChromaDB and Kuzu)
pip install "hy-memory[qdrant]==2.0.2" # + Qdrant
pip install "hy-memory[faiss]==2.0.2" # + FAISS
pip install "hy-memory[graph]==2.0.2" # + Neo4j
pip install "hy-memory[mysql]==2.0.2" # + MySQL cache
pip install "hy-memory[rerank]==2.0.2" # + local multilingual cross-encoder rerank
pip install "hy-memory[all]==2.0.2" # everything
```
### 2. Configuration
+8 -8
View File
@@ -32,7 +32,7 @@
## 最新动态
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.3`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.3/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.2`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.2/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** 长期记忆写入模型及[技术报告](https://arxiv.org/abs/2610.22196)!
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **基准与方法发布** 推出长周期与交错式编程记忆评测基准 **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)**(已投稿 **ICLR 2027**)。
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **论文录用** 双系统认知记忆架构论文 **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)** 被 **EMNLP 2026** 接收!
@@ -63,13 +63,13 @@
### 1. 安装
```bash
pip install hy-memory==2.0.3 # 核心(含 ChromaDB 与 Kuzu)
pip install "hy-memory[qdrant]==2.0.3" # + Qdrant
pip install "hy-memory[faiss]==2.0.3" # + FAISS
pip install "hy-memory[graph]==2.0.3" # + Neo4j
pip install "hy-memory[mysql]==2.0.3" # + MySQL 缓存
pip install "hy-memory[rerank]==2.0.3" # + 本地多语言 cross-encoder rerank
pip install "hy-memory[all]==2.0.3" # 全部
pip install hy-memory==2.0.2 # 核心(含 ChromaDB 与 Kuzu)
pip install "hy-memory[qdrant]==2.0.2" # + Qdrant
pip install "hy-memory[faiss]==2.0.2" # + FAISS
pip install "hy-memory[graph]==2.0.2" # + Neo4j
pip install "hy-memory[mysql]==2.0.2" # + MySQL 缓存
pip install "hy-memory[rerank]==2.0.2" # + 本地多语言 cross-encoder rerank
pip install "hy-memory[all]==2.0.2" # 全部
```
### 2. 配置
+1 -1
View File
@@ -37,7 +37,7 @@ do not belong in the public source tree.
- Root GitHub workflows run SDK and MemHub checks from their respective working
directories. Native lifecycle workflows produce reviewable Actions artifacts;
tag releases remain fail-closed on public identity, signing and notarization.
- Hy Memory is published as `hy-memory==2.0.3`. MemHub pins that public
- Hy Memory is published as `hy-memory==2.0.2`. MemHub pins that public
distribution; its selected npm package is `memhub-cli`, which is not yet
published.
- Hy-Memory source uses Apache-2.0 at the repository root. Third-party
+1 -1
View File
@@ -5,7 +5,7 @@
## 安装与快速开始
```bash
pip install hy-memory==2.0.3
pip install hy-memory==2.0.2
# 可选后端
pip install 'hy-memory[qdrant]==2.0.1'
-2
View File
@@ -1123,7 +1123,6 @@ class Extractor:
max_tokens=self._llm_config.agent_max_tokens,
tools=None,
tool_choice=None,
audit_stage="extract",
)
self._call_count += 1
total_tokens += response.tokens_used
@@ -1234,7 +1233,6 @@ class Extractor:
max_tokens=self._llm_config.agent_max_tokens,
tools=None,
tool_choice=None,
audit_stage="extract",
)
self._call_count += 1
total_tokens += response.tokens_used
+20 -35
View File
@@ -34,7 +34,6 @@ import os
import weakref
from ..config import MemoryConfig
from ..utils.model_call_context import model_call_stage
from ..core.workload import llm_slot
logger = logging.getLogger(__name__)
@@ -349,7 +348,6 @@ class LLMProvider:
stop: List[str] = None,
tools: Optional[List[Dict[str, Any]]] = None,
tool_choice: Optional[Any] = None,
audit_stage: Optional[str] = None,
**kwargs
) -> LLMResponse:
"""
@@ -388,16 +386,15 @@ class LLMProvider:
len(prompt), _content_fingerprint(prompt),
)
with model_call_stage(audit_stage):
response = await self._call_llm(
prompt=prompt,
max_tokens=max_tokens,
temperature=temperature,
stop=stop,
tools=tools,
tool_choice=tool_choice,
**kwargs
)
response = await self._call_llm(
prompt=prompt,
max_tokens=max_tokens,
temperature=temperature,
stop=stop,
tools=tools,
tool_choice=tool_choice,
**kwargs
)
self._total_calls += 1
self._total_tokens += response.tokens_used
@@ -499,21 +496,11 @@ class LLMProvider:
clients = self._clients.setdefault(loop, {})
client = clients.get(client_key)
if client is None:
client_options: Dict[str, Any] = {
"api_key": base_api_key,
"base_url": base_url,
"timeout": self._llm_config.timeout,
"max_retries": 0,
}
transport_factory = getattr(self.config, "http_transport_factory", None)
if callable(transport_factory):
import httpx
client_options["http_client"] = httpx.AsyncClient(
transport=transport_factory()
)
client = AsyncOpenAI(
**client_options,
api_key=base_api_key,
base_url=base_url,
timeout=self._llm_config.timeout,
max_retries=0,
)
clients[client_key] = client
@@ -598,7 +585,6 @@ class LLMProvider:
temperature: Optional[float] = None,
tools: Optional[List[Dict[str, Any]]] = None,
tool_choice: Optional[Any] = None,
audit_stage: Optional[str] = None,
) -> LLMResponse:
"""
多轮 messages 调用(支持 tool results 回传)。
@@ -626,14 +612,13 @@ class LLMProvider:
f"msgs={len(messages)} tools={len(tools) if tools else 0}"
)
with model_call_stage(audit_stage):
response = await self._chat_completions(
messages,
max_tokens=max_tokens,
temperature=temperature,
tools=tools,
tool_choice=tool_choice,
)
response = await self._chat_completions(
messages,
max_tokens=max_tokens,
temperature=temperature,
tools=tools,
tool_choice=tool_choice,
)
self._total_calls += 1
self._total_tokens += response.tokens_used
-2
View File
@@ -669,7 +669,6 @@ class MemoryReconciler:
response = await llm.complete(
prompt=prompt,
max_tokens=500,
audit_stage="reconcile",
)
text = response.content.strip()
if "```json" in text:
@@ -1384,7 +1383,6 @@ class MemoryReconciler:
response = await llm.complete(
prompt=prompt,
max_tokens=dyn_max_tokens,
audit_stage="reconcile",
)
except Exception as e:
# LLM 彻底不可用(provider 已按 max_retries 重试过,网关 500 /
-1
View File
@@ -919,7 +919,6 @@ class CodingMemoryExtractor:
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=self.max_tokens,
audit_stage="coding_extract",
)
except Exception as e:
logger.warning(f"[coding-extract] LLM call failed: {e}")
-2
View File
@@ -571,7 +571,6 @@ class WorkingFactReconciler:
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=self.max_tokens,
audit_stage="coding_fact_reconcile",
)
return self._parse_ops(resp.content)
@@ -638,7 +637,6 @@ class WorkingFactReconciler:
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=_CHAIN_SUMMARY_MAX_TOKENS,
audit_stage="coding_fact_summary",
)
except Exception as e:
logger.warning(f"[fact-reconcile] chain summary failed: {e}")
+3 -11
View File
@@ -29,9 +29,6 @@ logger = logging.getLogger(__name__)
_JUDGE_MAX_TURNS = 20
_JUDGE_HEAD_TURNS = 5
_JUDGE_USER_CONTENT_LIMIT = 1_000
_JUDGE_USER_CONTENT_HEAD = 200
_JUDGE_USER_CONTENT_TAIL = 200
_JUDGE_MAX_QUERY_CHARS = 300
_JUDGE_MAX_QUERIES = 20
@@ -189,7 +186,6 @@ async def classify_messages_scene(
prompt=prompt,
max_tokens=max_tokens,
temperature=temperature,
audit_stage="coding_judge",
)
_elapsed = round((time.perf_counter() - _t0) * 1000, 2)
result = _parse_scene_json(resp.content)
@@ -635,13 +631,9 @@ def _format_turns(summary: List[Dict[str, Any]]) -> str:
rows = []
for t in summary:
user = t.get("user", "") or ""
if len(user) > _JUDGE_USER_CONTENT_LIMIT:
omitted = len(user) - _JUDGE_USER_CONTENT_HEAD - _JUDGE_USER_CONTENT_TAIL
user = (
user[:_JUDGE_USER_CONTENT_HEAD]
+ f"\n[...{omitted} chars omitted...]\n"
+ user[-_JUDGE_USER_CONTENT_TAIL:]
)
# 单 turn 用户文本截到 200 字符
if len(user) > 200:
user = user[:200] + "..."
tools = t.get("tools", []) or []
rows.append(
f"- turn {t.get('turn', '?')}: user={user!r}, tools={tools}"
-1
View File
@@ -310,7 +310,6 @@ class CodingMemoryReconciler:
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=self.max_tokens,
audit_stage="coding_reconcile",
)
return self._parse_ops(resp.content), prompt, resp.content, resp
+1 -8
View File
@@ -6,7 +6,7 @@ Agent Memory - 配置管理
import os
from dataclasses import dataclass, field
from typing import Optional, Dict, Any, Callable, List
from typing import Optional, Dict, Any, List
from pathlib import Path
@@ -1252,13 +1252,6 @@ class MemoryConfig:
enable_agent: bool = False
debug: bool = False
metrics_enabled: bool = True
# Runtime-only HTTP transport factory. MemHub uses this for request-cost
# budgeting without restricting the embedded SDK's local store access.
http_transport_factory: Optional[Callable[[], Any]] = field(
default=None,
repr=False,
compare=False,
)
def __post_init__(self):
# enable_graph 跟随 mode 走:mode=ultra 时自动启用,无需独立环境变量
+4 -14
View File
@@ -356,21 +356,11 @@ class EmbedService:
key = self._client_cache_key(base_url, api_key, timeout)
client = self._client_cache.get(key)
if client is None:
client_options = {
"api_key": api_key,
"base_url": base_url or None,
"timeout": timeout,
"max_retries": 0,
}
transport_factory = getattr(self.config, "http_transport_factory", None)
if callable(transport_factory):
import httpx
client_options["http_client"] = httpx.AsyncClient(
transport=transport_factory()
)
client = AsyncOpenAI(
**client_options,
api_key=api_key,
base_url=base_url or None,
timeout=timeout,
max_retries=0,
)
self._client_cache[key] = client
return client
-24
View File
@@ -1,24 +0,0 @@
"""Context propagated to host-provided HTTP transports for cost accounting."""
from collections.abc import Iterator
from contextlib import contextmanager
from contextvars import ContextVar, Token
_model_call_stage: ContextVar[str] = ContextVar(
"hy_memory_model_call_stage",
default="other",
)
def current_model_call_stage() -> str:
return _model_call_stage.get()
@contextmanager
def model_call_stage(stage: str | None) -> Iterator[None]:
normalized = (stage or "other").strip().lower() or "other"
token: Token[str] = _model_call_stage.set(normalized)
try:
yield
finally:
_model_call_stage.reset(token)
+1 -5
View File
@@ -80,11 +80,7 @@ class PipelineLogWriter:
split_namespace_user,
)
# Providers define total_tokens as prompt+completion. Adding all
# three fields doubled every cost estimate in this JSONL.
tokens = (total_tokens or 0) or (
(prompt_tokens or 0) + (completion_tokens or 0)
)
tokens = (prompt_tokens or 0) + (completion_tokens or 0) + (total_tokens or 0)
namespace, uid = split_namespace_user(user_id or "")
if not is_llm_content_logging_enabled():
prompt = redact_content(prompt)
+7 -10
View File
@@ -15,12 +15,12 @@ write、search,并把搜索结果格式化为可交给模型的 memory text ch
```text
distribution: hy-memory
version: 2.0.3
version: 2.0.2
mode: 按运行时配置选择
reader: hybrid
```
仓库 pin、lockfile 和运行时版本守卫均固定到公开的 `hy-memory==2.0.3`。
仓库 pin、lockfile 和运行时版本守卫均固定到公开的 `hy-memory==2.0.2`。
MemHub 在本地 Runtime 内嵌入 SDK,但不绕过自身的来源授权、模型授权、Evidence、
Governance 和撤权边界。
@@ -305,9 +305,9 @@ Runtime 应长期复用 client,不为每次请求重新初始化数据库和
```text
Evidence
-> HyMemoryBridge
-> embedded Hy Memory SDK
-> native Vector / Coding / Graph stores
-> ModelProcessingPermissionLease
-> HyPermissionTransport
-> HyMemoryBridge.ingest / ingest_extracted / search_ids
-> write result verification
-> Governance
-> Recall Broker / UI
@@ -316,11 +316,8 @@ Evidence
不要在新业务模块里直接创建 `HyMemoryClient`,否则会绕过:
- 来源读取授权;
- 共享 Runtime、模型配置、API Key 安全存储与请求预算;
- 模型内容处理授权;
- 发送前与响应后的撤权复核;
- Evidence 引用与作者归属;
- 写后回读验证;
- Governance 状态、版本、更正和删除级联。
Hy Memory SDK 是 MemHub 的内置可信组件,可自由访问和管理本机的全部
Hy Memory 数据。来源授权仍决定哪些宿主数据可以进入本机;宿主 Recall Grant
仍决定哪些外部 AI 应用可以读取治理后的记忆。这两类边界不限制内置 SDK 的本地 CRUD。
+4 -6
View File
@@ -6,13 +6,13 @@ SDK 和记忆引擎。One Memory / `one-memory` 是历史模块、数据目录
## 发行与平台合同
公共 Python SDK 为 `hy-memory==2.0.3`。MemHub 源码版本为 `1.0.3`
(build 7),GitHub 仓库为
公共 Python SDK 为 `hy-memory==2.0.2`。MemHub 源码版本为 `1.0.1`
(build 5),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
`v1.0.3` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
`v1.0.1` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
@@ -42,8 +42,6 @@ Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命
架构详情见 [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) 和
[`docs/contracts/runtime-unification.md`](docs/contracts/runtime-unification.md)。
产品、测试和一线支持排查导入与记忆沉淀时,使用
[`docs/PRODUCT_LOG_TROUBLESHOOTING.md`](docs/PRODUCT_LOG_TROUBLESHOOTING.md)。
## 开发环境
@@ -101,7 +99,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
## Hy Memory
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.3`。SDK 配置、权限边界及 search
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.2`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+3 -3
View File
@@ -3,9 +3,9 @@
"dataLicense": "CC0-1.0",
"SPDXID": "SPDXRef-DOCUMENT",
"name": "one-memory",
"documentNamespace": "https://one-memory.local/spdx/2026-10-17T13:52:48Z",
"documentNamespace": "https://one-memory.local/spdx/2026-10-15T17:26:47Z",
"creationInfo": {
"created": "2026-10-17T13:52:48Z",
"created": "2026-10-15T17:26:47Z",
"creators": [
"Tool: one-memory-generate-sbom"
]
@@ -2335,7 +2335,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.4",
"versionInfo": "2.0.2",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+2 -2
View File
@@ -2,14 +2,14 @@
MemHub itself is licensed under Apache-2.0 (see `LICENSE`).
It bundles the runtime dependencies listed below.
Generated from the installed dependency tree on 2026-10-17T13:52:48Z
Generated from the installed dependency tree on 2026-10-15T17:26:47Z
by `python3 scripts/generate_sbom.py`. Do not edit by hand.
Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.4
## hy-memory 2.0.2
License: Apache-2.0
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/cli",
"version": "1.0.4",
"version": "1.0.1",
"description": "MemHub bootstrap CLI for installing and managing the local-first shared memory runtime",
"license": "Apache-2.0",
"repository": {
@@ -32,6 +32,7 @@ const REQUIRED_COMMAND_SURFACE = [
"calls",
"import-file",
"notes",
"permission",
"extraction",
"graph",
"models",
@@ -76,7 +77,7 @@ const REQUIRED_MARKERS = [
/**
* Sub-commands the UI-write parity work added. A package can declare the
* parent command (`models`) while still missing the new verb, so
* parent command (`permission`, `models`) while still missing the new verb, so
* checking command names alone would pass a stale bundle. The marker is the
* exact literal the catalog usage string contains.
*/
@@ -87,6 +88,9 @@ const REQUIRED_SUBCOMMANDS = [
{ command: "import-file", marker: "memhub import-file discard" },
{ command: "notes", marker: "memhub notes add" },
{ command: "notes", marker: "memhub notes process" },
{ command: "permission", marker: "memhub permission status" },
{ command: "permission", marker: "memhub permission allow" },
{ command: "permission", marker: "memhub permission deny" },
{ command: "extraction", marker: "memhub extraction set" },
{ command: "graph", marker: "memhub graph width" },
{ command: "models", marker: "memhub models edit" },
@@ -149,6 +153,8 @@ const STOPPED_DESTRUCTIVE_PROBES = [
"--confirm",
],
},
{ name: "permission_allow", args: ["permission", "allow"] },
{ name: "permission_deny", args: ["permission", "deny"] },
{
name: "extraction_set_apply",
args: ["extraction", "set", "--window", "6m", "--confirm"],
+71 -31
View File
@@ -81,7 +81,6 @@ import {
readMemoryWrite,
readScanRoots,
removeScanRoot,
resetOnboarding,
submitBackup,
submitMemoryWrite,
submitScan,
@@ -191,6 +190,13 @@ import {
renderMemoryDetailText,
renderMemoryPageText,
} from "./memory-report.js";
import {
type ModelContentPermissionDocument,
modelContentPermissionExitCode,
readModelContentPermission,
renderModelContentPermissionText,
setModelContentPermission,
} from "./model-content-permission-flow.js";
import {
executeModelProbe,
inspectModelProbePlan,
@@ -211,10 +217,12 @@ import {
executeModelProfileActivate,
executeModelProfileEdit,
type ModelWriteDocument,
type ModelWritePlan,
modelWriteExitCode,
planModelProfileActivate,
planModelProfileEdit,
renderModelWriteText,
validateModelWritePlan,
} from "./model-write-flows.js";
import { claimPlanAttempt } from "./operation-journal.js";
import {
@@ -689,36 +697,6 @@ async function runCommand(
// `path list` and `adapter list` are read-only: they have no plan to confirm,
// so they are answered here rather than falling into the two-phase branch
// (which would reject them for having nothing to apply).
// Back to the cold-start screens on purpose. Deleting the app cannot do this
// (the completion flag lives with the memories, by design), and deleting the
// data to get there would cost the user everything they have stored.
if (named === "onboarding" && arguments_.includes("reset")) {
if (
arguments_.filter((item) => item === "--json").length > 1 ||
arguments_.filter((item) => item !== "--json").join(" ") !==
"onboarding reset"
) {
return usage();
}
const attachment = await attachToRuntime(
resolveRuntimeCliConfig(environment, process.argv[1]).dataDir,
);
if (attachment === null) {
return runtimeUnavailable(descriptor, asJson);
}
const state = await resetOnboarding(attachment);
if (state === null) {
return reject(descriptor, asJson, "引导状态端点不可用。");
}
if (asJson) {
process.stdout.write(`${JSON.stringify(state)}\n`);
return EXIT_OK;
}
process.stdout.write(
`引导流程已回到首次运行(status=${state.status}),记忆数据未改动。\n`,
);
return EXIT_OK;
}
if (named === "path" && arguments_.includes("list")) {
if (
arguments_.filter((item) => item === "--json").length > 1 ||
@@ -2484,6 +2462,45 @@ async function runCommand(
);
return uiWriteExitCode(document);
}
if (named === "permission") {
const parsed = parsePermissionCommand(arguments_.slice(1));
if (!parsed) return usage();
const config = resolveRuntimeCliConfig(environment, process.argv[1]);
const runtime = await inspectRuntimeStatus({
dataDir: config.dataDir,
probeIdentity,
requireServiceReceipt: true,
});
const attached =
runtime.state === "running"
? await attachClients(config.dataDir)
: { reader: null, operations: null, failure: runtime.code };
if (
runtime.state !== "running" ||
(parsed === "status" ? !attached.reader : !attached.operations)
) {
const document: ModelContentPermissionDocument = {
schema_version: 1,
product: PRODUCT,
state: "unknown",
code:
runtime.state !== "running"
? runtime.code
: (attached.failure ?? "runtime_attach_failed"),
};
emitModelContentPermissionDocument(document, asJson);
return runtime.state === "stopped" ? 3 : 1;
}
const document =
parsed === "status"
? await readModelContentPermission(attached.reader!)
: await setModelContentPermission(
attached.operations!,
parsed === "allow",
);
emitModelContentPermissionDocument(document, asJson);
return modelContentPermissionExitCode(document);
}
if (named === "notes" || named === "extraction" || named === "graph") {
const parsed = parseExtendedWriteCommand(
named as "notes" | "extraction" | "graph",
@@ -5107,6 +5124,18 @@ function parseHeadlessCommand(
: null;
}
function parsePermissionCommand(
rest: readonly string[],
): "status" | "allow" | "deny" | null {
if (rest.filter((item) => item === "--json").length > 1) return null;
const tokens = rest.filter((item) => item !== "--json");
const action = tokens[0];
return tokens.length === 1 &&
(action === "status" || action === "allow" || action === "deny")
? action
: null;
}
function unwrapPlan(value: unknown): unknown {
if (
value &&
@@ -5294,6 +5323,17 @@ function emitModelReportDocument(
);
}
function emitModelContentPermissionDocument(
document: ModelContentPermissionDocument,
asJson: boolean,
): void {
process.stdout.write(
asJson
? `${JSON.stringify(document)}\n`
: renderModelContentPermissionText(document),
);
}
function emitSourceSyncDocument(
document: SourceSyncFlowDocument,
asJson: boolean,
+8 -8
View File
@@ -493,6 +493,14 @@ export const COMMANDS: readonly CommandDescriptor[] = [
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
},
{
name: "permission",
summary: "读取或直接设置全局模型内容外发许可。",
usage:
"memhub permission status [--json]\n memhub permission allow [--json]\n memhub permission deny [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
},
{
name: "extraction",
summary: "设置提取窗口;默认只出计划,--confirm 后才写入。",
@@ -651,14 +659,6 @@ export const COMMANDS: readonly CommandDescriptor[] = [
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
},
{
name: "onboarding",
summary:
"onboarding reset 回到首次运行的引导流程。只清除引导完成状态:不删记忆数据,也不需要卸载重装。",
usage: "memhub onboarding reset [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE],
},
{
name: "help",
summary: "显示总帮助或单个命令的用法。",
+2 -9
View File
@@ -81,6 +81,7 @@ const COMMANDS = [
"calls",
"import-file",
"notes",
"permission",
"extraction",
"graph",
"host",
@@ -92,7 +93,6 @@ const COMMANDS = [
"uninstall",
"upgrade",
"delete-data",
"onboarding",
"help",
"version",
// A6 product commands. Appended, not inserted: the catalog is a stable
@@ -259,14 +259,7 @@ describe("CLI-22 shared output contract", () => {
describe("argument validation", () => {
it("fails unknown commands without printing data or running anything", async () => {
for (const argv of [
["unknown"],
["unknown", "--help"],
["--typo"],
["permission", "status"],
["permission", "allow"],
["permission", "deny"],
]) {
for (const argv of [["unknown"], ["unknown", "--help"], ["--typo"]]) {
const streams = captureStreams();
try {
expect(await run(argv, {})).toBe(2);
@@ -384,101 +384,6 @@ describe("unified context search", () => {
expect(streams.stdout()).not.toContain("must-not-print");
});
it("counts characters as code points so an emoji hit stays grounded", async () => {
// Core reports used_characters with Python's len(), which counts code
// points: this text is 10 characters there and 12 UTF-16 units here.
const text = "消费保 📍 扩展 📡";
expect(text.length).toBe(12);
expect(Array.from(text).length).toBe(10);
transport.post.mockResolvedValue({
status: 200,
data: {
...RESPONSE,
paths: {
...RESPONSE.paths,
memory: {
...RESPONSE.paths.memory,
status: "empty",
returned_count: 0,
},
},
items: [
{
...(RESPONSE.items[0] as Record<string, unknown>),
messages: [
{
...(
RESPONSE.items[0] as {
messages: Array<Record<string, unknown>>;
}
).messages[0],
text,
},
],
},
],
used_characters: 10,
},
});
const streams = captureStreams();
try {
expect(await run(["recall", "test", "--json"])).toBe(0);
} finally {
streams.restore();
}
expect(JSON.parse(streams.stdout())).toMatchObject({
state: "grounded",
used_characters: 10,
items: [{ kind: "raw", messages: [{ text }] }],
});
});
it("still rejects a response whose character total does not match", async () => {
// The same text counted as UTF-16 units is exactly what Core does not mean,
// so the cross-check must keep failing instead of accepting both totals.
const text = "消费保 📍 扩展 📡";
transport.post.mockResolvedValue({
status: 200,
data: {
...RESPONSE,
paths: {
...RESPONSE.paths,
memory: {
...RESPONSE.paths.memory,
status: "empty",
returned_count: 0,
},
},
items: [
{
...(RESPONSE.items[0] as Record<string, unknown>),
messages: [
{
...(
RESPONSE.items[0] as {
messages: Array<Record<string, unknown>>;
}
).messages[0],
text,
},
],
},
],
used_characters: 12,
},
});
const streams = captureStreams();
try {
expect(await run(["recall", "test", "--json"])).toBe(1);
} finally {
streams.restore();
}
expect(JSON.parse(streams.stdout())).toMatchObject({
state: "invalid",
code: "response_invalid",
});
});
it("reports endpoint denial without exposing the response body", async () => {
transport.post.mockResolvedValue({
status: 403,
+2 -13
View File
@@ -181,9 +181,9 @@ function validateContextResponse(
items.push(item);
characters +=
item.kind === "memory"
? characterCount(item.content)
? String(item.content).length
: (item.messages as Array<Record<string, unknown>>).reduce(
(sum, message) => sum + characterCount(message.text),
(sum, message) => sum + String(message.text).length,
0,
);
}
@@ -330,17 +330,6 @@ function record(value: unknown): Record<string, unknown> | null {
: null;
}
/**
* Count characters the way Core does. Python's len() counts Unicode code
* points; String.prototype.length counts UTF-16 code units, so an emoji outside
* the BMP is one character to Core and two here. Comparing the two totals made
* every response containing an emoji fail the used_characters cross-check and
* surface as response_invalid.
*/
function characterCount(value: unknown): number {
return Array.from(String(value)).length;
}
function boundedText(
value: unknown,
maximum: number,
+1 -35
View File
@@ -8,6 +8,7 @@ import axios from "axios";
*/
const READ_ONLY_PATHS = [
"/api/v1/call-events",
"/api/v1/model-content-permission",
"/api/v1/home-summary",
"/api/v1/import-jobs",
"/api/v1/live-sync",
@@ -397,41 +398,6 @@ export async function readMemoryWrite(
}
}
export interface OnboardingState {
status: string;
selected_source_ids: string[];
started_at: string | null;
completed_at: string | null;
}
/** Send onboarding back to its first-run state.
*
* Returns null when the endpoint is not there, so an older runtime degrades to
* "not available" instead of failing the command. Only the completion flag
* moves: memories, sources and settings live in other tables, which is what
* makes this the honest way to see the cold-start screens again.
*/
export async function resetOnboarding(
attachment: RuntimeAttachmentLike,
): Promise<OnboardingState | null> {
try {
const response = await axios.post<OnboardingState>(
`${baseUrlOf(attachment)}/api/v1/onboarding/reset`,
undefined,
{
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout: 15_000,
proxy: false,
maxRedirects: 0,
maxContentLength: MAX_RESPONSE_BYTES,
},
);
return response.data;
} catch {
return null;
}
}
export async function readScanRoots(
attachment: RuntimeAttachmentLike,
): Promise<{
+2
View File
@@ -30,6 +30,7 @@ const IMPORT_FILE =
const MEMORY_NOTES_PATH = "/api/v1/memory-notes";
const MEMORY_NOTE_PROCESS =
/^\/api\/v1\/memory-notes\/[A-Za-z0-9][\w.:-]{0,127}\/process$/;
const MODEL_CONTENT_PERMISSION_PATH = "/api/v1/model-content-permission";
const EXTRACTION_SETTINGS_PATH = "/api/v1/extraction-settings";
const GRAPH_WIDTH_PATH = "/api/v1/graph-width";
const MANUAL_EXTRACT_PATH = "/api/v1/memory-processing/manual-extract";
@@ -133,6 +134,7 @@ export function createCoreOperations(attachment: {
!IMPORT_FILE_COMMIT.test(path) &&
path !== MEMORY_NOTES_PATH &&
!MEMORY_NOTE_PROCESS.test(path) &&
path !== MODEL_CONTENT_PERMISSION_PATH &&
path !== GRAPH_WIDTH_PATH &&
path !== MANUAL_EXTRACT_PATH &&
path !== ONBOARDING_COMPLETE_PATH &&
+12 -5
View File
@@ -259,7 +259,7 @@ describe("A2 dual-mode init", () => {
expect(streams.value()).toContain("处理:已开启");
});
it("continues the TTY wizard through model setup and probe", async () => {
it("continues the TTY wizard through model probe and content permission", async () => {
const profileId = "22222222-2222-4222-8222-222222222222";
const profile = {
profile_id: profileId,
@@ -361,6 +361,12 @@ describe("A2 dual-mode init", () => {
};
}
if (url.endsWith("/activate")) return { status: 200, data: profile };
if (url === "/api/v1/model-content-permission") {
return {
status: 200,
data: { allowed: body.allowed },
};
}
if (url === "/api/v1/onboarding/complete") {
return {
status: 200,
@@ -377,7 +383,7 @@ describe("A2 dual-mode init", () => {
},
);
const prompt = extendedPrompter(
["", "", "", "", "", "", "", "n"],
["", "", "", "", "", "", "", "", "n"],
"sk-never-print-this",
);
const streams = captureStdout();
@@ -396,6 +402,7 @@ describe("A2 dual-mode init", () => {
streams.restore();
}
expect(streams.value()).toContain("已接入来源:codex");
expect(prompt.written.join("\n")).toContain("模型内容外发:已允许");
expect(streams.value()).toContain("首次设置完成。");
expect(`${streams.value()}\n${prompt.written.join("\n")}`).not.toContain(
"sk-never-print-this",
@@ -405,10 +412,10 @@ describe("A2 dual-mode init", () => {
expect.objectContaining({ api_key: "sk-never-print-this" }),
undefined,
);
expect(transport.post).not.toHaveBeenCalledWith(
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/model-content-permission",
expect.anything(),
expect.anything(),
{ allowed: true },
undefined,
);
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/onboarding/complete",
@@ -132,6 +132,12 @@ function clients(options: { failedProbes?: number } = {}) {
if (path.endsWith("/activate")) {
return { state: "committed" as const, data: profile(true) };
}
if (path === "/api/v1/model-content-permission") {
return {
state: "committed" as const,
data: { allowed: body.allowed },
};
}
return {
state: "rejected" as const,
code: "unexpected_path",
@@ -154,8 +160,8 @@ function clients(options: { failedProbes?: number } = {}) {
}
describe("interactive model setup", () => {
it("uses App defaults and keeps the key out of output", async () => {
const prompt = prompter(["", "", ""]);
it("uses App defaults, keeps the key out of output, and allows model content", async () => {
const prompt = prompter(["", "", "", "", "", ""]);
const api = clients();
const result = await runInteractiveModelSetup(prompt.value, api, {
@@ -164,7 +170,10 @@ describe("interactive model setup", () => {
installHosts: false,
});
expect(result).toMatchObject({ state: "configured" });
expect(result).toMatchObject({
state: "configured",
model_content_allowed: true,
});
expect(api.post).toHaveBeenCalledWith(
"/api/v1/model-profiles",
expect.objectContaining({
@@ -175,10 +184,9 @@ describe("interactive model setup", () => {
embedding_provider_kind: "local_onnx",
}),
);
expect(api.post).not.toHaveBeenCalledWith(
"/api/v1/model-content-permission",
expect.anything(),
);
expect(api.post).toHaveBeenCalledWith("/api/v1/model-content-permission", {
allowed: true,
});
expect(prompt.written.join("\n")).not.toContain("sk-do-not-print");
});
@@ -197,7 +205,7 @@ describe("interactive model setup", () => {
});
it("removes an unverified profile before skipping after a failed probe", async () => {
const prompt = prompter(["", "", "", "0"]);
const prompt = prompter(["", "", "", "n", "0"]);
const api = clients();
api.post.mockImplementation(
async (path: string, body: Record<string, unknown>) => {
@@ -272,7 +280,7 @@ describe("interactive model setup", () => {
});
it("cleans a failed probe and lets the user re-enter the model settings", async () => {
const prompt = prompter(["", "", "", "", "", "", ""]);
const prompt = prompter(["", "", "", "", "", "", "", "", "", "", ""]);
const api = clients({ failedProbes: 1 });
const result = await runInteractiveModelSetup(prompt.value, api, {
@@ -281,7 +289,10 @@ describe("interactive model setup", () => {
installHosts: false,
});
expect(result).toMatchObject({ state: "configured" });
expect(result).toMatchObject({
state: "configured",
model_content_allowed: true,
});
expect(api.remove).toHaveBeenCalledTimes(1);
expect(
api.post.mock.calls.filter(([path]) => path === "/api/v1/model-profiles"),
@@ -294,7 +305,7 @@ describe("interactive model setup", () => {
const prompt = prompter(["", "", ""]);
prompt.value.ask = async () => {
asked += 1;
if (asked === 3) throw new Error("interrupted");
if (asked === 4) throw new Error("interrupted");
return "";
};
const api = clients();
@@ -311,7 +322,7 @@ describe("interactive model setup", () => {
});
it("installs selected detected hosts through preview and fingerprint apply", async () => {
const prompt = prompter(["", "", "", "", ""]);
const prompt = prompter(["", "", "", "n", "", ""]);
const api = clients();
const read = vi
.fn()
@@ -349,6 +360,7 @@ describe("interactive model setup", () => {
expect(result).toMatchObject({
state: "configured",
model_content_allowed: false,
installed_hosts: ["codex"],
});
expect(read).toHaveBeenNthCalledWith(
@@ -361,7 +373,7 @@ describe("interactive model setup", () => {
});
it("lists and installs only hosts that are actually present", async () => {
const prompt = prompter(["", "", "", "", ""]);
const prompt = prompter(["", "", "", "n", "", ""]);
const api = clients();
const read = vi
.fn()
@@ -408,7 +420,7 @@ describe("interactive model setup", () => {
});
it("completes the wizard and keeps onboarding reachable when one host install fails", async () => {
const prompt = prompter(["", "", "", "", ""]);
const prompt = prompter(["", "", "", "n", "", ""]);
const api = clients();
const read = vi
.fn()
@@ -457,6 +469,7 @@ describe("interactive model setup", () => {
// calling the install a failure.
expect(result).toMatchObject({
state: "configured",
model_content_allowed: false,
installed_hosts: ["codex"],
});
const summary = prompt.written.join("\n");
@@ -466,7 +479,7 @@ describe("interactive model setup", () => {
});
it("tells the user to install App-only hosts in the App instead of calling them failed", async () => {
const prompt = prompter(["", "", "", "", ""]);
const prompt = prompter(["", "", "", "n", "", ""]);
const api = clients();
const read = vi
.fn()
@@ -493,6 +506,7 @@ describe("interactive model setup", () => {
// only the App can install, so nothing counts as "installed".
expect(result).toMatchObject({
state: "configured",
model_content_allowed: false,
installed_hosts: [],
});
const summary = prompt.written.join("\n");
@@ -11,6 +11,7 @@ import {
} from "./headless-flows.js";
import { approvalFingerprint, listHosts, readHost } from "./host-flow.js";
import { parseSelection, type WizardPrompter } from "./init-wizard.js";
import { setModelContentPermission } from "./model-content-permission-flow.js";
import { executeModelProbe, planModelProbe } from "./model-probe-flow.js";
import {
executeModelProfileActivate,
@@ -121,6 +122,7 @@ export type InteractiveModelSetupState =
| {
state: "configured";
profile: ModelProfile;
model_content_allowed: boolean;
installed_hosts: string[];
}
| { state: "skipped" }
@@ -287,11 +289,15 @@ async function installHosts(
function writeSummary(
prompter: WizardPrompter,
profile: ModelProfile,
modelContentAllowed: boolean,
installedHosts: readonly string[],
failedHosts: readonly { hostId: string; label: string }[],
): void {
prompter.write("配置摘要:");
prompter.write(` 模型:${profile.name} / ${profile.model}`);
prompter.write(
` 模型内容外发:${modelContentAllowed ? "已允许" : "已禁止"}`,
);
prompter.write(
` 应用接入:${installedHosts.length > 0 ? installedHosts.join("、") : "已跳过"}`,
);
@@ -347,6 +353,14 @@ export async function runInteractiveModelSetup(
prompter.write("未填写 API Key。输入 back 返回,或重新选择模型服务。");
continue;
}
const permissionAnswer = (
await prompter.ask("允许将对话内容发送给模型以整理记忆?[Y/n]: ")
)
.trim()
.toLowerCase();
const modelContentAllowed = !["n", "no", "否", "0"].includes(
permissionAnswer,
);
const hostApi = options.hostApi ?? { list: listHosts, read: readHost };
const selectedHosts = options.installHosts
? await chooseDetectedHosts(prompter, options.hostHome, hostApi)
@@ -438,6 +452,13 @@ export async function runInteractiveModelSetup(
return { state: "failed", code: "model_activate_failed" };
}
const permission = await setModelContentPermission(
clients.operations,
modelContentAllowed,
);
if (permission.state !== "ok") {
return { state: "failed", code: "model_content_permission_set_failed" };
}
const { installed, failed } = await installHosts(
options.dataDir,
selectedHosts,
@@ -446,6 +467,7 @@ export async function runInteractiveModelSetup(
writeSummary(
prompter,
profile,
modelContentAllowed,
installed,
failed.map((hostId) => ({
hostId,
@@ -455,6 +477,7 @@ export async function runInteractiveModelSetup(
return {
state: "configured",
profile,
model_content_allowed: modelContentAllowed,
installed_hosts: installed,
};
}
-3
View File
@@ -180,8 +180,5 @@ describe("import-job write allow-list", () => {
await expect(
operations.post("/api/v1/import-jobs/job-1/delete", {}),
).rejects.toThrow("core_operation_path_not_allowed");
await expect(
operations.post("/api/v1/model-content-permission", { allowed: true }),
).rejects.toThrow("core_operation_path_not_allowed");
});
});
@@ -0,0 +1,154 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
}));
vi.mock("axios", () => ({
default: { create: vi.fn(() => transport), get: vi.fn() },
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
beforeEach(() => {
vi.clearAllMocks();
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue({
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
});
});
describe("model content permission CLI", () => {
it("documents only final-state permission and manual extraction commands", async () => {
const permission = await capturedRun(["help", "permission"]);
expect(permission.code).toBe(0);
expect(permission.stdout).toContain("memhub permission allow");
expect(permission.stdout).toContain("memhub permission deny");
expect(permission.stdout).not.toMatch(
/\b(preview|confirm|process|revoke)\b/,
);
expect(permission.stdout).not.toContain("revision");
expect(permission.stdout).not.toContain("receipt");
const processing = await capturedRun(["help", "processing"]);
expect(processing.code).toBe(0);
expect(processing.stdout).toContain("memhub processing extract");
expect(processing.stdout).not.toMatch(/\b(plan|apply|receipt)\b/);
expect(processing.stdout).not.toContain("backlog");
});
it("reads the final state", async () => {
transport.get.mockResolvedValue({ status: 200, data: { allowed: true } });
const result = await capturedRun(["permission", "status", "--json"]);
expect(result.code).toBe(0);
expect(JSON.parse(result.stdout)).toMatchObject({
state: "ok",
allowed: true,
});
expect(transport.get).toHaveBeenCalledWith(
"/api/v1/model-content-permission",
undefined,
);
});
it.each([
["allow", true],
["deny", false],
] as const)("sets %s directly with only allowed", async (verb, allowed) => {
transport.post.mockResolvedValue({ status: 200, data: { allowed } });
const result = await capturedRun(["permission", verb, "--json"]);
expect(result.code).toBe(0);
expect(JSON.parse(result.stdout)).toMatchObject({ state: "ok", allowed });
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/model-content-permission",
{ allowed },
undefined,
);
});
it.each(["preview", "confirm", "process", "revoke"])(
"rejects removed %s commands before runtime access",
async (verb) => {
const result = await capturedRun(["permission", verb, "--json"]);
expect(result.code).toBe(2);
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
expect(transport.get).not.toHaveBeenCalled();
expect(transport.post).not.toHaveBeenCalled();
},
);
it.each(["plan", "apply", "receipt"])(
"rejects removed processing %s commands before runtime access",
async (verb) => {
const result = await capturedRun(["processing", verb, "--json"]);
expect(result.code).toBe(2);
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
expect(transport.get).not.toHaveBeenCalled();
expect(transport.post).not.toHaveBeenCalled();
},
);
});
async function capturedRun(arguments_: string[]) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return {
code: await run(arguments_, {
ONE_MEMORY_DATA_DIR: "/synthetic/data",
}),
stdout,
stderr,
};
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
@@ -0,0 +1,66 @@
import { describe, expect, it, vi } from "vitest";
import type { CoreReader } from "./core-client.js";
import type { CoreOperations } from "./core-operations.js";
import {
readModelContentPermission,
setModelContentPermission,
} from "./model-content-permission-flow.js";
describe("model content permission flow", () => {
it("reads the single final-state boolean", async () => {
const reader = vi.fn().mockResolvedValue({
state: "ok",
data: { allowed: true },
});
await expect(
readModelContentPermission(reader as CoreReader),
).resolves.toMatchObject({ state: "ok", allowed: true });
expect(reader).toHaveBeenCalledWith("/api/v1/model-content-permission");
});
it.each([
["allow", true],
["deny", false],
] as const)(
"sets %s directly without revision or receipt fields",
async (_, allowed) => {
const post = vi.fn().mockResolvedValue({
state: "committed",
data: { allowed },
});
await expect(
setModelContentPermission({ post } as CoreOperations, allowed),
).resolves.toMatchObject({ state: "ok", allowed });
expect(post).toHaveBeenCalledWith("/api/v1/model-content-permission", {
allowed,
});
},
);
it("does not accept a malformed or mismatched success response", async () => {
const malformed = vi.fn().mockResolvedValue({
state: "committed",
data: { allowed: true, revision: 1 },
});
await expect(
setModelContentPermission({ post: malformed } as CoreOperations, true),
).resolves.toMatchObject({
state: "unknown",
code: "model_content_permission_response_invalid",
});
const mismatched = vi.fn().mockResolvedValue({
state: "committed",
data: { allowed: false },
});
await expect(
setModelContentPermission({ post: mismatched } as CoreOperations, true),
).resolves.toMatchObject({
state: "unknown",
code: "model_content_permission_state_mismatch",
});
});
});
@@ -0,0 +1,96 @@
import type { CoreReader } from "./core-client.js";
import type { CoreOperations } from "./core-operations.js";
import { PRODUCT } from "./version.js";
export type ModelContentPermissionDocument =
| {
schema_version: 1;
product: string;
state: "ok";
allowed: boolean;
}
| {
schema_version: 1;
product: string;
state: "rejected" | "unknown";
code: string;
};
export async function readModelContentPermission(
reader: CoreReader,
): Promise<ModelContentPermissionDocument> {
const outcome = await reader<unknown>("/api/v1/model-content-permission");
if (outcome.state !== "ok") {
return failure("unknown", outcome.code);
}
return permissionState(
outcome.data,
"model_content_permission_response_invalid",
);
}
export async function setModelContentPermission(
operations: CoreOperations,
allowed: boolean,
): Promise<ModelContentPermissionDocument> {
const outcome = await operations.post<unknown>(
"/api/v1/model-content-permission",
{ allowed },
);
if (outcome.state === "rejected") {
return failure("rejected", outcome.code);
}
if (outcome.state === "unknown") {
return failure("unknown", outcome.code);
}
const document = permissionState(
outcome.data,
"model_content_permission_response_invalid",
);
return document.state === "ok" && document.allowed !== allowed
? failure("unknown", "model_content_permission_state_mismatch")
: document;
}
export function modelContentPermissionExitCode(
document: ModelContentPermissionDocument,
): number {
return document.state === "ok" ? 0 : 1;
}
export function renderModelContentPermissionText(
document: ModelContentPermissionDocument,
): string {
if (document.state === "ok") {
return `模型内容外发:${document.allowed ? "允许" : "禁止"}\n`;
}
return `模型内容权限未更新:${document.state}(${document.code})\n`;
}
function permissionState(
value: unknown,
invalidCode: string,
): ModelContentPermissionDocument {
if (
value === null ||
typeof value !== "object" ||
Array.isArray(value) ||
Object.keys(value).length !== 1 ||
typeof (value as Record<string, unknown>).allowed !== "boolean"
) {
return failure("unknown", invalidCode);
}
return {
schema_version: 1,
product: PRODUCT,
state: "ok",
allowed: (value as { allowed: boolean }).allowed,
};
}
function failure(
state: "rejected" | "unknown",
code: string,
): ModelContentPermissionDocument {
return { schema_version: 1, product: PRODUCT, state, code };
}
+11 -6
View File
@@ -137,8 +137,8 @@ describe("package matrix harness", () => {
};
expect(document.state).toBe("verified");
const results = document.commands.stopped_destructive.results;
// 11 destructive commands, each unknown/runtime_not_running with exit 3.
expect(Object.keys(results)).toHaveLength(11);
// 13 destructive commands, each unknown/runtime_not_running with exit 3.
expect(Object.keys(results)).toHaveLength(13);
for (const [name, result] of Object.entries(results)) {
expect(result, `${name} did not fail closed`).toEqual({
exit: 3,
@@ -162,7 +162,7 @@ describe("package matrix harness", () => {
const shim = [
"#!/usr/bin/env node",
"const a=process.argv.slice(2);",
"if(['calls','import-file','notes','extraction','graph','models'].includes(a[0])||(a[0]==='processing'&&a[1]==='extract')){",
"if(['calls','import-file','notes','permission','extraction','graph','models'].includes(a[0])||(a[0]==='processing'&&a[1]==='extract')){",
' process.stdout.write(\'{"schema_version":1,"product":"memhub","state":"committed","operation_id":"x","receipt":{}}\\n\');',
" process.exit(0);",
"}",
@@ -188,7 +188,7 @@ describe("package matrix harness", () => {
expect(failure?.state).toBe("failed");
expect(
failure?.failures.filter((item) => item.includes("did not fail closed")),
).toHaveLength(11);
).toHaveLength(13);
});
it("can require an expected bundle SHA and reports bundle metadata", {
@@ -280,8 +280,9 @@ describe("package matrix harness", () => {
});
it("rejects a catalog that declares the parent command but not the new verb", async () => {
// A bundle can still declare `models` while missing `models activate`.
// Names alone would pass it; the usage marker must not.
// A bundle can still declare `permission`/`models` while missing
// `permission allow` or `models activate`. Names alone would pass it; the
// usage marker must not.
const declared = [
"install",
"init",
@@ -291,6 +292,7 @@ describe("package matrix harness", () => {
"calls",
"import-file",
"notes",
"permission",
"extraction",
"graph",
"models",
@@ -340,6 +342,9 @@ describe("package matrix harness", () => {
},
};
const failures = harness.validateCatalog(catalog);
expect(failures).toContain(
'command "permission" is missing usage "memhub permission allow"',
);
expect(failures).toContain(
'command "models" is missing usage "memhub models activate"',
);
@@ -151,6 +151,9 @@ describe("memhub short-command contract", () => {
"memhub import-file discard",
"memhub notes add",
"memhub notes process",
"memhub permission status",
"memhub permission allow",
"memhub permission deny",
"memhub models setup",
"memhub models activate",
"memhub models edit",
@@ -160,9 +163,6 @@ describe("memhub short-command contract", () => {
]) {
expect(flattened).toContain(shortForm);
}
expect(COMMANDS.some((command) => command.name === "permission")).toBe(
false,
);
});
it("ships a memhub bin entry pointing at a real bundle path", () => {
@@ -840,7 +840,6 @@ describe("CLI-21 memory reads", () => {
expect(isReadOnlyPath("/api/v1/memories/graph")).toBe(false);
expect(isReadOnlyPath("/api/v1/memories/ask")).toBe(false);
expect(isReadOnlyPath("/api/v1/memories/m1")).toBe(true);
expect(isReadOnlyPath("/api/v1/model-content-permission")).toBe(false);
expect(
isReadOnlyPath(
"/api/v1/memory-mutations/123e4567-e89b-12d3-a456-426614174000",
+1 -1
View File
@@ -11,7 +11,7 @@ folder_layout = layout == 'folder'
datas = [('src/one_memory/embedding_assets', 'one_memory/embedding_assets')]
binaries = []
# Loaded lazily by the API; static dependency analysis cannot discover it.
hiddenimports = []
hiddenimports = ['one_memory.hy_permission_context']
tmp_ret = collect_all('hy_memory')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('chromadb')
+2 -2
View File
@@ -4,7 +4,7 @@ build-backend = "uv_build"
[project]
name = "one-memory"
version = "1.0.4"
version = "1.0.1"
description = "MemHub local-first memory core"
readme = "README.md"
requires-python = ">=3.12"
@@ -12,7 +12,7 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.4",
"hy-memory==2.0.2",
# Retrieval's derived vector index directly consumes these locked versions.
"chromadb==1.5.9",
"numpy==2.5.2",
+77 -323
View File
@@ -20,6 +20,7 @@ from copy import copy
from datetime import UTC, datetime, timedelta
from enum import StrEnum
from functools import partial
from importlib import import_module
from pathlib import Path
from typing import Any, Literal, cast
from uuid import UUID
@@ -100,13 +101,7 @@ from one_memory.imports.access import (
SourceAccessError,
root_identity_matches,
)
from one_memory.imports.adapters import (
ScanFailure,
ScannedEvidence,
ScanOptions,
SourceAdapter,
SyncCompleteness,
)
from one_memory.imports.adapters import SourceAdapter, SyncCompleteness
from one_memory.imports.candidates import (
SourceCandidateError,
SourceCandidateSelection,
@@ -117,11 +112,7 @@ from one_memory.imports.connections import (
ConnectionIntentReceipt,
SourceConnectionStore,
)
from one_memory.imports.grants import (
SourceAuthorizationState,
SourceGrantReceipt,
SourceGrantStore,
)
from one_memory.imports.grants import SourceGrantReceipt, SourceGrantStore
from one_memory.imports.host_packages import CapabilityState, HostPackageCapabilities
from one_memory.imports.jobs import (
ImportIssueRecord,
@@ -173,6 +164,7 @@ from one_memory.memory_library import (
)
from one_memory.memory_notes import MemoryNoteStore, memory_note_router
from one_memory.memory_processing import (
MemoryBridgeProtocol,
MemoryProcessingSupervisor,
_conversation_chunks,
memory_task_timeout_seconds,
@@ -186,8 +178,10 @@ from one_memory.memory_queue import (
)
from one_memory.memory_server.config_state import ConfigStore
from one_memory.memory_tasks import MemoryTask, MemoryTaskStatus, MemoryTaskStore
from one_memory.model_content_routes import model_content_router
from one_memory.model_presets import seed_preset_profile
from one_memory.model_probe import ModelProber, ModelProbeResult
from one_memory.model_processing_permission_guard import ModelProcessingPermissionLease
from one_memory.model_profiles import (
ActiveModelProfileError,
ModelCreationOperationConflictError,
@@ -200,7 +194,6 @@ from one_memory.model_profiles import (
ModelProfileStore,
ModelProfileUnavailableError,
)
from one_memory.model_request_audit import AuditedModelTransport, ModelRequestAuditStore
from one_memory.model_request_budget import BudgetedModelTransport, ModelRequestBudget
from one_memory.onboarding import OnboardingState, OnboardingStore
from one_memory.recall import GovernedRecallTarget, RecallReceipt, RecallRequest
@@ -292,9 +285,6 @@ class SourceResponse(BaseModel):
candidate_revision: int = 0
candidate_change_allowed: bool = False
candidate_change_reason: str | None = None
#: Why `authorized` is false, so the UI can ask for the one thing that
#: would fix it instead of offering every recovery at once.
authorization_state: SourceAuthorizationState = SourceAuthorizationState.OK
class PreviewSourceCandidateRequest(BaseModel):
@@ -327,22 +317,6 @@ class SourceSyncState(BaseModel):
sync_enabled: bool
class SedimentationRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
enabled: StrictBool
class SedimentationState(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
enabled: bool
#: What the switch means for the model bill and for what still runs. Read
#: by the app so the setting can say so without hardcoding the wording.
pending_retained: bool = True
imports_unaffected: bool = True
class DetectionResponse(BaseModel):
source_id: str
detected: bool
@@ -369,7 +343,7 @@ class ConnectSourcesRequest(BaseModel):
import_upload_id: UUID | None = None
ONBOARDING_LARGE_HISTORY_USER_TURNS = 200
ONBOARDING_LARGE_HISTORY_SESSIONS = 200
ONBOARDING_HISTORY_WINDOW = timedelta(days=3)
@@ -844,13 +818,6 @@ def create_app(
evidence=app.state.evidence_store,
)
app.state.memory_write_queue = memory_write_queue
# Builds through 1.0.2 parked pre-invocation jobs behind the
# retired model-content consent bit. The embedded SDK is trusted
# now, so make those durable jobs claimable on the first startup.
await asyncio.to_thread(
memory_write_queue.readmit_parked,
states=("waiting_for_permission",),
)
repaired_identity_failures = await asyncio.to_thread(
memory_write_queue.repair_pre_send_identity_failures
)
@@ -916,7 +883,6 @@ def create_app(
)
app.state.import_job_store = ImportJobStore(database_path)
app.state.onboarding_store = OnboardingStore(database_path)
app.state.history_import_gate_released = False
app.state.extraction_settings_store = ExtractionSettingsStore(database_path)
app.state.source_grant_store = SourceGrantStore(
database_path,
@@ -939,7 +905,6 @@ def create_app(
app.state.source_permission_locks = {
source_id: asyncio.Lock() for source_id in app.state.source_catalog
}
app.state.sedimentation_lock = asyncio.Lock()
_follow_workbuddy_home(app, runtime_settings.source_home)
app.state.source_detection = {}
app.state.secret_store = secret_store or KeyringSecretStore()
@@ -995,7 +960,7 @@ def create_app(
# Not fatal: without the tables every read checks exactly, as
# it does today.
app.state.memory_visibility_error = str(error)
model_request_budget = (
app.state.model_request_budget = (
ModelRequestBudget(
database_path,
request_limit=runtime_settings.model_request_limit,
@@ -1005,8 +970,6 @@ def create_app(
if runtime_settings.model_request_limit is not None
else None
)
app.state.model_request_budget = model_request_budget
app.state.model_request_audit_store = ModelRequestAuditStore(database_path)
app.state.memory_answerer = MemoryAnswerer(
governance=app.state.memory_governance_store,
filter_memories=app.state.source_read_access.memories,
@@ -1045,23 +1008,6 @@ def create_app(
governance=app.state.memory_governance_store,
data_dir=runtime_settings.data_dir,
extraction_settings=app.state.extraction_settings_store,
bridge_factory=lambda profile, api_key, data_dir: HyMemoryBridge(
profile=profile,
api_key=api_key,
data_dir=data_dir,
http_transport_factory=lambda: AuditedModelTransport(
app.state.model_request_audit_store,
profile_id=profile.profile_id,
profile_revision=profile.revision,
provider_kind=str(profile.provider_kind),
model=profile.model,
inner=(
BudgetedModelTransport(model_request_budget)
if model_request_budget is not None
else None
),
),
),
# `None` means "whatever the active profile allows". Pinned to
# 1 in `a59f4b5` as temporary load-shedding for an acceptance
# package, and left pinned after it: a hardcoded 1 is rule 9's
@@ -1070,9 +1016,12 @@ def create_app(
# profile concurrency 4, recall latency is unchanged while
# precipitation goes 4x.
max_concurrent_tasks=runtime_settings.precipitation_max_concurrent_tasks,
model_content_config=app.state.config_store,
permission_bridge_factory=lambda lease, secret, directory: _permission_context(
app, lease, secret, directory
),
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
service_levels=app.state.service_levels,
sedimentation_enabled=lambda: _config_store(app).sedimentation_enabled(),
)
# C2 owns the single production budget (handle 5s + reserve 250ms,
# total 5.25s, lease 10s, renew 1s, claim 250ms). The composition
@@ -1090,27 +1039,18 @@ def create_app(
job.job_id,
)
await asyncio.to_thread(
memory_write_queue.retry_first_unknown_after_reconcile,
older_than=datetime.now(UTC) - timedelta(seconds=10),
memory_write_queue.retry_timed_out_unknown_after_reconcile,
older_than=datetime.now(UTC) - timedelta(minutes=5),
limit=32,
)
await asyncio.to_thread(
app.state.memory_task_store.retry_first_unknown_after_reconcile,
older_than=datetime.now(UTC) - timedelta(seconds=10),
limit=128,
)
async def admit_hy_invocation(
source_id: str,
mark_invoked: Callable[[], Awaitable[bool]],
) -> bool:
"""Serialize pause with the last boundary before an HY call."""
source_lock = app.state.source_permission_locks.setdefault(
source_id, asyncio.Lock()
)
async with app.state.sedimentation_lock, source_lock:
if not _config_store(app).sedimentation_enabled():
raise RuntimeError("model_sedimentation_disabled")
lock = app.state.source_permission_locks.setdefault(source_id, asyncio.Lock())
async with lock:
if not _source_writes_on(app, source_id):
raise RuntimeError("model_content_source_disabled")
return await mark_invoked()
@@ -1148,18 +1088,8 @@ def create_app(
+ 10
),
reconcile_unknown=reconcile_unknown_writes,
source_writes_permitted=lambda source_id: _sedimentation_claim_enabled(
app, source_id
),
# The claim gate has to be the same question the write gate
# asks, not a second answer to it. It used to be
# `sync_enabled`, the per-app toggle, and `one-memory-manual` is
# not a connected app: it has no entry in the configuration
# document, so that predicate answers False for it while the
# write gate admits it by its grant. Every manual note's job was
# therefore skipped on every drain tick — the note stayed
# "saved, waiting" with no task behind it, forever.
source_claim_enabled=lambda source_id: _sedimentation_claim_enabled(app, source_id),
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
source_claim_enabled=lambda source_id: _config_store(app).sync_enabled(source_id),
)
app.state.memory_write_drain = memory_write_drain
app.state.memory_write_drain_task = watched_task(
@@ -1188,7 +1118,10 @@ def create_app(
for chunk in _conversation_chunks(conversation, max_turns=12, completed_only=True):
if (
record not in chunk
or not any(item.role == "user" for item in chunk)
or not any(
item.role == "user" and item.written_by.value == "person"
for item in chunk
)
or not any(item.role == "assistant" for item in chunk)
):
continue
@@ -1201,6 +1134,7 @@ def create_app(
app.state.matter_discovery.processor = app.state.memory_processing_supervisor
app.state.matter_discovery.write_intake = app.state.memory_write_intake
app.state.matter_discovery.write_queue = memory_write_queue
app.state.processing_coordinator = None
app.state.memory_note_store = MemoryNoteStore(
app.state.evidence_store,
app.state.memory_processing_supervisor,
@@ -1512,6 +1446,7 @@ def create_app(
app.include_router(matter_router(protected))
app.include_router(memory_note_router(protected))
app.include_router(file_import_router(protected))
app.include_router(model_content_router(protected))
app.include_router(context_router(protected))
def require_workbuddy_boundary(request: Request) -> None:
@@ -1801,30 +1736,6 @@ def create_app(
"events": events,
}
@app.get("/api/v1/diagnostics/model-usage", dependencies=protected)
def diagnostic_model_usage(
since: datetime | None = Query(default=None), # noqa: B008
request_id: str | None = Query( # noqa: B008
default=None, min_length=1, max_length=512
),
limit: int = Query(default=200, ge=1, le=2_000), # noqa: B008
) -> dict[str, object]:
"""Return content-free precipitation usage for provider reconciliation."""
normalized_since = (
(since if since.tzinfo is not None else since.replace(tzinfo=UTC))
if since is not None
else None
)
report = app.state.model_request_audit_store.report(
since=normalized_since.astimezone(UTC).isoformat()
if normalized_since is not None
else None,
request_id=request_id,
limit=limit,
)
return {"schema_version": 1, **report}
@app.get(
"/api/v1/runtime/identity",
response_model=HealthResponse,
@@ -1962,20 +1873,6 @@ def create_app(
"""Record UI completion only; finishing or skipping is never model consent."""
return await asyncio.to_thread(_onboarding_store(app).complete)
@app.post(
"/api/v1/onboarding/reset",
response_model=OnboardingState,
dependencies=protected,
)
async def reset_onboarding() -> OnboardingState:
"""Go back to the first-run screens. Only the completion flag moves.
Memories, sources and settings live in other tables and are untouched,
which is the point: someone who wants to see the cold-start flow again
should not have to delete anything to do it.
"""
return await asyncio.to_thread(_onboarding_store(app).reset)
@app.get(
"/api/v1/sources",
response_model=list[SourceResponse],
@@ -2013,11 +1910,6 @@ def create_app(
candidate_change_reason=app.state.source_candidate_store.change_reason(
registration
),
authorization_state=grant_store.authorization_state(
user_id="local",
source_id=source_id,
candidate=registration.candidate.path,
),
)
for source_id, registration in _catalog(app).items()
]
@@ -2329,39 +2221,24 @@ def create_app(
raise HTTPException(status_code=409, detail=error.code) from None
if grant is None:
raise HTTPException(status_code=409, detail="grant_required")
try:
await _prepare_source_resume(source_id)
except BaseException as error:
raise _sync_refused("sync_prepare_failed") from error
await _prepare_source_resume(source_id)
else:
try:
await _live_sync(app).stop_source(source_id)
except BaseException as error:
raise _sync_refused("sync_prepare_failed") from error
await _live_sync(app).stop_source(source_id)
try:
await asyncio.to_thread(_config_store(app).set_sync, source_id, request.enabled)
except BaseException as error:
except BaseException:
if request.enabled:
await _live_sync(app).stop_source(source_id)
else:
await _prepare_source_resume(source_id)
await _activate_source_resume(source_id)
await _refresh_live_watch(app)
raise _sync_refused("sync_config_write_failed") from error
raise
try:
if request.enabled:
try:
await _activate_source_resume(source_id)
except BaseException as error:
raise _sync_refused("sync_resume_failed") from error
try:
await _refresh_live_watch(app)
except BaseException as error:
raise _sync_refused("sync_watcher_failed") from error
await _activate_source_resume(source_id)
await _refresh_live_watch(app)
except BaseException:
# Rolls back the desired state and settles the watcher exactly as
# before; the only change is that the raise above now carries a
# code saying which stage failed.
await asyncio.to_thread(_config_store(app).set_sync, source_id, previous)
if request.enabled:
await _live_sync(app).stop_source(source_id)
@@ -2451,7 +2328,7 @@ def create_app(
await asyncio.to_thread(
app.state.memory_task_store.park_source_for_permission,
source_id=source_id,
error_code="hy_memory_source_revoked",
error_code="model_content_permission_required",
)
await _live_sync(app).stop_source(source_id)
_supervisor(app).unregister_adapter(source_id)
@@ -2492,8 +2369,6 @@ def create_app(
async def connect_sources(
request: ConnectSourcesRequest,
) -> ConnectSourcesResponse:
if request.onboarding:
app.state.history_import_gate_released = False
source_ids = tuple(dict.fromkeys(request.source_ids))
registrations = [_require_registration(app, source_id) for source_id in source_ids]
if request.request_id is None:
@@ -3659,48 +3534,6 @@ def create_app(
def put_extraction_settings(request: ExtractionWindowUpdate) -> ExtractionSettings:
return _memory_processing_supervisor(app).set_extraction_window(request.window_preset)
@app.get(
"/api/v1/sedimentation",
response_model=SedimentationState,
dependencies=protected,
)
def get_sedimentation() -> SedimentationState:
"""Whether the model may still turn conversations into memories."""
return SedimentationState(enabled=_config_store(app).sedimentation_enabled())
@app.put(
"/api/v1/sedimentation",
response_model=SedimentationState,
dependencies=protected,
)
async def put_sedimentation(request: SedimentationRequest) -> SedimentationState:
"""Start or stop model-driven sedimentation, keeping the configuration.
Stopping is the user's cost control: nothing further goes to the model
and already queued work waits rather than failing, so starting again
resumes it. Importing conversations is not affected either way.
"""
store = _config_store(app)
changed = False
async with app.state.sedimentation_lock:
previous = store.sedimentation_enabled()
if previous is not request.enabled:
await asyncio.to_thread(store.set_sedimentation_enabled, request.enabled)
changed = True
diagnostic_event(
level="info",
component="memory.sedimentation",
event="memory_sedimentation.toggled",
outcome="succeeded",
details={"enabled": request.enabled},
)
if changed and request.enabled:
# Work parked by the stop is picked up again rather than waiting
# for the next natural dispatch, but not on this request's time:
# see _resume_sedimentation.
_resume_sedimentation(app)
return SedimentationState(enabled=store.sedimentation_enabled())
@app.post(
"/api/v1/memory-processing/manual-extract",
response_model=ManualExtractReceipt,
@@ -4061,8 +3894,12 @@ async def _authorize_answer_generation(
# The explicit question is this operation's input, not a standing permission for history.
if not question.strip() or not grounding:
raise MemoryAnswerError("memory_answer_generation_not_authorized")
if not await asyncio.to_thread(_config_store(app).model_content_allowed):
raise MemoryAnswerError("memory_answer_generation_not_authorized")
def check() -> None:
if not _config_store(app).model_content_allowed():
raise MemoryAnswerError("memory_answer_generation_not_authorized")
if _model_profile_store(app).active_identity() != (
profile.profile_id,
profile.revision,
@@ -4086,6 +3923,29 @@ async def _authorize_answer_generation(
return recheck
@asynccontextmanager
async def _permission_context(
app: FastAPI,
lease: ModelProcessingPermissionLease,
secret: str | None,
directory: Path,
) -> AsyncIterator[MemoryBridgeProtocol]:
module = await asyncio.to_thread(import_module, "one_memory.hy_permission_context")
budget = cast(ModelRequestBudget | None, app.state.model_request_budget)
source_id = lease.records[0].source_id if lease.records else "manual"
factory = module.permission_bridge_factory(
lambda profile, api_key, data_dir: _memory_processing_supervisor(app)._bridge(
profile,
api_key,
data_dir,
scope_key=source_id,
),
**({"http_transport": BudgetedModelTransport(budget)} if budget else {}),
)
async with factory(lease, secret, directory) as bridge:
yield cast(MemoryBridgeProtocol, bridge)
class _LazyASGIApp:
"""Compatibility ASGI app that is built only when a server imports it.
@@ -4117,16 +3977,6 @@ class _LazyASGIApp:
app = _LazyASGIApp()
def _sync_refused(code: str) -> HTTPException:
"""A sync refusal that says which stage refused.
Turning the switch on takes four steps: authorize, prepare, write, resume.
The UI used to be told only that the switch had sprung back, and every
stage needs a different fix, so each one gets its own code.
"""
return HTTPException(status_code=409, detail=code)
def _granted_roots(
source_ids: Iterable[str],
grant_store: SourceGrantStore,
@@ -4178,7 +4028,6 @@ def _source_response(
totals: tuple[int, int],
*,
candidate_change_reason: str | None = None,
authorization_state: SourceAuthorizationState = SourceAuthorizationState.OK,
) -> SourceResponse:
location = inspect_source_location(
registration.descriptor.source_id, registration.candidate.path
@@ -4391,7 +4240,7 @@ async def _onboarding_history_after(
home: Path,
now: datetime | None = None,
) -> datetime | None:
"""Return one shared cutoff when any selected harness has large, old history.
"""Return one shared cutoff when any selected harness has large history.
The cutoff is decided before any history job is created, then persisted on
every accepted job. This is deliberately global: independently limiting
@@ -4400,8 +4249,6 @@ async def _onboarding_history_after(
"""
if not request.onboarding:
return None
anchor = now or datetime.now(UTC)
cutoff = anchor - ONBOARDING_HISTORY_WINDOW
revisions = request.source_candidate_revisions or {}
for registration in registrations:
source_id = registration.descriptor.source_id
@@ -4429,52 +4276,29 @@ async def _onboarding_history_after(
home=home,
)
adapter = registration.create_adapter(grant)
if await _has_large_old_history(adapter, cutoff=cutoff):
return cutoff
count_sessions = cast(
Callable[..., Awaitable[int]] | None,
getattr(adapter, "count_sessions", None),
)
if count_sessions is not None:
session_count = await count_sessions(limit=ONBOARDING_LARGE_HISTORY_SESSIONS + 1)
else:
pending = _install_state(adapter)
if pending is None:
continue
session_count = (await pending).session_count
except (OSError, SourceAccessError):
# A supported source whose metadata count cannot be completed is
# treated conservatively. The later connection still owns the
# precise authorization error, while onboarding never falls back
# to an unbounded history import.
return cutoff
session_count = ONBOARDING_LARGE_HISTORY_SESSIONS + 1
if session_count > ONBOARDING_LARGE_HISTORY_SESSIONS:
anchor = now or datetime.now(UTC)
return anchor - ONBOARDING_HISTORY_WINDOW
return None
async def _has_large_old_history(adapter: SourceAdapter, *, cutoff: datetime) -> bool:
"""Probe one authorized Harness with the same parsers the import will use.
Stop as soon as both policy facts are proven: more than 200 genuine person
turns and at least one record older than the three-day cutoff. Counting
parsed Evidence instead of transcript files prevents one giant coding
session from bypassing the limit, while reusing the adapters keeps host
envelopes and tool-shaped user records out of the count.
"""
user_turns = 0
has_old_data = False
async for event in adapter.scan(ScanOptions()):
if isinstance(event, ScanFailure):
# A partial probe must never turn into an unbounded first import.
return True
if not isinstance(event, ScannedEvidence):
continue
observed = event.evidence.created_at
if observed is None and event.cursor.observed_modified_ns > 0:
observed = datetime.fromtimestamp(
event.cursor.observed_modified_ns / 1_000_000_000,
tz=UTC,
)
if observed is not None:
if observed.tzinfo is None:
observed = observed.replace(tzinfo=UTC)
has_old_data = has_old_data or observed.astimezone(UTC) < cutoff
if event.evidence.role == "user":
user_turns += 1
if user_turns > ONBOARDING_LARGE_HISTORY_USER_TURNS and has_old_data:
return True
return False
def _install_state(adapter: object) -> Awaitable[SourceInstallState] | None:
"""Ask an adapter for the two-level answer when it has one.
@@ -4766,7 +4590,7 @@ async def _keep_loop_heartbeat(watchdog: LoopWatchdog) -> None:
async def _emit_resource_snapshots(interval_seconds: float = 60.0) -> None:
while True:
diagnostic_event(
level="debug",
level="info",
component="core.resources",
event="core.resource_snapshot",
outcome="observed",
@@ -4791,33 +4615,6 @@ def _start_backfill(app: FastAPI) -> None:
)
def _resume_sedimentation(app: FastAPI) -> None:
"""Pick the work parked by a stop back up, off the caller's clock.
Turning the switch back on is a decision, not a job the caller waits for:
the choice is already stored, and what follows it is maintenance over every
unfinished task followed by a scan of the whole queue. On a real backlog
that runs for many minutes, so awaiting it here left the PUT unanswered
until the client gave up — the renderer showed "启用未能确认" for a switch
that had in fact been saved, and the switch only appeared to turn on once
the follow-up read came back. Measured: one toggle, twenty-plus minutes,
no response, client timeout at thirty seconds.
Repeated toggles reuse the running walk; it reads the switch as it goes, so
a stop part-way through is honoured on the next batch instead of needing a
cancel.
"""
running = cast("asyncio.Task[None] | None", getattr(app.state, "sedimentation_resume", None))
if running is not None and not running.done():
return
app.state.sedimentation_resume = watched_task(
asyncio.get_running_loop(),
lambda: _memory_processing_supervisor(app).resume_allowed(),
component="memory.sedimentation",
name="one-memory-sedimentation-resume",
)
def _start_visibility_recompute(app: FastAPI) -> None:
"""Keep the stored visibility conclusions current, in the background.
@@ -4926,6 +4723,8 @@ async def _run_backfill(app: FastAPI) -> None:
app.state.memory_backfill_error = None
if not app.state.settings.automatic_memory_processing:
return
if not await asyncio.to_thread(_config_store(app).model_content_allowed):
return
try:
queue = app.state.memory_write_queue
intake = app.state.memory_write_intake
@@ -5061,51 +4860,6 @@ def _source_writes_on(app: FastAPI, source_id: str) -> bool:
return _source_processing_enabled(app, source_id)
def _sedimentation_claim_enabled(app: FastAPI, source_id: str) -> bool:
"""Claim queued model work only while both global and source gates allow it."""
return (
_config_store(app).sedimentation_enabled()
and _history_import_gate_released(app)
and _source_writes_on(app, source_id)
)
def _history_import_gate_released(app: FastAPI) -> bool:
"""Hold first sedimentation until every accepted history import is terminal."""
if getattr(app.state, "history_import_gate_released", False):
return True
onboarding = getattr(app.state, "onboarding_store", None)
jobs = getattr(app.state, "import_job_store", None)
if onboarding is None or jobs is None:
return True
state = onboarding.get()
if state.status != "importing" or not state.selected_source_ids:
app.state.history_import_gate_released = True
return True
selected = set(state.selected_source_ids)
matching = [
job
for job in jobs.list_jobs()
if job.operation == "history"
and job.source_id in selected
and (state.started_at is None or job.created_at >= state.started_at)
]
if {job.source_id for job in matching} != selected:
return False
active = {
ImportStatus.PENDING,
ImportStatus.RUNNING,
ImportStatus.PAUSED,
ImportStatus.INTERRUPTED,
}
released = all(job.status not in active for job in matching)
if released:
app.state.history_import_gate_released = True
return released
def _model_prober(app: FastAPI) -> ModelProber:
return cast(ModelProber, app.state.model_prober)
@@ -175,22 +175,6 @@ async def in_interactive_read[T](
return await in_read_pool(interactive_read_pool(), work, *args, **kwargs)
def _retrieve[T](future: "asyncio.Future[T]") -> None:
"""Mark a shielded read's outcome as seen.
`in_read_pool` shields the wrapped worker, so a caller that gives up and
cancels leaves the read running to completion. When that read then fails,
nothing awaits the future any more: asyncio reports the orphan at GC as
"Future exception was never retrieved", the loop handler turns it into a
`core.loop.task_exception` error diagnostic, and a lane that degraded on
its own budget -- correctly, and already reported -- looks like a crashed
task on top of it. Retrieving here keeps the diagnostics honest. Awaiting
still raises for callers that are still there, so no failure is hidden.
"""
if not future.cancelled():
future.exception()
async def in_read_pool[T](
pool: ThreadPoolExecutor,
work: Callable[..., T],
@@ -209,9 +193,7 @@ async def in_read_pool[T](
_interactive_reads -= 1
worker.add_done_callback(released)
pending = asyncio.wrap_future(worker)
pending.add_done_callback(_retrieve)
return await asyncio.shield(pending)
return await asyncio.shield(asyncio.wrap_future(worker))
async def wait_for_interactive_reads() -> None:
+3 -21
View File
@@ -266,13 +266,7 @@ def serve(*, maintenance: bool = False, reader: bool = False) -> None:
)
application = create_reader_app(reader_settings)
print(f"reader_port:{port}", file=sys.stderr, flush=True)
uvicorn.run(
application,
host=settings.host,
port=port,
reload=False,
access_log=False,
)
uvicorn.run(application, host=settings.host, port=port, reload=False)
return
from one_memory.api import create_app
@@ -303,21 +297,9 @@ def serve(*, maintenance: bool = False, reader: bool = False) -> None:
)
if maintenance:
with CoreRuntime(settings, publish_receipt=False):
uvicorn.run(
application,
host=settings.host,
port=settings.port,
reload=False,
access_log=False,
)
uvicorn.run(application, host=settings.host, port=settings.port, reload=False)
else:
uvicorn.run(
application,
host=settings.host,
port=settings.port,
reload=False,
access_log=False,
)
uvicorn.run(application, host=settings.host, port=settings.port, reload=False)
if __name__ == "__main__":
@@ -196,7 +196,7 @@ def read_raw_context(
if _prefers_declared_facts(request.query)
else "1"
)
person_authored = "CASE WHEN e.role='user' THEN 1 ELSE 0 END"
person_authored = "CASE WHEN e.written_by='person' THEN 1 ELSE 0 END"
query_echo = (
"CASE WHEN trim(e.verbatim)=? OR trim(e.verbatim)=? THEN 0 ELSE 1 END"
if _prefers_declared_facts(request.query)
@@ -470,8 +470,9 @@ def revalidate_raw_context(
)
except (ValueError, TypeError):
break
if (row["role"], created) != (
if (row["role"], row["written_by"], created) != (
message.role,
message.written_by,
message.created_at,
):
break
@@ -66,12 +66,12 @@ def _field(text: str) -> ResumeField | None:
return None
def _raw_class(role: str, evidence_id: str, accepted: frozenset[str]) -> EvidenceClass:
if role == "user":
def _raw_class(written_by: str, evidence_id: str, accepted: frozenset[str]) -> EvidenceClass:
if written_by == "person":
return "person_statement"
if role == "assistant":
if written_by == "agent":
return "agent_report"
if role == "tool" and evidence_id in accepted:
if written_by == "tool" and evidence_id in accepted:
return "system_acceptance"
return "tool_output"
@@ -118,7 +118,7 @@ def assemble_resume_packet(
field=_field(message.text) or "unclassified",
text=message.text,
evidence_class=_raw_class(
message.role,
message.written_by,
message.evidence_id,
verified_system_evidence_ids,
),
@@ -345,7 +345,7 @@ def _lexical_raw_rows(
if context_raw._prefers_declared_facts(query.query)
else "1"
)
person_authored = "CASE WHEN e.role='user' THEN 1 ELSE 0 END"
person_authored = "CASE WHEN e.written_by='person' THEN 1 ELSE 0 END"
query_echo = (
"CASE WHEN trim(e.verbatim)=? OR trim(e.verbatim)=? THEN 0 ELSE 1 END"
if context_raw._prefers_declared_facts(query.query)
@@ -507,7 +507,7 @@ def _raw_item(row: sqlite3.Row, start1: int, matched_by: tuple[str, ...]) -> Raw
"source_id": row["source_id"],
"conversation_id": row["conversation_id"],
"role": row["role"],
"written_by": "person" if row["role"] == "user" else "agent",
"written_by": "person" if row["written_by"] == "person" else "agent",
"event_time": event_time.isoformat() if event_time else None,
"spans": [{"start_cp": start, "end_cp": start + len(text), "text": text}],
"matched_by": list(matched_by),
@@ -1039,7 +1039,7 @@ def close_main_databases() -> None:
def log_main_database_stats(moment: str) -> None:
"""One line in core.log: how many connections are open and who waited."""
stats = main_database_stats()
logger.debug(
logger.info(
"main_database_connections moment=%s %s",
moment,
" ".join(f"{key}={value}" for key, value in stats.items()),
@@ -19,7 +19,6 @@ from __future__ import annotations
import atexit
import json
import logging
import os
import re
import sys
@@ -140,10 +139,6 @@ _DETAIL_KEYS = frozenset(
"accounted_ms",
"unaccounted_ms",
"conversation_hash",
# The sedimentation switch: a bounded boolean, the same shape as
# `spawned` and `index_complete`. Left out of this list it did not
# drop a log line, it turned the switch's own request into a 500.
"enabled",
}
)
_trace_id: ContextVar[str | None] = ContextVar("one_memory_trace_id", default=None)
@@ -654,24 +649,12 @@ def diagnostic_event(**values: object) -> None:
active = _active
if active is None:
return
if values.get("level") == "debug" and not logging.getLogger().isEnabledFor(
logging.DEBUG
):
return
try:
active.write(**values) # type: ignore[arg-type]
except OSError:
# Diagnostics must not become a new product outage when a disk is full
# or the log directory is temporarily unavailable.
return
except ValueError:
# Same promise, same reason, for a rejected payload instead of a failed
# write. An unlisted `details` key once turned every external recall
# into a 500 and then killed the process, and the sedimentation switch
# into a permanent failure. The payload itself is still wrong and is
# still caught in CI by tests/test_diagnostic_payload_keys.py; what
# must not happen is the caller paying for it.
return
def current_session_id() -> str | None:
@@ -21,6 +21,7 @@ from pydantic import BaseModel, ConfigDict, Field
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception, elapsed_ms
from one_memory.hy_lifecycle import HyLifecycle, HyLifecycleUnreadable, read_hy_lifecycle
from one_memory.hy_permission_store import native_store_read_slot
from one_memory.hy_presence import HyPresence, confirm_deleted
from one_memory.hy_runtime_config import (
HY_MEMORY_MODE,
@@ -31,9 +32,7 @@ from one_memory.hy_runtime_config import (
diagnostic_log_level,
)
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
from one_memory.hy_store_concurrency import native_store_read_slot
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind, attach_disable_thinking
from one_memory.model_request_audit import audit_precipitation_request
from one_memory.semantic_deadline import SemanticMemoryHit
#: D102: how relevant HY has to call a hit before it counts. Raised from 0.25
@@ -144,22 +143,6 @@ class HyMemoryBridgeError(RuntimeError):
pass
MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES = 512
def coding_tool_result_max_bytes() -> int:
"""Return the per-tool payload ceiling used by MemHub's HY client."""
raw = os.environ.get("MEMORY_CODING_TOOL_RESULT_MAX_BYTES")
if raw is None:
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
try:
value = int(raw)
except ValueError:
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
return value if value > 0 else MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
def _exclusive[**P, T](
method: Callable[P, Coroutine[Any, Any, T]],
) -> Callable[P, Coroutine[Any, Any, T]]:
@@ -393,13 +376,11 @@ class HyMemoryBridge:
api_key: str | None,
data_dir: Path,
client: HyMemoryClientProtocol | None = None,
http_transport_factory: Callable[[], Any] | None = None,
) -> None:
self.profile = profile
self.api_key = api_key
self.data_dir = data_dir
self._client = client
self._http_transport_factory = http_transport_factory
self._client_lock = asyncio.Lock()
self._write_lock = asyncio.Lock()
self._last_bindings: HyWriteBindings | None = None
@@ -431,9 +412,8 @@ class HyMemoryBridge:
@_exclusive
async def ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
with audit_precipitation_request(request.request_id):
async with self._write_lock:
return await self._ingest(request)
async with self._write_lock:
return await self._ingest(request)
async def _ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
started = monotonic()
@@ -447,11 +427,6 @@ class HyMemoryBridge:
)
client = await self._get_client()
guards = getattr(client, "_one_memory_write_guards", ())
from one_memory.hy_response_guard import HyReconcileGuard
reconcile_guards = tuple(guard for guard in guards if isinstance(guard, HyReconcileGuard))
for guard in reconcile_guards:
guard.begin_write()
before = [len(guard.failures) for guard in guards]
bindings = HyWriteBindings(request.source_context) if request.source_context else None
if bindings:
@@ -525,12 +500,6 @@ class HyMemoryBridge:
if len(guard.failures) > count:
raise HyMemoryBridgeError(guard.failures[count])
result = await self._result(client, request, raw, expected_contents=None, bindings=bindings)
result = await _with_native_mutation_delta(
client,
result,
raw,
reconcile_guards,
)
if bindings:
outputs = []
for memory in result.memories:
@@ -560,9 +529,8 @@ class HyMemoryBridge:
request: HyMemoryIngestRequest,
contents: Sequence[str],
) -> HyMemoryIngestResult:
with audit_precipitation_request(request.request_id):
async with self._write_lock:
return await self._ingest_extracted(request, contents)
async with self._write_lock:
return await self._ingest_extracted(request, contents)
async def _ingest_extracted(
self,
@@ -585,8 +553,6 @@ class HyMemoryBridge:
from one_memory.hy_response_guard import HyReconcileGuard
report_guards = tuple(guard for guard in guards if isinstance(guard, HyReconcileGuard))
for guard in report_guards:
guard.begin_write()
expected_contents = frozenset(content.strip() for content in contents if content.strip())
for guard in report_guards:
guard.imported_contents = expected_contents
@@ -633,12 +599,6 @@ class HyMemoryBridge:
raw,
expected_contents=expected_contents,
)
result = await _with_native_mutation_delta(
client,
result,
raw,
report_guards,
)
diagnostic_event(
level="info",
component="hy.bridge",
@@ -1127,7 +1087,6 @@ class HyMemoryBridge:
self.profile,
api_key=self.api_key,
data_dir=self.data_dir,
http_transport_factory=self._http_transport_factory,
)
await _joined_thread(initialize)
@@ -1154,7 +1113,7 @@ class HyMemoryBridge:
)
raise HyMemoryBridgeError("hy_memory_init_failed") from error
diagnostic_event(
level="debug",
level="info",
component="hy.client",
event="hy.client_initialization.completed",
outcome="succeeded",
@@ -1177,7 +1136,6 @@ def create_hy_memory_client(
*,
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None = None,
) -> HyMemoryClientProtocol:
configure_hy_memory_environment(data_dir)
config = build_hy_memory_config(
@@ -1185,7 +1143,6 @@ def create_hy_memory_client(
api_key=api_key,
data_dir=data_dir,
)
config.http_transport_factory = http_transport_factory
reattach_legacy_vector_collection(profile, config)
mismatch = inspect_graph_width_mismatch(config)
if mismatch is not None:
@@ -1330,72 +1287,6 @@ async def _coding_result(
return HyMemoryIngestResult(success=True, memories=tuple(outputs))
async def _with_native_mutation_delta(
client: HyMemoryClientProtocol,
result: HyMemoryIngestResult,
raw: Mapping[str, object],
reconcile_guards: Sequence[Any],
) -> HyMemoryIngestResult:
"""Map native SDK mutations back into MemHub governance.
Store permission wrappers used to discover this by intercepting every
native CRUD call. The embedded SDK now owns unrestricted local CRUD, so
chat targets come from the reconcile observer and coding targets come from
the SDK's public result document.
"""
retired = set(result.retired_memory_ids)
superseded = {key: tuple(values) for key, values in result.superseded_memory_ids.items()}
chat_targets = {
key
for guard in reconcile_guards
for key in getattr(guard, "mutation_target_ids", ())
if isinstance(key, str) and key
}
operations = raw.get("ops")
for item in operations if isinstance(operations, list) else ():
if not isinstance(item, Mapping):
continue
if item.get("action") == "DELETE":
key = item.get("target_experience_id")
if isinstance(key, str) and key:
retired.add(key)
fact_operations = raw.get("fact_ops")
for item in fact_operations if isinstance(fact_operations, list) else ():
if not isinstance(item, Mapping) or item.get("action") != "SUPERSEDE":
continue
old, new = item.get("fact_id"), item.get("superseded_by")
if isinstance(old, str) and old:
retired.add(old)
if isinstance(new, str) and new:
superseded[old] = (new,)
for key in chat_targets:
record = await _joined_thread(_read_hy_record, client, key)
if record is None:
retired.add(key)
continue
status = str(record.get("status", "")).lower()
if status in {"deleted", "shadow", "superseded"}:
retired.add(key)
replacement = record.get("superseded_by")
if isinstance(replacement, str) and replacement:
superseded[key] = (replacement,)
elif isinstance(replacement, list):
values = tuple(value for value in replacement if isinstance(value, str) and value)
if values:
superseded[key] = values
return result.model_copy(
update={
"retired_memory_ids": tuple(sorted(retired)),
"superseded_memory_ids": superseded,
}
)
def _read_hy_record(
client: HyMemoryClientProtocol,
memory_id: str,
@@ -1712,7 +1603,6 @@ def build_hy_memory_config(
config.history.db_path = str(memory_root / "history.sqlite3")
config.coding.enable = True
config.coding.db_path = str(memory_root / "coding.sqlite3")
config.coding.tool_result_max_bytes = coding_tool_result_max_bytes()
config.recall.episode_recall_enabled = True
config.recall.episode_summary_enabled = False
config.recall.dynamic_episode_enabled = True
@@ -0,0 +1,607 @@
"""Operation-bound read/write capability for the pinned native HY coding store.
HY's coding route writes two layers, experiences and working facts. Rows this
lease cannot attest stay in the same backend but are invisible here, so they
never reach a coding prompt and cannot be updated, superseded or linked.
HY's reconcilers catch store exceptions and carry on, so a denial latches and
recheck() raises it again at finish; a swallowed denial still fails the task.
Any native method not implemented here is denied rather than passed through.
"""
from __future__ import annotations
import asyncio
import json
from collections.abc import Iterator, Mapping, Sequence
from copy import deepcopy
from hashlib import sha256
from types import MappingProxyType
from typing import Any, NoReturn
from hy_memory.coding.types import CodingMemory, WorkingFact # type: ignore[import-untyped]
from one_memory.hy_permission_store import HyNodePermission
from one_memory.model_processing_permission_guard import (
ModelProcessingPermissionLease,
permission_error,
)
_NATIVE_WRITES = frozenset(
{
"insert",
"update",
"delete",
"insert_fact",
"supersede_fact",
"set_fact_derived_experiences",
"set_experience_source_facts",
# Trace rows are raw conversation: writes must be gated like any other.
"append_trace",
}
)
_TIMESTAMPS = frozenset({"created_at", "updated_at"})
# SDK 2.0.2 reads `config` off the coding store while routing a coding
# turn. The wrapper must answer it, but the native `MemoryConfig` carries
# credentials (llm/embedder api keys, graph and cache passwords) and is a live
# handle into the backend, so it is never handed out. Only this redacted
# projection is exposed: credential-bearing keys collapse to None.
_CONFIG_REDACTED_KEYS = frozenset(
{
"api_key",
"password",
"username",
"mysql_password",
"mysql_user",
"extra_headers",
"extra_body",
}
)
class _RedactedConfig(Mapping[str, Any]):
"""Read-only config view: readable by key and attribute, never the native one.
The SDK reaches config values both ways, so this keeps attribute access
working. It is deliberately not the native MemoryConfig: it exposes only
plain copied values, so no backend handle, connection or callable escapes.
"""
__slots__ = ("_values",)
def __init__(self, values: Mapping[str, Any]) -> None:
object.__setattr__(self, "_values", MappingProxyType(dict(values)))
def __getitem__(self, key: str) -> Any:
return self._values[key]
def __iter__(self) -> Iterator[str]:
return iter(self._values)
def __len__(self) -> int:
return len(self._values)
def __getattr__(self, name: str) -> Any:
values = object.__getattribute__(self, "_values")
try:
return values[name]
except KeyError:
raise AttributeError(name) from None
def __setattr__(self, name: str, value: Any) -> NoReturn:
raise AttributeError("redacted config is read-only")
def __repr__(self) -> str:
return f"_RedactedConfig({sorted(self._values)})"
def _redact_config(value: Any, depth: int = 0) -> Any:
"""Copy config data into read-only views, collapsing credentials to None.
The SDK reads this both by attribute and by key, so each config section
becomes a mapping-like view rather than a plain dict: attribute access stays
available while the native object itself is never handed out.
"""
if depth > 4:
return None
if value is None or isinstance(value, (str, int, float, bool)):
return value
if isinstance(value, Mapping):
return _RedactedConfig(
{
key: None if key in _CONFIG_REDACTED_KEYS else _redact_config(item, depth + 1)
for key, item in value.items()
}
)
if isinstance(value, (list, tuple)):
return [_redact_config(item, depth + 1) for item in value]
if hasattr(value, "__dict__") and type(value).__module__.startswith("hy_memory"):
return _redact_config(vars(value), depth + 1)
return None
def _config_projection(native: Any) -> Any:
try:
return _redact_config(native.config)
except Exception:
return None
def hy_coding_fingerprint(row: CodingMemory | WorkingFact) -> str:
"""Hash the complete native row as read back, including links and timestamps."""
if type(row) is CodingMemory:
kind = "experience"
elif type(row) is WorkingFact:
kind = "fact"
else:
raise permission_error("hy_coding_store_row_shape")
payload = {"kind": kind, **row.to_dict()}
return sha256(
json.dumps(payload, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
).hexdigest()
def _row_id(row: CodingMemory | WorkingFact) -> str:
return str(row.experience_id if type(row) is CodingMemory else row.fact_id)
class HyPermissionCodingStore:
"""Closed, task-exclusive adapter; initialize/close never own the native store.
node_permissions is the same attested ledger the chat vector store uses;
coding rows are keyed by experience or fact id. written_nodes carries the
whole lease footprint for each row written here, as the chat store does.
"""
def __init__(
self,
native: Any,
lease: ModelProcessingPermissionLease,
*,
agent_id: str,
session_id: str,
node_permissions: Mapping[str, HyNodePermission] | None = None,
) -> None:
self._native = native
self._lease = lease
self._agent_id = agent_id
self._session_id = session_id
self._user_id = lease.user_id
self._failure: str | None = None
self._closed = False
self._lock = asyncio.Lock()
self._deleted: set[str] = set()
self._written: dict[str, HyNodePermission] = {}
self._superseded: dict[str, str] = {}
self._evidence_ids = tuple(record.evidence_id for record in lease.records)
self._permissions = {
key: proof
for key, proof in (node_permissions or {}).items()
if set(proof.evidence_ids) <= set(self._evidence_ids)
}
self.recheck()
if not agent_id or agent_id == "*" or not session_id or not self._evidence_ids:
self._deny("scope")
def __getattr__(self, name: str) -> NoReturn:
if name.startswith("__"):
raise AttributeError(name)
self._deny("unsupported_operation")
@property
def written_nodes(self) -> Mapping[str, HyNodePermission]:
self.recheck()
return MappingProxyType(dict(self._written))
@property
def config(self) -> Any:
"""Read-only, redacted view of the native coding config.
SDK 2.0.2 reads `config` when it routes a coding turn. Answering
it is safe only as a projection: the native MemoryConfig is a live
backend handle and carries credentials, so callers get plain values
with every credential-bearing key collapsed to None. This carries no
identity, permission or tenant state and is not the native object.
"""
self._assert_open()
return _config_projection(self._native)
@property
def retired_ids(self) -> frozenset[str]:
"""Experiences HY deleted in this operation."""
self.recheck()
return frozenset(self._deleted)
@property
def superseded_ids(self) -> Mapping[str, tuple[str, ...]]:
"""Facts HY superseded in this operation, with the fact that replaced each.
A successor linked before it existed must have been written here by
now; otherwise the old fact points at a row nobody attested.
"""
self.recheck()
if any(new not in self._written for new in self._superseded.values()):
self._deny("link_denied")
return MappingProxyType({old: (new,) for old, new in self._superseded.items()})
def recheck(self) -> None:
self._assert_open()
try:
self._lease.recheck()
except Exception as error:
code = str(error)
self._failure = (
code
if code.startswith("model_content_") and len(code) <= 100
else "model_content_hy_coding_store_lease_failed"
)
raise permission_error(self._failure) from None
def _assert_open(self) -> None:
if self._closed:
self._deny("closed")
if self._failure:
raise permission_error(self._failure)
async def close(self) -> None:
async with self._lock:
self._closed = True
self._permissions.clear()
self._written.clear()
def _deny(self, suffix: str) -> NoReturn:
self._failure = self._failure or f"model_content_hy_coding_store_{suffix}"
raise permission_error(self._failure)
def _scope(self, user_id: str | None) -> None:
if user_id not in (None, self._user_id):
self._deny("scope")
async def _invoke(self, method: str, *args: Any, **kwargs: Any) -> Any:
writing = method in _NATIVE_WRITES
self.recheck() if writing else self._assert_open()
try:
result = await getattr(self._native, method)(*args, **kwargs)
except Exception:
self._deny("backend_failed")
self.recheck() if writing else self._assert_open()
return result
def _visible(self, row: Any, expected: Any) -> bool:
if row is None:
return False
if type(row) is not expected:
self._deny("row_shape")
key = _row_id(row)
proof = self._permissions.get(key)
if proof is None or key in self._deleted or row.user_id != self._user_id:
return False
if hy_coding_fingerprint(row) != proof.fingerprint:
self._deny("row_changed")
return True
async def _experience(self, key: str) -> CodingMemory | None:
if key not in self._permissions or key in self._deleted:
return None
row = await self._invoke("get_by_id", key, user_id=self._user_id)
return deepcopy(row) if self._visible(row, CodingMemory) else None
async def _fact(self, key: str) -> WorkingFact | None:
if key not in self._permissions or key in self._deleted:
return None
rows = await self._invoke("get_facts", [key], user_id=self._user_id)
row = rows[0] if rows else None
return deepcopy(row) if self._visible(row, WorkingFact) else None
def _new_row_in_scope(self, row: CodingMemory | WorkingFact) -> bool:
return bool(
row.user_id == self._user_id
and row.agent_id == self._agent_id
and row.session_id == self._session_id
)
async def _capture(
self, key: str, kind: Any, expected: CodingMemory | WorkingFact | None = None
) -> None:
row: Any
if kind is CodingMemory:
row = await self._invoke("get_by_id", key, user_id=self._user_id)
else:
rows = await self._invoke("get_facts", [key], user_id=self._user_id)
row = rows[0] if rows else None
if type(row) is not kind or row.user_id != self._user_id:
self._deny("write_mismatch")
if expected is not None:
stored, planned = row.to_dict(), expected.to_dict()
for field in _TIMESTAMPS:
stored.pop(field, None)
planned.pop(field, None)
if stored != planned:
self._deny("write_mismatch")
proof = HyNodePermission(hy_coding_fingerprint(row), self._evidence_ids)
self._permissions[key] = proof
self._written[key] = proof
async def _plan_new(self, row: CodingMemory | WorkingFact, kind: Any) -> None:
if type(row) is not kind or not self._new_row_in_scope(row):
self._deny("scope")
key = _row_id(row)
if not key or key in self._permissions or key in self._deleted:
self._deny("scope")
if kind is CodingMemory:
existing = await self._invoke("get_by_id", key)
else:
existing = next(iter(await self._invoke("get_facts", [key])), None)
if existing is not None:
self._deny("scope")
# Reads
async def get_by_id(
self, experience_id: str, user_id: str | None = None
) -> CodingMemory | None:
async with self._lock:
self._scope(user_id)
return await self._experience(experience_id)
async def get_many(
self, experience_ids: list[str], user_id: str | None = None
) -> list[CodingMemory]:
async with self._lock:
self._scope(user_id)
wanted = [
key
for key in dict.fromkeys(experience_ids)
if key in self._permissions and key not in self._deleted
]
if not wanted:
return []
rows = await self._invoke("get_many", wanted, user_id=self._user_id)
return [deepcopy(row) for row in rows if self._visible(row, CodingMemory)]
async def search_by_query_embedding(
self,
query_embedding: list[float],
user_id: str,
*,
workspace_id: str | None = None,
branch: str | None = None,
top: int = 40,
) -> list[dict[str, Any]]:
async with self._lock:
self._scope(user_id)
hits = await self._invoke(
"search_by_query_embedding",
deepcopy(query_embedding),
self._user_id,
workspace_id=workspace_id,
branch=branch,
top=top * 4,
)
result = []
for hit in hits:
key = hit.get("experience_id")
if not isinstance(key, str) or await self._experience(key) is None:
continue
result.append(
{
"experience_id": key,
"score": float(hit.get("score", 0.0)),
"matched_key_kind": hit.get("matched_key_kind"),
"matched_key_text": hit.get("matched_key_text"),
}
)
return result[:top]
async def get_fact(self, fact_id: str, user_id: str | None = None) -> WorkingFact | None:
async with self._lock:
self._scope(user_id)
return await self._fact(fact_id)
async def get_facts(self, fact_ids: list[str], user_id: str | None = None) -> list[WorkingFact]:
async with self._lock:
self._scope(user_id)
found = [await self._fact(key) for key in dict.fromkeys(fact_ids)]
return [row for row in found if row is not None]
async def search_facts(
self,
query_embedding: list[float],
user_id: str,
*,
workspace_id: str | None = None,
branch: str | None = None,
top: int = 5,
query_text: str | None = None,
min_score: float = 0.0,
) -> list[tuple[WorkingFact, float]]:
async with self._lock:
self._scope(user_id)
hits = await self._invoke(
"search_facts",
deepcopy(query_embedding),
self._user_id,
workspace_id=workspace_id,
branch=branch,
top=top * 4,
query_text=query_text,
min_score=min_score,
)
result = []
for row, score in hits:
if self._visible(row, WorkingFact):
result.append((deepcopy(row), float(score)))
return result[:top]
async def get_fact_ancestors(
self, fact_id: str, *, user_id: str | None = None, max_depth: int = 8
) -> list[WorkingFact]:
async with self._lock:
self._scope(user_id)
if await self._fact(fact_id) is None:
return []
rows = await self._invoke(
"get_fact_ancestors", fact_id, user_id=self._user_id, max_depth=max_depth
)
# Oldest first; the chain is only as long as its nearest unapproved link.
kept: list[WorkingFact] = []
for row in reversed(rows):
if not self._visible(row, WorkingFact):
break
kept.append(deepcopy(row))
return list(reversed(kept))
# Writes
async def insert(self, memory: CodingMemory) -> None:
async with self._lock:
self.recheck()
await self._plan_new(memory, CodingMemory)
await self._invoke("insert", memory)
await self._capture(memory.experience_id, CodingMemory, memory)
async def update(self, memory: CodingMemory) -> None:
async with self._lock:
self.recheck()
if type(memory) is not CodingMemory or not self._new_row_in_scope(memory):
self._deny("scope")
if await self._experience(memory.experience_id) is None:
self._deny("target_denied")
await self._invoke("update", memory)
await self._capture(memory.experience_id, CodingMemory, memory)
async def delete(self, experience_id: str, user_id: str) -> bool:
async with self._lock:
self._scope(user_id)
if await self._experience(experience_id) is None:
self._deny("target_denied")
if await self._invoke("delete", experience_id, user_id=self._user_id) is not True:
self._deny("write_failed")
if await self._invoke("get_by_id", experience_id) is not None:
self._deny("write_mismatch")
self._deleted.add(experience_id)
self._permissions.pop(experience_id, None)
self._written.pop(experience_id, None)
return True
async def insert_fact(self, fact: WorkingFact) -> None:
async with self._lock:
self.recheck()
await self._plan_new(fact, WorkingFact)
await self._invoke("insert_fact", fact)
await self._capture(fact.fact_id, WorkingFact, fact)
async def supersede_fact(self, old_fact_id: str, new_fact_id: str) -> bool:
async with self._lock:
self.recheck()
if await self._fact(old_fact_id) is None or old_fact_id == new_fact_id:
self._deny("target_denied")
# HY links the old fact before it inserts the new one. A successor
# that does not exist yet may only become a row this lease writes;
# an existing row it cannot attest stays out of reach.
if await self._fact(new_fact_id) is None:
if not new_fact_id or new_fact_id in self._deleted:
self._deny("target_denied")
if next(iter(await self._invoke("get_facts", [new_fact_id])), None) is not None:
self._deny("target_denied")
changed = await self._invoke("supersede_fact", old_fact_id, new_fact_id)
await self._capture(old_fact_id, WorkingFact)
if changed:
self._superseded[old_fact_id] = new_fact_id
return bool(changed)
async def set_fact_derived_experiences(self, fact_id: str, experience_ids: list[str]) -> None:
async with self._lock:
self.recheck()
if await self._fact(fact_id) is None:
self._deny("target_denied")
for key in experience_ids:
if await self._experience(key) is None:
self._deny("link_denied")
await self._invoke("set_fact_derived_experiences", fact_id, list(experience_ids))
await self._capture(fact_id, WorkingFact)
async def set_experience_source_facts(self, experience_id: str, fact_ids: list[str]) -> None:
async with self._lock:
self.recheck()
if await self._experience(experience_id) is None:
self._deny("target_denied")
for key in fact_ids:
if await self._fact(key) is None:
self._deny("link_denied")
await self._invoke("set_experience_source_facts", experience_id, list(fact_ids))
await self._capture(experience_id, CodingMemory)
async def get_recent_trace_messages(
self,
*,
user_id: str,
agent_id: str = "default_agent",
max_user_turns: int = 5,
) -> list[Any]:
"""Read recent raw trace rows, still bounded by this lease.
Public SDK 2.0.2 asks for recent history before a coding write. The native call is
scoped to the lease's own user and agent, and rows this lease cannot
attest are filtered out, so unapproved conversations never reach a
coding prompt through this path.
"""
self._scope(user_id)
if agent_id not in (self._agent_id, "*"):
self._deny("scope")
rows = await self._invoke(
"get_recent_trace_messages",
user_id=self._user_id,
agent_id=self._agent_id,
max_user_turns=max_user_turns,
)
return [row for row in (rows or []) if self._trace_visible(row)]
async def append_trace(
self,
messages: Sequence[Any],
*,
request_id: str,
user_id: str,
agent_id: str = "default_agent",
session_id: str | None = None,
workspace_id: str | None = None,
branch: str | None = None,
) -> None:
"""Append raw trace rows for this lease only.
Trace rows are raw conversation, so this is a write path: the lease is
rechecked around the call and every message must belong to the leased
user and session. Nothing outside the lease may be recorded.
"""
self._scope(user_id)
if agent_id not in (self._agent_id, "*"):
self._deny("scope")
if session_id not in (None, self._session_id):
self._deny("scope")
for row in messages or []:
if not self._trace_visible(row):
self._deny("scope")
await self._invoke(
"append_trace",
list(messages or []),
request_id=request_id,
user_id=self._user_id,
agent_id=self._agent_id,
session_id=self._session_id,
workspace_id=workspace_id,
branch=branch,
)
def _trace_visible(self, row: Any) -> bool:
"""A trace row may be read or written only inside this lease's scope."""
if row is None:
return False
owner = getattr(row, "user_id", None)
session = getattr(row, "session_id", None)
return not (
(owner is not None and owner != self._user_id)
or (session is not None and session != self._session_id)
)
@@ -0,0 +1,582 @@
"""Task-exclusive HY wiring; provenance metadata stays beside the existing Core DB."""
from __future__ import annotations
import asyncio
import json
from collections.abc import AsyncIterator, Callable
from contextlib import asynccontextmanager, nullcontext, suppress
from pathlib import Path
from typing import Any, cast
import httpx
from one_memory.database import take_the_write_lock_first
from one_memory.database_access import main_database
from one_memory.hy_memory_bridge import (
HyMemoryBridge,
HyMemoryBridgeError,
HyMemoryIngestRequest,
HyMemoryIngestResult,
HyMemoryOutput,
LocalHyEmbedService,
_joined_thread,
)
from one_memory.hy_permission_coding_store import HyPermissionCodingStore
from one_memory.hy_permission_store import (
HyNodePermission,
HyPermissionGraphStore,
HyPermissionVectorStore,
)
from one_memory.hy_permission_transport import HyPermissionTransport, HyPermissionTransportError
from one_memory.hy_response_guard import HyResponseGuard
from one_memory.hy_step_timing import HyStepTimer
from one_memory.memory_server.migrations import hy_node_permissions_ddl
from one_memory.memory_tasks import evidence_ref
from one_memory.model_processing_permission_guard import ModelProcessingPermissionLease
class _NativeProofs:
def __init__(self, database: Path) -> None:
self.database = database
self._loaded_states: dict[str, tuple[int, str]] | None = None
with main_database(database).write() as db, db:
# Schema owned by `memory_server.migrations` (migration 26).
db.execute(hy_node_permissions_ddl("sqlite"))
def load(self, lease: ModelProcessingPermissionLease) -> dict[str, HyNodePermission]:
allowed = {record.evidence_id: evidence_ref(record) for record in lease.records}
with main_database(self.database).read() as db:
rows = db.execute(
"SELECT p.node_id,p.fingerprint,p.evidence_json,g.revision,g.status "
"FROM hy_node_permissions p LEFT JOIN governed_memories g ON g.memory_id=p.node_id "
# Superseded nodes stay attested: every later head links back to them.
"WHERE g.status IS NULL OR g.status IN ('confirmed','superseded')"
).fetchall()
proofs = {}
states = {}
for identifier, fingerprint, evidence_json, revision, status in rows:
refs = json.loads(evidence_json)
if (
not isinstance(refs, list)
or not refs
or not all(isinstance(ref, dict) for ref in refs)
):
continue
if all(allowed.get(ref.get("evidence_id")) == ref for ref in refs):
proofs[identifier] = HyNodePermission(
fingerprint, tuple(ref["evidence_id"] for ref in refs)
)
if revision is not None:
states[identifier] = (revision, status)
self._loaded_states = states
return proofs
def guard(self, values: dict[str, HyNodePermission]) -> Callable[[], None]:
identifiers = tuple(values)
def state() -> dict[str, tuple[int, str]]:
found: dict[str, tuple[int, str]] = {}
with main_database(self.database).read() as db:
for start in range(0, len(identifiers), 900):
batch = identifiers[start : start + 900]
placeholders = ",".join("?" for _ in batch)
for identifier, revision, status in db.execute(
"SELECT memory_id,revision,status FROM governed_memories "
f"WHERE memory_id IN ({placeholders})",
batch,
):
found[identifier] = (revision, status)
return found
if self._loaded_states is None:
raise HyMemoryBridgeError("model_content_context_unverified")
before = dict(self._loaded_states)
def check() -> None:
if state() != before:
raise HyMemoryBridgeError("model_content_context_changed")
check()
return check
def save(
self, lease: ModelProcessingPermissionLease, values: dict[str, HyNodePermission]
) -> None:
refs = {record.evidence_id: evidence_ref(record) for record in lease.records}
with main_database(self.database).write(autocommit=True) as db:
take_the_write_lock_first(db)
with db:
lease.recheck()
db.executemany(
"INSERT INTO hy_node_permissions VALUES (?,?,?) ON CONFLICT(node_id) "
"DO UPDATE SET fingerprint=excluded.fingerprint,"
"evidence_json=excluded.evidence_json",
[
(
identifier,
proof.fingerprint,
json.dumps([refs[key] for key in proof.evidence_ids]),
)
for identifier, proof in values.items()
],
)
def refresh(
self, lease: ModelProcessingPermissionLease, node_id: str
) -> HyNodePermission | None:
del lease
with main_database(self.database).read() as db:
row = db.execute(
"SELECT fingerprint,evidence_json FROM hy_node_permissions WHERE node_id=?",
(node_id,),
).fetchone()
if row is None:
return None
fingerprint = str(row["fingerprint"])
refs = json.loads(row["evidence_json"])
if (
len(fingerprint) != 64
or any(value not in "0123456789abcdef" for value in fingerprint)
or not isinstance(refs, list)
or not refs
or not all(isinstance(ref, dict) for ref in refs)
):
return None
evidence_ids = tuple(ref.get("evidence_id") for ref in refs)
if any(not isinstance(value, str) or not value for value in evidence_ids):
return None
return HyNodePermission(fingerprint, evidence_ids)
class _BoundLease:
def __init__(
self,
lease: ModelProcessingPermissionLease,
proof_check: Callable[[], None],
step_timer: HyStepTimer | None = None,
) -> None:
self.lease = lease
self.step_timer = step_timer
self.user_id = lease.user_id
self.profile = lease.profile
self.task_id = lease.task_id
self.records = lease.records
self.source_context = lease.source_context
self.proof_check = proof_check
self.vector: HyPermissionVectorStore | None = None
self.coding: HyPermissionCodingStore | None = None
def recheck(self) -> None:
timer = self.step_timer
with timer.step("permission_check") if timer is not None else nullcontext():
self.lease.recheck()
self.proof_check()
if self.vector is not None:
self.vector.recheck()
if self.coding is not None:
self.coding.recheck()
class _Wiring:
def __init__(
self,
client: Any,
lease: ModelProcessingPermissionLease,
proofs_store: _NativeProofs,
proofs: dict[str, HyNodePermission],
agent_id: str,
session_id: str,
transport: httpx.AsyncBaseTransport | None,
proof_check: Callable[[], None],
step_timer: HyStepTimer,
) -> None:
self.client, self.lease, self.proofs_store, self.proofs = (
client,
lease,
proofs_store,
proofs,
)
self.step_timer = step_timer
self.agent_id, self.session_id, self.http_transport = agent_id, session_id, transport
self.cross_evidence_scope = False
self.proof_check = proof_check
self.saved: list[tuple[Any, str, Any]] = []
self.transport: HyPermissionTransport
self.vector: HyPermissionVectorStore
self.graph: HyPermissionGraphStore | None = None
self.coding: HyPermissionCodingStore
self.active = False
def replace(self, target: Any, name: str, value: Any) -> None:
self.saved.append((target, name, getattr(target, name)))
setattr(target, name, value)
async def enter(self) -> None:
client, registry = self.client, self.client._registry
config = client._config
if config.llm.enable_summary or config.llm.enable_query_rewrite:
raise HyMemoryBridgeError("model_content_context_unverified")
bound = _BoundLease(self.lease, self.proof_check, self.step_timer)
self.transport = await HyPermissionTransport(
bound, http_transport=self.http_transport, step_timer=self.step_timer
).__aenter__()
self.active = True
try:
for guard in getattr(client, "_one_memory_write_guards", ()):
if hasattr(type(guard), "step_timer"):
self.replace(guard, "step_timer", self.step_timer)
system2 = await registry.get_system2_writer()
writers = (await registry.get_writer(), system2._writer)
native_vector = client._vector_store
store_lease = _BoundLease(self.lease, self.proof_check, self.step_timer)
self.vector = HyPermissionVectorStore(
native_vector,
cast(ModelProcessingPermissionLease, store_lease),
agent_id=self.agent_id,
session_id=self.session_id,
cross_evidence_scope=self.cross_evidence_scope,
node_permissions=self.proofs,
refresh_permission=lambda node_id: asyncio.to_thread(
self.proofs_store.refresh, self.lease, node_id
),
step_timer=self.step_timer,
)
bound.vector = self.vector
embed = client._embed_service
if isinstance(embed, LocalHyEmbedService):
self.transport.bind_local_embed_service(embed)
else:
self.transport.bind_embed_service(embed)
for writer in writers:
provider = writer.mem_agent.llm_provider
self.transport.bind_provider(
provider.provider if isinstance(provider, HyResponseGuard) else provider
)
reconciler = writer.reconciler
provider = reconciler._get_llm()
self.transport.bind_provider(
provider.provider if isinstance(provider, HyResponseGuard) else provider
)
self.replace(writer, "_external_vector_store", self.vector)
self.replace(writer, "_vector_store", self.vector)
self.replace(writer, "_vector_store_initialized", True)
self.replace(writer, "_cache", None)
self.replace(client, "_vector_store", self.vector)
native_graph = client._graph_store
if self.cross_evidence_scope and native_graph is not None:
self.graph = HyPermissionGraphStore(native_graph, self.proof_check)
self.replace(client, "_graph_store", self.graph)
else:
self.replace(client, "_graph_store", None)
self.replace(registry, "_shared_vector_store", self.vector)
self.replace(
registry,
"_shared_graph_store",
self.graph if self.graph is not None else None,
)
self.replace(registry, "_shared_cache", None)
self.replace(system2, "_vector_store", self.vector)
self.replace(
system2,
"_graph_store",
self.graph if self.graph is not None else None,
)
self.replace(system2, "_cache", None)
# Result fallback may call search; rebuild only its non-generative reader.
self.replace(registry, "_reader", None)
self.replace(registry, "_readers", {})
await self._enter_coding(bound, embed)
self.transport.assert_allowed()
except BaseException:
await self.exit()
raise
async def _enter_coding(self, bound: _BoundLease, embed: Any) -> None:
"""HY judges and writes the coding route itself; every seam it uses is bound here."""
client = self.client
# Created now, so the judge and every coding stage share one bound provider.
writer = await client._get_coding_writer()
stages = (writer, writer.extractor, writer.reconciler, writer.fact_reconciler)
native_store = client._coding_store
if any(
stage.store is not native_store
for stage in (writer, writer.reconciler, writer.fact_reconciler)
) or any(
item.embed_service is not embed
for item in (native_store, writer, writer.reconciler, writer.fact_reconciler)
):
raise HyMemoryBridgeError("model_content_context_unverified")
providers = {
id(p): p for p in (client._coding_llm_provider, *(s.llm_provider for s in stages))
}
for provider in providers.values():
self.transport.bind_provider(provider)
self.coding = HyPermissionCodingStore(
native_store,
cast(
ModelProcessingPermissionLease,
_BoundLease(self.lease, self.proof_check, self.step_timer),
),
agent_id=self.agent_id,
session_id=self.session_id,
node_permissions=self.proofs,
)
bound.coding = self.coding
for stage in (writer, writer.reconciler, writer.fact_reconciler):
self.replace(stage, "store", self.coding)
self.replace(client, "_coding_store", self.coding)
for stage in (writer, writer.extractor, writer.reconciler):
self.replace(stage, "cache", None)
# The judge logs its prompt through the client's own cache.
self.replace(client, "_cache", None)
async def finish(self) -> dict[str, HyNodePermission]:
self.transport.assert_allowed()
self.vector.recheck()
self.coding.recheck()
return {**self.vector.written_nodes, **self.coding.written_nodes}
async def retired(self) -> tuple[tuple[str, ...], dict[str, tuple[str, ...]]]:
self.transport.assert_allowed()
return (
tuple(sorted(self.vector.retired_node_ids | self.coding.retired_ids)),
{**self.vector.superseded_nodes, **self.coding.superseded_ids},
)
async def exit(self) -> None:
if not self.active:
return
self.active = False
try:
await self.transport.__aexit__(None, None, None)
finally:
for target, name, value in reversed(self.saved):
setattr(target, name, value)
self.saved.clear()
if hasattr(self, "vector"):
await self.vector.close()
if self.graph is not None:
await self.graph.close()
if hasattr(self, "coding"):
await self.coding.close()
class _LeasedBridge:
def __init__(
self,
bridge: HyMemoryBridge,
lease: ModelProcessingPermissionLease,
proofs: _NativeProofs,
transport: httpx.AsyncBaseTransport | None,
) -> None:
self.bridge, self.lease, self.proofs, self.transport = bridge, lease, proofs, transport
self.wiring: _Wiring | None = None
self.finished = False
self.step_timer = HyStepTimer()
self.request_id: str | None = None
def _step(self, name: str) -> Any:
return self.step_timer.step(name)
async def _recheck(self) -> None:
def recheck() -> None:
with self._step("permission_check"):
self.lease.recheck()
await _joined_thread(recheck)
async def _start(self, request: HyMemoryIngestRequest) -> None:
if self.wiring is not None or self.finished:
raise HyMemoryBridgeError("model_content_context_reused")
if request.user_id != self.lease.user_id or set(request.evidence_ids) - {
item.evidence_id for item in self.lease.records
}:
raise HyMemoryBridgeError("model_content_context_unverified")
await self._start_operation(
user_id=request.user_id,
agent_id=request.agent_id,
request_id=request.request_id,
session_id=request.session_id,
)
async def _start_operation(
self,
*,
user_id: str,
agent_id: str,
request_id: str,
session_id: str,
cross_evidence_scope: bool = False,
) -> None:
if self.wiring is not None or self.finished:
raise HyMemoryBridgeError("model_content_context_reused")
if user_id != self.lease.user_id:
raise HyMemoryBridgeError("model_content_context_unverified")
self.request_id = request_id
await self._recheck()
client = cast(Any, await self.bridge._get_client())
with self._step("permission_store_read"):
known = await asyncio.to_thread(self.proofs.load, self.lease)
proof_check = await asyncio.to_thread(self.proofs.guard, known)
self.wiring = _Wiring(
client,
self.lease,
self.proofs,
known,
agent_id,
session_id,
self.transport,
proof_check,
self.step_timer,
)
self.wiring.cross_evidence_scope = cross_evidence_scope
await _joined_thread(client._loop_thread.run, self.wiring.enter())
async def list_agent_ids(self, *, user_id: str, limit: int = 1_000) -> tuple[str, ...]:
if user_id != self.lease.user_id or self.wiring is not None or self.finished:
raise HyMemoryBridgeError("model_content_context_unverified")
await self._recheck()
result = await self.bridge.list_agent_ids(user_id=user_id, limit=limit)
await self._recheck()
return result
async def digest(self, *, user_id: str, agent_id: str) -> dict[str, object]:
await self._start_operation(
user_id=user_id,
agent_id=agent_id,
request_id=f"digest:{self.lease.task_id}:{agent_id}",
session_id=f"digest:{self.lease.task_id}",
cross_evidence_scope=True,
)
try:
result = dict(await self.bridge.digest(user_id=user_id, agent_id=agent_id))
except HyMemoryBridgeError:
await self._check()
raise
proofs = await self._check()
with self._step("permission_store_write"):
await asyncio.to_thread(self.proofs.save, self.lease, proofs)
await self.close()
await self._recheck()
return result
async def graph_snapshot(self, *, user_id: str, agent_id: str, limit: int = 300) -> Any:
await self._start_operation(
user_id=user_id,
agent_id=agent_id,
request_id=f"digest-graph:{self.lease.task_id}:{agent_id}",
session_id=f"digest:{self.lease.task_id}",
cross_evidence_scope=True,
)
try:
result = await self.bridge.graph_snapshot(
user_id=user_id,
agent_id=agent_id,
limit=limit,
)
except HyMemoryBridgeError:
await self._check()
raise
await self._check()
await self.close()
await self._recheck()
return result
async def ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
await self._start(request)
try:
result = await self.bridge.ingest(request)
except HyMemoryBridgeError:
await self._check()
raise
return await self._with_retired(result)
async def ingest_extracted(
self, request: HyMemoryIngestRequest, contents: tuple[str, ...]
) -> HyMemoryIngestResult:
await self._start(request)
try:
result = await self.bridge.ingest_extracted(request, contents)
except HyMemoryBridgeError:
await self._check()
raise
return await self._with_retired(result)
async def _with_retired(self, result: HyMemoryIngestResult) -> HyMemoryIngestResult:
await self._check()
if self.wiring is None:
raise HyMemoryBridgeError("model_content_context_unverified")
try:
retired, superseded = await _joined_thread(
self.wiring.client._loop_thread.run, self.wiring.retired()
)
except HyPermissionTransportError as error:
await self._recheck()
raise HyMemoryBridgeError(error.code) from None
return result.model_copy(
update={"retired_memory_ids": retired, "superseded_memory_ids": superseded}
)
async def _check(self) -> dict[str, HyNodePermission]:
if self.wiring is None or self.finished:
raise HyMemoryBridgeError("model_content_context_unverified")
await self._recheck()
try:
return await _joined_thread(self.wiring.client._loop_thread.run, self.wiring.finish())
except HyPermissionTransportError as error:
await self._recheck()
raise HyMemoryBridgeError(error.code) from None
async def profile_memories(self, *, user_id: str, agent_id: str) -> tuple[HyMemoryOutput, ...]:
if self.wiring is None or user_id != self.lease.user_id or agent_id != self.wiring.agent_id:
raise HyMemoryBridgeError("model_content_context_unverified")
result = await self.bridge.profile_memories(user_id=user_id, agent_id=agent_id)
proofs = await self._check()
with self._step("permission_store_write"):
await asyncio.to_thread(self.proofs.save, self.lease, proofs)
# SDK context closes before governance can update a previously known memory.
await self.close()
return result
async def close(self) -> None:
if self.wiring is not None and not self.finished:
self.finished = True
try:
await _joined_thread(self.wiring.client._loop_thread.run, self.wiring.exit())
finally:
self.step_timer.emit(request_id=self.request_id, task_id=self.lease.task_id)
def permission_bridge_factory(
resolve_bridge: Callable[..., Any],
*,
http_transport: httpx.AsyncBaseTransport | None = None,
) -> Callable[..., Any]:
@asynccontextmanager
async def factory(
lease: ModelProcessingPermissionLease, secret: str | None, data_dir: Path
) -> AsyncIterator[Any]:
bridge = resolve_bridge(lease.profile, secret, data_dir)
if not isinstance(bridge, HyMemoryBridge):
raise HyMemoryBridgeError("model_content_context_unverified")
proofs = _NativeProofs(data_dir / "one-memory.sqlite3")
async with bridge.exclusive_access():
await bridge.prepare()
bound = _LeasedBridge(bridge, lease, proofs, http_transport)
try:
yield bound
except BaseException as error:
with suppress(Exception):
await bound.close()
if isinstance(error, HyPermissionTransportError):
await _joined_thread(lease.recheck)
raise HyMemoryBridgeError(error.code) from None
raise
else:
try:
await bound.close()
except HyPermissionTransportError as error:
raise HyMemoryBridgeError(error.code) from None
return factory
@@ -0,0 +1,880 @@
"""Operation-bound read/write capability for the pinned native HY chat vector store.
No semantic storage or extraction lives here. Pre-existing nodes require a full
native-node attestation with complete approved Evidence provenance; known content
alone cannot authorize tags, custom fields, vectors or links. Unknown nodes stay
in the same HY backend but are not exposed to this operation.
The caller must exclusively own the native backend's writes during execution:
its API has no compare-and-swap, and a swallowed get_by_id error can look missing.
This component cannot make that check/upsert interval atomic. Factory integration
must replace client/registry/both writers/cached-reader references, bind physical
LLM/embedding sends to the lease AND this denial latch, and guard publication.
Coding rows go through hy_permission_coding_store instead; graph, entities,
digest, sweeper and summary remain unsupported.
"""
from __future__ import annotations
import asyncio
import json
import math
import struct
import weakref
from collections.abc import AsyncIterator, Awaitable, Callable, Mapping
from contextlib import asynccontextmanager, nullcontext
from copy import deepcopy
from dataclasses import dataclass
from hashlib import sha256
from types import MappingProxyType
from typing import Any, NoReturn
from hy_memory.models.memory import ( # type: ignore[import-untyped]
MemoryLayer,
MemoryNode,
MemoryStatus,
)
from one_memory.hy_step_timing import HyStepTimer
from one_memory.model_processing_permission_guard import (
ModelProcessingPermissionLease,
permission_error,
)
_WRITE_LOCKS: weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
weakref.WeakKeyDictionary()
)
def _native_write_lock() -> asyncio.Lock:
loop = asyncio.get_running_loop()
lock = _WRITE_LOCKS.get(loop)
if lock is None:
lock = asyncio.Lock()
_WRITE_LOCKS[loop] = lock
return lock
@asynccontextmanager
async def native_store_read_slot() -> AsyncIterator[bool]:
"""Admit one native HY read only while no write is active.
Chroma's native reader may retain the GIL after an asyncio timeout. Running
it concurrently with HY's multi-step store writes has deadlocked the Core
under sustained imports. Reads therefore share the physical store lock,
but never wait behind a writer: host recall can immediately use the local
governance ledger instead.
"""
lock = _native_write_lock()
if lock.locked():
yield False
return
await lock.acquire()
try:
yield True
finally:
lock.release()
@dataclass(frozen=True)
class HyNodePermission:
"""Trusted caller's full-node attestation, not permission inferred from an ID."""
fingerprint: str
evidence_ids: tuple[str, ...]
def hy_node_fingerprint(node: MemoryNode) -> str:
"""Hash the complete native serialized node, including vectors and metadata.
Timestamp precision follows HY's to_dict contract (integer Unix seconds).
Attest the node read from the backend, not an unpersisted input object.
"""
if type(node) is not MemoryNode:
raise permission_error("hy_store_node_shape")
return sha256(
json.dumps(node.to_dict(), sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
).hexdigest()
_CHAT_LAYERS = frozenset(
{
MemoryLayer.L0_BASIC_INFO,
MemoryLayer.L1_RAW,
MemoryLayer.L2_FACT,
MemoryLayer.L4_IDENTITY,
MemoryLayer.L7_INTENTION,
}
)
_UPDATE_FIELDS = frozenset(
{
"content",
"embedding",
"tags",
"tag_ners",
"layer",
"memory_at",
"gmt_modified",
"is_latest",
"status",
"supersedes",
"superseded_by",
"source_raw_memory_id",
# SDK 2.0.2 sends both of these on an in-place reconcile UPDATE.
# They are metadata only and must never carry identity, permission or
# tenant state: `owner` is a provenance label whose only values are
# "user" / "agent" (who said this line), and `custom` is the SDK's own
# free-form notes field. Neither may hold a user_id, scope, grant or
# tenancy marker, so allowing the field names does not widen the
# cross-user boundary that the store still enforces per node.
"owner",
"custom",
}
)
_DEDUP_AUDIT_FIELDS = frozenset({"shadow_reason", "dedup_kept_id"})
_DIGEST_COUNTER_FIELD = "s2_evidence_count"
def _write_view(node: MemoryNode) -> dict[str, Any]:
"""Only native Chroma's empty-string and float32 storage normalization."""
payload: dict[str, Any] = node.to_dict()
payload["source_session_id"] = payload["source_session_id"] or ""
# The pinned public SDK serializes tag_ners as JSON, but its Chroma read path omits this
# field from the JSON decode list. Decode only the exact list-of-strings
# shape that the write path produced; malformed values still fail the
# complete-node comparison below.
tag_ners = payload.get("tag_ners")
if (
isinstance(tag_ners, list)
and tag_ners
and all(isinstance(value, str) and len(value) == 1 for value in tag_ners)
and tag_ners[0] == "["
and tag_ners[-1] == "]"
):
tag_ners = "".join(tag_ners)
if isinstance(tag_ners, str):
try:
decoded = json.loads(tag_ners)
except (json.JSONDecodeError, TypeError):
decoded = None
if isinstance(decoded, list) and all(isinstance(value, str) for value in decoded):
payload["tag_ners"] = decoded
if payload.get("tag_ners") in ([], ["[", "]"]):
payload["tag_ners"] = []
# HY Chroma encodes None as ""; these JSON fields keep "" when json.loads
# fails (vector_store_chroma.py:741-788). Only an absent link is equivalent.
for field in ("supersedes", "superseded_by"):
if payload[field] == "":
payload[field] = None
if payload["embedding"] is not None:
payload["embedding"] = [
struct.unpack("f", struct.pack("f", n))[0] for n in payload["embedding"]
]
if not all(math.isfinite(n) for n in payload["embedding"]):
raise ValueError("nonfinite_embedding")
return payload
def _same_written_vector(actual: list[float] | None, expected: list[float] | None) -> bool:
if actual == expected:
return True
if actual is None or expected is None or len(actual) != len(expected):
return False
# Chroma's cosine index can renormalize float32 vectors on readback. Bound
# only this write roundtrip; the saved full-node proof uses exact readback.
return all(
math.isclose(left, right, rel_tol=4 * 2**-23, abs_tol=0.0)
for left, right in zip(actual, expected, strict=True)
)
_NATIVE_WRITES = frozenset(
{"upsert", "upsert_batch", "update_payload", "update_embedding", "delete"}
)
class HyPermissionGraphStore:
"""Lease-gated graph store for HY System 2's digest transaction.
The native graph is deliberately absent from ordinary ingest leases. Digest
needs it to write L6 schemas, but every operation must remain inside the
same frozen permission lease rather than exposing the raw store.
"""
def __init__(self, native: Any, recheck: Any) -> None:
self._native = native
self._recheck = recheck
self._closed = False
def recheck(self) -> None:
if self._closed:
raise permission_error("hy_graph_store_closed")
self._recheck()
async def close(self) -> None:
self._closed = True
def __getattr__(self, name: str) -> Any:
if name.startswith("_") and name != "_run":
raise AttributeError(name)
attribute = getattr(self._native, name)
if not callable(attribute):
return attribute
async def guarded(*args: Any, **kwargs: Any) -> Any:
self.recheck()
result = await attribute(*args, **kwargs)
self.recheck()
return result
return guarded
class HyPermissionVectorStore:
"""Closed, task-exclusive adapter; initialize/close never own the native store.
node_permissions may attest additional native raw/profile nodes not represented
in known_memories. Every attestation must cover all its contributors, and all
Evidence IDs must already be in this lease. Known entries also constrain exact
content/owner. Without a full attestation even a known entry is denied.
written_nodes records successful writes with the entire lease footprint, a
conservative provenance bound, not a claim of semantic citation accuracy.
These transient fingerprints must be persisted by the integrating factory if
future operations are to reuse raw/profile nodes. No content is stored here.
"""
__slots__ = (
"__native",
"__lease",
"__agent_id",
"__session_id",
"__cross_evidence_scope",
"__user_id",
"__known",
"__permissions",
"__written",
"__evidence_ids",
"__deleted",
"__shadowed",
"__superseded",
"__heads",
"__lock",
"__closed",
"__failure",
"__step_timer",
"__refresh_permission",
)
def __init__(
self,
native: Any,
lease: ModelProcessingPermissionLease,
*,
agent_id: str,
session_id: str,
cross_evidence_scope: bool = False,
node_permissions: Mapping[str, HyNodePermission] | None = None,
refresh_permission: Callable[[str], Awaitable[HyNodePermission | None]] | None = None,
step_timer: HyStepTimer | None = None,
) -> None:
self.__step_timer = step_timer
self.__native = native
self.__lease = lease
self.__agent_id = agent_id
self.__session_id = session_id
self.__cross_evidence_scope = cross_evidence_scope
self.__user_id = lease.user_id
self.__failure: str | None = None
self.__closed = False
self.__lock = asyncio.Lock()
self.__deleted: set[str] = set()
self.__written: dict[str, HyNodePermission] = {}
self.__shadowed: set[str] = set()
self.__superseded: dict[str, tuple[str, ...]] = {}
self.__heads: dict[str, tuple[str, ...]] = {}
self.__refresh_permission = refresh_permission
self.recheck()
self.__evidence_ids = tuple(record.evidence_id for record in lease.records)
current = lease.source_context.current if lease.source_context else lease.records
conversation_mismatch = any(r.conversation_id != session_id for r in current)
if (
not lease.task_id
or not agent_id
or agent_id == "*"
or not session_id
or not current
or (conversation_mismatch and not cross_evidence_scope)
or not self.__evidence_ids
or len(set(self.__evidence_ids)) != len(lease.records)
or any(r.user_id != self.__user_id for r in lease.records)
):
self.__deny("scope")
known = lease.source_context.known_memories if lease.source_context else ()
self.__known = {item.hy_memory_id: item for item in known}
if len(self.__known) != len(known):
self.__deny("provenance")
self.__permissions = dict(node_permissions or {})
for key, proof in self.__permissions.items():
if (
not key
or not isinstance(proof, HyNodePermission)
or len(proof.fingerprint) != 64
or any(c not in "0123456789abcdef" for c in proof.fingerprint)
or not isinstance(proof.evidence_ids, tuple)
or not proof.evidence_ids
or len(set(proof.evidence_ids)) != len(proof.evidence_ids)
or not set(proof.evidence_ids) <= set(self.__evidence_ids)
):
self.__deny("provenance")
for key, item in self.__known.items():
if not item.evidence_ids or not set(item.evidence_ids) <= set(self.__evidence_ids):
self.__deny("provenance")
known_proof = self.__permissions.get(key)
if known_proof and not set(item.evidence_ids) <= set(known_proof.evidence_ids):
self.__deny("provenance")
@property
def written_nodes(self) -> Mapping[str, HyNodePermission]:
"""Latest verified write proofs; each access returns a new immutable snapshot.
Export after SDK writes have drained and before close/governance projection.
Merge these rows into the lineage ledger, then remove deleted_node_ids;
this write delta does not include untouched pre-existing permissions.
"""
self.recheck()
return MappingProxyType(dict(self.__written))
@property
def deleted_node_ids(self) -> frozenset[str]:
"""Verified native deletions in this operation, for lineage-ledger cleanup."""
self.recheck()
return frozenset(self.__deleted)
@property
def retired_node_ids(self) -> frozenset[str]:
"""Nodes HY deleted or shadowed (its logical delete) in this operation."""
self.recheck()
return frozenset(self.__deleted | self.__shadowed)
@property
def superseded_nodes(self) -> Mapping[str, tuple[str, ...]]:
"""Nodes HY marked superseded in this operation, with what replaced them.
HY writes the two directions separately and may fail either one, so a
replacement counts whether it was recorded on the old node or only on
the new head.
"""
self.recheck()
result = {}
for key, by in self.__superseded.items():
heads = set(by) | {head for head, olds in self.__heads.items() if key in olds}
heads -= self.__deleted | {key}
if key not in self.__deleted and heads:
result[key] = tuple(sorted(heads))
return MappingProxyType(result)
@property
def supports_fulltext(self) -> bool:
return False
def recheck(self) -> None:
"""Must also gate transport/publication after HY swallows a store failure."""
self.__assert_open()
try:
self.__lease.recheck()
except Exception as error:
code = str(error)
self.__failure = (
code
if code.startswith("model_content_") and len(code) <= 100
else "model_content_hy_store_lease_failed"
)
raise permission_error(self.__failure) from None
def __assert_open(self) -> None:
if self.__closed:
self.__deny("closed")
if self.__failure:
raise permission_error(self.__failure)
async def initialize(self) -> None:
self.recheck()
async def close(self) -> None:
"""Revoke this wrapper locally without rechecking context or closing HY.
Cleanup must remain possible after governance projection changes the
lease's known-memory revisions, or after permission has been revoked.
"""
async with self.__lock:
self.__closed = True
self.__permissions.clear()
self.__written.clear()
async def has_tag_embedding(self, user_id: str, tag: str) -> bool:
self.recheck()
# Native chat calls this best-effort helper. NotImplemented prevents it
# from embedding tags or falling through to an unguarded entity store.
raise NotImplementedError("hy_permission_store_tag_index_unsupported")
def __deny(self, suffix: str) -> NoReturn:
self.__failure = self.__failure or f"model_content_hy_store_{suffix}"
raise permission_error(self.__failure)
async def __invoke(self, method: str, *args: Any, **kwargs: Any) -> Any:
writing = method in _NATIVE_WRITES
self.recheck() if writing else self.__assert_open()
step = "vector_store_write" if writing else "vector_store_read"
first = args[0] if args else None
try:
with (
self.__step_timer.step(step, items=len(first) if isinstance(first, list) else 1)
if self.__step_timer is not None
else nullcontext()
):
result = await getattr(self.__native, method)(*args, **kwargs)
except Exception:
self.__deny("backend_failed")
self.recheck() if writing else self.__assert_open()
return result
async def __read_native(self, key: str) -> MemoryNode | None:
value = await self.__invoke("get_by_id", key)
if value is not None and (type(value) is not MemoryNode or value.node_id != key):
self.__deny("node_shape")
return deepcopy(value)
def __fingerprint(self, node: MemoryNode) -> str:
try:
return hy_node_fingerprint(node)
except (TypeError, ValueError, AttributeError, OverflowError):
self.__deny("node_shape")
def __write_view(self, node: MemoryNode) -> dict[str, Any]:
self.__fingerprint(node)
try:
return _write_view(node)
except (TypeError, ValueError, OverflowError, struct.error):
self.__deny("node_shape")
def __in_scope(self, node: MemoryNode) -> bool:
if node.user_id != self.__user_id or node.layer not in _CHAT_LAYERS:
return False
if node.layer is MemoryLayer.L0_BASIC_INFO:
# Public SDK 2.0.2 migrates verified legacy agent-level profiles into one
# user-level chain. Existing nodes still require a full proof in
# __load; accepting their historical agent id does not authorize
# unproven cross-agent memories.
return True
if self.__cross_evidence_scope:
return bool(node.agent_id == self.__agent_id)
return bool(node.agent_id == self.__agent_id and node.session_id == self.__session_id)
def __new_node_in_scope(self, node: MemoryNode) -> bool:
if not self.__in_scope(node):
return False
if node.layer is MemoryLayer.L0_BASIC_INFO:
return node.agent_id in ("", self.__agent_id) and node.session_id in (
"",
self.__session_id,
)
return True
def __references(self, node: MemoryNode) -> list[str]:
refs = []
for field in ("supersedes", "superseded_by", "evidence_chain"):
values = getattr(node, field)
if field != "evidence_chain" and values == "":
continue
if values is not None:
if not isinstance(values, list) or any(
not isinstance(v, str) or not v for v in values
):
self.__deny("node_shape")
refs.extend(values)
if node.source_raw_memory_id is not None:
if not isinstance(node.source_raw_memory_id, str) or not node.source_raw_memory_id:
self.__deny("node_shape")
refs.append(node.source_raw_memory_id)
return refs
async def __load(
self,
key: str,
first: MemoryNode | None = None,
*,
allow_deleted_links: bool = False,
) -> MemoryNode | None:
pending = [key]
checked: dict[str, MemoryNode] = {}
while pending:
current = pending.pop()
if current in checked:
continue
# HY dedup deletes head then ancestors. Only this operation's
# verified tombstones may terminate a closure during deletion.
if allow_deleted_links and current != key and current in self.__deleted:
continue
proof = self.__permissions.get(current)
if proof is None or current in self.__deleted:
return None
value = (
first if current == key and first is not None else await self.__read_native(current)
)
if value is None or not self.__in_scope(value):
return None
if self.__fingerprint(value) != proof.fingerprint:
refreshed = (
await self.__refresh_permission(current)
if self.__refresh_permission is not None
else None
)
if refreshed is None or self.__fingerprint(value) != refreshed.fingerprint:
self.__deny("node_changed")
self.__permissions[current] = refreshed
proof = refreshed
known = self.__known.get(current) if current not in self.__written else None
if known and (value.content != known.content or value.owner != known.owner):
self.__deny("node_changed")
checked[current] = value
pending.extend(self.__references(value))
return deepcopy(checked[key])
async def get_by_id(self, node_id: str) -> MemoryNode | None:
async with self.__lock:
self.recheck()
result = await self.__load(node_id)
self.recheck()
return result
async def get_by_ids(self, node_ids: list[str]) -> list[MemoryNode]:
async with self.__lock:
self.recheck()
result = [
n for key in dict.fromkeys(node_ids) if (n := await self.__load(key)) is not None
]
self.recheck()
return result
async def get_embeddings(self, node_ids: list[str]) -> dict[str, list[float]]:
return {
n.node_id: list(n.embedding)
for n in await self.get_by_ids(node_ids)
if n.embedding is not None
}
def __scope(self, user_id: str | None, agent_id: str | None) -> None:
if user_id not in (None, self.__user_id) or agent_id not in (None, self.__agent_id):
self.__deny("scope")
async def list_by_user(
self,
user_id: str | None = None,
agent_id: str | None = None,
limit: int = 10000,
status_filter: list[Any] | None = None,
layers: list[Any] | None = None,
user_ids: list[str] | None = None,
agent_ids: list[str] | None = None,
only_latest: bool = False,
with_vectors: bool = True,
allow_raw_layer: bool = False,
) -> list[MemoryNode]:
async with self.__lock:
self.recheck()
effective_users = user_ids or ([user_id] if user_id else [])
effective_agents = agent_ids or ([agent_id] if agent_id else [])
if effective_users != [self.__user_id] or any(
value != self.__agent_id for value in effective_agents
):
self.__deny("scope")
rows = await self.__invoke(
"list_by_user",
user_id=self.__user_id,
agent_id=agent_id,
limit=limit,
status_filter=status_filter,
layers=layers,
user_ids=[self.__user_id],
agent_ids=effective_agents or None,
only_latest=only_latest,
with_vectors=with_vectors,
allow_raw_layer=allow_raw_layer,
)
result = []
for row in rows:
if type(row) is not MemoryNode:
self.__deny("node_shape")
value = await self.__load(row.node_id)
if (
value is not None
and (not layers or value.layer in layers or value.layer.value in layers)
and (
not status_filter
or value.status in status_filter
or value.status.value in status_filter
)
):
result.append(value)
self.recheck()
return result[:limit]
async def search(
self,
query_embedding: list[float],
isolation_key: str = "",
isolation_keys: list[str] | None = None,
user_id: str | None = None,
user_ids: list[str] | None = None,
agent_ids: list[str] | None = None,
layers: list[Any] | None = None,
limit: int = 10,
score_threshold: float = 0.0,
status_filter: list[Any] | None = None,
only_latest: bool = True,
tags_match_any: list[str] | None = None,
created_after: float | None = None,
) -> list[dict[str, Any]]:
async with self.__lock:
self.recheck()
self.__scope(user_id, None)
key = MemoryNode.build_isolation_key(self.__user_id, self.__agent_id, self.__session_id)
if (
any(uid != self.__user_id for uid in user_ids or [])
or any(aid != self.__agent_id for aid in agent_ids or [])
or isolation_key not in ("", key)
or any(ik != key for ik in isolation_keys or [])
):
self.__deny("scope")
hits = await self.__invoke(
"search",
query_embedding=deepcopy(query_embedding),
isolation_key=isolation_key,
isolation_keys=isolation_keys,
user_id=self.__user_id,
user_ids=[self.__user_id],
agent_ids=[self.__agent_id],
layers=layers,
limit=limit,
score_threshold=score_threshold,
status_filter=status_filter,
only_latest=only_latest,
tags_match_any=tags_match_any,
created_after=created_after,
)
result = []
for hit in hits:
value = await self.__load(hit["node_id"])
if value is None:
continue
score = float(hit["score"])
if not math.isfinite(score):
self.__deny("node_shape")
# Do not forward arbitrary backend hit metadata. Rehydrate the
# full fingerprinted node; native search often omits embedding.
result.append({"node_id": value.node_id, "node": value, "score": score})
self.recheck()
return result[:limit]
async def __target(
self,
key: str,
first: MemoryNode | None = None,
*,
allow_deleted_links: bool = False,
) -> MemoryNode:
value = await self.__load(key, first, allow_deleted_links=allow_deleted_links)
if value is None:
self.__deny("target_denied")
return value
async def __check_links(self, value: MemoryNode) -> None:
for ref in self.__references(value):
if await self.__load(ref) is None:
self.__deny("link_denied")
async def __plan_upsert(self, value: MemoryNode) -> MemoryNode:
value = deepcopy(value)
if type(value) is not MemoryNode or not value.node_id or not self.__in_scope(value):
self.__deny("scope")
existing = await self.__read_native(value.node_id)
if existing is not None:
await self.__target(value.node_id, existing)
elif (
not self.__new_node_in_scope(value)
or value.node_id in self.__permissions
or value.node_id in self.__known
or value.node_id in self.__deleted
):
self.__deny("scope")
await self.__check_links(value)
self.__write_view(value)
return value
async def __capture(
self,
expected: MemoryNode,
*,
allow_native_modified_time: bool = False,
) -> None:
actual = await self.__read_native(expected.node_id)
if actual is None:
self.__deny("write_mismatch")
actual_view, expected_view = self.__write_view(actual), self.__write_view(expected)
if allow_native_modified_time:
if actual.gmt_modified < expected.gmt_modified:
self.__deny("write_mismatch")
actual_view["gmt_modified"] = expected_view["gmt_modified"]
actual_vector = actual_view.pop("embedding")
expected_vector = expected_view.pop("embedding")
if actual_view != expected_view or not _same_written_vector(actual_vector, expected_vector):
self.__deny("write_mismatch")
proof = HyNodePermission(self.__fingerprint(actual), self.__evidence_ids)
self.__permissions[actual.node_id] = proof
self.__written[actual.node_id] = proof
status = getattr(actual.status, "value", actual.status)
if status == MemoryStatus.SHADOW.value:
self.__shadowed.add(actual.node_id)
else:
self.__shadowed.discard(actual.node_id)
if status == MemoryStatus.SUPERSEDED.value:
by = actual.superseded_by
self.__superseded[actual.node_id] = tuple(by) if isinstance(by, list) else ()
else:
self.__superseded.pop(actual.node_id, None)
olds = actual.supersedes
if isinstance(olds, list) and olds:
self.__heads[actual.node_id] = tuple(olds)
else:
self.__heads.pop(actual.node_id, None)
async def upsert(self, node: MemoryNode) -> str:
async with _native_write_lock(), self.__lock:
self.recheck()
expected = await self.__plan_upsert(node)
result = await self.__invoke("upsert", deepcopy(expected))
if result != expected.node_id:
self.__deny("write_mismatch")
await self.__capture(expected)
return str(result)
async def upsert_batch(self, nodes: list[MemoryNode]) -> list[str]:
async with _native_write_lock(), self.__lock:
self.recheck()
if len({n.node_id for n in nodes}) != len(nodes):
self.__deny("node_shape")
# No forward links to uncommitted batch members. Preflight every
# existing target before the backend can make a partial write.
planned = [await self.__plan_upsert(n) for n in nodes]
ids = [n.node_id for n in planned]
result = await self.__invoke("upsert_batch", deepcopy(planned))
if result != ids:
self.__deny("write_mismatch")
for expected in planned:
await self.__capture(expected)
return ids
async def update_payload(self, node_id: str, updates: dict[str, Any]) -> bool:
async with _native_write_lock(), self.__lock:
self.recheck()
current = await self.__target(node_id)
updates = deepcopy(updates)
dedup_audit = False
if set(updates) & _DEDUP_AUDIT_FIELDS:
if (
not set(updates) <= (_UPDATE_FIELDS | _DEDUP_AUDIT_FIELDS)
or updates.get("status") != MemoryStatus.SHADOW.value
or updates.get("is_latest") is not False
or not isinstance(updates.get("shadow_reason"), str)
or not updates["shadow_reason"].startswith("dedup:")
or not isinstance(updates.get("dedup_kept_id"), str)
or not updates["dedup_kept_id"]
):
self.__deny("update_fields")
await self.__target(updates["dedup_kept_id"])
custom = deepcopy(current.custom)
custom["hy_dedup"] = {
"reason": updates.pop("shadow_reason"),
"kept_id": updates.pop("dedup_kept_id"),
}
updates["custom"] = custom
dedup_audit = True
allowed = _UPDATE_FIELDS | ({"custom"} if dedup_audit else set())
if self.__cross_evidence_scope and set(updates) == {"custom"}:
custom = updates.get("custom")
if not isinstance(custom, dict):
self.__deny("update_fields")
current_custom = current.custom if isinstance(current.custom, dict) else {}
previous = current_custom.get(_DIGEST_COUNTER_FIELD, 0)
proposed = custom.get(_DIGEST_COUNTER_FIELD)
if (
not isinstance(previous, int)
or isinstance(previous, bool)
or not isinstance(proposed, int)
or isinstance(proposed, bool)
or proposed != previous + 1
or set(custom) != set(current_custom) | {_DIGEST_COUNTER_FIELD}
or any(custom[key] != value for key, value in current_custom.items())
):
self.__deny("update_fields")
allowed |= {"custom"}
# `custom` is an allowed field name (SDK 2.0.2 sends it on a
# reconcile UPDATE), but the System 2 evidence counter it can hold
# stays digest-only: only the digest scope may advance it, and only
# by one per write. Letting the name through must not let an
# ordinary write move that counter.
if not self.__cross_evidence_scope and _DIGEST_COUNTER_FIELD in (
updates.get("custom") or {}
):
self.__deny("update_fields")
if not set(updates) <= allowed:
self.__deny("update_fields")
try:
if "layer" in updates:
MemoryLayer(updates["layer"])
if "status" in updates:
MemoryStatus(updates["status"])
if "tag_ners" in updates and (
not isinstance(updates["tag_ners"], list)
or any(not isinstance(value, str) for value in updates["tag_ners"])
):
self.__deny("node_shape")
expected = MemoryNode.from_dict({**current.to_dict(), **updates})
self.__fingerprint(expected)
except (TypeError, ValueError):
self.__deny("node_shape")
if not self.__in_scope(expected):
self.__deny("scope")
await self.__check_links(expected)
self.__write_view(expected)
if await self.__invoke("update_payload", node_id, updates) is not True:
self.__deny("write_failed")
await self.__capture(expected, allow_native_modified_time=True)
return True
async def update_embedding(self, node_id: str, embedding: list[float]) -> bool:
async with _native_write_lock(), self.__lock:
self.recheck()
expected = await self.__target(node_id)
expected.embedding = deepcopy(embedding)
self.__write_view(expected)
if await self.__invoke("update_embedding", node_id, deepcopy(embedding)) is not True:
self.__deny("write_failed")
await self.__capture(expected, allow_native_modified_time=True)
return True
async def delete(self, node_id: str) -> bool:
async with _native_write_lock(), self.__lock:
self.recheck()
await self.__target(node_id, allow_deleted_links=True)
if await self.__invoke("delete", node_id) is not True:
self.__deny("write_failed")
if await self.__read_native(node_id) is not None:
self.__deny("write_mismatch")
self.__deleted.add(node_id)
self.__permissions.pop(node_id, None)
self.__written.pop(node_id, None)
self.__superseded.pop(node_id, None)
self.__heads.pop(node_id, None)
return True
File diff suppressed because it is too large. Load diff
@@ -144,10 +144,6 @@ class HyReconcileGuard:
self.failures: list[str] = []
self.bindings: HyWriteBindings | None = None
self.imported_contents: frozenset[str] | None = None
self.mutation_target_ids: set[str] = set()
def begin_write(self) -> None:
self.mutation_target_ids.clear()
def __getattr__(self, name: str) -> Any:
return getattr(self.reconciler, name)
@@ -214,15 +210,6 @@ class HyReconcileGuard:
for operation in result.ops:
raw_name = getattr(operation, "op", "")
name = str(getattr(raw_name, "value", raw_name)).upper()
if name in {"UPDATE", "SUPERSEDE", "DELETE", "SHADOW"}:
target = getattr(operation, "memory_id", None)
if isinstance(target, str) and target:
self.mutation_target_ids.add(target)
self.mutation_target_ids.update(
key
for key in getattr(operation, "memory_ids", ())
if isinstance(key, str) and key
)
if name == "ADD":
operation_counts["ADD"] += 1
elif name in {"UPDATE", "MERGE", "NOOP"}:
@@ -245,6 +232,20 @@ class HyReconcileGuard:
"result_count": len(result.ops),
},
)
timer = self.step_timer
try:
with (
timer.step("write_verification", items=len(result.ops))
if timer is not None
else nullcontext()
):
await self._check_targets(result, kwargs)
except Exception as error:
code = str(error)
if not code.startswith("hy_memory_write_verification_") or len(code) > 100:
code = "hy_memory_write_verification_failed"
self.failures.append(code)
raise ValueError(code) from error
return result
async def _check_targets(self, result: Any, kwargs: dict[str, Any]) -> None:
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.4"
HY_MEMORY_VERSION = "2.0.2"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -31,12 +31,11 @@ _COLLECTION_LOGGERS = (
"httpx",
"httpcore",
)
_NOISY_LIBRARY_LOGGERS = ("httpx", "httpcore", "watchfiles", "jieba")
# One Memory deliberately stays on ultra because its existing, user-visible
# profile/graph flow calls digest(), which the pinned public SDK exposes only
# for ultra clients. Reader and non-reranker behavior remain pinned for
# compatibility with public SDK 2.0.4.
# compatibility with public SDK 2.0.2.
HY_MEMORY_MODE = "ultra"
# The SDK only recognises `legacy` and `hybrid` (see
# hy_memory/pipelines/_retrieval/config.py: `ALL_READERS = (READER_LEGACY,
@@ -140,18 +139,6 @@ def apply_diagnostic_log_level(level: DiagnosticLogLevel) -> None:
for name in tuple(logging.Logger.manager.loggerDict):
if any(name == root or name.startswith(f"{root}.") for root in _COLLECTION_LOGGERS):
logging.getLogger(name).setLevel(selected)
# Successful HTTP exchanges, file-watch notifications and tokenizer
# initialization are useful only while actively debugging. At INFO they
# dominated core.log and the in-app panel without describing product work.
noisy_level = selected if selected == logging.DEBUG else max(selected, logging.WARNING)
for root in _NOISY_LIBRARY_LOGGERS:
logging.getLogger(root).setLevel(noisy_level)
for name in tuple(logging.Logger.manager.loggerDict):
if any(
name == root or name.startswith(f"{root}.")
for root in _NOISY_LIBRARY_LOGGERS
):
logging.getLogger(name).setLevel(noisy_level)
os.environ["MEMORY_LOG_LEVEL"] = logging.getLevelName(selected)
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "true" if level == "debug" else "false"
@@ -7,7 +7,7 @@ from typing import Any, Literal
from pydantic import BaseModel, ConfigDict
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord, WrittenBy
from one_memory.input_sanitization import sanitize_model_input
@@ -61,7 +61,17 @@ class HyWriteBindings:
seed.workspace,
)
or record.content_kind is not EvidenceContentKind.RAW_CONVERSATION
or record.role not in {"user", "assistant", "tool"}
or (record.role, record.written_by)
not in {
# D98: the same change as the turn assembly. A sub agent's
# question is a `user` message marked written by the
# agent; excluding it here would make the source context
# disagree with what was actually sent.
("user", WrittenBy.PERSON),
("user", WrittenBy.AGENT),
("assistant", WrittenBy.AGENT),
("tool", WrittenBy.TOOL),
}
):
raise ValueError("hy_memory_write_verification_context_scope")
self.all_sources: list[dict[str, Any]] = [
@@ -152,7 +152,7 @@ class HyStepTimer:
if value is None:
continue
diagnostic_event(
level="debug",
level="info",
component=STEP_COMPONENTS[name],
event="hy.step.completed",
outcome="succeeded",
@@ -168,7 +168,7 @@ class HyStepTimer:
)
model = measured.get("model")
diagnostic_event(
level="debug",
level="info",
component="hy.bridge",
event="hy.steps.summary",
outcome="succeeded",
@@ -1,41 +0,0 @@
"""Process-local coordination for HY's native stores."""
from __future__ import annotations
import asyncio
import weakref
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
_WRITE_LOCKS: weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
weakref.WeakKeyDictionary()
)
def native_store_write_lock() -> asyncio.Lock:
"""Return the write lock for the current event loop."""
loop = asyncio.get_running_loop()
lock = _WRITE_LOCKS.get(loop)
if lock is None:
lock = asyncio.Lock()
_WRITE_LOCKS[loop] = lock
return lock
@asynccontextmanager
async def native_store_read_slot() -> AsyncIterator[bool]:
"""Admit a native HY read only while no write is active.
Chroma's native reader may retain the GIL after an asyncio timeout. Reads
therefore share the physical-store lock, but never wait behind a writer:
host recall can immediately fall back to the governance ledger.
"""
lock = native_store_write_lock()
if lock.locked():
yield False
return
await lock.acquire()
try:
yield True
finally:
lock.release()
@@ -105,7 +105,7 @@ async def iterate_in_worker[T](
# was unwinding with, and a diagnostic is not worth an import.
try:
diagnostic_event(
level="debug",
level="info",
component="import.async_bridge",
event="import.source_read_finished",
outcome="succeeded",
@@ -1,7 +1,6 @@
import sqlite3
from contextlib import AbstractContextManager
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
from pydantic import BaseModel, ConfigDict
@@ -21,23 +20,6 @@ CURSOR_MAX_SOURCE_FILE_BYTES = 64 * 1024 * 1024 * 1024
GRANT_SCHEMA = source_grants_ddl("sqlite")
class SourceAuthorizationState(StrEnum):
"""Why a source cannot be read right now, or `ok` when it can.
The API used to answer `authorized=false` for all of these at once, which
is how a revoked grant, a rebuilt history directory and a moved candidate
all looked like the same bug and all got the same advice. The values carry
no paths and no user content, because they are written to diagnostics.
"""
OK = "ok"
MISSING_GRANT = "missing_grant"
REVOKED_GRANT = "revoked_grant"
SOURCE_IDENTITY_CHANGED = "source_identity_changed"
SOURCE_ROOT_UNAVAILABLE = "source_root_unavailable"
CANDIDATE_CHANGED = "candidate_changed"
class SourceGrantReceipt(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
@@ -164,46 +146,6 @@ class SourceGrantStore:
raise SourceAccessError(SourceAccessErrorCode.SOURCE_IDENTITY_CHANGED)
return authorized
def authorization_state(
self,
*,
user_id: str,
source_id: str,
candidate: Path,
) -> SourceAuthorizationState:
"""Why reading this source would be refused.
Read-only and fails the same way `authorized_root` does: this names a
state for the UI and for diagnostics, it does not widen access. The
candidate is compared by path, so a grant that still points somewhere
else reads as a moved candidate rather than as a rebuilt directory —
the two need different fixes.
"""
with self._connect() as connection:
row = connection.execute(
"SELECT * FROM source_grants WHERE user_id=? AND source_id=?",
(user_id, source_id),
).fetchone()
if row is None:
return SourceAuthorizationState.MISSING_GRANT
if row["revoked_at"] is not None or row["revision"] <= 0:
return SourceAuthorizationState.REVOKED_GRANT
granted = Path(row["requested_path"])
if not granted.exists():
return SourceAuthorizationState.SOURCE_ROOT_UNAVAILABLE
try:
if not root_identity_matches(granted, row["root_identity"]):
return SourceAuthorizationState.SOURCE_IDENTITY_CHANGED
except SourceAccessError:
# Unreadable or gone between the two calls: there is no grant to
# repair, only a directory to make available again.
return SourceAuthorizationState.SOURCE_ROOT_UNAVAILABLE
# Same directory, but the app now looks somewhere else. Re-authorizing
# the old grant would not help; the candidate has to be granted.
if granted != candidate:
return SourceAuthorizationState.CANDIDATE_CHANGED
return SourceAuthorizationState.OK
def revoke(self, *, user_id: str, source_id: str) -> SourceGrantReceipt:
now = datetime.now(UTC).isoformat()
with self._connect(writing=True) as connection:
@@ -106,7 +106,7 @@ class PollingLiveSyncSupervisor:
level=(
"warn"
if value.state in {LiveSyncState.DEGRADED, LiveSyncState.UNAVAILABLE}
else "debug"
else "info"
),
component="import.live_sync",
event="live_sync.state_changed",
@@ -156,7 +156,7 @@ class FileWatchLiveSync:
if source is not None
):
diagnostic_event(
level="debug",
level="info",
component="import.live_watch",
event="live_sync.hook_received",
outcome="queued",
@@ -190,7 +190,7 @@ class FileWatchLiveSync:
try:
await self._trigger_past(source_id, observed)
diagnostic_event(
level="debug",
level="info",
component="import.live_watch",
event="live_sync.hook_completed",
outcome="written",
@@ -79,8 +79,6 @@ class EvidenceInput(BaseModel):
source_ordinal: str | None = Field(default=None, min_length=1, max_length=256)
content_kind: EvidenceContentKind = EvidenceContentKind.RAW_CONVERSATION
role: str = Field(min_length=1, max_length=128)
# Legacy import/storage metadata only. Protocol `role` is authoritative;
# this value must not affect identity, admission, scheduling or ownership.
written_by: WrittenBy
created_at: datetime | None = None
verbatim: str = Field(min_length=1, max_length=MAX_VERBATIM_CHARACTERS)
@@ -151,22 +149,17 @@ class EvidenceInput(BaseModel):
)
)
else:
# `role` is the protocol-level speaker identity. `written_by` was a
# parser guess and differed between two imports of the same host
# message (notably sub-agent/user and unknown/user), minting two
# Evidence rows. Keep it out of every new identity.
#
# No workspace and no source_ordinal in the derived key: the hook
# lane has neither (it reports no workspace and a constant ordinal),
# so including either gave the same message a different key per lane
# and stored it twice. Both are record metadata here, not identity.
identity = "\0".join(
(
"derived-v4",
"derived-v3",
self.user_id,
self.source_id,
self.conversation_id,
self.role,
self.written_by,
self.content_hash,
)
)
@@ -175,54 +168,6 @@ class EvidenceInput(BaseModel):
# two rows. The capture time is record metadata, not identity.
return hashlib.sha256(identity.encode()).hexdigest()
@property
def written_by_derived_v3_idempotency_key(self) -> str:
"""The last identity that treated the parser's author guess as canonical.
New builds key by protocol role. During an upgrade this key lets a
rescan find and reuse the row already stored by derived-v3 instead of
inserting the same host message under a second Evidence id.
"""
identity = "\0".join(
(
"derived-v3",
self.user_id,
self.source_id,
self.conversation_id,
self.written_by,
self.content_hash,
)
)
return hashlib.sha256(identity.encode()).hexdigest()
@property
def written_by_derived_v3_idempotency_keys(self) -> tuple[str, ...]:
"""Every legacy author guess this protocol role may have carried."""
guesses = {
"user": (
WrittenBy.PERSON,
WrittenBy.AGENT,
WrittenBy.UNKNOWN,
),
"assistant": (WrittenBy.AGENT, WrittenBy.UNKNOWN, WrittenBy.PERSON),
"tool": (WrittenBy.TOOL, WrittenBy.UNKNOWN),
}.get(self.role, tuple(WrittenBy))
ordered = (self.written_by, *guesses)
keys: list[str] = []
for written_by in dict.fromkeys(ordered):
identity = "\0".join(
(
"derived-v3",
self.user_id,
self.source_id,
self.conversation_id,
written_by,
self.content_hash,
)
)
keys.append(hashlib.sha256(identity.encode()).hexdigest())
return tuple(keys)
@property
def legacy_message_idempotency_key(self) -> str | None:
"""The pre-workspace key for a message-shaped input, when one exists.
@@ -267,13 +212,7 @@ class EvidenceInput(BaseModel):
"""
return self.derived_v2_idempotency_key("derived-v2", workspace_agnostic=False)
def derived_v2_idempotency_key(
self,
label: str,
*,
workspace_agnostic: bool,
written_by: WrittenBy | None = None,
) -> str:
def derived_v2_idempotency_key(self, label: str, *, workspace_agnostic: bool) -> str:
identity = "\0".join(
(
label,
@@ -281,7 +220,7 @@ class EvidenceInput(BaseModel):
self.source_id,
*([] if workspace_agnostic else [self.workspace or "\uffffnull"]),
self.conversation_id,
written_by or self.written_by,
self.written_by,
self.created_at.isoformat() if self.created_at else "",
self.source_ordinal or "",
# v2 hashed the cleaned body; v1 predates raw_verbatim entirely,
@@ -291,26 +230,6 @@ class EvidenceInput(BaseModel):
)
return hashlib.sha256(identity.encode()).hexdigest()
@property
def older_written_by_idempotency_keys(self) -> tuple[str, ...]:
keys: list[str] = []
for written_by in WrittenBy:
keys.append(
self.derived_v2_idempotency_key(
"derived-v1",
workspace_agnostic=True,
written_by=written_by,
)
)
keys.append(
self.derived_v2_idempotency_key(
"derived-v2",
workspace_agnostic=False,
written_by=written_by,
)
)
return tuple(keys)
@property
def evidence_id(self) -> str:
return str(uuid5(EVIDENCE_NAMESPACE, self.idempotency_key))
@@ -330,7 +249,6 @@ class EvidenceRecord(BaseModel):
source_ordinal: str | None
content_kind: EvidenceContentKind
role: str
# Retained while old databases and export formats still carry the column.
written_by: WrittenBy
created_at: datetime | None
verbatim: str
@@ -199,7 +199,7 @@ class ImportRunner:
raise ValueError("Import job source does not match adapter source")
job, runner_token = await in_the_background(self.job_store.claim, job_id)
diagnostic_event(
level="debug",
level="info",
component="import.parser",
event="import.parse.started",
outcome="started",
@@ -310,7 +310,7 @@ class ImportRunner:
if event.kind == "progress" and str(event.phase) != last_phase:
phase = str(event.phase)
diagnostic_event(
level="debug",
level="info",
component="import.parser",
event="import.phase_changed",
outcome="changed",
@@ -516,7 +516,7 @@ class ImportRunner:
async def _emit_parse_result(self, job: ImportJob, started: float) -> None:
succeeded = job.status is ImportStatus.COMPLETED
diagnostic_event(
level="debug" if succeeded else "warn",
level="info" if succeeded else "warn",
component="import.parser",
event="import.parse.completed",
outcome="succeeded" if succeeded else str(job.status),
@@ -141,18 +141,11 @@ def parse_codex_record(
if role == "user":
written_by = _user_author(record, payload, session)
# Machine inputs may quote a whole transcript, including user_query tags.
# Protocol role is authoritative; host author metadata does not change
# how a user message is cleaned.
if not (
content.startswith(
"The following is the Codex agent history whose request action you are assessing."
)
and ">>> TRANSCRIPT START" in content
and ">>> TRANSCRIPT END" in content
):
# Session provenance is not proof of who wrote any particular future turn.
if written_by is WrittenBy.PERSON:
content = strip_host_wrappers(content)
if not content:
return ParsedSkip(ScanIssueCode.NO_TEXT)
if not content:
return ParsedSkip(ScanIssueCode.NO_TEXT)
return ParsedMessage(
conversation_id=conversation_from_relative_path(context.relative_path),
message_id=first_string(payload.get("id"), record.get("id")),
@@ -355,7 +355,7 @@ class JsonlSourceAdapter:
source_file, checkpoint, header = await asyncio.to_thread(
self._prepare, listed, options
)
_log.debug(
_log.info(
"import_file_read source=%s file=%d/%d file_bytes=%d planned_bytes=%d",
self.descriptor.source_id,
index,
@@ -376,7 +376,7 @@ class JsonlSourceAdapter:
else DEFAULT_IMPORT_MAX_FILE_BYTES
)
if source_file.size > budget:
_log.debug(
_log.info(
"import_file_budgeted source=%s file=%d/%d file_bytes=%d budget=%d path=%s",
self.descriptor.source_id,
index,
@@ -7,7 +7,7 @@ import sqlite3
import stat
from collections.abc import AsyncIterator, Iterator
from dataclasses import dataclass, replace
from datetime import UTC, datetime, timedelta, timezone
from datetime import UTC, datetime, timedelta
from functools import partial
from pydantic import JsonValue
@@ -82,11 +82,6 @@ _CURSOR_HOST_CONTEXT_START = re.compile(
)
_CURSOR_USER_QUERY = re.compile(r"<user_query\b")
_CURSOR_TIMESTAMP_PREFIX = re.compile(r"<timestamp>[\s\S]*?</timestamp>\s*")
_CURSOR_TIMESTAMP_VALUE = re.compile(
r"<timestamp>\s*"
r"([A-Za-z]+,\s+[A-Za-z]{3}\s+\d{1,2},\s+\d{4},\s+\d{1,2}:\d{2}\s+[AP]M)"
r"\s+\(UTC([+-]\d{1,2})(?::(\d{2}))?\)\s*</timestamp>",
)
_CURSOR_IMAGE_FILES_PREFIX = re.compile(r"\[Image\]\s*<image_files>[\s\S]*?</image_files>\s*")
_CURSOR_BARE_IMAGE_PREFIX = re.compile(r"(?:\[Image\]\s*)+")
@@ -128,24 +123,6 @@ def _strip_cursor_timestamp(content: str | None) -> str | None:
return _CURSOR_TIMESTAMP_PREFIX.sub("", content, count=1) if content else None
def _cursor_host_timestamp(content: str | None) -> datetime | None:
"""Parse Cursor's user-visible timestamp envelope into UTC."""
if not content:
return None
match = _CURSOR_TIMESTAMP_VALUE.search(content)
if match is None:
return None
try:
local = datetime.strptime(match[1], "%A, %b %d, %Y, %I:%M %p")
hours = int(match[2])
minutes = int(match[3] or "0")
sign = -1 if hours < 0 else 1
offset = timedelta(hours=hours, minutes=sign * minutes)
return local.replace(tzinfo=timezone(offset)).astimezone(UTC)
except (ValueError, OverflowError):
return None
def _strip_cursor_host_prefix(content: str | None) -> str | None:
if not content:
return content
@@ -421,7 +398,10 @@ class CursorUnifiedSourceAdapter:
if (
isinstance(event, ScannedEvidence)
and event.evidence.conversation_id in transcript_ids
and not is_host_generated_user_turn(event.evidence.verbatim)
and not (
event.evidence.written_by is WrittenBy.AGENT
and is_host_generated_user_turn(event.evidence.verbatim)
)
):
yield ScanSkip(
cursor=event.cursor,
@@ -446,7 +426,10 @@ def _shadowed_by_transcript(event: ScanEvent, transcript_ids: set[str]) -> ScanE
if (
isinstance(event, ScannedEvidence)
and event.evidence.conversation_id in transcript_ids
and not is_host_generated_user_turn(event.evidence.verbatim)
and not (
event.evidence.written_by is WrittenBy.AGENT
and is_host_generated_user_turn(event.evidence.verbatim)
)
):
return ScanSkip(
cursor=event.cursor,
@@ -720,7 +703,6 @@ def _parse_transcript_record(
project_name = path_parts[path_parts.index("projects") + 1] if "projects" in path_parts else ""
session_name = context.relative_path.rsplit("/", 1)[-1].removesuffix(".jsonl")
is_subagent = f"{project_name}\0{session_name}" in subagent_ids
host_timestamp = _cursor_host_timestamp(content) if role == "user" else None
created_at = _transcript_fallback_time(context)
native_message_id = record.get("id") or message.get("id")
message_id = (
@@ -746,16 +728,7 @@ def _parse_transcript_record(
created_at=created_at,
workspace=workspace,
record_type="cursor_transcript",
origin_metadata=OriginMetadata(
raw_meta={
"time_basis": "file_mtime",
**(
{"turn_event_at": host_timestamp.isoformat()}
if host_timestamp is not None
else {}
),
}
),
origin_metadata=OriginMetadata(raw_meta={"time_basis": "file_mtime"}),
),
tool_blocks_evidence(message.get("content"), block_types=TOOL_BLOCKS),
)
@@ -846,13 +819,6 @@ def _read_store_messages(
reason_code=ScanIssueCode.UNKNOWN_RECORD,
)
continue
host_timestamp = (
_cursor_host_timestamp(content) if role == "user" else None
)
fallback_timestamp = datetime.fromtimestamp(
source_file.modified_ns / 1_000_000_000,
tz=UTC,
) + timedelta(microseconds=rowid)
parsed = ParsedMessage(
conversation_id=conversation_id,
message_id=str(record.get("id") or row["id"]),
@@ -863,19 +829,8 @@ def _read_store_messages(
else written_by_user
),
verbatim=cleaned,
created_at=fallback_timestamp,
workspace=_store_workspace(source_file),
record_type="cursor_store_blob",
origin_metadata=OriginMetadata(
raw_meta={
"time_basis": "file_mtime",
**(
{"turn_event_at": host_timestamp.isoformat()}
if host_timestamp is not None
else {}
),
}
),
)
yield ScannedEvidence(
cursor=cursor,
@@ -57,8 +57,7 @@ _UNANSWERED_PREFIXES = (
def _is_visible_user_row(role: str, written_by: str, verbatim: str) -> bool:
del written_by # Legacy storage metadata; protocol role is authoritative.
if role != "user":
if role != "user" or written_by != WrittenBy.PERSON.value:
return False
return not verbatim.lstrip().lower().startswith(_UNANSWERED_PREFIXES)
@@ -469,54 +468,16 @@ class EvidenceStore:
if anchored is not None:
self._write_origins(connection, anchored, evidence.origin_meta, now)
workspace, project = self._resolve_conversation_attribution(connection, evidence)
existing = connection.execute(
"SELECT role,revision FROM evidence WHERE evidence_id=?",
(anchored,),
).fetchone()
if existing is None:
raise RuntimeError("anchored evidence disappeared")
if str(existing["role"]) != evidence.role:
was_revised = True
revision = int(existing["revision"]) + 1
connection.execute(
"""
UPDATE evidence
SET workspace=COALESCE(workspace,?),
project=COALESCE(project,?),
content_kind=?,role=?,created_at=COALESCE(?,created_at),
verbatim=?,content_hash=?,revision=?,updated_at=?
WHERE evidence_id=?
""",
(
workspace,
project,
evidence.content_kind,
evidence.role,
evidence.created_at.isoformat() if evidence.created_at else None,
evidence.verbatim,
evidence.content_hash,
revision,
now,
anchored,
),
)
self._insert_current_version(
connection,
anchored,
revision=revision,
changed_at=now,
)
else:
connection.execute(
"""
UPDATE evidence
SET workspace=COALESCE(workspace,?),
project=COALESCE(project,?),
updated_at=?
WHERE evidence_id=?
""",
(workspace, project, now, anchored),
)
connection.execute(
"""
UPDATE evidence
SET workspace = COALESCE(workspace, ?),
project = COALESCE(project, ?),
updated_at = ?
WHERE evidence_id = ?
""",
(workspace, project, now, anchored),
)
record = self._get_with_connection(connection, anchored)
if record is None:
raise RuntimeError("anchored evidence disappeared")
@@ -524,11 +485,7 @@ class EvidenceStore:
from one_memory import ingest_journal
ingest_journal.record(connection, anchored, cause="import")
return PutEvidenceResult(
record=record,
was_created=False,
was_revised=was_revised,
)
return PutEvidenceResult(record=record, was_created=False, was_revised=False)
insert = connection.execute(
"""
INSERT INTO evidence (
@@ -846,20 +803,27 @@ class EvidenceStore:
and same_scope[0]["idempotency_key"] != evidence.idempotency_key
):
return str(same_scope[0]["evidence_id"])
elif (
legacy_id := next(
(
anchored
for key in (
*evidence.written_by_derived_v3_idempotency_keys,
*evidence.older_written_by_idempotency_keys,
)
if (anchored := claim_legacy_key(key)) is not None
),
None,
)
) is not None:
elif (legacy_id := claim_legacy_key(evidence.legacy_derived_idempotency_key)) is not None:
return legacy_id
if not evidence.message_id and not evidence.correlation_id:
# BUG-03 moved the derived key off the capture time and onto the
# pre-cleaning text. A replay of a row stored under the old scheme
# has to be recognized as the same message.
replayed = connection.execute(
"""
SELECT evidence_id FROM evidence
WHERE idempotency_key = ? AND user_id = ? AND source_id = ?
AND conversation_id = ?
""",
(
evidence.prior_derived_idempotency_key,
evidence.user_id,
evidence.source_id,
evidence.conversation_id,
),
).fetchone()
if replayed is not None:
return str(replayed["evidence_id"])
if (
EvidenceOrigin.HOOK in evidence.origins
and evidence.message_id is None
@@ -874,7 +838,7 @@ class EvidenceStore:
ON hook.evidence_id=e.evidence_id AND hook.origin='hook'
WHERE e.user_id=? AND e.source_id=? AND e.conversation_id=?
AND (e.workspace IS ? OR e.workspace IS NULL OR ? IS NULL)
AND e.role=? AND e.content_hash=?
AND e.role=? AND e.written_by=? AND e.content_hash=?
AND e.message_id IS NOT NULL AND hook.evidence_id IS NULL
AND ABS((julianday(e.created_at)-julianday(?))*86400.0) <= 5.0
ORDER BY ABS((julianday(e.created_at)-julianday(?))*86400.0), e.evidence_id
@@ -887,6 +851,7 @@ class EvidenceStore:
evidence.workspace,
evidence.workspace,
evidence.role,
evidence.written_by,
evidence.content_hash,
evidence.created_at.isoformat(),
evidence.created_at.isoformat(),
@@ -907,7 +872,7 @@ class EvidenceStore:
AND (e.workspace IS ? OR e.workspace IS NULL OR ? IS NULL)
-- Unary plus keeps the coverage index from looking better than
-- the message index, which narrows to this exact text.
AND e.role = ? AND e.content_hash = ?
AND e.role = ? AND +e.written_by = ? AND e.content_hash = ?
AND e.message_id IS NULL
""",
(
@@ -917,6 +882,7 @@ class EvidenceStore:
evidence.workspace,
evidence.workspace,
evidence.role,
evidence.written_by,
evidence.content_hash,
),
).fetchall()
@@ -1281,7 +1247,7 @@ class EvidenceStore:
rows = connection.execute(
"""
SELECT evidence_id FROM evidence
WHERE user_id = 'local' AND role = 'user'
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
ORDER BY imported_at, evidence_id
LIMIT ?
""",
@@ -1335,7 +1301,7 @@ class EvidenceStore:
rows = connection.execute(
f"""
SELECT evidence_id FROM evidence
WHERE user_id = 'local' AND role = 'user'
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
AND ({predicates})
ORDER BY imported_at, evidence_id
LIMIT ?
@@ -1367,7 +1333,7 @@ class EvidenceStore:
rows = connection.execute(
f"""
SELECT evidence_id FROM evidence
WHERE user_id = 'local' AND role = 'user'
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
AND workspace IS NOT NULL
AND ({predicates})
ORDER BY imported_at, evidence_id
@@ -1510,7 +1476,7 @@ class EvidenceStore:
)
values: list[object] = [user_id, *(source_ids or ())]
visible_user = """
role='user' AND NOT (
role='user' AND written_by='person' AND NOT (
lower(ltrim(verbatim)) LIKE '<codex_internal_context%' OR
lower(ltrim(verbatim)) LIKE '<environment_context>%' OR
lower(ltrim(verbatim)) LIKE '<heartbeat>%' OR
@@ -1519,7 +1485,7 @@ class EvidenceStore:
)
"""
selected_visible_user = """
e.role='user' AND NOT (
e.role='user' AND e.written_by='person' AND NOT (
lower(ltrim(e.verbatim)) LIKE '<codex_internal_context%' OR
lower(ltrim(e.verbatim)) LIKE '<environment_context>%' OR
lower(ltrim(e.verbatim)) LIKE '<heartbeat>%' OR
@@ -1672,7 +1638,7 @@ class EvidenceStore:
if seed is None:
return ConversationTurnPage((), {}, 0, offset, limit)
visible_user = """
role='user' AND NOT (
role='user' AND written_by='person' AND NOT (
lower(ltrim(verbatim)) LIKE '<codex_internal_context%' OR
lower(ltrim(verbatim)) LIKE '<environment_context>%' OR
lower(ltrim(verbatim)) LIKE '<heartbeat>%' OR
@@ -1765,7 +1731,7 @@ class EvidenceStore:
AND first.conversation_id = evidence.conversation_id
AND first.workspace IS evidence.workspace
AND (
first.role = 'user'
(first.role = 'user' AND first.written_by = 'person')
OR first.content_kind = 'memory_content'
)
ORDER BY julianday(COALESCE(first.created_at, first.imported_at)),
@@ -2159,6 +2125,8 @@ class EvidenceStore:
(
existing["content_kind"] != evidence.content_kind,
existing["role"] != evidence.role,
evidence.written_by is not WrittenBy.UNKNOWN
and existing["written_by"] != evidence.written_by,
existing["content_hash"] != evidence.content_hash,
)
)
@@ -111,7 +111,7 @@ class ImportSupervisor:
)
)
diagnostic_event(
level="debug",
level="info",
component="import.supervisor",
event="import.queued",
outcome="queued",
@@ -198,7 +198,7 @@ class ImportSupervisor:
started = monotonic()
failure_type: str | None = None
diagnostic_event(
level="debug",
level="info",
component="import.supervisor",
event="import.started",
outcome="started",
@@ -316,10 +316,6 @@ class ImportSupervisor:
"source_id": current.source_id,
"mode": current.operation,
"count": current.stored,
"stored_count": current.stored,
"revised_count": current.revised,
"skipped_count": current.skipped,
"error_count": current.errors,
},
)
@@ -85,14 +85,15 @@ def _state(connection: sqlite3.Connection, evidence_id: str) -> dict[str, str |
value = evidence.model_dump(mode="json")
# The stored normalized content_hash is insufficient for exact-content leases.
content = _digest(value["verbatim"])
material = _digest({key: value[key] for key in ("verbatim", "role", "content_kind")})
material = _digest(
{key: value[key] for key in ("verbatim", "role", "written_by", "content_kind")}
)
origins = _digest(value["origin_meta"])
identity = _digest(
{
key: item
for key, item in value.items()
if key
not in {"verbatim", "written_by", "origin_meta", "imported_at", "updated_at"}
if key not in {"verbatim", "origin_meta", "imported_at", "updated_at"}
}
)
return dict(
@@ -3,7 +3,6 @@
import json
import time
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
from typing import TextIO, cast
@@ -105,13 +104,7 @@ def capture_hook(source_id: str, payload: object, database: Path) -> bool:
# Cursor also runs ~/.claude/settings.json hooks; its own hook already files the turn.
if source_id != "cursor" and "cursor_version" in payload:
return False
if not database.is_file():
return False
rejection = capture_rejection(database, source_id)
if rejection is not None:
# Silent, and the reason is counted rather than printed: a hook that
# spoke would put this into the host's turn as text.
record_capture_rejection(database, source_id, rejection)
if not database.is_file() or not _sync_open(database, source_id):
return False
if source_id == "pi" and _is_automated_session(first_string(payload.get("cwd"))):
return False
@@ -146,79 +139,17 @@ def capture_hook(source_id: str, payload: object, database: Path) -> bool:
def _sync_open(database: Path, source_id: str) -> bool:
return capture_rejection(database, source_id) is None
class CaptureRejection(StrEnum):
"""Why a hook's turn was dropped, or nothing at all.
Capture fails open on purpose: a hook that raises or prints would be read
back into the host's turn, so a refusal has to be silent and exit 0. It was
also completely invisible, which is why "持续同步开着但一条都没进" could not be
told apart from "capture 坏了". These values name the reason without any
payload, path, session or message — the only part that may be recorded.
"""
SYNC_DISABLED = "sync_disabled"
GRANT_MISSING = "grant_missing"
GRANT_REVOKED = "grant_revoked"
def capture_rejection(database: Path, source_id: str) -> CaptureRejection | None:
"""None when this turn may be captured, otherwise why it may not."""
from one_memory.memory_server.config_state import ConfigStore
if not ConfigStore(database.parent).sync_enabled(source_id):
return CaptureRejection.SYNC_DISABLED
return False
with main_database(database).read() as connection:
row = connection.execute(
"SELECT revoked_at FROM source_grants WHERE user_id='local' AND source_id=?",
"SELECT 1 FROM source_grants WHERE user_id='local' AND source_id=? "
"AND revoked_at IS NULL",
(source_id,),
).fetchone()
if row is None:
return CaptureRejection.GRANT_MISSING
# Indexed, not named: this connection has no row factory, and a hook is
# not the place to add one.
if row[0] is not None:
return CaptureRejection.GRANT_REVOKED
return None
def record_capture_rejection(
database: Path,
source_id: str,
reason: CaptureRejection,
*,
window_seconds: float = 600.0,
) -> None:
"""Note one dropped turn, at most once per window per source and reason.
Best effort by construction: a hook runs inside somebody's turn, so a sink
that is slow, full or read-only must never surface here. The record is a
count and nothing else — no payload, no path, no session or conversation
id — because this file is meant to be handed to support as is.
One small file per source and reason, rather than a line per turn: capture
runs on every prompt, and a log that grew with the conversation would be a
privacy problem wearing a diagnosis as a disguise.
"""
try:
directory = database.parent / "logs"
directory.mkdir(parents=True, exist_ok=True)
marker = directory / f"capture-rejection-{source_id}-{reason.value}.json"
now = time.time()
payload: dict[str, object] = {"source_id": source_id, "reason": reason.value}
if marker.exists() and now - marker.stat().st_mtime < window_seconds:
payload = json.loads(marker.read_text(encoding="utf8"))
previous = payload.get("count", 1)
payload["count"] = int(previous) + 1 if isinstance(previous, (str, int, float)) else 2
else:
payload["count"] = 1
payload["window_started_at"] = datetime.fromtimestamp(now, UTC).isoformat()
payload["last_at"] = payload["window_started_at"]
marker.write_text(json.dumps(payload, sort_keys=True), encoding="utf8")
except Exception:
return
return row is not None
def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed: bool) -> None:
@@ -236,7 +167,7 @@ def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed
):
if (
record in chunk
and any(item.role == "user" for item in chunk)
and any(item.role == "user" and item.written_by is WrittenBy.PERSON for item in chunk)
and any(item.role == "assistant" for item in chunk)
):
intake.submit_records(chunk, priority="live")
@@ -160,9 +160,7 @@ class MatterDiscovery:
task_by_id: dict[str, MemoryTask] = {}
for chunk in chunks:
for record in chunk:
task = await asyncio.to_thread(
processor.tasks.task_for_evidence, record.evidence_id
)
task = await asyncio.to_thread(processor.tasks.task_for_evidence, record.evidence_id)
if task is not None and task.status is MemoryTaskStatus.COMPLETED:
task_by_id[task.task_id] = task
tasks = tuple(sorted(task_by_id.values(), key=lambda task: (task.created_at, task.task_id)))
@@ -270,7 +270,7 @@ async def reconcile_current(
(
r
for r in records
if r.role == "user"
if r.written_by == "person"
and r.created_at
and (r.source_id, r.conversation_id) in related
),
@@ -51,19 +51,14 @@ def validate_claim(claim: Claim, records: Mapping[str, EvidenceRecord | SampleRe
text = record.text if isinstance(record, SampleRecord) else record.verbatim
if not citation.quote.strip() or citation.quote not in text:
raise ValueError("citation quote is not in the source")
author = {
"user": "person",
"assistant": "agent",
"tool": "tool",
}.get(record.role, "unknown")
authors.add(author)
authors.add(str(record.written_by))
created = record.created_at
citations.append(
citation.model_copy(
update={
"revision": record.revision,
"source_id": record.source_id,
"written_by": author,
"written_by": str(record.written_by),
"created_at": created.isoformat() if isinstance(created, datetime) else created,
}
)
@@ -290,7 +290,7 @@ def read_memory_library(
COALESCE(
MIN(
CASE WHEN e.role = 'user'
AND e.role = 'user'
AND e.written_by IN ('person', 'user')
THEN e.created_at END
),
MIN(e.created_at)
@@ -10,6 +10,7 @@ from fastapi.params import Depends
from pydantic import BaseModel, ConfigDict, Field, field_validator
from one_memory.database_access import main_database
from one_memory.hy_memory_bridge import HyMemoryBridgeError
from one_memory.imports.models import (
EvidenceInput,
EvidenceOrigin,
@@ -185,10 +186,6 @@ class MemoryNoteStore:
]
if task is None or task.status == MemoryTaskStatus.WAITING_FOR_MODEL:
state = "waiting_for_model"
elif task.status == MemoryTaskStatus.WAITING_FOR_PERMISSION:
# Upgrade compatibility: old rows are claimable and will leave
# this state as soon as the direct SDK runner claims them.
state = "waiting_for_permission"
elif task.status == MemoryTaskStatus.COMPLETED:
state = "ready" if memory_ids else "no_memories"
elif task.status == MemoryTaskStatus.RUNNING:
@@ -198,6 +195,22 @@ class MemoryNoteStore:
else:
state = "failed"
error = task.last_error_code if task else None
if self.processor.model_content_config is not None and state in {
"waiting_for_model",
"failed",
}:
try:
active = self.processor.profiles.active(self.processor.secrets)
if active is not None:
self.processor._guard_records((record,), active)
except HyMemoryBridgeError as failure:
if str(failure).startswith("model_content_"):
state = "waiting_for_permission"
error = str(failure)
else:
raise
except SecretStoreUnavailableError:
error = "secure_storage_unavailable"
return MemoryNoteReceipt(
evidence_id=record.evidence_id,
text=sanitize_model_input(record.verbatim),
@@ -261,8 +274,11 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
# unchanged; replaying the same request returns the original
# `entry:<evidence_id>` job instead of a second HY call.
if readmit:
# `/{evidence_id}/process` is the explicit retry signal.
# It also revives pre-upgrade permission waiters.
# `/{evidence_id}/process` is the explicit "the condition
# that parked it may have changed" signal: the person just
# granted consent or fixed the model. `submit_evidence`
# alone would return an existing parked row unchanged, so
# without this readmit the note stays parked forever.
await asyncio.to_thread(notes.intake.queue.readmit_parked)
submitted = await asyncio.to_thread(
notes.intake.submit_evidence,
@@ -280,6 +296,10 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
last_round=1,
priority="live",
)
except HyMemoryBridgeError as error:
if not str(error).startswith("model_content_"):
raise
return await asyncio.to_thread(notes.receipt, record.evidence_id)
except SecretStoreUnavailableError:
saved = await asyncio.to_thread(notes.receipt, record.evidence_id)
return saved.model_copy(
File diff suppressed because it is too large. Load diff
@@ -29,13 +29,8 @@ class ConfigDocument(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
schema_version: int = Field(default=1, ge=1, le=1)
model_content_allowed: StrictBool = False
sources: dict[str, SourceConfig] = Field(default_factory=dict)
#: Whether the configured model may turn imported conversations into
#: memories. Defaulting to on keeps every existing install doing what it
#: did before the switch existed; stopping it is the user's cost control
#: and leaves the model configuration itself untouched, so starting again
#: needs no re-entry of a key.
sedimentation_enabled: StrictBool = True
@field_validator("sources")
@classmethod
@@ -69,18 +64,9 @@ class ConfigStore:
source = self._document.sources.get(source_id)
return bool(source and source.sync_enabled)
def sedimentation_enabled(self) -> bool:
def model_content_allowed(self) -> bool:
with self._lock:
return bool(self._document.sedimentation_enabled)
def set_sedimentation_enabled(self, enabled: bool) -> ConfigDocument:
if type(enabled) is not bool:
raise ValueError("invalid_sedimentation_config")
with self._lock:
updated = self._document.model_copy(update={"sedimentation_enabled": enabled})
self._write(updated)
self._document = updated
return updated
return self._document.model_content_allowed
def set_sync(self, source_id: str, enabled: bool) -> ConfigDocument:
return self.set_sync_many({source_id: enabled})
@@ -100,17 +86,20 @@ class ConfigStore:
self._document = updated
return updated
def set_model_content_allowed(self, allowed: bool) -> ConfigDocument:
if type(allowed) is not bool:
raise ValueError("invalid_model_content_config")
with self._lock:
updated = self._document.model_copy(update={"model_content_allowed": allowed})
self._write(updated)
self._document = updated
return updated
def _read(self) -> ConfigDocument:
if not self.path.exists():
return ConfigDocument()
try:
value = json.loads(self.path.read_text(encoding="utf-8"))
if isinstance(value, dict):
# Builds through 1.0.2 persisted this global consent bit.
# The embedded SDK is now trusted to use all local memory, so
# accept existing files while discarding the retired setting.
value.pop("model_content_allowed", None)
return ConfigDocument.model_validate(value)
return ConfigDocument.model_validate_json(self.path.read_text(encoding="utf-8"))
except (OSError, ValueError) as error:
raise ConfigFileError("config_invalid") from error
@@ -23,13 +23,10 @@ import hashlib
import hmac
import os
import uuid
from datetime import datetime
from enum import StrEnum
from typing import Annotated, Final, Literal
from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator
from one_memory.imports.models import EvidenceRecord
from pydantic import BaseModel, ConfigDict, Field, field_validator
#: Bumped when a derivation rule changes, so old and new ids never compare equal.
SCHEMA_VERSION: Final[str] = "ms1"
@@ -169,6 +166,7 @@ class CanonicalMessage(BaseModel):
message_id: str | None = Field(default=None, min_length=1, max_length=512)
ordinal: int = Field(ge=0)
role: str = Field(min_length=1, max_length=128)
written_by: str = Field(min_length=1, max_length=64)
content: str = Field(min_length=1)
workspace: str | None = Field(default=None, max_length=1024)
@@ -215,13 +213,8 @@ class WriteRequest(BaseModel):
source_id: str | None = Field(default=None, min_length=1, max_length=128)
batch_identity: BatchIdentity
evidence_ids: list[str] = Field(min_length=1)
# New jobs freeze the complete input in their own payload. The default is
# solely for reading pre-upgrade rows whose payload only carried ids.
input_records: tuple[EvidenceRecord, ...] = ()
idempotency_key: str = Field(min_length=1, max_length=256)
priority: Literal["live", "normal"] = "normal"
event_at: datetime | None = None
time_quality: Literal["exact", "anchored", "session", "approximate"] = "approximate"
@field_validator("evidence_ids")
@classmethod
@@ -234,16 +227,6 @@ class WriteRequest(BaseModel):
raise ValueError("evidence_ids must be unique")
return value
@field_validator("input_records")
@classmethod
def _input_records_match_ids(
cls, value: tuple[EvidenceRecord, ...], info: ValidationInfo
) -> tuple[EvidenceRecord, ...]:
evidence_ids = info.data.get("evidence_ids")
if value and evidence_ids != [record.evidence_id for record in value]:
raise ValueError("input_records must match evidence_ids in order")
return value
@property
def request_key(self) -> str:
"""Stable dedup key. Same batch + same intent must not enqueue twice."""
@@ -30,7 +30,7 @@ from uuid import uuid4
from one_memory.background_work import in_the_background
from one_memory.database_access import main_database
from one_memory.diagnostic_log import diagnostic_event
from one_memory.imports.models import EvidenceOrigin, EvidenceRecord
from one_memory.imports.models import EvidenceRecord
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_server.contracts import WriteRequest
from one_memory.memory_server.data_safety import MIN_FREE_BYTES # noqa: F401 - re-exported
@@ -249,7 +249,6 @@ MODEL_NOT_READY_CODES = frozenset(
#: nothing was sent and retrying the same intent changes nothing — a terminal
#: `failed`, not `unknown` (which would wrongly imply a possibly-applied
#: write) and not a retry (which would fail identically).
#:
PRE_SEND_REFUSAL_CODES = frozenset(
{
"hy_memory_input_exceeds_window",
@@ -283,6 +282,11 @@ _VALID_JOB_STATES = frozenset(
#: A row in one of these states can never be moved back to a live one.
_TERMINAL_WRITE_STATES = frozenset({COMPLETED, FAILED, DEAD, CANCELLED})
#: A job in one of these states can never run again, so its payload is dead
#: weight. `unknown` is deliberately absent: it is non-terminal and still needs
#: its intent for reconciliation. `running` is obviously still live.
TERMINAL_PAYLOAD_STATES = (COMPLETED, FAILED, DEAD, CANCELLED)
#: Columns a current schema declares but an older `memory_write_jobs` may lack.
#: `CREATE TABLE IF NOT EXISTS` is a no-op for an existing table, so without
#: this an upgraded install would keep an older queue that every claim/lease
@@ -300,12 +304,9 @@ LEGACY_MEMORY_WRITE_JOB_COLUMNS = (
("lease_expires_at", "TEXT"),
("hy_invoked", "INTEGER NOT NULL DEFAULT 0"),
("last_error", "TEXT"),
("event_at", "TEXT"),
("time_quality", "TEXT NOT NULL DEFAULT 'approximate'"),
)
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)
EVENT_TIME_WINDOW = timedelta(minutes=10)
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL, UNKNOWN)
@dataclass(frozen=True)
@@ -392,42 +393,6 @@ def _matches_payload_except_priority(stored: str, payload: str) -> bool:
return old == new
def _matches_legacy_reference_replay(stored: str, payload: str) -> bool:
"""Match an old id-only caller to the same job's frozen payload."""
try:
old = json.loads(stored)
new = json.loads(payload)
except (TypeError, ValueError):
return False
if not isinstance(old, dict) or not isinstance(new, dict):
return False
old_inputs = old.get("input_records")
new_inputs = new.get("input_records")
if bool(old_inputs) == bool(new_inputs):
return False
old.pop("input_records", None)
new.pop("input_records", None)
return old == new
def _matches_reaped_legacy_payload(stored: str, payload: str) -> bool:
"""Compare a current frozen request with a pre-snapshot reaped payload."""
try:
old = json.loads(stored)
current = json.loads(payload)
except (TypeError, ValueError):
return False
if (
not isinstance(old, dict)
or not isinstance(current, dict)
or not isinstance(old.get(REAPED_PAYLOAD_MARKER), str)
):
return False
current.pop("input_records", None)
legacy_payload = json.dumps(current, sort_keys=True)
return bool(old[REAPED_PAYLOAD_MARKER] == _payload_digest(legacy_payload))
class DurableQueueSchemaUnavailableError(RuntimeError):
"""The durable queue schema has not been created yet.
@@ -480,7 +445,6 @@ class DurableWriteQueue:
if not initialize:
return
with main_database(database_path).write() as connection, connection:
connection.row_factory = sqlite3.Row
# First create the base table if absent. Then widen an older table
# before replaying indexes that reference the new columns; a bare
# CREATE INDEX would fail on a pre-priority queue.
@@ -491,7 +455,6 @@ class DurableWriteQueue:
for statement in memory_write_jobs_sqlite_schema()[1:]:
connection.execute(statement)
self._backfill_legacy_references(connection)
self._backfill_legacy_event_times(connection)
# A process that died while holding a lease cannot run its own
# recovery. Opening the queue is the first safe point to recover any
# rows it left behind, before any new worker claims them.
@@ -516,7 +479,8 @@ class DurableWriteQueue:
"UPDATE memory_write_jobs SET source_key=session_key WHERE source_key=''"
)
legacy_rows = connection.execute(
"SELECT job_id FROM memory_write_jobs WHERE queue_seq=0 ORDER BY created_at, rowid"
"SELECT job_id FROM memory_write_jobs WHERE queue_seq=0 "
"ORDER BY created_at, rowid"
).fetchall()
next_seq = int(
connection.execute(
@@ -550,49 +514,6 @@ class DurableWriteQueue:
if columns and columns != ["state", "priority", "created_at"]:
connection.execute("DROP INDEX idx_write_jobs_claim")
@staticmethod
def _backfill_legacy_event_times(connection: sqlite3.Connection) -> None:
if (
connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='evidence'"
).fetchone()
is None
):
connection.execute(
"UPDATE memory_write_jobs SET event_at=created_at WHERE event_at IS NULL"
)
return
evidence_columns = {
str(row[1]) for row in connection.execute("PRAGMA table_info(evidence)").fetchall()
}
if not {"created_at", "imported_at", "role"} <= evidence_columns:
connection.execute(
"UPDATE memory_write_jobs SET event_at=created_at WHERE event_at IS NULL"
)
return
connection.execute(
"""
UPDATE memory_write_jobs
SET event_at = COALESCE(
(
SELECT MAX(COALESCE(e.created_at, e.imported_at))
FROM memory_write_job_evidence AS linked
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
WHERE linked.job_id = memory_write_jobs.job_id
AND e.role = 'user'
),
(
SELECT MIN(COALESCE(e.created_at, e.imported_at))
FROM memory_write_job_evidence AS linked
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
WHERE linked.job_id = memory_write_jobs.job_id
),
created_at
)
WHERE event_at IS NULL
"""
)
@staticmethod
def _backfill_legacy_references(connection: sqlite3.Connection) -> None:
"""Populate the reference table after its schema exists."""
@@ -689,20 +610,8 @@ class DurableWriteQueue:
replacing the earlier intent.
"""
self._refuse_if_below_reserve()
request = self._freeze_request_inputs(request)
payload = json.dumps(
request.model_dump(
mode="json",
exclude={"event_at", "time_quality"},
),
sort_keys=True,
)
payload = json.dumps(request.model_dump(mode="json"), sort_keys=True)
now = datetime.now(UTC).isoformat()
resolved_event, resolved_quality = self._event_time_for_request(request)
event = request.event_at or resolved_event or datetime.fromisoformat(now)
if event.tzinfo is None:
event = event.replace(tzinfo=UTC)
event_at = event.astimezone(UTC).isoformat()
key = request.request_key
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
@@ -715,22 +624,16 @@ class DurableWriteQueue:
(key,),
).fetchone()
if existing is not None:
# A replay of work that already finished is a hit, not a
# conflict. Terminal rows store a reaped digest instead of the
# original payload, and the caller's key is derived from the
# evidence's revision, which can move on between replays while
# the payload's evidence ids stay the same. Refusing that pair
# turned every startup replay of one settled job into an
# exception: 236 of them in a single afternoon, all one key.
already_settled = (
str(existing["state"]) == COMPLETED and int(existing["hy_invoked"]) == 1
)
stored_payload = str(existing["payload_json"])
priority_replay = _matches_payload_except_priority(stored_payload, payload)
legacy_reaped_match = bool(
already_settled
and _matches_reaped_legacy_payload(stored_payload, payload)
)
if (
not _matches_stored_payload(stored_payload, payload)
and not priority_replay
and not _matches_legacy_reference_replay(stored_payload, payload)
and not legacy_reaped_match
):
raise ValueError(f"idempotency key {key} already exists with different content")
if already_settled:
return DurableWriteRecord(
job_id=str(existing["job_id"]),
@@ -741,6 +644,14 @@ class DurableWriteQueue:
updated_at=str(existing["updated_at"]),
detail="already completed; replay treated as a hit",
)
priority_replay = _matches_payload_except_priority(
str(existing["payload_json"]), payload
)
if (
not _matches_stored_payload(str(existing["payload_json"]), payload)
and not priority_replay
):
raise ValueError(f"idempotency key {key} already exists with different content")
if priority_replay and request.priority == "live":
connection.execute(
"UPDATE memory_write_jobs SET priority='live', updated_at=? "
@@ -800,13 +711,11 @@ class DurableWriteQueue:
)
connection.execute(
"INSERT INTO memory_write_jobs ("
" job_id, queue_seq, session_key, source_key, batch_id,"
" idempotency_key, payload_json,"
" job_id, queue_seq, session_key, source_key, batch_id, idempotency_key, payload_json,"
" evidence_ids_json,"
" state, priority, attempts, max_attempts, lease_owner,"
" lease_expires_at, hy_invoked, last_error, event_at, time_quality,"
" created_at, updated_at"
") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
" lease_expires_at, hy_invoked, last_error, created_at, updated_at"
") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
(
job_id,
queue_seq,
@@ -824,8 +733,6 @@ class DurableWriteQueue:
None,
0,
None,
event_at,
request.time_quality if request.event_at is not None else resolved_quality,
now,
now,
),
@@ -834,18 +741,6 @@ class DurableWriteQueue:
"INSERT INTO memory_write_job_evidence (job_id, evidence_id) VALUES (?,?)",
((job_id, str(evidence_id)) for evidence_id in request.evidence_ids),
)
diagnostic_event(
level="info",
component="memory.write_queue",
event="memory_write_job.queued",
outcome="queued",
job_id=job_id,
details={
"source_id": request.source_id or request.internal_session_id,
"mode": request.priority,
"count": len(request.evidence_ids),
},
)
return DurableWriteRecord(
job_id=job_id,
request_id=key,
@@ -855,45 +750,6 @@ class DurableWriteQueue:
updated_at=now,
)
def _freeze_request_inputs(self, request: WriteRequest) -> WriteRequest:
if request.input_records or self.evidence is None:
return request
records: list[EvidenceRecord] = []
for evidence_id in request.evidence_ids:
record = self.evidence.get(evidence_id)
if record is None:
return request
records.append(record)
return request.model_copy(update={"input_records": tuple(records)})
def _event_time_for_request(self, request: WriteRequest) -> tuple[datetime | None, str]:
records = list(request.input_records)
if not records and self.evidence is not None:
records = [
record
for evidence_id in request.evidence_ids
if (record := self.evidence.get(evidence_id)) is not None
]
if not records:
return None, "approximate"
opener = next(
(record for record in records if record.role == "user"),
records[0],
)
if opener.created_at is None:
return opener.imported_at, "approximate"
origin = opener.origin_meta.get(EvidenceOrigin.FILE)
basis = origin.raw_meta.get("time_basis") if origin is not None else None
turn_event_at = origin.raw_meta.get("turn_event_at") if origin is not None else None
if isinstance(turn_event_at, str):
try:
return datetime.fromisoformat(turn_event_at), "exact"
except ValueError:
pass
if basis == "file_mtime":
return opener.created_at, "session"
return opener.created_at, "exact"
def repair_pre_send_identity_failures(self, *, limit: int = 128) -> int:
"""Refresh stale Evidence journals and requeue only provably unsent jobs."""
if self.evidence is None:
@@ -1003,55 +859,11 @@ class DurableWriteQueue:
raise ReconcileSchemaUnavailableError(
"evidence table is unavailable; run migrations before reconcile"
)
has_versions = (
connection.execute(
"SELECT 1 FROM sqlite_master "
"WHERE type='table' AND name='evidence_versions'"
).fetchone()
is not None
where = (
"WHERE evidence.imported_at IS NOT NULL "
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence ref "
"WHERE ref.evidence_id=evidence.evidence_id)"
)
evidence_columns = {
str(row[1])
for row in connection.execute("PRAGMA table_info(evidence)").fetchall()
}
if {"revision", "content_hash", "updated_at"} <= evidence_columns:
revision_time = (
"COALESCE((SELECT version.changed_at FROM evidence_versions version "
"WHERE version.evidence_id=evidence.evidence_id "
"AND version.revision=evidence.revision),"
"evidence.updated_at,evidence.imported_at)"
if has_versions
else "COALESCE(evidence.updated_at,evidence.imported_at)"
)
where = (
"WHERE evidence.imported_at IS NOT NULL "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_job_evidence linked "
"JOIN memory_write_jobs job ON job.job_id=linked.job_id, "
"json_each(job.payload_json,'$.input_records') frozen "
"WHERE linked.evidence_id=evidence.evidence_id "
"AND json_extract(frozen.value,'$.evidence_id')=evidence.evidence_id "
"AND json_extract(frozen.value,'$.revision')=evidence.revision "
"AND json_extract(frozen.value,'$.content_hash')=evidence.content_hash"
") "
# Pre-snapshot builds only recorded an Evidence id. Treat
# that as coverage only when its Job was created after this
# exact revision.
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_job_evidence legacy "
"JOIN memory_write_jobs legacy_job ON legacy_job.job_id=legacy.job_id "
"WHERE legacy.evidence_id=evidence.evidence_id "
"AND COALESCE(json_array_length("
"json_extract(legacy_job.payload_json,'$.input_records')),0)=0 "
f"AND legacy_job.created_at>=({revision_time}))"
)
else:
# A genuinely old/minimal schema has no revision identity.
where = (
"WHERE evidence.imported_at IS NOT NULL "
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence legacy "
"WHERE legacy.evidence_id=evidence.evidence_id)"
)
params: tuple[Any, ...] = ()
cursor_clause = ""
if after_cursor is not None:
@@ -1283,7 +1095,6 @@ class DurableWriteQueue:
with main_database(self.database_path).write() as connection, connection:
updated = connection.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,"
"attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,"
"lease_expires_at=NULL,updated_at=? WHERE " + where,
(PENDING, now, *params),
)
@@ -1351,10 +1162,45 @@ class DurableWriteQueue:
return state
def clear_terminal_payloads(self, *, limit: int = 512) -> int:
"""Deprecated compatibility hook; durable inputs are immutable audit data."""
"""Replace terminal rows' request bodies with a content digest.
A completed/failed/dead/cancelled row can never run again, so the full
serialized `WriteRequest` is unbounded dead weight. A digest keeps the
idempotency contract intact: a replay of the same intent still resolves
to the same row, while the same key with different content still
conflicts. `unknown` is left alone because it is non-terminal and its
intent is needed for reconciliation.
Bounded by `limit` so a large backlog drains over several calls instead
of one long write on the single writer.
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
return 0
placeholders = ",".join("?" for _ in TERMINAL_PAYLOAD_STATES)
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
rows = connection.execute(
"SELECT job_id, payload_json FROM memory_write_jobs "
f"WHERE state IN ({placeholders}) "
"AND instr(payload_json, ?) = 0 "
"LIMIT ?",
(*TERMINAL_PAYLOAD_STATES, REAPED_PAYLOAD_MARKER, limit),
).fetchall()
if not rows:
return 0
for row in rows:
connection.execute(
"UPDATE memory_write_jobs SET payload_json=? "
f"WHERE job_id=? AND state IN ({placeholders})",
(
_reaped_payload(_payload_digest(str(row["payload_json"]))),
str(row["job_id"]),
*TERMINAL_PAYLOAD_STATES,
),
)
return len(rows)
def readmit_parked(self, *, states: tuple[str, ...] = PARKED_STATES) -> int:
"""Move parked jobs back to `pending` for a fresh claim.
@@ -1556,13 +1402,7 @@ class DurableWriteQueue:
cross back to pending, so this cannot duplicate a possibly-sent write.
"""
now = datetime.now(UTC).isoformat()
payload = json.dumps(
request.model_dump(
mode="json",
exclude={"event_at", "time_quality"},
),
sort_keys=True,
)
payload = json.dumps(request.model_dump(mode="json"), sort_keys=True)
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
@@ -1652,59 +1492,6 @@ class DurableWriteQueue:
)
return len(job_ids)
def retry_first_unknown_after_reconcile(
self,
*,
older_than: datetime,
limit: int = 32,
) -> int:
"""Retry one unresolved unknown outcome once, after reconciliation.
The caller first asks HY and checks linked governed output. Rows still
unknown may have lost the response after the SDK call, so one bounded
retry is useful. `attempts=1` is the guard: the retry claim increments
it to two, and a second unknown is left for explicit user recovery.
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
now = datetime.now(UTC).isoformat()
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
rows = connection.execute(
"SELECT job_id FROM memory_write_jobs "
"WHERE state=? AND attempts=1 AND attempts < max_attempts "
"AND updated_at <= ? ORDER BY updated_at, job_id LIMIT ?",
(UNKNOWN, older_than.isoformat(), limit),
).fetchall()
job_ids = tuple(str(row["job_id"]) for row in rows)
if not job_ids:
return 0
placeholders = ",".join("?" for _ in job_ids)
connection.execute(
f"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
f"lease_expires_at=NULL, hy_invoked=0, "
f"last_error='model_content_unknown_auto_retry', updated_at=? "
f"WHERE state=? AND attempts=1 AND job_id IN ({placeholders})",
(PENDING, now, UNKNOWN, *job_ids),
)
if connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_tasks'"
).fetchone():
connection.execute(
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
f"last_error_code='model_content_unknown_auto_retry', updated_at=? "
f"WHERE status=? AND task_id IN ("
f"SELECT inputs.task_id FROM memory_task_inputs AS inputs "
f"JOIN memory_write_job_evidence AS linked ON EXISTS ("
f"SELECT 1 FROM json_each(inputs.refs_json) AS ref "
f"WHERE json_extract(ref.value, '$.evidence_id')=linked.evidence_id) "
f"WHERE linked.job_id IN ({placeholders}))",
(MemoryTaskStatus.PENDING, now, MemoryTaskStatus.UNKNOWN, *job_ids),
)
return len(job_ids)
def resolve_unknown_from_linked_output(self, job_id: str) -> bool:
"""Close an unknown durable row when local governance output proves success."""
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
@@ -1901,12 +1688,9 @@ class DurableWriteRegistry:
def evidence_for_request(store: EvidenceStore, request: WriteRequest) -> tuple[EvidenceRecord, ...]:
"""The Evidence a stored write points at, in the request's own order.
Current jobs are self-contained: their immutable records are the model
input even if a Harness later rewinds or edits its live transcript. The
store lookup remains only for payloads written by pre-upgrade builds.
A write may only point at Evidence that still exists; a missing id is a
rejection rather than a silent partial send.
"""
if request.input_records:
return request.input_records
records = []
for evidence_id in request.evidence_ids:
record = store.get(evidence_id)
@@ -2013,14 +1797,7 @@ async def execute_job(
# Never persist the raw message: it may be a provider response or the
# user's own text, and `last_error` is shown over HTTP as `detail`.
code = safe_error_code(error)
if (
code
in {
"model_content_source_disabled",
"model_sedimentation_disabled",
}
and not hy_invoked
):
if code == "model_content_source_disabled" and not hy_invoked:
await in_the_background(queue.release_unstarted, job_id, owner=owner)
return
# The task layer classifies these before any bridge is opened, so HY
@@ -2031,11 +1808,7 @@ async def execute_job(
# - transient model/provider trouble -> waiting_for_model (parked).
# Everything else may have crossed into HY, and HY has no idempotency
# key, so it is `unknown` rather than a blind retry.
if code in PRE_SEND_REFUSAL_CODES:
# The frozen input window is checked before the payload reaches
# HY. This one refusal therefore has a proven pre-send boundary.
state, invoked_marker = FAILED, False
elif hy_invoked:
if hy_invoked:
# Error names do not prove execution phase: permission and
# evidence rechecks also run after HY returns. Once the durable
# marker is set, every failure is possibly-sent and must stay out
@@ -2047,6 +1820,10 @@ async def execute_job(
state, invoked_marker = WAITING_FOR_PERMISSION, False
elif code in MODEL_NOT_READY_CODES:
state, invoked_marker = WAITING_FOR_MODEL, False
elif code in PRE_SEND_REFUSAL_CODES:
# Provably before the bridge: the frozen input is stale or too
# large. Nothing was sent, so this is terminal rather than unknown.
state, invoked_marker = FAILED, False
elif code == "model_content_transport_request_failed":
# The generation or embedding HTTP exchange can fail after HY has
# already persisted a node. Treating that as a pre-send failure
@@ -2119,13 +1896,6 @@ class DurableWriteDrain:
#: Reaping is a write; it runs at most once per this interval on the
#: idle path so an empty queue does not issue one write per poll.
self._reap_interval = reap_interval
# Unknown recovery is user-visible and lightweight enough to check
# sooner than general database housekeeping. Tests that set the common
# interval to zero keep immediate maintenance.
self._unknown_reconcile_interval = min(
reap_interval,
timedelta(seconds=10),
)
#: The maintenance writes below are recoverability/retention work, not
#: idle housekeeping, so each has its own interval and runs at the start
#: of every drain tick. Sharing one timestamp would let a busy tick
@@ -2135,6 +1905,7 @@ class DurableWriteDrain:
#: sustained backlog must not keep finished rows' request bodies in the
#: business database indefinitely. Its own gate so neither interval can
#: starve the other.
self._last_terminal_reap: datetime | None = None
#: `waiting_for_model` re-admission is also time-based recoverability:
#: a model outage clears on its own, and a queue that never goes idle
#: must still re-admit the write once the wait has elapsed.
@@ -2185,7 +1956,7 @@ class DurableWriteDrain:
now: datetime | None = None,
budget_ms: int | None = None,
) -> tuple[str, ...]:
"""Claim event-time heads, with one running job per enabled Harness."""
"""Claim at most one FIFO head per enabled Harness."""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
now = now or datetime.now(UTC)
@@ -2227,7 +1998,9 @@ class DurableWriteDrain:
(PENDING,),
).fetchall()
enabled_sources = {
str(row[0]) for row in source_rows if self.source_writes_permitted(str(row[0]))
str(row[0])
for row in source_rows
if self.source_writes_permitted(str(row[0]))
}
if not enabled_sources:
return ()
@@ -2246,142 +2019,38 @@ class DurableWriteDrain:
placeholders = ",".join("?" for _ in enabled_sources)
source_clause = f"AND candidate.source_key IN ({placeholders}) "
source_params = tuple(sorted(enabled_sources))
window = connection.execute(
"SELECT window_anchor,window_end FROM memory_write_schedule_state WHERE singleton=1"
).fetchone()
if window is not None:
anchor_text, end_text = str(window[0]), str(window[1])
active_source_clause = ""
active_source_params: tuple[Any, ...] = ()
if enabled_sources is not None:
placeholders = ",".join("?" for _ in enabled_sources)
active_source_clause = f"AND active.source_key IN ({placeholders}) "
active_source_params = tuple(sorted(enabled_sources))
# The contract retires a window when every *eligible*
# pending/running job in it has settled or parked. A pending row
# that scheduling already holds back (an unexpired lease, a
# busy Harness, or an earlier event in its Session) is not eligible, so it
# must not keep the anchor alive: those blockers usually sit
# outside this ten-minute range and can never be claimed from
# inside it, which pinned the anchor and starved the whole
# queue. Reuse the claim's own predicate so "the window is
# busy" and "the window can still be worked" agree.
active = connection.execute(
"SELECT 1 FROM memory_write_jobs AS active "
"WHERE active.event_at>=? AND active.event_at<? AND ("
"active.state=? OR (active.state=? "
"AND (active.lease_expires_at IS NULL "
"OR active.lease_expires_at < ?) " + active_source_clause + "AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=active.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL "
"AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=active.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<active.event_at OR "
"(earlier.event_at=active.event_at "
"AND earlier.queue_seq<active.queue_seq))) "
")) LIMIT 1",
(
anchor_text,
end_text,
RUNNING,
PENDING,
now.isoformat(),
*active_source_params,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchone()
if active is None:
connection.execute("DELETE FROM memory_write_schedule_state WHERE singleton=1")
window = None
if window is None:
anchor = connection.execute(
"SELECT candidate.event_at "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL "
"OR candidate.lease_expires_at < ?) " + source_clause + "AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL "
"AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.event_at,candidate.queue_seq LIMIT 1",
(
PENDING,
now.isoformat(),
*source_params,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchone()
if anchor is None:
return ()
anchor_at = datetime.fromisoformat(str(anchor[0]))
if anchor_at.tzinfo is None:
anchor_at = anchor_at.replace(tzinfo=UTC)
anchor_text = anchor_at.astimezone(UTC).isoformat()
end_text = (anchor_at + EVENT_TIME_WINDOW).astimezone(UTC).isoformat()
connection.execute(
"INSERT INTO memory_write_schedule_state "
"(singleton,window_anchor,window_end,updated_at) "
"VALUES (1,?,?,?)",
(anchor_text, end_text, now.isoformat()),
)
rows = connection.execute(
"SELECT candidate.job_id,candidate.source_key "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ "AND candidate.event_at>=? AND candidate.event_at<? "
+ source_clause +
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
"WHERE earlier.source_key=candidate.source_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.event_at,candidate.queue_seq",
"AND earlier.queue_seq<candidate.queue_seq) "
"ORDER BY candidate.queue_seq",
(
PENDING,
now.isoformat(),
*source_params,
anchor_text,
end_text,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchall()
selected: list[str] = []
selected_sources: set[str] = set()
for row in rows:
source_key = str(row["source_key"])
if source_key in selected_sources:
continue
if self.source_claim_enabled is not None and not self.source_claim_enabled(
source_key
str(row["source_key"])
):
continue
selected.append(str(row["job_id"]))
selected_sources.add(source_key)
if len(selected) == min(limit, self.max_concurrent_sources):
break
if not selected:
@@ -2610,6 +2279,19 @@ class DurableWriteDrain:
self.queue.reclaim_expired,
operation_name="reclaim_expired",
)
# Terminal-payload reaping is a bounded retention write: a
# finished row's full request body must not sit in the business
# database forever just because the queue never went idle. Its
# own interval keeps it off every poll.
if (
self._last_terminal_reap is None
or tick_started - self._last_terminal_reap >= self._reap_interval
):
self._last_terminal_reap = tick_started
await self._run_idle_maintenance(
self.queue.clear_terminal_payloads,
operation_name="clear_terminal_payloads",
)
# A model outage clears on its own, so `waiting_for_model` jobs
# are re-admitted on a timer. That must not depend on idleness: a
# queue busy for longer than the wait window would otherwise keep
@@ -2626,8 +2308,7 @@ class DurableWriteDrain:
)
if self.reconcile_unknown is not None and (
self._last_unknown_reconcile is None
or tick_started - self._last_unknown_reconcile
>= self._unknown_reconcile_interval
or tick_started - self._last_unknown_reconcile >= self._reap_interval
):
self._last_unknown_reconcile = tick_started
try:
@@ -3,7 +3,7 @@
The result decides whether the `unknown` job state can ever be retired, so it
is probed against the installed SDK rather than written down from memory.
What was verified against `hy-memory==2.0.4`:
What was verified against `hy-memory==2.0.2`:
- `HyMemoryClient.add(..., request_id=...)` accepts a caller-supplied id and
echoes it back. It is described as a request/receipt handle, and the SDK
@@ -151,8 +151,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
lease_expires_at TEXT,
hy_invoked INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
event_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
time_quality TEXT NOT NULL DEFAULT 'approximate',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
@@ -166,8 +164,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
ON memory_write_jobs(source_key, state, queue_seq);
CREATE INDEX IF NOT EXISTS idx_write_jobs_fifo
ON memory_write_jobs(queue_seq);
CREATE INDEX IF NOT EXISTS idx_write_jobs_event_claim
ON memory_write_jobs(state, event_at, queue_seq);
CREATE INDEX IF NOT EXISTS idx_write_jobs_source_recent
ON memory_write_jobs(source_key, updated_at DESC);
CREATE TABLE IF NOT EXISTS memory_write_job_evidence (
@@ -176,13 +172,7 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
PRIMARY KEY (job_id, evidence_id)
);
CREATE INDEX IF NOT EXISTS idx_write_job_evidence_evidence
ON memory_write_job_evidence(evidence_id);
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
window_anchor TEXT NOT NULL,
window_end TEXT NOT NULL,
updated_at TEXT NOT NULL
)
ON memory_write_job_evidence(evidence_id)
"""
return """
CREATE TABLE IF NOT EXISTS memory_write_jobs (
@@ -202,8 +192,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
lease_expires_at DATETIME(6) NULL,
hy_invoked TINYINT(1) NOT NULL DEFAULT 0,
last_error TEXT NULL,
event_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
time_quality VARCHAR(32) NOT NULL DEFAULT 'approximate',
created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6)
ON UPDATE CURRENT_TIMESTAMP(6),
@@ -213,7 +201,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
KEY idx_write_jobs_source_claim (state, source_key, priority, created_at),
KEY idx_write_jobs_source_fifo (source_key, state, queue_seq),
KEY idx_write_jobs_fifo (queue_seq),
KEY idx_write_jobs_event_claim (state, event_at, queue_seq),
KEY idx_write_jobs_source_recent (source_key, updated_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
@@ -222,13 +209,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
evidence_id VARCHAR(128) NOT NULL,
PRIMARY KEY (job_id, evidence_id),
KEY idx_write_job_evidence_evidence (evidence_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
singleton INT PRIMARY KEY CHECK (singleton = 1),
window_anchor DATETIME(6) NOT NULL,
window_end DATETIME(6) NOT NULL,
updated_at DATETIME(6) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
@@ -1251,68 +1231,6 @@ CREATE TABLE IF NOT EXISTS model_request_budget_events (
"""
def model_request_audit_ddl(dialect: str = "mysql") -> str:
"""Content-free accounting for physical precipitation model requests."""
if dialect == "sqlite":
return """
CREATE TABLE IF NOT EXISTS model_request_audit (
audit_id TEXT PRIMARY KEY,
request_id TEXT NOT NULL,
profile_id TEXT NOT NULL,
profile_revision INTEGER NOT NULL,
provider_kind TEXT NOT NULL,
model TEXT NOT NULL,
request_kind TEXT NOT NULL,
prompt_stage TEXT NOT NULL DEFAULT 'other',
started_at TEXT NOT NULL,
completed_at TEXT,
outcome TEXT NOT NULL CHECK(outcome IN ('in_flight','succeeded','failed')),
http_status INTEGER,
error_type TEXT,
provider_request_id TEXT,
prompt_tokens INTEGER,
completion_tokens INTEGER,
total_tokens INTEGER,
cached_prompt_tokens INTEGER,
reasoning_tokens INTEGER
);
CREATE INDEX IF NOT EXISTS model_request_audit_request
ON model_request_audit(request_id, started_at);
CREATE INDEX IF NOT EXISTS model_request_audit_started
ON model_request_audit(started_at);
"""
return """
CREATE TABLE IF NOT EXISTS model_request_audit (
audit_id VARCHAR(128) PRIMARY KEY,
request_id VARCHAR(512) NOT NULL,
profile_id VARCHAR(128) NOT NULL,
profile_revision INT NOT NULL,
provider_kind VARCHAR(64) NOT NULL,
model VARCHAR(512) NOT NULL,
request_kind VARCHAR(64) NOT NULL,
prompt_stage VARCHAR(128) NOT NULL DEFAULT 'other',
started_at DATETIME(6) NOT NULL,
completed_at DATETIME(6) NULL,
outcome VARCHAR(32) NOT NULL CHECK(outcome IN ('in_flight','succeeded','failed')),
http_status INT NULL,
error_type VARCHAR(128) NULL,
provider_request_id VARCHAR(512) NULL,
prompt_tokens BIGINT NULL,
completion_tokens BIGINT NULL,
total_tokens BIGINT NULL,
cached_prompt_tokens BIGINT NULL,
reasoning_tokens BIGINT NULL,
KEY model_request_audit_request (request_id, started_at),
KEY model_request_audit_started (started_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
async def _052_model_request_audit(cursor: Any) -> None:
for statement in _split_statements(model_request_audit_ddl(_dialect(cursor))):
await cursor.execute(statement)
def model_probe_operations_ddl(dialect: str = "mysql") -> str:
"""Probe operations; `state` is CHECKed to a known set.
@@ -1486,105 +1404,7 @@ async def _024_matter_model_cache(cursor: Any) -> None:
async def _026_hy_node_permissions(cursor: Any) -> None:
# Historical migration retained as a ledger slot. Fresh databases no
# longer create this retired proof table; migration 51 removes it from
# databases that ran the original v26 body.
return None
async def _051_drop_hy_node_permissions(cursor: Any) -> None:
await cursor.execute("DROP TABLE IF EXISTS hy_node_permissions")
async def _053_memory_write_event_time(cursor: Any) -> None:
"""Add event-time scheduling without re-keying any existing write."""
dialect = _dialect(cursor)
columns = await cursor.columns("memory_write_jobs")
added_event_at = "event_at" not in columns
if added_event_at:
await cursor.execute(
"ALTER TABLE memory_write_jobs ADD COLUMN event_at "
+ ("TEXT" if dialect == "sqlite" else "DATETIME(6) NULL")
)
if "time_quality" not in columns:
await cursor.execute(
"ALTER TABLE memory_write_jobs ADD COLUMN time_quality "
+ (
"TEXT NOT NULL DEFAULT 'approximate'"
if dialect == "sqlite"
else "VARCHAR(32) NOT NULL DEFAULT 'approximate'"
)
)
await cursor.execute(
"""
UPDATE memory_write_jobs
SET event_at = COALESCE(
(
SELECT MAX(COALESCE(e.created_at, e.imported_at))
FROM memory_write_job_evidence AS linked
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
WHERE linked.job_id = memory_write_jobs.job_id
AND e.role = 'user'
),
(
SELECT MIN(COALESCE(e.created_at, e.imported_at))
FROM memory_write_job_evidence AS linked
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
WHERE linked.job_id = memory_write_jobs.job_id
),
created_at
)
WHERE event_at IS NULL
"""
)
if dialect == "sqlite":
await cursor.execute(
"CREATE INDEX IF NOT EXISTS idx_write_jobs_event_claim "
"ON memory_write_jobs(state, event_at, queue_seq)"
)
await cursor.execute(
"""
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
window_anchor TEXT NOT NULL,
window_end TEXT NOT NULL,
updated_at TEXT NOT NULL
)
"""
)
else:
if added_event_at:
await cursor.execute(
"CREATE INDEX idx_write_jobs_event_claim "
"ON memory_write_jobs(state, event_at, queue_seq)"
)
await cursor.execute(
"""
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
singleton INT PRIMARY KEY CHECK (singleton = 1),
window_anchor DATETIME(6) NOT NULL,
window_end DATETIME(6) NOT NULL,
updated_at DATETIME(6) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
async def _054_legacy_possibly_sent_tasks(cursor: Any) -> None:
"""Preserve the old sent marker before removing its retired permission table."""
columns = await cursor.columns("memory_task_continuous_permissions")
if "task_id" not in columns:
return
await cursor.execute(
"""
UPDATE memory_tasks
SET hy_invocation_count=CASE
WHEN hy_invocation_count<1 THEN 1 ELSE hy_invocation_count END
WHERE task_id IN (SELECT task_id FROM memory_task_continuous_permissions)
"""
)
await cursor.execute("DROP TABLE memory_task_continuous_permissions")
await cursor.execute(hy_node_permissions_ddl(_dialect(cursor)))
def model_profiles_ddl(dialect: str = "mysql") -> str:
@@ -2853,11 +2673,7 @@ MIGRATIONS: tuple[Migration, ...] = (
2,
"memory_write_jobs",
_002_write_jobs,
(
"memory_write_jobs",
"memory_write_job_evidence",
"memory_write_schedule_state",
),
("memory_write_jobs", "memory_write_job_evidence"),
),
Migration(3, "source_file_cursors", _003_scanner_cursors, ("source_file_cursors",)),
Migration(4, "change_outbox", _004_change_outbox, ("change_outbox",)),
@@ -2940,7 +2756,7 @@ MIGRATIONS: tuple[Migration, ...] = (
),
Migration(23, "source_candidates", _023_source_candidates, ("source_candidates",)),
Migration(24, "matter_model_cache", _024_matter_model_cache, ("matter_model_cache",)),
Migration(26, "hy_node_permissions", _026_hy_node_permissions),
Migration(26, "hy_node_permissions", _026_hy_node_permissions, ("hy_node_permissions",)),
Migration(
27,
"model_profiles",
@@ -3025,22 +2841,4 @@ MIGRATIONS: tuple[Migration, ...] = (
"model_probe_operations_reserved_at",
_050_model_probe_operations_reserved_at,
),
Migration(51, "drop_hy_node_permissions", _051_drop_hy_node_permissions),
Migration(
52,
"model_request_audit",
_052_model_request_audit,
("model_request_audit",),
),
Migration(
53,
"memory_write_event_time",
_053_memory_write_event_time,
("memory_write_schedule_state",),
),
Migration(
54,
"legacy_possibly_sent_tasks",
_054_legacy_possibly_sent_tasks,
),
)
@@ -107,8 +107,6 @@ def build_request(
first_round: int,
last_round: int,
evidence_ids: list[str],
input_records: tuple[EvidenceRecord, ...] = (),
source_id: str | None = None,
priority: Literal["live", "normal"] = "normal",
idempotency_key: str | None = None,
) -> WriteRequest:
@@ -127,8 +125,6 @@ def build_request(
internal_session_id=internal_session_id,
batch_identity=batch,
evidence_ids=evidence_ids,
input_records=input_records,
source_id=source_id,
idempotency_key=idempotency_key or batch.batch_id,
priority=priority,
)
@@ -275,8 +271,6 @@ def build_request_from_evidence(
first_round=first_round,
last_round=last_round,
evidence_ids=list(ordered),
input_records=tuple(ordered.values()),
source_id=seed.source_id,
priority=priority,
idempotency_key=idempotency_key,
)
@@ -18,11 +18,10 @@ from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass
from datetime import datetime
from hashlib import sha256
from typing import Literal
from one_memory.imports.models import EvidenceOrigin, EvidenceRecord
from one_memory.imports.models import EvidenceRecord
from one_memory.memory_server.contracts import (
BatchIdentity,
WriteRequest,
@@ -125,12 +124,6 @@ class UnifiedWriteIntake:
priority: WritePriority,
) -> WriteRequest:
session_id = internal_session_id_for(record.conversation_id, record.source_id)
event_at, time_quality = _job_time((record,))
request_key = (
f"entry:{record.evidence_id}"
if record.revision == 1
else (f"entry:v2:{record.evidence_id}:{record.revision}:{record.content_hash}")
)
return WriteRequest(
internal_session_id=session_id,
source_id=record.source_id,
@@ -140,11 +133,8 @@ class UnifiedWriteIntake:
last_round=last_round,
),
evidence_ids=[record.evidence_id],
input_records=(record,),
idempotency_key=request_key,
idempotency_key=f"entry:{record.evidence_id}",
priority=priority,
event_at=event_at,
time_quality=time_quality,
)
def submit_records(
@@ -171,7 +161,6 @@ class UnifiedWriteIntake:
)
request_id = f"turn:v1:{sha256(material.encode()).hexdigest()}"
session_id = internal_session_id_for(first.conversation_id, first.source_id)
event_at, time_quality = _job_time(records)
durable = self.queue.submit(
WriteRequest(
internal_session_id=session_id,
@@ -182,11 +171,8 @@ class UnifiedWriteIntake:
last_round=1,
),
evidence_ids=[record.evidence_id for record in records],
input_records=records,
idempotency_key=request_id,
priority=priority,
event_at=event_at,
time_quality=time_quality,
)
)
return UnifiedWriteReceipt(
@@ -203,29 +189,3 @@ class SourceWriteDisabledError(RuntimeError):
def __init__(self, evidence_id: str) -> None:
super().__init__("model_content_source_disabled")
self.evidence_id = evidence_id
def _job_time(
records: tuple[EvidenceRecord, ...],
) -> tuple[datetime, Literal["exact", "anchored", "session", "approximate"]]:
"""Use the turn opener's wall clock, with an explicit fallback quality."""
opener = next(
(record for record in records if record.role == "user"),
records[0],
)
if opener.created_at is None:
return opener.imported_at, "approximate"
file_origin = opener.origin_meta.get(EvidenceOrigin.FILE)
basis = file_origin.raw_meta.get("time_basis") if file_origin is not None else None
turn_event_at = file_origin.raw_meta.get("turn_event_at") if file_origin is not None else None
if isinstance(turn_event_at, str):
try:
return datetime.fromisoformat(turn_event_at), "exact"
except ValueError:
pass
if basis == "file_mtime":
return opener.created_at, "session"
if EvidenceOrigin.HOOK in opener.origins:
return opener.created_at, "exact"
return opener.created_at, "exact"
+13 -117
View File
@@ -17,7 +17,7 @@ from pydantic import BaseModel, ConfigDict, Field
from one_memory.database import take_the_write_lock_first
from one_memory.database_access import main_database
from one_memory.imports.models import EvidenceRecord, WrittenBy
from one_memory.imports.models import EvidenceRecord
from one_memory.memory_server.migrations import memory_tasks_ddl
from one_memory.model_profiles import ModelProfile, ModelProfileSnapshot
@@ -174,60 +174,29 @@ def evidence_ref(record: EvidenceRecord) -> TaskEvidenceRef:
"segment_end": int(marker["end"]),
"segment_hash": str(marker["hash"]),
}
identity = record.model_dump(mode="json", exclude={"origin_meta", "imported_at", "updated_at"})
return {
"evidence_id": record.evidence_id,
"revision": record.revision,
"content_hash": record.content_hash,
"identity_hash": _evidence_identity_hash(record, include_written_by=False),
"identity_hash": sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest(),
}
def _evidence_identity_hash(record: EvidenceRecord, *, include_written_by: bool) -> str:
excluded = {"origin_meta", "imported_at", "updated_at"}
if not include_written_by:
excluded.add("written_by")
identity = record.model_dump(mode="json", exclude=excluded)
return sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest()
def _identity_hash_matches(record: EvidenceRecord, expected: object) -> bool:
candidates = {_evidence_identity_hash(record, include_written_by=False)}
# Tasks created by pre-role-only builds keep their original proof. The
# legacy metadata could also have changed in place without a revision, so
# accept the old hash under every historical author guess.
for written_by in WrittenBy:
candidates.add(
_evidence_identity_hash(
record.model_copy(update={"written_by": written_by}),
include_written_by=True,
)
)
return expected in candidates
def matches_evidence_ref(record: EvidenceRecord, ref: TaskEvidenceRef) -> bool:
if "segment_hash" in ref:
base = evidence_ref(record)
if any(
base.get(key) != ref.get(key)
for key in ("evidence_id", "revision", "content_hash")
for key in ("evidence_id", "revision", "content_hash", "identity_hash")
):
return False
if not _identity_hash_matches(record, ref.get("identity_hash")):
return False
start, end = ref.get("segment_start"), ref.get("segment_end")
if not isinstance(start, int) or not isinstance(end, int) or not (0 <= start < end):
return False
return sha256(record.verbatim[start:end].encode()).hexdigest() == ref["segment_hash"]
current = evidence_ref(record)
return all(
(
_identity_hash_matches(record, value)
if key == "identity_hash"
else current.get(key) == value
)
for key, value in ref.items()
)
return all(current.get(key) == value for key, value in ref.items())
def segment_evidence_record(
@@ -298,13 +267,15 @@ class MemoryTaskOutput(BaseModel):
MEMORY_TASK_SCHEMA = memory_tasks_ddl("sqlite")
#: Legacy transport refusals raised before the physical send, so provably
#: nothing reached HY: nothing can have been written and retrying cannot
#: duplicate. Kept to recover rows written by builds through 1.0.2.
#: Transport refusals raised before the physical send, so provably nothing
#: reached HY: nothing can have been written and retrying cannot duplicate.
#:
#: The retired request hook denied every name below before opening a socket.
#: This stays explicit: other old transport codes may have happened after HY
#: persisted a node and must not be blindly requeued.
#: Every name here is denied by `HyPermissionTransport._check_request` (the
#: httpx request hook) or by `_PhysicalTransport.handle_async_request` before
#: it counts the request — both run inside httpx, before the socket. Explicit
#: on purpose (D88): the transport also denies `budget_exceeded`, `redirect`,
#: `lease_*` and `request_failed`, all of which can arrive after HY persisted
#: a node, so a prefix rule would requeue work that may already exist.
_PRE_SEND_REFUSALS = (
"model_content_transport_headers_mismatch",
"model_content_transport_body_mismatch",
@@ -1008,44 +979,6 @@ class MemoryTaskStore:
error_code=error_code,
)
def release_unstarted(
self,
task_id: str,
*,
runner_token: str,
error_code: str,
) -> MemoryTask:
"""Return a claimed task to pending when a user pause wins before HY.
Claim increments ``attempts``. A pause is not an execution failure, so
the attempt is returned as well; repeatedly pausing and resuming can
never exhaust an otherwise healthy task.
"""
now = datetime.now(UTC).isoformat()
with self._connect(writing=True) as connection:
changed = connection.execute(
"""
UPDATE memory_tasks
SET status=?, runner_token=NULL,
attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,
last_error_code=?, duration_seconds=NULL, updated_at=?
WHERE task_id=? AND status=? AND runner_token=?
""",
(
MemoryTaskStatus.PENDING,
error_code,
now,
task_id,
MemoryTaskStatus.RUNNING,
runner_token,
),
)
if changed.rowcount != 1:
raise MemoryTaskOwnershipError(
"Memory task runner no longer owns this task"
)
return self.require(task_id)
def park_for_permission(self, task_id: str, *, error_code: str) -> bool:
"""Park unstarted work until a later permission snapshot admits it."""
now = datetime.now(UTC).isoformat()
@@ -2230,43 +2163,6 @@ class MemoryTaskStore:
)
return changed.rowcount
def retry_first_unknown_after_reconcile(
self,
*,
older_than: datetime,
limit: int = 128,
) -> int:
"""Automatically retry a task-only unknown exactly once.
Reconciliation runs before this transition. Attempts are preserved, so
the retry claim moves `attempts=1` to two and another unknown is left
for explicit user recovery.
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
now = datetime.now(UTC).isoformat()
with self._connect(writing=True) as connection:
changed = connection.execute(
"""
UPDATE memory_tasks
SET status=?, runner_token=NULL,
last_error_code='model_content_unknown_auto_retry', updated_at=?
WHERE task_id IN (
SELECT task_id FROM memory_tasks
WHERE status=? AND attempts=1 AND updated_at <= ?
ORDER BY updated_at, task_id LIMIT ?
)
""",
(
MemoryTaskStatus.PENDING,
now,
MemoryTaskStatus.UNKNOWN,
older_than.isoformat(),
limit,
),
)
return changed.rowcount
def recover_running(self) -> int:
"""Return tasks orphaned by a previous process to the pending queue."""
return self.recover_orphan_running()
@@ -0,0 +1,49 @@
"""One global final-state permission for remote model content egress."""
import asyncio
from collections.abc import Sequence
from typing import Annotated, cast
from fastapi import APIRouter, Request
from fastapi.params import Depends
from pydantic import BaseModel, ConfigDict, StrictBool
from one_memory.memory_server.config_state import ConfigStore
class ModelContentPermissionState(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
allowed: bool
class SetModelContentPermission(BaseModel):
model_config = ConfigDict(extra="forbid")
allowed: StrictBool
def model_content_router(dependencies: Sequence[Depends]) -> APIRouter:
router = APIRouter(prefix="/api/v1/model-content-permission", dependencies=dependencies)
def config(request: Request) -> ConfigStore:
return cast(ConfigStore, request.app.state.config_store)
Config = Annotated[ConfigStore, Depends(config)]
@router.get("", response_model=ModelContentPermissionState)
async def current(store: Config) -> ModelContentPermissionState:
return ModelContentPermissionState(allowed=store.model_content_allowed())
@router.post("", response_model=ModelContentPermissionState)
async def set_permission(
request: Request,
value: SetModelContentPermission,
store: Config,
) -> ModelContentPermissionState:
await asyncio.to_thread(store.set_model_content_allowed, value.allowed)
if value.allowed:
request.app.state.start_memory_backfill()
return ModelContentPermissionState(allowed=value.allowed)
return router
@@ -0,0 +1,180 @@
"""Live global permission checks for remote model-content egress.
``config.json`` is the only durable permission state. Tasks do not carry a
permission receipt, revision, source scope, or model pin. Every remote send
and every publication boundary comes back through the lease and reads the
current final-state boolean.
"""
from __future__ import annotations
from collections.abc import Iterator
from contextlib import contextmanager
from dataclasses import dataclass, field
from datetime import UTC, datetime
from one_memory.hy_source_context import HySourceContext
from one_memory.imports.models import EvidenceRecord
from one_memory.memory_governance import MemoryGovernanceStore, MemoryStatus
from one_memory.memory_server.config_state import ConfigStore
from one_memory.memory_tasks import MemoryTask, MemoryTaskStatus
from one_memory.model_profiles import ModelProfileSnapshot, ModelProfileStore
def permission_error(code: str) -> RuntimeError:
from one_memory.hy_memory_bridge import HyMemoryBridgeError
return HyMemoryBridgeError(
code if code.startswith("model_content_") else f"model_content_{code}"
)
def error_is_policy_stop(code: str) -> bool:
"""A disabled final-state permission is an intentional stop."""
return code in {
"model_content_source_disabled",
"hy_memory_evidence_changed",
}
@dataclass(frozen=True)
class ModelProcessingPermissionLease:
task_id: str
user_id: str
profile: ModelProfileSnapshot
records: tuple[EvidenceRecord, ...]
source_context: HySourceContext | None
_guard: ModelProcessingPermissionGuard = field(repr=False, compare=False)
def recheck(self) -> None:
"""Read current config/profile immediately before an egress boundary."""
self.recheck_permission()
self._guard.check_context(self.records, self.source_context)
def recheck_permission(self) -> None:
self._guard.check(self.profile, self.records)
class ModelProcessingPermissionGuard:
def __init__(
self,
config: ConfigStore,
governance: MemoryGovernanceStore,
profiles: ModelProfileStore,
) -> None:
self.config = config
self.governance = governance
self.profiles = profiles
def require_allowed(self) -> None:
if not self.config.model_content_allowed():
raise permission_error("permission_required")
def check(
self,
profile: ModelProfileSnapshot,
records: tuple[EvidenceRecord, ...] = (),
) -> None:
"""Validate only live global consent and the currently active model."""
self.require_allowed()
if self.profiles.active_identity() != (profile.profile_id, profile.revision):
raise permission_error("profile_changed")
if records and any(record.user_id != records[0].user_id for record in records):
raise permission_error("context_scope_invalid")
def lease(
self,
task: MemoryTask,
records: tuple[EvidenceRecord, ...],
context: HySourceContext | None,
) -> ModelProcessingPermissionLease:
combined = self.context_records(records, context)
lease = ModelProcessingPermissionLease(
task_id=task.task_id,
user_id=task.user_id,
profile=task.profile_snapshot,
records=combined,
source_context=context,
_guard=self,
)
lease.recheck()
return lease
@staticmethod
def context_records(
records: tuple[EvidenceRecord, ...],
context: HySourceContext | None,
) -> tuple[EvidenceRecord, ...]:
by_id = {record.evidence_id: record for record in records}
if len(by_id) != len(records):
raise permission_error("context_scope_invalid")
if context is not None:
for record in (*context.current, *context.history, *context.supporting):
previous = by_id.get(record.evidence_id)
if previous is not None and previous != record:
raise permission_error("evidence_changed")
by_id[record.evidence_id] = record
return tuple(by_id.values())
def check_context(
self,
records: tuple[EvidenceRecord, ...],
context: HySourceContext | None,
) -> None:
if context is None:
return
ids = {record.evidence_id for record in records}
if not context.current:
raise permission_error("context_scope_invalid")
context_records = (*context.current, *context.history, *context.supporting)
if len({record.evidence_id for record in context_records}) != len(context_records):
raise permission_error("context_scope_invalid")
for known in context.known_memories:
if not known.evidence_ids or not set(known.evidence_ids) <= ids:
raise permission_error("context_support_missing")
memory = self.governance.get(
known.governed_memory_id,
user_id=context.current[0].user_id,
)
if (
memory is None
or memory.revision != known.revision
or memory.content != known.content
or set(memory.evidence_ids) != set(known.evidence_ids)
or memory.status is not MemoryStatus.CONFIRMED
):
raise permission_error("context_changed")
def reject_unstarted(self, task_id: str, code: str) -> None:
stopped = error_is_policy_stop(code)
with self._task_write() as tasks:
tasks.execute(
"UPDATE memory_tasks SET status=?, last_error_code=?, updated_at=? "
"WHERE task_id=? AND status IN (?,?,?,?)",
(
MemoryTaskStatus.CANCELLED if stopped else MemoryTaskStatus.FAILED,
str(permission_error(code)),
datetime.now(UTC).isoformat(),
task_id,
MemoryTaskStatus.PENDING,
MemoryTaskStatus.WAITING_FOR_PERMISSION,
MemoryTaskStatus.WAITING_FOR_MODEL,
MemoryTaskStatus.DEFERRED_OUTSIDE_WINDOW,
),
)
@contextmanager
def _task_write(self) -> Iterator[object]:
from one_memory.database import take_the_write_lock_first
from one_memory.database_access import main_database
with main_database(self.governance.database_path).write(
autocommit=True,
foreign_keys=True,
) as connection:
take_the_write_lock_first(connection)
with connection:
yield connection
@@ -1,404 +0,0 @@
"""Persistent, content-free accounting for precipitation model requests."""
from __future__ import annotations
import asyncio
import json
import logging
from collections.abc import Iterator
from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass
from datetime import UTC, datetime
from importlib import import_module
from pathlib import Path
from typing import TypedDict
from uuid import uuid4
import httpx
from one_memory.database_access import main_database
from one_memory.memory_server.migrations import model_request_audit_ddl
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class ModelRequestAuditContext:
request_id: str
_audit_context: ContextVar[ModelRequestAuditContext | None] = ContextVar(
"one_memory_model_request_audit_context",
default=None,
)
@contextmanager
def audit_precipitation_request(request_id: str) -> Iterator[None]:
"""Enable physical-request accounting for one memory precipitation."""
token = _audit_context.set(ModelRequestAuditContext(request_id=request_id))
try:
yield
finally:
_audit_context.reset(token)
def _current_prompt_stage() -> str:
try:
resolver = getattr( # noqa: B009 - optional SDK version boundary
import_module("hy_memory.utils.model_call_context"),
"current_model_call_stage",
)
except (ImportError, AttributeError):
return "other"
return str(resolver())
class ModelRequestAuditStore:
def __init__(self, database: Path) -> None:
self.database = database
with main_database(database).write() as connection, connection:
for statement in model_request_audit_ddl("sqlite").split(";"):
if statement.strip():
connection.execute(statement)
def start(
self,
*,
request_id: str,
profile_id: str,
profile_revision: int,
provider_kind: str,
model: str,
request_kind: str,
prompt_stage: str,
) -> str:
audit_id = str(uuid4())
with main_database(self.database).write() as connection, connection:
connection.execute(
"""
INSERT INTO model_request_audit (
audit_id,request_id,profile_id,profile_revision,provider_kind,
model,request_kind,prompt_stage,started_at,outcome
) VALUES (?,?,?,?,?,?,?,?,?,?)
""",
(
audit_id,
request_id,
profile_id,
profile_revision,
provider_kind,
model,
request_kind,
prompt_stage,
datetime.now(UTC).isoformat(),
"in_flight",
),
)
return audit_id
def finish(
self,
audit_id: str,
*,
outcome: str,
http_status: int | None,
error_type: str | None = None,
provider_request_id: str | None = None,
prompt_tokens: int | None = None,
completion_tokens: int | None = None,
total_tokens: int | None = None,
cached_prompt_tokens: int | None = None,
reasoning_tokens: int | None = None,
) -> None:
with main_database(self.database).write() as connection, connection:
connection.execute(
"""
UPDATE model_request_audit
SET completed_at=?,outcome=?,http_status=?,error_type=?,
provider_request_id=?,prompt_tokens=?,completion_tokens=?,
total_tokens=?,cached_prompt_tokens=?,reasoning_tokens=?
WHERE audit_id=?
""",
(
datetime.now(UTC).isoformat(),
outcome,
http_status,
error_type,
provider_request_id,
prompt_tokens,
completion_tokens,
total_tokens,
cached_prompt_tokens,
reasoning_tokens,
audit_id,
),
)
def report(
self,
*,
since: str | None = None,
request_id: str | None = None,
limit: int = 200,
) -> dict[str, object]:
clauses: list[str] = []
params: list[object] = []
if since is not None:
clauses.append("started_at>=?")
params.append(since)
if request_id is not None:
clauses.append("request_id=?")
params.append(request_id)
where = f"WHERE {' AND '.join(clauses)}" if clauses else ""
with main_database(self.database).read() as connection:
summary = connection.execute(
f"""
SELECT COUNT(*) AS request_count,
SUM(CASE WHEN outcome='succeeded' THEN 1 ELSE 0 END) AS succeeded,
SUM(CASE WHEN outcome='failed' THEN 1 ELSE 0 END) AS failed,
SUM(CASE WHEN outcome='in_flight' THEN 1 ELSE 0 END) AS in_flight,
SUM(prompt_tokens) AS prompt_tokens,
SUM(completion_tokens) AS completion_tokens,
SUM(total_tokens) AS total_tokens,
SUM(cached_prompt_tokens) AS cached_prompt_tokens,
SUM(reasoning_tokens) AS reasoning_tokens,
AVG(prompt_tokens) AS avg_prompt_tokens,
AVG(completion_tokens) AS avg_completion_tokens
FROM model_request_audit {where}
""",
tuple(params),
).fetchone()
rows = connection.execute(
f"""
SELECT audit_id,request_id,profile_id,profile_revision,provider_kind,
model,request_kind,prompt_stage,started_at,completed_at,outcome,http_status,
error_type,provider_request_id,prompt_tokens,completion_tokens,
total_tokens,cached_prompt_tokens,reasoning_tokens
FROM model_request_audit {where}
ORDER BY started_at DESC,audit_id DESC LIMIT ?
""",
(*params, limit),
).fetchall()
minute_rows = connection.execute(
f"""
SELECT substr(started_at,1,16) || ':00Z' AS minute,
prompt_stage,
COUNT(*) AS request_count,
SUM(prompt_tokens) AS prompt_tokens,
SUM(completion_tokens) AS completion_tokens,
SUM(total_tokens) AS total_tokens,
AVG(prompt_tokens) AS avg_prompt_tokens,
AVG(completion_tokens) AS avg_completion_tokens
FROM model_request_audit {where}
GROUP BY substr(started_at,1,16),prompt_stage
ORDER BY minute,prompt_stage
""",
tuple(params),
).fetchall()
keys = (
"audit_id",
"request_id",
"profile_id",
"profile_revision",
"provider_kind",
"model",
"request_kind",
"prompt_stage",
"started_at",
"completed_at",
"outcome",
"http_status",
"error_type",
"provider_request_id",
"prompt_tokens",
"completion_tokens",
"total_tokens",
"cached_prompt_tokens",
"reasoning_tokens",
)
summary_keys = (
"request_count",
"succeeded",
"failed",
"in_flight",
"prompt_tokens",
"completion_tokens",
"total_tokens",
"cached_prompt_tokens",
"reasoning_tokens",
"avg_prompt_tokens",
"avg_completion_tokens",
)
return {
"summary": {
key: (
round(float(value), 2)
if key.startswith("avg_") and value is not None
else int(value)
if value is not None
else 0
)
for key, value in zip(summary_keys, summary, strict=True)
},
"requests": [
dict(zip(keys, tuple(row), strict=True))
for row in rows
],
"by_minute": [
{
"minute": str(row[0]),
"prompt_stage": str(row[1]),
"request_count": int(row[2]),
"prompt_tokens": int(row[3] or 0),
"completion_tokens": int(row[4] or 0),
"total_tokens": int(row[5] or 0),
"avg_prompt_tokens": round(float(row[6] or 0), 2),
"avg_completion_tokens": round(float(row[7] or 0), 2),
}
for row in minute_rows
],
}
def _non_negative_int(value: object) -> int | None:
return value if type(value) is int and value >= 0 else None
class _ResponseUsage(TypedDict):
provider_request_id: str | None
prompt_tokens: int | None
completion_tokens: int | None
total_tokens: int | None
cached_prompt_tokens: int | None
reasoning_tokens: int | None
def _empty_usage() -> _ResponseUsage:
return {
"provider_request_id": None,
"prompt_tokens": None,
"completion_tokens": None,
"total_tokens": None,
"cached_prompt_tokens": None,
"reasoning_tokens": None,
}
def _response_usage(content: bytes) -> _ResponseUsage:
try:
body = json.loads(content)
except (json.JSONDecodeError, UnicodeDecodeError):
return _empty_usage()
if not isinstance(body, dict):
return _empty_usage()
usage = body.get("usage")
if not isinstance(usage, dict):
usage = {}
prompt_details = usage.get("prompt_tokens_details")
completion_details = usage.get("completion_tokens_details")
cached = (
prompt_details.get("cached_tokens")
if isinstance(prompt_details, dict)
else usage.get("prompt_cache_hit_tokens")
)
reasoning = (
completion_details.get("reasoning_tokens")
if isinstance(completion_details, dict)
else None
)
return {
"provider_request_id": body.get("id") if isinstance(body.get("id"), str) else None,
"prompt_tokens": _non_negative_int(usage.get("prompt_tokens")),
"completion_tokens": _non_negative_int(usage.get("completion_tokens")),
"total_tokens": _non_negative_int(usage.get("total_tokens")),
"cached_prompt_tokens": _non_negative_int(cached),
"reasoning_tokens": _non_negative_int(reasoning),
}
def _request_kind(request: httpx.Request) -> str:
path = request.url.path.rstrip("/")
if path.endswith("/chat/completions"):
return "chat_completion"
if path.endswith("/embeddings"):
return "embedding"
return "model_http"
def _request_model(content: bytes, fallback: str) -> str:
try:
body = json.loads(content)
except (json.JSONDecodeError, UnicodeDecodeError):
return fallback
model = body.get("model") if isinstance(body, dict) else None
return model if isinstance(model, str) and model else fallback
class AuditedModelTransport(httpx.AsyncBaseTransport):
"""Record one row for each physical precipitation HTTP attempt."""
def __init__(
self,
store: ModelRequestAuditStore,
*,
profile_id: str,
profile_revision: int,
provider_kind: str,
model: str,
inner: httpx.AsyncBaseTransport | None = None,
) -> None:
self.store = store
self.profile_id = profile_id
self.profile_revision = profile_revision
self.provider_kind = provider_kind
self.model = model
self.inner = inner or httpx.AsyncHTTPTransport(retries=0, trust_env=False)
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
context = _audit_context.get()
if context is None:
return await self.inner.handle_async_request(request)
content = await request.aread()
audit_id = await asyncio.to_thread(
self.store.start,
request_id=context.request_id,
profile_id=self.profile_id,
profile_revision=self.profile_revision,
provider_kind=self.provider_kind,
model=_request_model(content, self.model),
request_kind=_request_kind(request),
prompt_stage=_current_prompt_stage(),
)
try:
response = await self.inner.handle_async_request(request)
except BaseException as error:
await asyncio.to_thread(
self.store.finish,
audit_id,
outcome="failed",
http_status=None,
error_type=type(error).__name__,
)
raise
content = await response.aread()
usage = _response_usage(content)
try:
await asyncio.to_thread(
self.store.finish,
audit_id,
outcome="succeeded" if response.is_success else "failed",
http_status=response.status_code,
error_type=None if response.is_success else f"http_{response.status_code}",
**usage,
)
except Exception:
# The durable in_flight row is more truthful than hiding the model
# response after it has already incurred cost.
logger.exception("model_request_audit_finish_failed audit_id=%s", audit_id)
return response
async def aclose(self) -> None:
await self.inner.aclose()
@@ -68,16 +68,6 @@ COMPLETE_INSERT_MYSQL = (
"singleton, status, selected_source_ids_json, started_at, completed_at, updated_at"
") VALUES (1, 'completed', '[]', NULL, %s, %s)"
)
RESET_UPDATE_SQLITE = (
"UPDATE onboarding_state SET status = 'not_started', "
"selected_source_ids_json = '[]', started_at = NULL, completed_at = NULL, "
"updated_at = ? WHERE singleton = 1"
)
RESET_UPDATE_MYSQL = (
"UPDATE onboarding_state SET status = 'not_started', "
"selected_source_ids_json = '[]', started_at = NULL, completed_at = NULL, "
"updated_at = %s WHERE singleton = 1"
)
class OnboardingState(BaseModel):
@@ -150,30 +140,6 @@ class OnboardingStore:
)
return self.get()
def reset(self) -> OnboardingState:
"""Return onboarding to its first-run state, touching nothing else.
The completion flag is stored beside the memories on purpose: a fresh
install of the app must not greet someone who already has memories as a
first-time user. That makes the way back to the cold-start screens an
explicit action rather than a side effect of reinstalling, and it is
the whole reason this exists: QA and long-time users need to see those
screens again without giving up a single memory.
Sources, memories and settings are separate tables and are not read or
written here.
"""
now = datetime.now(UTC).isoformat()
mysql = is_mysql(self._store)
with self._connect(writing=True) as connection:
execute(
connection,
mysql,
RESET_UPDATE_MYSQL if mysql else RESET_UPDATE_SQLITE,
(now,),
)
return self.get()
@contextmanager
def _connect(
self,
@@ -0,0 +1,81 @@
"""Continuous recovery monitor for model processing.
The monitor is intentionally retained. It observes the global config final
state plus ``memory_tasks`` and asks the supervisor to recover work; it owns no
second request/receipt state machine.
"""
import asyncio
import time
from collections.abc import Hashable
from one_memory.memory_processing import MemoryProcessingSupervisor
from one_memory.memory_server.config_state import ConfigStore
class ProcessingCoordinator:
RESCAN_SECONDS = 60.0
POLL_SECONDS = 2.0
def __init__(self, config: ConfigStore, supervisor: MemoryProcessingSupervisor):
self.config = config
self.supervisor = supervisor
self._stopping = False
self._monitor: asyncio.Task[None] | None = None
self._error: str | None = None
self._observed: tuple[Hashable, ...] | None = None
self._last_scan = 0.0
self._clock = time.monotonic
async def recover(self) -> None:
"""Start the one process-owned monitor, whether permission is on or off."""
if self._stopping:
return
if self._monitor is None or self._monitor.done():
self._monitor = asyncio.create_task(
self._monitor_loop(),
name="one-memory-processing-monitor",
)
def _identity(self) -> tuple[Hashable, ...]:
counts = self.supervisor.tasks.counts()
return (
self.config.model_content_allowed(),
tuple(sorted((str(status), count) for status, count in counts.items())),
)
async def _tick(self) -> None:
identity = await asyncio.to_thread(self._identity)
now = self._clock()
if identity[0] and (
identity != self._observed or now - self._last_scan >= self.RESCAN_SECONDS
):
await self.supervisor.resume_allowed()
self._last_scan = now
identity = await asyncio.to_thread(self._identity)
self._observed = identity
self._error = None
async def _monitor_loop(self) -> None:
while not self._stopping:
try:
await self._tick()
except asyncio.CancelledError:
raise
except Exception:
self._error = "model_content_recovery_blocked"
await asyncio.sleep(self.POLL_SECONDS)
def status(self) -> dict[str, object]:
return {
"monitoring": self._monitor is not None and not self._monitor.done(),
"allowed": self._observed[0] if self._observed else None,
"error": self._error,
}
async def shutdown(self) -> None:
self._stopping = True
if self._monitor is not None:
self._monitor.cancel()
await asyncio.gather(self._monitor, return_exceptions=True)
@@ -87,7 +87,7 @@ def build_acceptance_pack(
"""
SELECT DISTINCT e.evidence_id
FROM evidence e
WHERE e.role = 'user'
WHERE e.role = 'user' AND e.written_by = 'person'
AND LENGTH(e.verbatim) <= 300
AND (
e.verbatim LIKE '%我是一名%' OR e.verbatim LIKE '%我是一个%'
@@ -118,7 +118,7 @@ def build_acceptance_pack(
for row in connection.execute(
"""
SELECT evidence_id FROM evidence
WHERE role != 'user'
WHERE role = 'user' AND written_by != 'person'
"""
):
strata["attribution_edge"].add(row["evidence_id"])
@@ -138,7 +138,11 @@ def build_acceptance_pack(
quota,
)
if len(selected) < sample_size:
eligible = (evidence_id for evidence_id, row in evidence.items() if row["role"] == "user")
eligible = (
evidence_id
for evidence_id, row in evidence.items()
if row["role"] == "user" and row["written_by"] == "person"
)
_extend_unique(selected, _stable_order(eligible, seed=f"{seed}:fill"), sample_size)
selected = selected[:sample_size]
if len(selected) < sample_size:
@@ -159,11 +163,7 @@ def build_acceptance_pack(
row_id=f"{index:03d}",
evidence_id=evidence_id,
source_id=source["source_id"],
written_by=(
"person"
if source["role"] == "user"
else ("agent" if source["role"] == "assistant" else source["role"])
),
written_by=source["written_by"],
character_count=len(source["verbatim"]),
content_hash=source["content_hash"],
strata=tuple(
@@ -2,7 +2,7 @@
from collections.abc import Sequence
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord, WrittenBy
from one_memory.imports.sources.parsing import is_host_generated_user_turn, strip_host_wrappers
from one_memory.input_sanitization import sanitize_model_input
from one_memory.memory_processing import _is_hy_input_turn
@@ -15,7 +15,10 @@ def identity_content_ready(record: EvidenceRecord) -> bool:
# cleaning requires a source map and remains explicitly uncovered for now.
return (
record.content_kind is EvidenceContentKind.RAW_CONVERSATION
and record.role in {"user", "assistant"}
and (
(record.role == "user" and record.written_by is WrittenBy.PERSON)
or (record.role == "assistant" and record.written_by is WrittenBy.AGENT)
)
and _is_hy_input_turn(record)
and not is_host_generated_user_turn(record.verbatim)
and strip_host_wrappers(record.verbatim) == record.verbatim.strip()
@@ -10,6 +10,7 @@ from one_memory.imports.models import (
MAX_VERBATIM_CHARACTERS,
EvidenceContentKind,
EvidenceRecord,
WrittenBy,
)
from one_memory.imports.sources.parsing import (
is_host_generated_user_turn,
@@ -163,10 +164,10 @@ def prepare_mapped_content(record: EvidenceRecord) -> PreparedMappedContent:
The returned text is transient; only metadata is suitable for an outer manifest.
That manifest must independently bind Evidence identity/revision/hash and grants.
"""
if record.content_kind is not EvidenceContentKind.RAW_CONVERSATION or record.role not in {
"user",
"assistant",
}:
if record.content_kind is not EvidenceContentKind.RAW_CONVERSATION or not (
(record.role == "user" and record.written_by is WrittenBy.PERSON)
or (record.role == "assistant" and record.written_by is WrittenBy.AGENT)
):
raise MappedContentError("mapped_content_ineligible")
original = record.verbatim
if len(original) > MAX_VERBATIM_CHARACTERS:
@@ -11,7 +11,7 @@ from datetime import UTC
from one_memory import ingest_journal as journal
from one_memory.context_raw import _check, _connection
from one_memory.imports.models import EvidenceRecord
from one_memory.imports.models import EvidenceRecord, WrittenBy
from one_memory.imports.storage import EvidenceStore
from one_memory.retrieval.index_store import RetrievalIndexStore
from one_memory.retrieval.models import EvidenceSpan, RawEvidence
@@ -166,7 +166,7 @@ def materialize_raw_candidates(
source_id=record.source_id,
conversation_id=record.conversation_id,
role=record.role,
written_by="person" if record.role == "user" else "agent",
written_by="person" if record.written_by is WrittenBy.PERSON else "agent",
event_time=event_time,
spans=tuple(spans),
matched_by=("semantic",),
@@ -264,7 +264,7 @@ class ServiceLevelScheduler:
self._quiet_admission_logged_at[key] = now
if last is not None:
diagnostic_event(
level="debug",
level="info",
component="service_level.scheduler",
event="service_level.summary",
outcome="succeeded",
@@ -276,7 +276,7 @@ class ServiceLevelScheduler:
)
return
diagnostic_event(
level="debug",
level="info",
component="service_level.scheduler",
event="service_level.admitted",
outcome="succeeded",
@@ -56,10 +56,10 @@ def build_source_import_receipt(
"SELECT count(*) AS messages, "
"count(DISTINCT conversation_id || char(0) || COALESCE(workspace, '')) "
"AS conversations, "
"coalesce(sum(role='user'),0) AS person_messages, "
"coalesce(sum(role='assistant'),0) AS agent_messages, "
"coalesce(sum(role='tool'),0) AS tool_messages, "
"coalesce(sum(role NOT IN ('user','assistant','tool')),0) "
"coalesce(sum(written_by='person'),0) AS person_messages, "
"coalesce(sum(written_by='agent'),0) AS agent_messages, "
"coalesce(sum(written_by='tool'),0) AS tool_messages, "
"coalesce(sum(written_by NOT IN ('person','agent','tool')),0) "
"AS unknown_author_messages, "
"coalesce(sum(created_at IS NULL),0) AS unknown_time_messages "
"FROM evidence WHERE user_id=? AND source_id=?",
@@ -68,9 +68,7 @@ def build_source_import_receipt(
# Bound text reads before materializing snippets. These are samples, not a
# recency ranking; sorting every large original body is unnecessary here.
examples = db.execute(
"SELECT evidence_id, role, CASE role "
"WHEN 'user' THEN 'person' WHEN 'assistant' THEN 'agent' "
"WHEN 'tool' THEN 'tool' ELSE 'unknown' END AS written_by, created_at, "
"SELECT evidence_id, role, written_by, created_at, "
"substr(verbatim,1,600) AS excerpt, length(verbatim)>600 AS truncated "
"FROM evidence WHERE evidence_id IN ("
"SELECT evidence_id FROM evidence WHERE user_id=? AND source_id=? LIMIT 3)",
@@ -82,13 +80,10 @@ def build_source_import_receipt(
# Qualify before LIMIT: agent/system rows must not consume the three samples.
query_filter = " AND evidence_id=?" if query_evidence_id is not None else ""
queries = db.execute(
f"""SELECT evidence_id, revision AS evidence_revision, role,
CASE role WHEN 'user' THEN 'person' WHEN 'assistant' THEN 'agent'
WHEN 'tool' THEN 'tool' ELSE 'unknown' END AS written_by,
created_at,
CASE WHEN user_id=? AND source_id=? AND role='user'
f"""SELECT evidence_id, revision AS evidence_revision, role, written_by, created_at,
CASE WHEN user_id=? AND source_id=? AND role='user' AND written_by='person'
AND length(verbatim)<=65536
THEN display_query_projection(verbatim,role,'person')
THEN display_query_projection(verbatim,role,written_by)
ELSE NULL END AS projection
FROM evidence WHERE user_id=? AND source_id=?{query_filter}
AND projection IS NOT NULL
+53 -8
View File
@@ -13,7 +13,10 @@
| `test_archive_extract.py` | `contract` | Unpacking an archive a stranger sent, onto this user's disk. |
| `test_ask_workspace_path.py` | `contract` | Ask must honour the same workspace_path expansion recall uses. |
| `test_async_bridge.py` | `contract` | 覆盖 reader rejects parallel workers that could close during next、cancelling a read stops the worker before the batch finishes、starting a read does not stop the app answering 等 10 个场景。 |
| `test_async_dispatch_checks.py` | `contract` | 覆盖 cancelling quiet timer does not start any permission reader、dispatch check does not queue work after shutdown、dispatch permission check denies before queue after classification 等 6 个场景。 |
| `test_auth.py` | `integration` | 覆盖 local api requires bearer token、browser origin is checked in addition to token 场景。 |
| `test_backfill_acceptance_lifecycle.py` | `contract` | A processing acknowledgement belongs to one consent revision and live worker. |
| `test_backfill_conversation_isolation.py` | `contract` | One stale conversation cannot invalidate an entire backfill scan. |
| `test_background_work.py` | `contract` | 覆盖 a request does not wait behind a catch up、bulk work is kept to a few threads at a time、the pool outlives one hand off 等 4 个场景。 |
| `test_backup_destroy_restore_reconciliation.py` | `contract` | Backup -> destroy -> restore, reconciled row for row. |
| `test_backup_endpoint.py` | `integration` | The 'POST /api/v1/backup' endpoint the CLI's 'backup create' calls. |
@@ -61,6 +64,7 @@
| `test_cli_lifecycle.py` | `integration` | 覆盖 cli starts and stops without residual runtime 场景。 |
| `test_cli_restore.py` | `contract` | 覆盖 restore requires a source、restore refuses while the runtime is running、restore previews without confirmation 等 4 个场景。 |
| `test_codex_header_provenance.py` | `contract` | 覆盖 header workspace versions and native ids reach file origin、new adapter rereads header after a real checkpoint、machine and unknown session sources do not establish human or agent sender 等 14 个场景。 |
| `test_compact_model_content.py` | `contract` | Server-held previews preserve exact grants without round-tripping all refs. |
| `test_connection_lifetime.py` | `integration` | Every SQLite connection a request opens must be closed when the request ends. |
| `test_context_access.py` | `contract` | 覆盖 memory only never reads raw authority、raw permission does not grant memory and removes revoked sources、raw requires all apps and source 等 13 个场景。 |
| `test_context_api.py` | `integration` | 覆盖 no model local query returns real raw and governed items、existing single lane authorization never runs denied reader、authentication and origin remain required 等 14 个场景。 |
@@ -72,6 +76,12 @@
| `test_context_search.py` | `contract` | 覆盖 authorization and requested paths never invoke other lane、empty partial zero unavailable and safe failures、parallel common deadline and finalizers wait for both reads 等 24 个场景。 |
| `test_context_search_both_paths.py` | `integration` | The objective names 'recall' and requires BOTH paths: raw and memory. |
| `test_context_v2_contract_parity.py` | `unit` | Mechanical parity checks for the shared v2 coverage receipt contract. |
| `test_continuous_completed_input_review.py` | `contract` | Independent review of completed-input reuse; real stores and controlled HY only. |
| `test_continuous_coordinator.py` | `contract` | 覆盖 activation hook and startup recovery start single monitor then stop、failed preparation does not advance observed watermark、new journal event and source regrant wake replay without legacy request 等 6 个场景。 |
| `test_continuous_outcome_unknown.py` | `contract` | 覆盖 a timed out continuous task becomes unknown not pending、a timed out non continuous task also becomes unknown、an unsettled continuous send becomes unknown not failed 等 8 个场景。 |
| `test_continuous_policy.py` | `contract` | 覆盖 backlog snapshot over one hundred thousand is compact、caller rollback includes policy and backlog、enable idempotence rejects changed payload and superseded operation 等 19 个场景。 |
| `test_continuous_processing_bridge.py` | `contract` | 覆盖 bound source regrant refuses old task、continuous inactive never falls back to legacy、continuous prepare lease ingest has no legacy receipt or old context 等 17 个场景。 |
| `test_continuous_processing_integration.py` | `contract` | Real temporary Core/stores with controlled probe and HY boundaries, never real data. |
| `test_contract_parity.py` | `contract` | Python 的枚举和 TypeScript 里那份手写的联合类型必须一致。 |
| `test_conversation_search.py` | `integration` | Searching imported conversations before any generation model is configured. |
| `test_conversation_seed_ids.py` | `contract` | 覆盖 seed ids read metadata only without hydrating、seed eligibility keeps person and memory content only、latest activity includes nonseed turns and import time fallback 等 13 个场景。 |
@@ -87,6 +97,7 @@
| `test_database_contention.py` | `contract` | Writers meet in this one file, and the loser used to lose data. |
| `test_database_file_permissions.py` | `contract` | 覆盖 a new main database is readable by its owner alone、the wal sidecars are not left world readable、a database from an older build is tightened on open 场景。 |
| `test_db_session_utc.py` | `contract` | The MySQL session must be pinned to UTC. |
| `test_delete_no_conversation_freeze.py` | `contract` | 删掉一条记忆,不得让同一对话之后的新 turn 停止沉淀(RC10)。 |
| `test_delivery_board.py` | `contract` | 覆盖 parses only the marked structured record、verified requires evidence with a positive control、rejects unknown dependency 等 5 个场景。 |
| `test_derived_echo.py` | `contract` | A host that saves our injection back must not mint a new user decision. |
| `test_detailed_logging.py` | `contract` | 覆盖 detailed pipeline logging defaults off and survives restart、live log level applies to full hy and provider call chain、info collects hy info without enabling debug pipeline content 等 7 个场景。 |
@@ -100,6 +111,7 @@
| `test_dual_engine_helper.py` | `contract` | Tests for the dual-engine harness itself. |
| `test_event_loop_blocking.py` | `contract` | Proof, as a test, that blocking work on the loop freezes it — and the fix works. |
| `test_evidence_reconcile_primitive.py` | `integration` | A bounded, read-only discovery primitive for the H23 Evidence reconcile. |
| `test_evidence_revision_read_review.py` | `contract` | Read invalidation review: real guard/provenance, controlled HY, no model/network. |
| `test_evidence_store.py` | `contract` | 覆盖 same source message merges origins without duplication、identical text from different sources is not deduplicated、rescanning an unchanged file does not write the message again 等 26 个场景。 |
| `test_export_probe.py` | `contract` | What a probe may say about a stranger's chat history, and what it may not. |
| `test_external_endpoint_scopes.py` | `integration` | Core boundary tests using the broker's envelope, never a host's claimed identity. |
@@ -113,9 +125,10 @@
| `test_file_import_operations.py` | `integration` | Bounded synthetic uploads; operation identity is not a content dedup key. |
| `test_file_import_operations_review.py` | `contract` | Upload refusal/legacy compatibility review with bounded synthetic files only. |
| `test_file_publish_job_boundary.py` | `contract` | Staged uploads cannot widen an earlier directory import's uncaptured input. |
| `test_four_app_live_ingest.py` | `contract` | Actual source readers + OS watch + Core routes; no model or native host calls. |
| `test_four_app_live_ingest.py` | `integration` | Actual source readers + OS watch + Core routes; no model or native host calls. |
| `test_frozen_core_layout.py` | `contract` | 覆盖 single file layout remains available without private preset、folder layout keeps same core and all native dependencies、unknown layout does not silently build another package 等 5 个场景。 |
| `test_golden_corpus.py` | `contract` | A7-01: the golden corpus comparison. |
| `test_governance_authority_review.py` | `contract` | D04 authority and workspace scope through the real continuous permission guard. |
| `test_governance_permission_commit.py` | `contract` | 覆盖 permission guard is rechecked inside write transaction and rolls back、guard context does not leak into subsequent manual actions、guard check and commit hold database write authority 场景。 |
| `test_governance_read_path_does_not_take_the_writer.py` | `contract` | The recall read path must not acquire the single global writer. |
| `test_governance_store_root.py` | `contract` | 生产代码里治理 store 只在 Core 启动的组合根建一次。 |
@@ -131,19 +144,32 @@
| `test_hourly_c1_peak_replay_contract.py` | `contract` | Guard the package peak-replay probe's error accounting. |
| `test_hourly_c1_recall_stage_probe_contract.py` | `contract` | Guard the packaged recall stage probe's command and error semantics. |
| `test_hy_baseline_source.py` | `contract` | 覆盖 first import replay copies raw prefix without selecting message ids、frozen transcript keeps middle user message and excludes later records 场景。 |
| `test_hy_coding_lease.py` | `contract` | HY's own coding/working route under the permission lease, on the pinned SDK. |
| `test_hy_coding_reply.py` | `contract` | R-07: HY's coding-shaped replies read as "no memories at all". |
| `test_hy_lease_responsiveness.py` | `contract` | Blocked lease checks, local failing doubles only; no network or credentials. |
| `test_hy_lifecycle_sync.py` | `contract` | HY decides deletions; the ledger mirrors them. Pinned SDK, scripted HTTP. |
| `test_hy_log_privacy.py` | `contract` | HY SDK records are not rewritten by One Memory's logging setup. |
| `test_hy_memory_bridge.py` | `contract` | 覆盖 hy client initialization is serial across bridges、native hy clients share one runtime per data directory、ollama profile ignores ambient remote storage and models 等 41 个场景。 |
| `test_hy_metrics_lifecycle.py` | `contract` | What the pinned public SDK's MetricsCollector does, and what that costs us. |
| `test_hy_native_write_chain.py` | `contract` | HY writes with its own prompts and judgment; the product adds no audit of its own. |
| `test_hy_network_egress.py` | `contract` | Where a real HY client is allowed to send bytes, and where it is not. |
| `test_hy_permission_context.py` | `contract` | Whole native HY client with synthetic content and HTTP MockTransport, never cloud. |
| `test_hy_permission_context_races.py` | `contract` | Bounded factory races with a temporary native HY client and no model calls. |
| `test_hy_permission_raw_context.py` | `contract` | Real pinned HY SDK/Chroma and guards; scripted HTTP is not model acceptance. |
| `test_hy_permission_store.py` | `contract` | Native HY node/algorithm tests with synthetic data and an in-memory backend only. |
| `test_hy_permission_transport.py` | `contract` | Native HY entry points, synthetic inputs and MockTransport only. |
| `test_hy_response_guard.py` | `contract` | 覆盖 invalid response retries same hy provider without changing prompt、repeated empty response fails instead of becoming no op、failure receipt survives a later successful model call 等 23 个场景。 |
| `test_hy_response_shapes.py` | `contract` | HY response shape selection only; no provider, write routing or model calls. |
| `test_hy_runtime_config.py` | `contract` | MEMORY.md is the audited record of what we send HY; keep it answerable. |
| `test_hy_sdk_idempotency_probe.py` | `contract` | A0-03: probe whether the HY SDK can make a write idempotent. |
| `test_hy_source_context.py` | `contract` | 覆盖 bound context rejects a same named conversation in another workspace、bound context rejects a duplicated evidence id、bound context rejects a same conversation from another source 等 9 个场景。 |
| `test_hy_stage_trace.py` | `contract` | 覆盖 observer returns native results without editing or requesting models、observer redacts content without recording prompts or exception details 场景。 |
| `test_hy_state_source.py` | `contract` | RC4:记忆生命周期(有效 / 被取代 / 去重隐藏 / 过期)直接读 HY,产品账本不再逐类镜像。 |
| `test_hy_step_timing.py` | `contract` | Where an HY write spends the time that is not a model call. |
| `test_hy_step_trace_switch.py` | `contract` | Rule 15: the per-step HY trace is off unless the debug tier asks for it. |
| `test_hy_supersede_sync.py` | `contract` | HY decides supersessions; the ledger mirrors them. Pinned SDK, scripted HTTP. |
| `test_hy_trial_budget.py` | `contract` | Offline proofs for the real-model acceptance budget (no network, no Keychain). |
| `test_hy_turn_input.py` | `contract` | HY gets one turn at a time, exactly as it happened, and picks its own route. |
| `test_hy_writeback.py` | `integration` | HY 的结果只有一扇门进治理表,而且每次进门都带着调用开始时的证据快照。 |
| `test_import_api.py` | `integration` | 覆盖 resume during cleanup returns conflict without internal error、authorize detect import and revoke api flow、protected api rejects missing token and foreign origin 等 5 个场景。 |
| `test_import_control_races_review.py` | `contract` | Read-only review repro: real runner/store, synthetic delayed adapter cleanup. |
@@ -155,6 +181,7 @@
| `test_import_three_scenarios.py` | `contract` | The three import scenarios a user actually meets. |
| `test_import_three_scenarios_real.py` | `contract` | The three import scenarios, on a real Core at a real volume. |
| `test_imported_index_background.py` | `contract` | The ordering index is built off the startup path. |
| `test_incremental_memory_backfill.py` | `contract` | 覆盖 accept backfill returns after first batch while later history is still read、backfill budget yields between rounds and resumes same cursor、backfill holds an unanswered raw turn until its reply arrives 等 18 个场景。 |
| `test_ingest_journal.py` | `contract` | 覆盖 baseline and empty initialize are explicit transactional and idempotent、old date new import and revise noop and rollbacks、import metadata changes invalidate without authorizing content 等 12 个场景。 |
| `test_input_sanitization.py` | `unit` | 覆盖 redacts cli and assignment secrets without removing context、redacts bearer known tokens urls and private keys、redacts quoted json assignments and basic authorization 等 8 个场景。 |
| `test_installer.py` | `contract` | 覆盖 installer create verify and rollback is transactional、installer updates owned file and restores backup、installer refuses unowned or changed target 场景。 |
@@ -182,6 +209,7 @@
| `test_main_database_fairness.py` | `contract` | Waiting for the one database has to be a queue, not a scramble. |
| `test_main_database_isolation.py` | `contract` | What one checkout of the main database can do, and what it leaves for the next. |
| `test_managed_restore.py` | `contract` | 覆盖 restore rejects a source outside the managed backup directory、restore rejects a symlinked source、restore rejects a non one memory database 等 9 个场景。 |
| `test_manual_memory_via_hy.py` | `contract` | T04: what the person explicitly asks to remember is written through HY's import path. |
| `test_manual_transcript_import.py` | `integration` | 覆盖 manual json keeps authors unknown time and exact text、marked text keeps code fences and never invents timestamps、ambiguous partial or invalid manual data never leaves partial import 等 4 个场景。 |
| `test_matter_collection.py` | `integration` | 覆盖 hy windows cover every turn including leading and boundary assistants、oversize turn is split without silently dropping content、partial timestamps preserve source sequence 等 8 个场景。 |
| `test_matter_discovery.py` | `contract` | 覆盖 synthesis materializes only known verbatim citations、duplicate claims keep all governed refs and distinct states、synthesis cannot echo the original sixty claim list 等 27 个场景。 |
@@ -239,19 +267,30 @@
| `test_memory_write_registry.py` | `contract` | A3-01: acceptance and outcome recording for the unified write path. |
| `test_memory_write_routes.py` | `integration` | A3-01: the HTTP write entry, tested against a real FastAPI app. |
| `test_migrate_off_sqlite.py` | `contract` | A1-06: the migration is only done when reconciliation agrees. |
| `test_model_content_bulk.py` | `contract` | Bulk consent reads preserve exact identities and SQLite snapshot boundaries. |
| `test_model_content_parse_cost.py` | `contract` | Memoize parsing only; every permission decision still reads current SQLite state. |
| `test_model_content_permissions.py` | `contract` | Content permission tests use only a temporary database and synthetic sources. |
| `test_model_content_preview_races.py` | `contract` | Confirmation must tolerate append-only import without broadening consent. |
| `test_model_content_routes.py` | `integration` | 覆盖 permission route reads and writes only config final state、core composition starts with config permission and durable drain 场景。 |
| `test_model_content_scope_seals.py` | `contract` | Indexed scope invalidation retains complete-snapshot semantics without rereading it. |
| `test_model_content_snapshot_cost.py` | `contract` | Reuse derived fingerprints only after reading every raw row in this transaction. |
| `test_model_creation_api.py` | `integration` | Normal protected model creation routes with temporary data and in-memory secrets. |
| `test_model_creation_operations.py` | `contract` | 覆盖 lookup without table is read only and legacy create does not initialize、operation commits with profile and version without autoactivation、any replay conflicts before secret put 等 21 个场景。 |
| `test_model_creation_validation_boundary.py` | `integration` | Real ASGI validation must precede creation reservation and secret writes. |
| `test_model_policy_activation.py` | `contract` | 覆盖 activation helper is transactional and restores row factory、backlog scan does not hold the single writer、cloud credential is checked without keychain write 等 21 个场景。 |
| `test_model_policy_routes.py` | `contract` | 覆盖 all routes require auth、backlog is required not defaulted、enable and transition safe shape and callback once after commit 等 10 个场景。 |
| `test_model_probe.py` | `contract` | 覆盖 probe still measures embedding when structured output fails、probe records the real embedding width on the profile、openai compatible probe checks model and structured output 等 23 个场景。 |
| `test_model_probe_input_revision.py` | `contract` | 覆盖 input revision precedes measured width update、concurrent update still rejects width or capability cas、early returns record the actual starting revision 等 6 个场景。 |
| `test_model_probe_operations.py` | `integration` | Persisted probe identity, real SQLite/API and synthetic transport only. |
| `test_model_processing_permissions.py` | `contract` | 覆盖 lease reads current allowed before each boundary、lease refuses a task profile that is no longer active 场景。 |
| `test_model_profile_api.py` | `integration` | 覆盖 model profile api never returns key and enforces revision、model profile probe route returns only machine results、model profile validation never echoes api key 等 4 个场景。 |
| `test_model_profile_context_reset.py` | `integration` | 覆盖 create context default and inclusive bounds、explicit null clears override and preserves history and secret、omitted or same value is noop with unchanged revision and metadata 等 11 个场景。 |
| `test_model_profile_identity.py` | `contract` | Stable model editing and configured-vs-available credentials; no Keychain/network. |
| `test_model_profile_queue_handover.py` | `contract` | Queued work when the model changes or its configuration is deleted (D75). |
| `test_model_profiles.py` | `contract` | 覆盖 profile uses one model and keeps key out of database、api key updates reuse one stable secret reference、startup cleanup removes only secrets not referenced by current profiles 等 15 个场景。 |
| `test_model_request_audit.py` | `contract` | 覆盖 failure is recorded without error message、missing usage stays unknown in detail and zero in summary、records one physical request with provider usage and no content 等 5 个场景。 |
| `test_model_request_budget.py` | `contract` | 覆盖 persists failed request reservations without content、parallel leases cannot exceed request limit、unknown or excessive output and body are denied 等 6 个场景。 |
| `test_model_request_budget_integration.py` | `contract` | Real HY leaf retries plus durable budget accounting; no network or provider. |
| `test_model_scope_lifecycle.py` | `contract` | Bound derived/temporary data without changing consent or original history. |
| `test_mysql_lifecycle.py` | `contract` | A1-07: the decisions Runtime makes around the bundled MySQL. |
| `test_mysql_platform.py` | `contract` | A1-07: the machine-facing half of managing the bundled MySQL. |
| `test_obs03_health_probe.py` | `contract` | OBS-03: '/health' answered with the web app instead of a health payload. |
@@ -260,11 +299,14 @@
| `test_oversized_file_guard.py` | `contract` | One pass over a source file is bounded by a byte budget, not skipped. |
| `test_parent_watchdog.py` | `integration` | A Core that outlives its launcher makes the next launch impossible. |
| `test_policy_api_wiring.py` | `integration` | 覆盖 global model permission off does not backfill、backfill puts history on write jobs without scheduling memory tasks、source pause skips history until the source is reenabled 场景。 |
| `test_policy_operation_lookup.py` | `contract` | 覆盖 corrupt current row cannot make history or missing operation look safe、current operation uses safe receipt without callback or writes、lookup connection rejects writes and preserves database 等 12 个场景。 |
| `test_policy_production_entry.py` | `contract` | 覆盖 production serve passes only supported app options、maintenance serve does not own the runtime 场景。 |
| `test_precipitation_concurrency_setting.py` | `contract` | Rule 2/9: precipitation concurrency is configurable, not hardcoded to 1. |
| `test_primary_source_adapters.py` | `contract` | 覆盖 pipeline json is host generated even when truncated、workbuddy record parsing never blocks the event loop、cursor store user info envelope is not attributed to the person 等 34 个场景。 |
| `test_probe_code_contract_parity.py` | `contract` | 探测码在三处必须同时存在:python 枚举、TS 契约、界面文案表。 |
| `test_probe_payment_required.py` | `contract` | 覆盖 http 402 without a useful body is still understood as unpaid、a wrong address is still a missing model not an unpaid one 场景。 |
| `test_processing_coordinator.py` | `contract` | 覆盖 monitor is retained and observes config plus tasks 场景。 |
| `test_processing_merged_context.py` | `contract` | 覆盖 new chunk omits known memory with changed support but keeps original history 场景。 |
| `test_processing_requests.py` | `contract` | 覆盖 concurrent claim only one snapshot wins、concurrent create same grant has one id and profile、concurrent different profile never reuses another scope 等 14 个场景。 |
| `test_quality_acceptance.py` | `contract` | 覆盖 acceptance pack is deterministic and scores personal boundary、acceptance score fails missing personal memory 场景。 |
| `test_raw_search_without_model.py` | `integration` | Search imported conversations with no generative model configured at all. |
@@ -328,17 +370,17 @@
| `test_secret_store_error_codes.py` | `integration` | 3925a kernel item 3: keychain failure classification and backoff that heals. |
| `test_secure_storage.py` | `integration` | 覆盖 isolated keychain service namespace、custom keychain service is refused outside isolated tests、packaged keychain budget covers loaded process startup 等 20 个场景。 |
| `test_sedimentation_concurrency.py` | `contract` | Sedimentation runs several harnesses at once, but one harness at a time. |
| `test_sedimentation_switch.py` | `contract` | Stopping model-driven sedimentation without losing the configuration. |
| `test_service_level_pressure_acceptance.py` | `contract` | 覆盖 workbuddy recall remains available during 100k import 场景。 |
| `test_service_levels.py` | `contract` | 覆盖 background never consumes the realtime reservation、realtime is next for the same source after background yields、import history gets one slice after nine realtime admissions 等 8 个场景。 |
| `test_session_inbox.py` | `contract` | A session inbox is not a shared mailbox and a pull is not adoption. |
| `test_session_lease.py` | `contract` | A session file on disk is not proof that the session is online. |
| `test_session_registry.py` | `contract` | Trusted session identity is not a workspace path. |
| `test_settings_policy_lifecycle.py` | `contract` | Settings data-plane lifecycle: real temporary Core, fake probe/HY, no UI or package. |
| `test_simplified_model_permission_migrations.py` | `contract` | 覆盖 fresh schema omits removed permission state machines 场景。 |
| `test_snapshot_adapter.py` | `contract` | Selection-only binding reopens checksum-verified frozen copies. |
| `test_snapshot_pool.py` | `contract` | A whole-directory snapshot must not hold a thread everything else is waiting for. |
| `test_soak_probe_effectiveness.py` | `integration` | The soak probe must observe the load it reports on. |
| `test_source_access.py` | `contract` | 覆盖 source root requires explicit user grant、sensitive home directory is rejected without requesting os access、authorized file cannot escape root through a symlink 等 24 个场景。 |
| `test_source_authorization_state.py` | `unit` | Why a source cannot be read, named separately from whether it can. |
| `test_source_candidate_api.py` | `integration` | Candidate location changes must not implicitly grant or import anything. |
| `test_source_candidate_connections.py` | `contract` | 覆盖 empty candidate revisions preserve default fingerprint、persisted candidate requires exact revision、connection resolves candidate in transaction and prior replay never regrants 等 7 个场景。 |
| `test_source_candidates.py` | `contract` | 覆盖 preview is metadata only and cancel changes nothing、select persists only candidate and restarts with revision、compare and swap and aba cannot reuse old preview 等 9 个场景。 |
@@ -350,7 +392,7 @@
| `test_source_parsing.py` | `contract` | 覆盖 reminders are removed only at message edges、original supported outer context and query envelopes、attributed reminder is opaque before finding the outer query 等 11 个场景。 |
| `test_source_permission_races.py` | `integration` | Permission execution regressions using temporary stores and Core ASGI instances. |
| `test_source_query_preview.py` | `contract` | 覆盖 preserves exact user text and span、supported outer wrapper is stripped without paraphrasing、cursor host prefixes are not shown as the user query 等 8 个场景。 |
| `test_source_read_revocation.py` | `contract` | 覆盖 revocation during model wait prevents late answer、source revocation stops every outward read but keeps other sources 场景。 |
| `test_source_read_revocation.py` | `integration` | 覆盖 source revocation stops every outward read but keeps other sources、revocation during model wait prevents late answer 场景。 |
| `test_source_readers.py` | `contract` | 覆盖 jsonl is resumable and does not advance over an incomplete tail、sqlite reader uses monotonic row checkpoint、zstd jsonl reader resumes by completed record 等 7 个场景。 |
| `test_source_registry_and_grants.py` | `contract` | 覆盖 catalog resolves defaults and marks environment overrides、recall hosts that can append join the live sync poller、declarations with no reader are named rather than mistaken for behaviour 等 15 个场景。 |
| `test_source_revocation_parking.py` | `integration` | Revoking a source parks its unfinished extraction instead of failing it (D73). |
@@ -360,7 +402,7 @@
| `test_source_sync_idempotent.py` | `integration` | Final-state Harness sync is idempotent and resumes missed source work. |
| `test_source_sync_permissions.py` | `integration` | S1.3 source permission tests use temporary files only, never real host history. |
| `test_source_totals.py` | `contract` | 覆盖 the first ask is answered from the rows、same named sessions in two workspaces count as two conversations、insert during count is not marked as already counted 等 7 个场景。 |
| `test_source_upload_intents.py` | `contract` | Upload identity binds retries without replacing file receipts or directory consent. |
| `test_source_upload_intents.py` | `integration` | Upload identity binds retries without replacing file receipts or directory consent. |
| `test_source_write_switch.py` | `contract` | A frozen Harness produces no extraction tasks. |
| `test_sql_compat.py` | `contract` | The shared compatibility layer, especially 'rowcount'. |
| `test_sqlite_semantics_gate.py` | `contract` | D56: prevent SQLite implementation details spreading outside the data layer. |
@@ -369,10 +411,13 @@
| `test_store_binding.py` | `contract` | A2: which store is in force, and honest reporting of it. |
| `test_store_binding_ratchet.py` | `contract` | D56: ratchet the retired dual-engine 'store_for' seam out of the Core. |
| `test_store_port.py` | `contract` | A1-04/A2: the store port, and that real implementations satisfy it. |
| `test_subagent_sessions.py` | `contract` | D98: a sub agent's transcript is its own conversation, not part of the main one. |
| `test_subagent_turns.py` | `contract` | D98: a sub agent's turn is ordinary user-assistant data. |
| `test_subscription_visibility.py` | `unit` | A shared directory is not a read grant. |
| `test_sync_facade.py` | `contract` | A2: the sync facade must keep the sync shape call sites already use. |
| `test_tail_metadata_lane.py` | `contract` | Post-response metadata retries must be bounded and never block the route. |
| `test_terminal_payload_reaping.py` | `contract` | Rule 14: dead-end payloads are actually reaped, not just reapable. |
| `test_tool_evidence_hy_turns.py` | `contract` | Every adapter's tool records reach HY inside the turn they happened in. |
| `test_two_level_discovery.py` | `contract` | 3925a kernel item 1: two-level discovery. |
| `test_ui_session.py` | `contract` | 覆盖 exchange is single use and creates a session、expired exchange cannot create a session、exchange is bound to the core instance 等 5 个场景。 |
| `test_ui_session_api.py` | `integration` | 覆盖 ui session exchange sets httponly strict cookie、ui session requires bearer for exchange、ui session reports missing web assets 场景。 |
@@ -384,8 +429,8 @@
| `test_windows_process_control.py` | `contract` | R-29:Windows 上 'os.kill' 不是发信号,是无条件杀进程。 |
| `test_workbuddy_boundary.py` | `contract` | WorkBuddy only injects on SessionStart and UserPromptSubmit. |
| `test_workbuddy_parsing.py` | `contract` | 覆盖 automation preserves query without claiming fresh human authorship、automation context attributes do not depend on order or quote style、quoted full automation envelope is not scheduled provenance 等 17 个场景。 |
| `test_workbuddy_prompt_backfill.py` | `contract` | When the prompt hook's capture fails, the transcript file is what brings the words in. |
| `test_workbuddy_prompt_backfill.py` | `integration` | When the prompt hook's capture fails, the transcript file is what brings the words in. |
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
共 383 个测试文件。
共 428 个测试文件。
@@ -14,8 +14,8 @@ from pathlib import Path
import pytest
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
EXPECTED_TEST_FILE_COUNT = 383
EXPECTED_TEST_FILES_SHA256 = "3c1a3d3febaf478e1ba005fbcd22ad23d76fc5925b32b8d8740e4e65c554b922"
EXPECTED_TEST_FILE_COUNT = 428
EXPECTED_TEST_FILES_SHA256 = "bd3bf4fc9cda2a8e4427d106283c40d22dad026fd452db954046b6f28d21b896"
# Pure transformations, validation models, parity checks and state machines.
# This list is intentionally small: uncertain files default to contract.
@@ -35,7 +35,6 @@ UNIT_FILES = frozenset(
"test_memory_server_job_state.py",
"test_scan_roots.py",
"test_scanner_classifier.py",
"test_source_authorization_state.py",
"test_scanner_sessionizer.py",
"test_source_envelope_positions.py",
"test_source_lifecycle.py",
@@ -69,6 +68,7 @@ INTEGRATION_FILES = frozenset(
"test_external_installation_permissions.py",
"test_fault_matrix.py",
"test_file_import_operations.py",
"test_four_app_live_ingest.py",
"test_graph_width_endpoint.py",
"test_home_summary.py",
"test_hy_writeback.py",
@@ -100,6 +100,7 @@ INTEGRATION_FILES = frozenset(
"test_memory_supersede.py",
"test_memory_visibility.py",
"test_memory_write_routes.py",
"test_model_content_routes.py",
"test_model_creation_api.py",
"test_model_creation_validation_boundary.py",
"test_model_probe_operations.py",
@@ -123,14 +124,17 @@ INTEGRATION_FILES = frozenset(
"test_source_connection_intents.py",
"test_source_import_receipts.py",
"test_source_permission_races.py",
"test_source_read_revocation.py",
"test_source_revocation_parking.py",
"test_source_session_count_api.py",
"test_source_status_latency.py",
"test_source_sync_idempotent.py",
"test_source_sync_permissions.py",
"test_source_upload_intents.py",
"test_ui_session_api.py",
"test_unified_write_entry_wiring.py",
"test_visibility_background.py",
"test_workbuddy_prompt_backfill.py",
}
)
+61
View File
@@ -26,6 +26,67 @@ async def thaw_source(client, source_id, receipt):
return response.json()
async def confirm_model_content(
client,
*,
source_ids,
evidence_ids=(),
all_imported=None,
future_additions=True,
process=True,
):
"""Grant model processing the same way the product does: preview, then confirm.
Live suites need this because importing Evidence is not consent to send it
to a model; without it the processing guard refuses, which is the correct
product behaviour and not something a test may bypass by writing the store.
"""
profiles = await client.get("/api/v1/model-profiles")
profiles.raise_for_status()
active = next(item for item in profiles.json() if item["is_active"])
preview = await client.post(
"/api/v1/model-content-permission/preview",
json={
"profile_id": active["profile_id"],
"profile_revision": active["revision"],
"source_ids": list(source_ids),
"evidence_ids": list(evidence_ids),
# The manual source only accepts named originals, so "everything
# from these sources" is not a default the caller can ignore.
"all_imported_from_selected_sources": (
not evidence_ids if all_imported is None else all_imported
),
"include_future_additions": future_additions,
},
)
# Report the refusal code, not just the status: every content refusal is a
# named product decision and a live run that hides it cannot be diagnosed.
assert preview.status_code == 200, preview.text
current = await client.get("/api/v1/model-content-permission")
current.raise_for_status()
existing = current.json()
confirmed = await client.post(
"/api/v1/model-content-permission",
json={
"snapshot": preview.json(),
"expected_revision": existing["revision"] if existing else 0,
"user_confirmed": True,
},
)
confirmed.raise_for_status()
receipt = confirmed.json()
if process:
# What the product does after the user confirms: take the already
# imported Evidence through the pipeline instead of waiting for the
# next import to trigger a debounce.
accepted = await client.post(
"/api/v1/model-content-permission/process",
json={"expected_revision": receipt["revision"], "user_confirmed": True},
)
accepted.raise_for_status()
return receipt
async def connection_request(client, source_ids, **options):
roots = {}
for source_id in source_ids:
Loaded 100 of 316 files, more files were not shown because too many files have changed in this diff. Show more