Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8367667264 |
No files matched your search
+2
-45
@@ -2,26 +2,6 @@
|
||||
|
||||
Hy Memory and MemHub are versioned and released independently.
|
||||
|
||||
## Hy Memory 2.0.3 — 2026-10-17
|
||||
|
||||
- Added an optional runtime HTTP transport factory so embedded products can
|
||||
enforce request budgets without wrapping or restricting SDK local-store
|
||||
access.
|
||||
- Published
|
||||
[`hy-memory==2.0.3`](https://pypi.org/project/hy-memory/2.0.3/) with
|
||||
unchanged memory extraction, reconciliation, storage, and Coding/Working
|
||||
behavior.
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Removed MemHub's task-scoped HyPermission store wrappers and global
|
||||
model-content consent switch. The trusted embedded Hy Memory SDK now uses
|
||||
its native local stores directly.
|
||||
- Existing installations ignore the retired config flag, readmit previously
|
||||
parked pre-invocation work, and drop only the obsolete
|
||||
`hy_node_permissions` proof table. Evidence, governed memories, model
|
||||
profiles, SDK stores, and source/host grants are preserved.
|
||||
|
||||
## Hy Memory 2.0.2 — 2026-10-17
|
||||
|
||||
- Published
|
||||
@@ -36,34 +16,11 @@ Hy Memory and MemHub are versioned and released independently.
|
||||
Because PyPI release metadata is immutable, the next SDK patch release must
|
||||
carry the updated license metadata; `2.0.1` retains its published metadata.
|
||||
|
||||
## MemHub 1.0.3 — Unreleased
|
||||
## MemHub 1.0.1 — Unreleased
|
||||
|
||||
- Updated the source version to `1.0.3` (build 7).
|
||||
- Pinned the public SDK to `hy-memory==2.0.3`.
|
||||
- Removed the HyPermission store wrappers and global model-content consent
|
||||
switch; the trusted embedded SDK now uses native local stores.
|
||||
- Prevented uncertain writes from blocking later source work and added one
|
||||
reconciled automatic retry after ten seconds.
|
||||
- Added product-facing memory pipeline counters and the log troubleshooting
|
||||
runbook.
|
||||
- Allowed a verified packaged Desktop Runtime from a different commit to take
|
||||
over an equal-build install atomically while preserving downgrade and
|
||||
tamper refusal.
|
||||
|
||||
## MemHub 1.0.2 — 2026-10-17
|
||||
|
||||
- Updated the source version to `1.0.2` (build 6).
|
||||
- Updated the source version to `1.0.1` (build 5).
|
||||
- Pinned the public SDK to `hy-memory==2.0.2`.
|
||||
- Pinned `onnxruntime==1.23.2` to retain macOS x64 packaging support.
|
||||
- Fixed relaunch and upgrade failures when a prior managed Runtime exists.
|
||||
- Embedded canonical build identity when an external packaging pipeline omits
|
||||
release environment fields.
|
||||
- Added automatic Linux Runtime upload to tagged Draft Releases.
|
||||
|
||||
## MemHub 1.0.1 — Superseded before release
|
||||
|
||||
- The candidate tag was not published as a GitHub Release; `1.0.2` replaces it
|
||||
after the managed-Runtime relaunch fix.
|
||||
|
||||
## MemHub 1.0.0
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ English | [中文](README.zh.md)
|
||||
|
||||
## Latest News
|
||||
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.3`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.3/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.2`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.2/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** on Hugging Face along with its [technical report](https://arxiv.org/abs/2610.22196) on attributed reinforcement learning for long-term memory writing!
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **Released** **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)** (submitted to **ICLR 2027**).
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **Accepted** at **EMNLP 2026**: **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)**!
|
||||
@@ -63,13 +63,13 @@ Welcome to the **Hy Memory Research Family**, covering cognitive chat memory, lo
|
||||
### 1. Installation
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.3 # core (includes ChromaDB and Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.3" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.3" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.3" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.3" # + MySQL cache
|
||||
pip install "hy-memory[rerank]==2.0.3" # + local multilingual cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.3" # everything
|
||||
pip install hy-memory==2.0.2 # core (includes ChromaDB and Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.2" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.2" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.2" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.2" # + MySQL cache
|
||||
pip install "hy-memory[rerank]==2.0.2" # + local multilingual cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.2" # everything
|
||||
```
|
||||
|
||||
### 2. Configuration
|
||||
|
||||
+8
-8
@@ -32,7 +32,7 @@
|
||||
|
||||
## 最新动态
|
||||
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.3`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.3/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.2`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.2/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** 长期记忆写入模型及[技术报告](https://arxiv.org/abs/2610.22196)!
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **基准与方法发布** 推出长周期与交错式编程记忆评测基准 **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)**(已投稿 **ICLR 2027**)。
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **论文录用** 双系统认知记忆架构论文 **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)** 被 **EMNLP 2026** 接收!
|
||||
@@ -63,13 +63,13 @@
|
||||
### 1. 安装
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.3 # 核心(含 ChromaDB 与 Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.3" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.3" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.3" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.3" # + MySQL 缓存
|
||||
pip install "hy-memory[rerank]==2.0.3" # + 本地多语言 cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.3" # 全部
|
||||
pip install hy-memory==2.0.2 # 核心(含 ChromaDB 与 Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.2" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.2" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.2" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.2" # + MySQL 缓存
|
||||
pip install "hy-memory[rerank]==2.0.2" # + 本地多语言 cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.2" # 全部
|
||||
```
|
||||
|
||||
### 2. 配置
|
||||
|
||||
+1
-1
@@ -37,7 +37,7 @@ do not belong in the public source tree.
|
||||
- Root GitHub workflows run SDK and MemHub checks from their respective working
|
||||
directories. Native lifecycle workflows produce reviewable Actions artifacts;
|
||||
tag releases remain fail-closed on public identity, signing and notarization.
|
||||
- Hy Memory is published as `hy-memory==2.0.3`. MemHub pins that public
|
||||
- Hy Memory is published as `hy-memory==2.0.2`. MemHub pins that public
|
||||
distribution; its selected npm package is `memhub-cli`, which is not yet
|
||||
published.
|
||||
- Hy-Memory source uses Apache-2.0 at the repository root. Third-party
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@
|
||||
## 安装与快速开始
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.3
|
||||
pip install hy-memory==2.0.2
|
||||
|
||||
# 可选后端
|
||||
pip install 'hy-memory[qdrant]==2.0.1'
|
||||
|
||||
@@ -1123,7 +1123,6 @@ class Extractor:
|
||||
max_tokens=self._llm_config.agent_max_tokens,
|
||||
tools=None,
|
||||
tool_choice=None,
|
||||
audit_stage="extract",
|
||||
)
|
||||
self._call_count += 1
|
||||
total_tokens += response.tokens_used
|
||||
@@ -1234,7 +1233,6 @@ class Extractor:
|
||||
max_tokens=self._llm_config.agent_max_tokens,
|
||||
tools=None,
|
||||
tool_choice=None,
|
||||
audit_stage="extract",
|
||||
)
|
||||
self._call_count += 1
|
||||
total_tokens += response.tokens_used
|
||||
|
||||
@@ -34,7 +34,6 @@ import os
|
||||
import weakref
|
||||
|
||||
from ..config import MemoryConfig
|
||||
from ..utils.model_call_context import model_call_stage
|
||||
from ..core.workload import llm_slot
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -349,7 +348,6 @@ class LLMProvider:
|
||||
stop: List[str] = None,
|
||||
tools: Optional[List[Dict[str, Any]]] = None,
|
||||
tool_choice: Optional[Any] = None,
|
||||
audit_stage: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> LLMResponse:
|
||||
"""
|
||||
@@ -388,16 +386,15 @@ class LLMProvider:
|
||||
len(prompt), _content_fingerprint(prompt),
|
||||
)
|
||||
|
||||
with model_call_stage(audit_stage):
|
||||
response = await self._call_llm(
|
||||
prompt=prompt,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
stop=stop,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
**kwargs
|
||||
)
|
||||
response = await self._call_llm(
|
||||
prompt=prompt,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
stop=stop,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
self._total_calls += 1
|
||||
self._total_tokens += response.tokens_used
|
||||
@@ -499,21 +496,11 @@ class LLMProvider:
|
||||
clients = self._clients.setdefault(loop, {})
|
||||
client = clients.get(client_key)
|
||||
if client is None:
|
||||
client_options: Dict[str, Any] = {
|
||||
"api_key": base_api_key,
|
||||
"base_url": base_url,
|
||||
"timeout": self._llm_config.timeout,
|
||||
"max_retries": 0,
|
||||
}
|
||||
transport_factory = getattr(self.config, "http_transport_factory", None)
|
||||
if callable(transport_factory):
|
||||
import httpx
|
||||
|
||||
client_options["http_client"] = httpx.AsyncClient(
|
||||
transport=transport_factory()
|
||||
)
|
||||
client = AsyncOpenAI(
|
||||
**client_options,
|
||||
api_key=base_api_key,
|
||||
base_url=base_url,
|
||||
timeout=self._llm_config.timeout,
|
||||
max_retries=0,
|
||||
)
|
||||
clients[client_key] = client
|
||||
|
||||
@@ -598,7 +585,6 @@ class LLMProvider:
|
||||
temperature: Optional[float] = None,
|
||||
tools: Optional[List[Dict[str, Any]]] = None,
|
||||
tool_choice: Optional[Any] = None,
|
||||
audit_stage: Optional[str] = None,
|
||||
) -> LLMResponse:
|
||||
"""
|
||||
多轮 messages 调用(支持 tool results 回传)。
|
||||
@@ -626,14 +612,13 @@ class LLMProvider:
|
||||
f"msgs={len(messages)} tools={len(tools) if tools else 0}"
|
||||
)
|
||||
|
||||
with model_call_stage(audit_stage):
|
||||
response = await self._chat_completions(
|
||||
messages,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
)
|
||||
response = await self._chat_completions(
|
||||
messages,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
)
|
||||
|
||||
self._total_calls += 1
|
||||
self._total_tokens += response.tokens_used
|
||||
|
||||
@@ -669,7 +669,6 @@ class MemoryReconciler:
|
||||
response = await llm.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=500,
|
||||
audit_stage="reconcile",
|
||||
)
|
||||
text = response.content.strip()
|
||||
if "```json" in text:
|
||||
@@ -1384,7 +1383,6 @@ class MemoryReconciler:
|
||||
response = await llm.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=dyn_max_tokens,
|
||||
audit_stage="reconcile",
|
||||
)
|
||||
except Exception as e:
|
||||
# LLM 彻底不可用(provider 已按 max_retries 重试过,网关 500 /
|
||||
|
||||
@@ -919,7 +919,6 @@ class CodingMemoryExtractor:
|
||||
resp = await self.llm_provider.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=self.max_tokens,
|
||||
audit_stage="coding_extract",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"[coding-extract] LLM call failed: {e}")
|
||||
|
||||
@@ -571,7 +571,6 @@ class WorkingFactReconciler:
|
||||
resp = await self.llm_provider.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=self.max_tokens,
|
||||
audit_stage="coding_fact_reconcile",
|
||||
)
|
||||
return self._parse_ops(resp.content)
|
||||
|
||||
@@ -638,7 +637,6 @@ class WorkingFactReconciler:
|
||||
resp = await self.llm_provider.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=_CHAIN_SUMMARY_MAX_TOKENS,
|
||||
audit_stage="coding_fact_summary",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"[fact-reconcile] chain summary failed: {e}")
|
||||
|
||||
@@ -29,9 +29,6 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
_JUDGE_MAX_TURNS = 20
|
||||
_JUDGE_HEAD_TURNS = 5
|
||||
_JUDGE_USER_CONTENT_LIMIT = 1_000
|
||||
_JUDGE_USER_CONTENT_HEAD = 200
|
||||
_JUDGE_USER_CONTENT_TAIL = 200
|
||||
_JUDGE_MAX_QUERY_CHARS = 300
|
||||
_JUDGE_MAX_QUERIES = 20
|
||||
|
||||
@@ -189,7 +186,6 @@ async def classify_messages_scene(
|
||||
prompt=prompt,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
audit_stage="coding_judge",
|
||||
)
|
||||
_elapsed = round((time.perf_counter() - _t0) * 1000, 2)
|
||||
result = _parse_scene_json(resp.content)
|
||||
@@ -635,13 +631,9 @@ def _format_turns(summary: List[Dict[str, Any]]) -> str:
|
||||
rows = []
|
||||
for t in summary:
|
||||
user = t.get("user", "") or ""
|
||||
if len(user) > _JUDGE_USER_CONTENT_LIMIT:
|
||||
omitted = len(user) - _JUDGE_USER_CONTENT_HEAD - _JUDGE_USER_CONTENT_TAIL
|
||||
user = (
|
||||
user[:_JUDGE_USER_CONTENT_HEAD]
|
||||
+ f"\n[...{omitted} chars omitted...]\n"
|
||||
+ user[-_JUDGE_USER_CONTENT_TAIL:]
|
||||
)
|
||||
# 单 turn 用户文本截到 200 字符
|
||||
if len(user) > 200:
|
||||
user = user[:200] + "..."
|
||||
tools = t.get("tools", []) or []
|
||||
rows.append(
|
||||
f"- turn {t.get('turn', '?')}: user={user!r}, tools={tools}"
|
||||
|
||||
@@ -310,7 +310,6 @@ class CodingMemoryReconciler:
|
||||
resp = await self.llm_provider.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=self.max_tokens,
|
||||
audit_stage="coding_reconcile",
|
||||
)
|
||||
return self._parse_ops(resp.content), prompt, resp.content, resp
|
||||
|
||||
|
||||
+1
-8
@@ -6,7 +6,7 @@ Agent Memory - 配置管理
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional, Dict, Any, Callable, List
|
||||
from typing import Optional, Dict, Any, List
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -1252,13 +1252,6 @@ class MemoryConfig:
|
||||
enable_agent: bool = False
|
||||
debug: bool = False
|
||||
metrics_enabled: bool = True
|
||||
# Runtime-only HTTP transport factory. MemHub uses this for request-cost
|
||||
# budgeting without restricting the embedded SDK's local store access.
|
||||
http_transport_factory: Optional[Callable[[], Any]] = field(
|
||||
default=None,
|
||||
repr=False,
|
||||
compare=False,
|
||||
)
|
||||
|
||||
def __post_init__(self):
|
||||
# enable_graph 跟随 mode 走:mode=ultra 时自动启用,无需独立环境变量
|
||||
|
||||
@@ -356,21 +356,11 @@ class EmbedService:
|
||||
key = self._client_cache_key(base_url, api_key, timeout)
|
||||
client = self._client_cache.get(key)
|
||||
if client is None:
|
||||
client_options = {
|
||||
"api_key": api_key,
|
||||
"base_url": base_url or None,
|
||||
"timeout": timeout,
|
||||
"max_retries": 0,
|
||||
}
|
||||
transport_factory = getattr(self.config, "http_transport_factory", None)
|
||||
if callable(transport_factory):
|
||||
import httpx
|
||||
|
||||
client_options["http_client"] = httpx.AsyncClient(
|
||||
transport=transport_factory()
|
||||
)
|
||||
client = AsyncOpenAI(
|
||||
**client_options,
|
||||
api_key=api_key,
|
||||
base_url=base_url or None,
|
||||
timeout=timeout,
|
||||
max_retries=0,
|
||||
)
|
||||
self._client_cache[key] = client
|
||||
return client
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
"""Context propagated to host-provided HTTP transports for cost accounting."""
|
||||
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar, Token
|
||||
|
||||
_model_call_stage: ContextVar[str] = ContextVar(
|
||||
"hy_memory_model_call_stage",
|
||||
default="other",
|
||||
)
|
||||
|
||||
|
||||
def current_model_call_stage() -> str:
|
||||
return _model_call_stage.get()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def model_call_stage(stage: str | None) -> Iterator[None]:
|
||||
normalized = (stage or "other").strip().lower() or "other"
|
||||
token: Token[str] = _model_call_stage.set(normalized)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_model_call_stage.reset(token)
|
||||
@@ -80,11 +80,7 @@ class PipelineLogWriter:
|
||||
split_namespace_user,
|
||||
)
|
||||
|
||||
# Providers define total_tokens as prompt+completion. Adding all
|
||||
# three fields doubled every cost estimate in this JSONL.
|
||||
tokens = (total_tokens or 0) or (
|
||||
(prompt_tokens or 0) + (completion_tokens or 0)
|
||||
)
|
||||
tokens = (prompt_tokens or 0) + (completion_tokens or 0) + (total_tokens or 0)
|
||||
namespace, uid = split_namespace_user(user_id or "")
|
||||
if not is_llm_content_logging_enabled():
|
||||
prompt = redact_content(prompt)
|
||||
|
||||
+7
-10
@@ -15,12 +15,12 @@ write、search,并把搜索结果格式化为可交给模型的 memory text ch
|
||||
|
||||
```text
|
||||
distribution: hy-memory
|
||||
version: 2.0.3
|
||||
version: 2.0.2
|
||||
mode: 按运行时配置选择
|
||||
reader: hybrid
|
||||
```
|
||||
|
||||
仓库 pin、lockfile 和运行时版本守卫均固定到公开的 `hy-memory==2.0.3`。
|
||||
仓库 pin、lockfile 和运行时版本守卫均固定到公开的 `hy-memory==2.0.2`。
|
||||
MemHub 在本地 Runtime 内嵌入 SDK,但不绕过自身的来源授权、模型授权、Evidence、
|
||||
Governance 和撤权边界。
|
||||
|
||||
@@ -305,9 +305,9 @@ Runtime 应长期复用 client,不为每次请求重新初始化数据库和
|
||||
|
||||
```text
|
||||
Evidence
|
||||
-> HyMemoryBridge
|
||||
-> embedded Hy Memory SDK
|
||||
-> native Vector / Coding / Graph stores
|
||||
-> ModelProcessingPermissionLease
|
||||
-> HyPermissionTransport
|
||||
-> HyMemoryBridge.ingest / ingest_extracted / search_ids
|
||||
-> write result verification
|
||||
-> Governance
|
||||
-> Recall Broker / UI
|
||||
@@ -316,11 +316,8 @@ Evidence
|
||||
不要在新业务模块里直接创建 `HyMemoryClient`,否则会绕过:
|
||||
|
||||
- 来源读取授权;
|
||||
- 共享 Runtime、模型配置、API Key 安全存储与请求预算;
|
||||
- 模型内容处理授权;
|
||||
- 发送前与响应后的撤权复核;
|
||||
- Evidence 引用与作者归属;
|
||||
- 写后回读验证;
|
||||
- Governance 状态、版本、更正和删除级联。
|
||||
|
||||
Hy Memory SDK 是 MemHub 的内置可信组件,可自由访问和管理本机的全部
|
||||
Hy Memory 数据。来源授权仍决定哪些宿主数据可以进入本机;宿主 Recall Grant
|
||||
仍决定哪些外部 AI 应用可以读取治理后的记忆。这两类边界不限制内置 SDK 的本地 CRUD。
|
||||
+4
-6
@@ -6,13 +6,13 @@ SDK 和记忆引擎。One Memory / `one-memory` 是历史模块、数据目录
|
||||
|
||||
## 发行与平台合同
|
||||
|
||||
公共 Python SDK 为 `hy-memory==2.0.3`。MemHub 源码版本为 `1.0.3`
|
||||
(build 7),GitHub 仓库为
|
||||
公共 Python SDK 为 `hy-memory==2.0.2`。MemHub 源码版本为 `1.0.1`
|
||||
(build 5),GitHub 仓库为
|
||||
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
|
||||
`memhub-cli`。当前代码架构见
|
||||
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
|
||||
|
||||
`v1.0.3` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
|
||||
`v1.0.1` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
|
||||
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
|
||||
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
|
||||
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
|
||||
@@ -42,8 +42,6 @@ Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命
|
||||
|
||||
架构详情见 [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) 和
|
||||
[`docs/contracts/runtime-unification.md`](docs/contracts/runtime-unification.md)。
|
||||
产品、测试和一线支持排查导入与记忆沉淀时,使用
|
||||
[`docs/PRODUCT_LOG_TROUBLESHOOTING.md`](docs/PRODUCT_LOG_TROUBLESHOOTING.md)。
|
||||
|
||||
## 开发环境
|
||||
|
||||
@@ -101,7 +99,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
|
||||
|
||||
## Hy Memory
|
||||
|
||||
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.3`。SDK 配置、权限边界及 search
|
||||
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.2`。SDK 配置、权限边界及 search
|
||||
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
|
||||
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@
|
||||
"dataLicense": "CC0-1.0",
|
||||
"SPDXID": "SPDXRef-DOCUMENT",
|
||||
"name": "one-memory",
|
||||
"documentNamespace": "https://one-memory.local/spdx/2026-10-17T13:52:48Z",
|
||||
"documentNamespace": "https://one-memory.local/spdx/2026-10-15T17:26:47Z",
|
||||
"creationInfo": {
|
||||
"created": "2026-10-17T13:52:48Z",
|
||||
"created": "2026-10-15T17:26:47Z",
|
||||
"creators": [
|
||||
"Tool: one-memory-generate-sbom"
|
||||
]
|
||||
@@ -2335,7 +2335,7 @@
|
||||
{
|
||||
"SPDXID": "SPDXRef-Package-hy-memory",
|
||||
"name": "hy-memory",
|
||||
"versionInfo": "2.0.4",
|
||||
"versionInfo": "2.0.2",
|
||||
"downloadLocation": "NOASSERTION",
|
||||
"filesAnalyzed": false,
|
||||
"licenseConcluded": "Apache-2.0",
|
||||
|
||||
@@ -2,14 +2,14 @@
|
||||
|
||||
MemHub itself is licensed under Apache-2.0 (see `LICENSE`).
|
||||
It bundles the runtime dependencies listed below.
|
||||
Generated from the installed dependency tree on 2026-10-17T13:52:48Z
|
||||
Generated from the installed dependency tree on 2026-10-15T17:26:47Z
|
||||
by `python3 scripts/generate_sbom.py`. Do not edit by hand.
|
||||
|
||||
Per-package license texts are not reproduced here yet; the license
|
||||
column is filled in as part of release review, and the SBOM records
|
||||
`NOASSERTION` until then rather than guessing.
|
||||
|
||||
## hy-memory 2.0.4
|
||||
## hy-memory 2.0.2
|
||||
|
||||
License: Apache-2.0
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@one-memory/cli",
|
||||
"version": "1.0.4",
|
||||
"version": "1.0.1",
|
||||
"description": "MemHub bootstrap CLI for installing and managing the local-first shared memory runtime",
|
||||
"license": "Apache-2.0",
|
||||
"repository": {
|
||||
|
||||
@@ -32,6 +32,7 @@ const REQUIRED_COMMAND_SURFACE = [
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"models",
|
||||
@@ -76,7 +77,7 @@ const REQUIRED_MARKERS = [
|
||||
|
||||
/**
|
||||
* Sub-commands the UI-write parity work added. A package can declare the
|
||||
* parent command (`models`) while still missing the new verb, so
|
||||
* parent command (`permission`, `models`) while still missing the new verb, so
|
||||
* checking command names alone would pass a stale bundle. The marker is the
|
||||
* exact literal the catalog usage string contains.
|
||||
*/
|
||||
@@ -87,6 +88,9 @@ const REQUIRED_SUBCOMMANDS = [
|
||||
{ command: "import-file", marker: "memhub import-file discard" },
|
||||
{ command: "notes", marker: "memhub notes add" },
|
||||
{ command: "notes", marker: "memhub notes process" },
|
||||
{ command: "permission", marker: "memhub permission status" },
|
||||
{ command: "permission", marker: "memhub permission allow" },
|
||||
{ command: "permission", marker: "memhub permission deny" },
|
||||
{ command: "extraction", marker: "memhub extraction set" },
|
||||
{ command: "graph", marker: "memhub graph width" },
|
||||
{ command: "models", marker: "memhub models edit" },
|
||||
@@ -149,6 +153,8 @@ const STOPPED_DESTRUCTIVE_PROBES = [
|
||||
"--confirm",
|
||||
],
|
||||
},
|
||||
{ name: "permission_allow", args: ["permission", "allow"] },
|
||||
{ name: "permission_deny", args: ["permission", "deny"] },
|
||||
{
|
||||
name: "extraction_set_apply",
|
||||
args: ["extraction", "set", "--window", "6m", "--confirm"],
|
||||
|
||||
+71
-31
@@ -81,7 +81,6 @@ import {
|
||||
readMemoryWrite,
|
||||
readScanRoots,
|
||||
removeScanRoot,
|
||||
resetOnboarding,
|
||||
submitBackup,
|
||||
submitMemoryWrite,
|
||||
submitScan,
|
||||
@@ -191,6 +190,13 @@ import {
|
||||
renderMemoryDetailText,
|
||||
renderMemoryPageText,
|
||||
} from "./memory-report.js";
|
||||
import {
|
||||
type ModelContentPermissionDocument,
|
||||
modelContentPermissionExitCode,
|
||||
readModelContentPermission,
|
||||
renderModelContentPermissionText,
|
||||
setModelContentPermission,
|
||||
} from "./model-content-permission-flow.js";
|
||||
import {
|
||||
executeModelProbe,
|
||||
inspectModelProbePlan,
|
||||
@@ -211,10 +217,12 @@ import {
|
||||
executeModelProfileActivate,
|
||||
executeModelProfileEdit,
|
||||
type ModelWriteDocument,
|
||||
type ModelWritePlan,
|
||||
modelWriteExitCode,
|
||||
planModelProfileActivate,
|
||||
planModelProfileEdit,
|
||||
renderModelWriteText,
|
||||
validateModelWritePlan,
|
||||
} from "./model-write-flows.js";
|
||||
import { claimPlanAttempt } from "./operation-journal.js";
|
||||
import {
|
||||
@@ -689,36 +697,6 @@ async function runCommand(
|
||||
// `path list` and `adapter list` are read-only: they have no plan to confirm,
|
||||
// so they are answered here rather than falling into the two-phase branch
|
||||
// (which would reject them for having nothing to apply).
|
||||
// Back to the cold-start screens on purpose. Deleting the app cannot do this
|
||||
// (the completion flag lives with the memories, by design), and deleting the
|
||||
// data to get there would cost the user everything they have stored.
|
||||
if (named === "onboarding" && arguments_.includes("reset")) {
|
||||
if (
|
||||
arguments_.filter((item) => item === "--json").length > 1 ||
|
||||
arguments_.filter((item) => item !== "--json").join(" ") !==
|
||||
"onboarding reset"
|
||||
) {
|
||||
return usage();
|
||||
}
|
||||
const attachment = await attachToRuntime(
|
||||
resolveRuntimeCliConfig(environment, process.argv[1]).dataDir,
|
||||
);
|
||||
if (attachment === null) {
|
||||
return runtimeUnavailable(descriptor, asJson);
|
||||
}
|
||||
const state = await resetOnboarding(attachment);
|
||||
if (state === null) {
|
||||
return reject(descriptor, asJson, "引导状态端点不可用。");
|
||||
}
|
||||
if (asJson) {
|
||||
process.stdout.write(`${JSON.stringify(state)}\n`);
|
||||
return EXIT_OK;
|
||||
}
|
||||
process.stdout.write(
|
||||
`引导流程已回到首次运行(status=${state.status}),记忆数据未改动。\n`,
|
||||
);
|
||||
return EXIT_OK;
|
||||
}
|
||||
if (named === "path" && arguments_.includes("list")) {
|
||||
if (
|
||||
arguments_.filter((item) => item === "--json").length > 1 ||
|
||||
@@ -2484,6 +2462,45 @@ async function runCommand(
|
||||
);
|
||||
return uiWriteExitCode(document);
|
||||
}
|
||||
if (named === "permission") {
|
||||
const parsed = parsePermissionCommand(arguments_.slice(1));
|
||||
if (!parsed) return usage();
|
||||
const config = resolveRuntimeCliConfig(environment, process.argv[1]);
|
||||
const runtime = await inspectRuntimeStatus({
|
||||
dataDir: config.dataDir,
|
||||
probeIdentity,
|
||||
requireServiceReceipt: true,
|
||||
});
|
||||
const attached =
|
||||
runtime.state === "running"
|
||||
? await attachClients(config.dataDir)
|
||||
: { reader: null, operations: null, failure: runtime.code };
|
||||
if (
|
||||
runtime.state !== "running" ||
|
||||
(parsed === "status" ? !attached.reader : !attached.operations)
|
||||
) {
|
||||
const document: ModelContentPermissionDocument = {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
state: "unknown",
|
||||
code:
|
||||
runtime.state !== "running"
|
||||
? runtime.code
|
||||
: (attached.failure ?? "runtime_attach_failed"),
|
||||
};
|
||||
emitModelContentPermissionDocument(document, asJson);
|
||||
return runtime.state === "stopped" ? 3 : 1;
|
||||
}
|
||||
const document =
|
||||
parsed === "status"
|
||||
? await readModelContentPermission(attached.reader!)
|
||||
: await setModelContentPermission(
|
||||
attached.operations!,
|
||||
parsed === "allow",
|
||||
);
|
||||
emitModelContentPermissionDocument(document, asJson);
|
||||
return modelContentPermissionExitCode(document);
|
||||
}
|
||||
if (named === "notes" || named === "extraction" || named === "graph") {
|
||||
const parsed = parseExtendedWriteCommand(
|
||||
named as "notes" | "extraction" | "graph",
|
||||
@@ -5107,6 +5124,18 @@ function parseHeadlessCommand(
|
||||
: null;
|
||||
}
|
||||
|
||||
function parsePermissionCommand(
|
||||
rest: readonly string[],
|
||||
): "status" | "allow" | "deny" | null {
|
||||
if (rest.filter((item) => item === "--json").length > 1) return null;
|
||||
const tokens = rest.filter((item) => item !== "--json");
|
||||
const action = tokens[0];
|
||||
return tokens.length === 1 &&
|
||||
(action === "status" || action === "allow" || action === "deny")
|
||||
? action
|
||||
: null;
|
||||
}
|
||||
|
||||
function unwrapPlan(value: unknown): unknown {
|
||||
if (
|
||||
value &&
|
||||
@@ -5294,6 +5323,17 @@ function emitModelReportDocument(
|
||||
);
|
||||
}
|
||||
|
||||
function emitModelContentPermissionDocument(
|
||||
document: ModelContentPermissionDocument,
|
||||
asJson: boolean,
|
||||
): void {
|
||||
process.stdout.write(
|
||||
asJson
|
||||
? `${JSON.stringify(document)}\n`
|
||||
: renderModelContentPermissionText(document),
|
||||
);
|
||||
}
|
||||
|
||||
function emitSourceSyncDocument(
|
||||
document: SourceSyncFlowDocument,
|
||||
asJson: boolean,
|
||||
|
||||
@@ -493,6 +493,14 @@ export const COMMANDS: readonly CommandDescriptor[] = [
|
||||
],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
|
||||
},
|
||||
{
|
||||
name: "permission",
|
||||
summary: "读取或直接设置全局模型内容外发许可。",
|
||||
usage:
|
||||
"memhub permission status [--json]\n memhub permission allow [--json]\n memhub permission deny [--json]",
|
||||
options: [JSON_OPTION],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
|
||||
},
|
||||
{
|
||||
name: "extraction",
|
||||
summary: "设置提取窗口;默认只出计划,--confirm 后才写入。",
|
||||
@@ -651,14 +659,6 @@ export const COMMANDS: readonly CommandDescriptor[] = [
|
||||
],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
|
||||
},
|
||||
{
|
||||
name: "onboarding",
|
||||
summary:
|
||||
"onboarding reset 回到首次运行的引导流程。只清除引导完成状态:不删记忆数据,也不需要卸载重装。",
|
||||
usage: "memhub onboarding reset [--json]",
|
||||
options: [JSON_OPTION],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE],
|
||||
},
|
||||
{
|
||||
name: "help",
|
||||
summary: "显示总帮助或单个命令的用法。",
|
||||
|
||||
@@ -81,6 +81,7 @@ const COMMANDS = [
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"host",
|
||||
@@ -92,7 +93,6 @@ const COMMANDS = [
|
||||
"uninstall",
|
||||
"upgrade",
|
||||
"delete-data",
|
||||
"onboarding",
|
||||
"help",
|
||||
"version",
|
||||
// A6 product commands. Appended, not inserted: the catalog is a stable
|
||||
@@ -259,14 +259,7 @@ describe("CLI-22 shared output contract", () => {
|
||||
|
||||
describe("argument validation", () => {
|
||||
it("fails unknown commands without printing data or running anything", async () => {
|
||||
for (const argv of [
|
||||
["unknown"],
|
||||
["unknown", "--help"],
|
||||
["--typo"],
|
||||
["permission", "status"],
|
||||
["permission", "allow"],
|
||||
["permission", "deny"],
|
||||
]) {
|
||||
for (const argv of [["unknown"], ["unknown", "--help"], ["--typo"]]) {
|
||||
const streams = captureStreams();
|
||||
try {
|
||||
expect(await run(argv, {})).toBe(2);
|
||||
|
||||
@@ -384,101 +384,6 @@ describe("unified context search", () => {
|
||||
expect(streams.stdout()).not.toContain("must-not-print");
|
||||
});
|
||||
|
||||
it("counts characters as code points so an emoji hit stays grounded", async () => {
|
||||
// Core reports used_characters with Python's len(), which counts code
|
||||
// points: this text is 10 characters there and 12 UTF-16 units here.
|
||||
const text = "消费保 📍 扩展 📡";
|
||||
expect(text.length).toBe(12);
|
||||
expect(Array.from(text).length).toBe(10);
|
||||
transport.post.mockResolvedValue({
|
||||
status: 200,
|
||||
data: {
|
||||
...RESPONSE,
|
||||
paths: {
|
||||
...RESPONSE.paths,
|
||||
memory: {
|
||||
...RESPONSE.paths.memory,
|
||||
status: "empty",
|
||||
returned_count: 0,
|
||||
},
|
||||
},
|
||||
items: [
|
||||
{
|
||||
...(RESPONSE.items[0] as Record<string, unknown>),
|
||||
messages: [
|
||||
{
|
||||
...(
|
||||
RESPONSE.items[0] as {
|
||||
messages: Array<Record<string, unknown>>;
|
||||
}
|
||||
).messages[0],
|
||||
text,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
used_characters: 10,
|
||||
},
|
||||
});
|
||||
const streams = captureStreams();
|
||||
try {
|
||||
expect(await run(["recall", "test", "--json"])).toBe(0);
|
||||
} finally {
|
||||
streams.restore();
|
||||
}
|
||||
expect(JSON.parse(streams.stdout())).toMatchObject({
|
||||
state: "grounded",
|
||||
used_characters: 10,
|
||||
items: [{ kind: "raw", messages: [{ text }] }],
|
||||
});
|
||||
});
|
||||
|
||||
it("still rejects a response whose character total does not match", async () => {
|
||||
// The same text counted as UTF-16 units is exactly what Core does not mean,
|
||||
// so the cross-check must keep failing instead of accepting both totals.
|
||||
const text = "消费保 📍 扩展 📡";
|
||||
transport.post.mockResolvedValue({
|
||||
status: 200,
|
||||
data: {
|
||||
...RESPONSE,
|
||||
paths: {
|
||||
...RESPONSE.paths,
|
||||
memory: {
|
||||
...RESPONSE.paths.memory,
|
||||
status: "empty",
|
||||
returned_count: 0,
|
||||
},
|
||||
},
|
||||
items: [
|
||||
{
|
||||
...(RESPONSE.items[0] as Record<string, unknown>),
|
||||
messages: [
|
||||
{
|
||||
...(
|
||||
RESPONSE.items[0] as {
|
||||
messages: Array<Record<string, unknown>>;
|
||||
}
|
||||
).messages[0],
|
||||
text,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
used_characters: 12,
|
||||
},
|
||||
});
|
||||
const streams = captureStreams();
|
||||
try {
|
||||
expect(await run(["recall", "test", "--json"])).toBe(1);
|
||||
} finally {
|
||||
streams.restore();
|
||||
}
|
||||
expect(JSON.parse(streams.stdout())).toMatchObject({
|
||||
state: "invalid",
|
||||
code: "response_invalid",
|
||||
});
|
||||
});
|
||||
|
||||
it("reports endpoint denial without exposing the response body", async () => {
|
||||
transport.post.mockResolvedValue({
|
||||
status: 403,
|
||||
|
||||
@@ -181,9 +181,9 @@ function validateContextResponse(
|
||||
items.push(item);
|
||||
characters +=
|
||||
item.kind === "memory"
|
||||
? characterCount(item.content)
|
||||
? String(item.content).length
|
||||
: (item.messages as Array<Record<string, unknown>>).reduce(
|
||||
(sum, message) => sum + characterCount(message.text),
|
||||
(sum, message) => sum + String(message.text).length,
|
||||
0,
|
||||
);
|
||||
}
|
||||
@@ -330,17 +330,6 @@ function record(value: unknown): Record<string, unknown> | null {
|
||||
: null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count characters the way Core does. Python's len() counts Unicode code
|
||||
* points; String.prototype.length counts UTF-16 code units, so an emoji outside
|
||||
* the BMP is one character to Core and two here. Comparing the two totals made
|
||||
* every response containing an emoji fail the used_characters cross-check and
|
||||
* surface as response_invalid.
|
||||
*/
|
||||
function characterCount(value: unknown): number {
|
||||
return Array.from(String(value)).length;
|
||||
}
|
||||
|
||||
function boundedText(
|
||||
value: unknown,
|
||||
maximum: number,
|
||||
|
||||
@@ -8,6 +8,7 @@ import axios from "axios";
|
||||
*/
|
||||
const READ_ONLY_PATHS = [
|
||||
"/api/v1/call-events",
|
||||
"/api/v1/model-content-permission",
|
||||
"/api/v1/home-summary",
|
||||
"/api/v1/import-jobs",
|
||||
"/api/v1/live-sync",
|
||||
@@ -397,41 +398,6 @@ export async function readMemoryWrite(
|
||||
}
|
||||
}
|
||||
|
||||
export interface OnboardingState {
|
||||
status: string;
|
||||
selected_source_ids: string[];
|
||||
started_at: string | null;
|
||||
completed_at: string | null;
|
||||
}
|
||||
|
||||
/** Send onboarding back to its first-run state.
|
||||
*
|
||||
* Returns null when the endpoint is not there, so an older runtime degrades to
|
||||
* "not available" instead of failing the command. Only the completion flag
|
||||
* moves: memories, sources and settings live in other tables, which is what
|
||||
* makes this the honest way to see the cold-start screens again.
|
||||
*/
|
||||
export async function resetOnboarding(
|
||||
attachment: RuntimeAttachmentLike,
|
||||
): Promise<OnboardingState | null> {
|
||||
try {
|
||||
const response = await axios.post<OnboardingState>(
|
||||
`${baseUrlOf(attachment)}/api/v1/onboarding/reset`,
|
||||
undefined,
|
||||
{
|
||||
headers: { Authorization: `Bearer ${attachment.apiToken}` },
|
||||
timeout: 15_000,
|
||||
proxy: false,
|
||||
maxRedirects: 0,
|
||||
maxContentLength: MAX_RESPONSE_BYTES,
|
||||
},
|
||||
);
|
||||
return response.data;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function readScanRoots(
|
||||
attachment: RuntimeAttachmentLike,
|
||||
): Promise<{
|
||||
|
||||
@@ -30,6 +30,7 @@ const IMPORT_FILE =
|
||||
const MEMORY_NOTES_PATH = "/api/v1/memory-notes";
|
||||
const MEMORY_NOTE_PROCESS =
|
||||
/^\/api\/v1\/memory-notes\/[A-Za-z0-9][\w.:-]{0,127}\/process$/;
|
||||
const MODEL_CONTENT_PERMISSION_PATH = "/api/v1/model-content-permission";
|
||||
const EXTRACTION_SETTINGS_PATH = "/api/v1/extraction-settings";
|
||||
const GRAPH_WIDTH_PATH = "/api/v1/graph-width";
|
||||
const MANUAL_EXTRACT_PATH = "/api/v1/memory-processing/manual-extract";
|
||||
@@ -133,6 +134,7 @@ export function createCoreOperations(attachment: {
|
||||
!IMPORT_FILE_COMMIT.test(path) &&
|
||||
path !== MEMORY_NOTES_PATH &&
|
||||
!MEMORY_NOTE_PROCESS.test(path) &&
|
||||
path !== MODEL_CONTENT_PERMISSION_PATH &&
|
||||
path !== GRAPH_WIDTH_PATH &&
|
||||
path !== MANUAL_EXTRACT_PATH &&
|
||||
path !== ONBOARDING_COMPLETE_PATH &&
|
||||
|
||||
@@ -259,7 +259,7 @@ describe("A2 dual-mode init", () => {
|
||||
expect(streams.value()).toContain("处理:已开启");
|
||||
});
|
||||
|
||||
it("continues the TTY wizard through model setup and probe", async () => {
|
||||
it("continues the TTY wizard through model probe and content permission", async () => {
|
||||
const profileId = "22222222-2222-4222-8222-222222222222";
|
||||
const profile = {
|
||||
profile_id: profileId,
|
||||
@@ -361,6 +361,12 @@ describe("A2 dual-mode init", () => {
|
||||
};
|
||||
}
|
||||
if (url.endsWith("/activate")) return { status: 200, data: profile };
|
||||
if (url === "/api/v1/model-content-permission") {
|
||||
return {
|
||||
status: 200,
|
||||
data: { allowed: body.allowed },
|
||||
};
|
||||
}
|
||||
if (url === "/api/v1/onboarding/complete") {
|
||||
return {
|
||||
status: 200,
|
||||
@@ -377,7 +383,7 @@ describe("A2 dual-mode init", () => {
|
||||
},
|
||||
);
|
||||
const prompt = extendedPrompter(
|
||||
["", "", "", "", "", "", "", "n"],
|
||||
["", "", "", "", "", "", "", "", "n"],
|
||||
"sk-never-print-this",
|
||||
);
|
||||
const streams = captureStdout();
|
||||
@@ -396,6 +402,7 @@ describe("A2 dual-mode init", () => {
|
||||
streams.restore();
|
||||
}
|
||||
expect(streams.value()).toContain("已接入来源:codex");
|
||||
expect(prompt.written.join("\n")).toContain("模型内容外发:已允许");
|
||||
expect(streams.value()).toContain("首次设置完成。");
|
||||
expect(`${streams.value()}\n${prompt.written.join("\n")}`).not.toContain(
|
||||
"sk-never-print-this",
|
||||
@@ -405,10 +412,10 @@ describe("A2 dual-mode init", () => {
|
||||
expect.objectContaining({ api_key: "sk-never-print-this" }),
|
||||
undefined,
|
||||
);
|
||||
expect(transport.post).not.toHaveBeenCalledWith(
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
{ allowed: true },
|
||||
undefined,
|
||||
);
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v1/onboarding/complete",
|
||||
|
||||
@@ -132,6 +132,12 @@ function clients(options: { failedProbes?: number } = {}) {
|
||||
if (path.endsWith("/activate")) {
|
||||
return { state: "committed" as const, data: profile(true) };
|
||||
}
|
||||
if (path === "/api/v1/model-content-permission") {
|
||||
return {
|
||||
state: "committed" as const,
|
||||
data: { allowed: body.allowed },
|
||||
};
|
||||
}
|
||||
return {
|
||||
state: "rejected" as const,
|
||||
code: "unexpected_path",
|
||||
@@ -154,8 +160,8 @@ function clients(options: { failedProbes?: number } = {}) {
|
||||
}
|
||||
|
||||
describe("interactive model setup", () => {
|
||||
it("uses App defaults and keeps the key out of output", async () => {
|
||||
const prompt = prompter(["", "", ""]);
|
||||
it("uses App defaults, keeps the key out of output, and allows model content", async () => {
|
||||
const prompt = prompter(["", "", "", "", "", ""]);
|
||||
const api = clients();
|
||||
|
||||
const result = await runInteractiveModelSetup(prompt.value, api, {
|
||||
@@ -164,7 +170,10 @@ describe("interactive model setup", () => {
|
||||
installHosts: false,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({ state: "configured" });
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: true,
|
||||
});
|
||||
expect(api.post).toHaveBeenCalledWith(
|
||||
"/api/v1/model-profiles",
|
||||
expect.objectContaining({
|
||||
@@ -175,10 +184,9 @@ describe("interactive model setup", () => {
|
||||
embedding_provider_kind: "local_onnx",
|
||||
}),
|
||||
);
|
||||
expect(api.post).not.toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
expect.anything(),
|
||||
);
|
||||
expect(api.post).toHaveBeenCalledWith("/api/v1/model-content-permission", {
|
||||
allowed: true,
|
||||
});
|
||||
expect(prompt.written.join("\n")).not.toContain("sk-do-not-print");
|
||||
});
|
||||
|
||||
@@ -197,7 +205,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("removes an unverified profile before skipping after a failed probe", async () => {
|
||||
const prompt = prompter(["", "", "", "0"]);
|
||||
const prompt = prompter(["", "", "", "n", "0"]);
|
||||
const api = clients();
|
||||
api.post.mockImplementation(
|
||||
async (path: string, body: Record<string, unknown>) => {
|
||||
@@ -272,7 +280,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("cleans a failed probe and lets the user re-enter the model settings", async () => {
|
||||
const prompt = prompter(["", "", "", "", "", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", "", "", "", "", "", "", ""]);
|
||||
const api = clients({ failedProbes: 1 });
|
||||
|
||||
const result = await runInteractiveModelSetup(prompt.value, api, {
|
||||
@@ -281,7 +289,10 @@ describe("interactive model setup", () => {
|
||||
installHosts: false,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({ state: "configured" });
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: true,
|
||||
});
|
||||
expect(api.remove).toHaveBeenCalledTimes(1);
|
||||
expect(
|
||||
api.post.mock.calls.filter(([path]) => path === "/api/v1/model-profiles"),
|
||||
@@ -294,7 +305,7 @@ describe("interactive model setup", () => {
|
||||
const prompt = prompter(["", "", ""]);
|
||||
prompt.value.ask = async () => {
|
||||
asked += 1;
|
||||
if (asked === 3) throw new Error("interrupted");
|
||||
if (asked === 4) throw new Error("interrupted");
|
||||
return "";
|
||||
};
|
||||
const api = clients();
|
||||
@@ -311,7 +322,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("installs selected detected hosts through preview and fingerprint apply", async () => {
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -349,6 +360,7 @@ describe("interactive model setup", () => {
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: ["codex"],
|
||||
});
|
||||
expect(read).toHaveBeenNthCalledWith(
|
||||
@@ -361,7 +373,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("lists and installs only hosts that are actually present", async () => {
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -408,7 +420,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("completes the wizard and keeps onboarding reachable when one host install fails", async () => {
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -457,6 +469,7 @@ describe("interactive model setup", () => {
|
||||
// calling the install a failure.
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: ["codex"],
|
||||
});
|
||||
const summary = prompt.written.join("\n");
|
||||
@@ -466,7 +479,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("tells the user to install App-only hosts in the App instead of calling them failed", async () => {
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -493,6 +506,7 @@ describe("interactive model setup", () => {
|
||||
// only the App can install, so nothing counts as "installed".
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: [],
|
||||
});
|
||||
const summary = prompt.written.join("\n");
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
} from "./headless-flows.js";
|
||||
import { approvalFingerprint, listHosts, readHost } from "./host-flow.js";
|
||||
import { parseSelection, type WizardPrompter } from "./init-wizard.js";
|
||||
import { setModelContentPermission } from "./model-content-permission-flow.js";
|
||||
import { executeModelProbe, planModelProbe } from "./model-probe-flow.js";
|
||||
import {
|
||||
executeModelProfileActivate,
|
||||
@@ -121,6 +122,7 @@ export type InteractiveModelSetupState =
|
||||
| {
|
||||
state: "configured";
|
||||
profile: ModelProfile;
|
||||
model_content_allowed: boolean;
|
||||
installed_hosts: string[];
|
||||
}
|
||||
| { state: "skipped" }
|
||||
@@ -287,11 +289,15 @@ async function installHosts(
|
||||
function writeSummary(
|
||||
prompter: WizardPrompter,
|
||||
profile: ModelProfile,
|
||||
modelContentAllowed: boolean,
|
||||
installedHosts: readonly string[],
|
||||
failedHosts: readonly { hostId: string; label: string }[],
|
||||
): void {
|
||||
prompter.write("配置摘要:");
|
||||
prompter.write(` 模型:${profile.name} / ${profile.model}`);
|
||||
prompter.write(
|
||||
` 模型内容外发:${modelContentAllowed ? "已允许" : "已禁止"}`,
|
||||
);
|
||||
prompter.write(
|
||||
` 应用接入:${installedHosts.length > 0 ? installedHosts.join("、") : "已跳过"}`,
|
||||
);
|
||||
@@ -347,6 +353,14 @@ export async function runInteractiveModelSetup(
|
||||
prompter.write("未填写 API Key。输入 back 返回,或重新选择模型服务。");
|
||||
continue;
|
||||
}
|
||||
const permissionAnswer = (
|
||||
await prompter.ask("允许将对话内容发送给模型以整理记忆?[Y/n]: ")
|
||||
)
|
||||
.trim()
|
||||
.toLowerCase();
|
||||
const modelContentAllowed = !["n", "no", "否", "0"].includes(
|
||||
permissionAnswer,
|
||||
);
|
||||
const hostApi = options.hostApi ?? { list: listHosts, read: readHost };
|
||||
const selectedHosts = options.installHosts
|
||||
? await chooseDetectedHosts(prompter, options.hostHome, hostApi)
|
||||
@@ -438,6 +452,13 @@ export async function runInteractiveModelSetup(
|
||||
return { state: "failed", code: "model_activate_failed" };
|
||||
}
|
||||
|
||||
const permission = await setModelContentPermission(
|
||||
clients.operations,
|
||||
modelContentAllowed,
|
||||
);
|
||||
if (permission.state !== "ok") {
|
||||
return { state: "failed", code: "model_content_permission_set_failed" };
|
||||
}
|
||||
const { installed, failed } = await installHosts(
|
||||
options.dataDir,
|
||||
selectedHosts,
|
||||
@@ -446,6 +467,7 @@ export async function runInteractiveModelSetup(
|
||||
writeSummary(
|
||||
prompter,
|
||||
profile,
|
||||
modelContentAllowed,
|
||||
installed,
|
||||
failed.map((hostId) => ({
|
||||
hostId,
|
||||
@@ -455,6 +477,7 @@ export async function runInteractiveModelSetup(
|
||||
return {
|
||||
state: "configured",
|
||||
profile,
|
||||
model_content_allowed: modelContentAllowed,
|
||||
installed_hosts: installed,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -180,8 +180,5 @@ describe("import-job write allow-list", () => {
|
||||
await expect(
|
||||
operations.post("/api/v1/import-jobs/job-1/delete", {}),
|
||||
).rejects.toThrow("core_operation_path_not_allowed");
|
||||
await expect(
|
||||
operations.post("/api/v1/model-content-permission", { allowed: true }),
|
||||
).rejects.toThrow("core_operation_path_not_allowed");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,154 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { run } from "./cli.js";
|
||||
|
||||
const transport = vi.hoisted(() => ({
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
put: vi.fn(),
|
||||
patch: vi.fn(),
|
||||
delete: vi.fn(),
|
||||
}));
|
||||
vi.mock("axios", () => ({
|
||||
default: { create: vi.fn(() => transport), get: vi.fn() },
|
||||
}));
|
||||
|
||||
const runtime = vi.hoisted(() => ({
|
||||
inspectRuntimeStatus: vi.fn(),
|
||||
discoverRuntimeAttachment: vi.fn(),
|
||||
}));
|
||||
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
|
||||
...runtime,
|
||||
}));
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
runtime.inspectRuntimeStatus.mockResolvedValue({
|
||||
state: "running",
|
||||
code: "runtime_verified",
|
||||
instanceId: "instance-1",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
version: "1.2.3",
|
||||
});
|
||||
runtime.discoverRuntimeAttachment.mockResolvedValue({
|
||||
instanceId: "instance-1",
|
||||
apiToken: "synthetic-runtime-token",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
servicePid: 4242,
|
||||
health: {
|
||||
status: "ok",
|
||||
service: "one-memory-core",
|
||||
version: "1.2.3",
|
||||
instance_id: "instance-1",
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
describe("model content permission CLI", () => {
|
||||
it("documents only final-state permission and manual extraction commands", async () => {
|
||||
const permission = await capturedRun(["help", "permission"]);
|
||||
expect(permission.code).toBe(0);
|
||||
expect(permission.stdout).toContain("memhub permission allow");
|
||||
expect(permission.stdout).toContain("memhub permission deny");
|
||||
expect(permission.stdout).not.toMatch(
|
||||
/\b(preview|confirm|process|revoke)\b/,
|
||||
);
|
||||
expect(permission.stdout).not.toContain("revision");
|
||||
expect(permission.stdout).not.toContain("receipt");
|
||||
|
||||
const processing = await capturedRun(["help", "processing"]);
|
||||
expect(processing.code).toBe(0);
|
||||
expect(processing.stdout).toContain("memhub processing extract");
|
||||
expect(processing.stdout).not.toMatch(/\b(plan|apply|receipt)\b/);
|
||||
expect(processing.stdout).not.toContain("backlog");
|
||||
});
|
||||
|
||||
it("reads the final state", async () => {
|
||||
transport.get.mockResolvedValue({ status: 200, data: { allowed: true } });
|
||||
|
||||
const result = await capturedRun(["permission", "status", "--json"]);
|
||||
|
||||
expect(result.code).toBe(0);
|
||||
expect(JSON.parse(result.stdout)).toMatchObject({
|
||||
state: "ok",
|
||||
allowed: true,
|
||||
});
|
||||
expect(transport.get).toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["allow", true],
|
||||
["deny", false],
|
||||
] as const)("sets %s directly with only allowed", async (verb, allowed) => {
|
||||
transport.post.mockResolvedValue({ status: 200, data: { allowed } });
|
||||
|
||||
const result = await capturedRun(["permission", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(0);
|
||||
expect(JSON.parse(result.stdout)).toMatchObject({ state: "ok", allowed });
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
{ allowed },
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
|
||||
it.each(["preview", "confirm", "process", "revoke"])(
|
||||
"rejects removed %s commands before runtime access",
|
||||
async (verb) => {
|
||||
const result = await capturedRun(["permission", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(2);
|
||||
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
|
||||
expect(transport.get).not.toHaveBeenCalled();
|
||||
expect(transport.post).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["plan", "apply", "receipt"])(
|
||||
"rejects removed processing %s commands before runtime access",
|
||||
async (verb) => {
|
||||
const result = await capturedRun(["processing", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(2);
|
||||
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
|
||||
expect(transport.get).not.toHaveBeenCalled();
|
||||
expect(transport.post).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
async function capturedRun(arguments_: string[]) {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const stdoutSpy = vi
|
||||
.spyOn(process.stdout, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stdout += String(chunk);
|
||||
return true;
|
||||
});
|
||||
const stderrSpy = vi
|
||||
.spyOn(process.stderr, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stderr += String(chunk);
|
||||
return true;
|
||||
});
|
||||
try {
|
||||
return {
|
||||
code: await run(arguments_, {
|
||||
ONE_MEMORY_DATA_DIR: "/synthetic/data",
|
||||
}),
|
||||
stdout,
|
||||
stderr,
|
||||
};
|
||||
} finally {
|
||||
stdoutSpy.mockRestore();
|
||||
stderrSpy.mockRestore();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import type { CoreReader } from "./core-client.js";
|
||||
import type { CoreOperations } from "./core-operations.js";
|
||||
import {
|
||||
readModelContentPermission,
|
||||
setModelContentPermission,
|
||||
} from "./model-content-permission-flow.js";
|
||||
|
||||
describe("model content permission flow", () => {
|
||||
it("reads the single final-state boolean", async () => {
|
||||
const reader = vi.fn().mockResolvedValue({
|
||||
state: "ok",
|
||||
data: { allowed: true },
|
||||
});
|
||||
|
||||
await expect(
|
||||
readModelContentPermission(reader as CoreReader),
|
||||
).resolves.toMatchObject({ state: "ok", allowed: true });
|
||||
expect(reader).toHaveBeenCalledWith("/api/v1/model-content-permission");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["allow", true],
|
||||
["deny", false],
|
||||
] as const)(
|
||||
"sets %s directly without revision or receipt fields",
|
||||
async (_, allowed) => {
|
||||
const post = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed },
|
||||
});
|
||||
|
||||
await expect(
|
||||
setModelContentPermission({ post } as CoreOperations, allowed),
|
||||
).resolves.toMatchObject({ state: "ok", allowed });
|
||||
expect(post).toHaveBeenCalledWith("/api/v1/model-content-permission", {
|
||||
allowed,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("does not accept a malformed or mismatched success response", async () => {
|
||||
const malformed = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed: true, revision: 1 },
|
||||
});
|
||||
await expect(
|
||||
setModelContentPermission({ post: malformed } as CoreOperations, true),
|
||||
).resolves.toMatchObject({
|
||||
state: "unknown",
|
||||
code: "model_content_permission_response_invalid",
|
||||
});
|
||||
|
||||
const mismatched = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed: false },
|
||||
});
|
||||
await expect(
|
||||
setModelContentPermission({ post: mismatched } as CoreOperations, true),
|
||||
).resolves.toMatchObject({
|
||||
state: "unknown",
|
||||
code: "model_content_permission_state_mismatch",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
import type { CoreReader } from "./core-client.js";
|
||||
import type { CoreOperations } from "./core-operations.js";
|
||||
import { PRODUCT } from "./version.js";
|
||||
|
||||
export type ModelContentPermissionDocument =
|
||||
| {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
state: "ok";
|
||||
allowed: boolean;
|
||||
}
|
||||
| {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
state: "rejected" | "unknown";
|
||||
code: string;
|
||||
};
|
||||
|
||||
export async function readModelContentPermission(
|
||||
reader: CoreReader,
|
||||
): Promise<ModelContentPermissionDocument> {
|
||||
const outcome = await reader<unknown>("/api/v1/model-content-permission");
|
||||
if (outcome.state !== "ok") {
|
||||
return failure("unknown", outcome.code);
|
||||
}
|
||||
return permissionState(
|
||||
outcome.data,
|
||||
"model_content_permission_response_invalid",
|
||||
);
|
||||
}
|
||||
|
||||
export async function setModelContentPermission(
|
||||
operations: CoreOperations,
|
||||
allowed: boolean,
|
||||
): Promise<ModelContentPermissionDocument> {
|
||||
const outcome = await operations.post<unknown>(
|
||||
"/api/v1/model-content-permission",
|
||||
{ allowed },
|
||||
);
|
||||
if (outcome.state === "rejected") {
|
||||
return failure("rejected", outcome.code);
|
||||
}
|
||||
if (outcome.state === "unknown") {
|
||||
return failure("unknown", outcome.code);
|
||||
}
|
||||
const document = permissionState(
|
||||
outcome.data,
|
||||
"model_content_permission_response_invalid",
|
||||
);
|
||||
return document.state === "ok" && document.allowed !== allowed
|
||||
? failure("unknown", "model_content_permission_state_mismatch")
|
||||
: document;
|
||||
}
|
||||
|
||||
export function modelContentPermissionExitCode(
|
||||
document: ModelContentPermissionDocument,
|
||||
): number {
|
||||
return document.state === "ok" ? 0 : 1;
|
||||
}
|
||||
|
||||
export function renderModelContentPermissionText(
|
||||
document: ModelContentPermissionDocument,
|
||||
): string {
|
||||
if (document.state === "ok") {
|
||||
return `模型内容外发:${document.allowed ? "允许" : "禁止"}\n`;
|
||||
}
|
||||
return `模型内容权限未更新:${document.state}(${document.code})\n`;
|
||||
}
|
||||
|
||||
function permissionState(
|
||||
value: unknown,
|
||||
invalidCode: string,
|
||||
): ModelContentPermissionDocument {
|
||||
if (
|
||||
value === null ||
|
||||
typeof value !== "object" ||
|
||||
Array.isArray(value) ||
|
||||
Object.keys(value).length !== 1 ||
|
||||
typeof (value as Record<string, unknown>).allowed !== "boolean"
|
||||
) {
|
||||
return failure("unknown", invalidCode);
|
||||
}
|
||||
return {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
state: "ok",
|
||||
allowed: (value as { allowed: boolean }).allowed,
|
||||
};
|
||||
}
|
||||
|
||||
function failure(
|
||||
state: "rejected" | "unknown",
|
||||
code: string,
|
||||
): ModelContentPermissionDocument {
|
||||
return { schema_version: 1, product: PRODUCT, state, code };
|
||||
}
|
||||
@@ -137,8 +137,8 @@ describe("package matrix harness", () => {
|
||||
};
|
||||
expect(document.state).toBe("verified");
|
||||
const results = document.commands.stopped_destructive.results;
|
||||
// 11 destructive commands, each unknown/runtime_not_running with exit 3.
|
||||
expect(Object.keys(results)).toHaveLength(11);
|
||||
// 13 destructive commands, each unknown/runtime_not_running with exit 3.
|
||||
expect(Object.keys(results)).toHaveLength(13);
|
||||
for (const [name, result] of Object.entries(results)) {
|
||||
expect(result, `${name} did not fail closed`).toEqual({
|
||||
exit: 3,
|
||||
@@ -162,7 +162,7 @@ describe("package matrix harness", () => {
|
||||
const shim = [
|
||||
"#!/usr/bin/env node",
|
||||
"const a=process.argv.slice(2);",
|
||||
"if(['calls','import-file','notes','extraction','graph','models'].includes(a[0])||(a[0]==='processing'&&a[1]==='extract')){",
|
||||
"if(['calls','import-file','notes','permission','extraction','graph','models'].includes(a[0])||(a[0]==='processing'&&a[1]==='extract')){",
|
||||
' process.stdout.write(\'{"schema_version":1,"product":"memhub","state":"committed","operation_id":"x","receipt":{}}\\n\');',
|
||||
" process.exit(0);",
|
||||
"}",
|
||||
@@ -188,7 +188,7 @@ describe("package matrix harness", () => {
|
||||
expect(failure?.state).toBe("failed");
|
||||
expect(
|
||||
failure?.failures.filter((item) => item.includes("did not fail closed")),
|
||||
).toHaveLength(11);
|
||||
).toHaveLength(13);
|
||||
});
|
||||
|
||||
it("can require an expected bundle SHA and reports bundle metadata", {
|
||||
@@ -280,8 +280,9 @@ describe("package matrix harness", () => {
|
||||
});
|
||||
|
||||
it("rejects a catalog that declares the parent command but not the new verb", async () => {
|
||||
// A bundle can still declare `models` while missing `models activate`.
|
||||
// Names alone would pass it; the usage marker must not.
|
||||
// A bundle can still declare `permission`/`models` while missing
|
||||
// `permission allow` or `models activate`. Names alone would pass it; the
|
||||
// usage marker must not.
|
||||
const declared = [
|
||||
"install",
|
||||
"init",
|
||||
@@ -291,6 +292,7 @@ describe("package matrix harness", () => {
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"models",
|
||||
@@ -340,6 +342,9 @@ describe("package matrix harness", () => {
|
||||
},
|
||||
};
|
||||
const failures = harness.validateCatalog(catalog);
|
||||
expect(failures).toContain(
|
||||
'command "permission" is missing usage "memhub permission allow"',
|
||||
);
|
||||
expect(failures).toContain(
|
||||
'command "models" is missing usage "memhub models activate"',
|
||||
);
|
||||
|
||||
@@ -151,6 +151,9 @@ describe("memhub short-command contract", () => {
|
||||
"memhub import-file discard",
|
||||
"memhub notes add",
|
||||
"memhub notes process",
|
||||
"memhub permission status",
|
||||
"memhub permission allow",
|
||||
"memhub permission deny",
|
||||
"memhub models setup",
|
||||
"memhub models activate",
|
||||
"memhub models edit",
|
||||
@@ -160,9 +163,6 @@ describe("memhub short-command contract", () => {
|
||||
]) {
|
||||
expect(flattened).toContain(shortForm);
|
||||
}
|
||||
expect(COMMANDS.some((command) => command.name === "permission")).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("ships a memhub bin entry pointing at a real bundle path", () => {
|
||||
|
||||
@@ -840,7 +840,6 @@ describe("CLI-21 memory reads", () => {
|
||||
expect(isReadOnlyPath("/api/v1/memories/graph")).toBe(false);
|
||||
expect(isReadOnlyPath("/api/v1/memories/ask")).toBe(false);
|
||||
expect(isReadOnlyPath("/api/v1/memories/m1")).toBe(true);
|
||||
expect(isReadOnlyPath("/api/v1/model-content-permission")).toBe(false);
|
||||
expect(
|
||||
isReadOnlyPath(
|
||||
"/api/v1/memory-mutations/123e4567-e89b-12d3-a456-426614174000",
|
||||
|
||||
@@ -11,7 +11,7 @@ folder_layout = layout == 'folder'
|
||||
datas = [('src/one_memory/embedding_assets', 'one_memory/embedding_assets')]
|
||||
binaries = []
|
||||
# Loaded lazily by the API; static dependency analysis cannot discover it.
|
||||
hiddenimports = []
|
||||
hiddenimports = ['one_memory.hy_permission_context']
|
||||
tmp_ret = collect_all('hy_memory')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('chromadb')
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "uv_build"
|
||||
|
||||
[project]
|
||||
name = "one-memory"
|
||||
version = "1.0.4"
|
||||
version = "1.0.1"
|
||||
description = "MemHub local-first memory core"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
@@ -12,7 +12,7 @@ dependencies = [
|
||||
"fastapi>=0.141.1",
|
||||
"filelock>=3.32.4",
|
||||
# Public SDK version validated against MemHub's permission and recall contracts.
|
||||
"hy-memory==2.0.4",
|
||||
"hy-memory==2.0.2",
|
||||
# Retrieval's derived vector index directly consumes these locked versions.
|
||||
"chromadb==1.5.9",
|
||||
"numpy==2.5.2",
|
||||
|
||||
@@ -20,6 +20,7 @@ from copy import copy
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from enum import StrEnum
|
||||
from functools import partial
|
||||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, cast
|
||||
from uuid import UUID
|
||||
@@ -100,13 +101,7 @@ from one_memory.imports.access import (
|
||||
SourceAccessError,
|
||||
root_identity_matches,
|
||||
)
|
||||
from one_memory.imports.adapters import (
|
||||
ScanFailure,
|
||||
ScannedEvidence,
|
||||
ScanOptions,
|
||||
SourceAdapter,
|
||||
SyncCompleteness,
|
||||
)
|
||||
from one_memory.imports.adapters import SourceAdapter, SyncCompleteness
|
||||
from one_memory.imports.candidates import (
|
||||
SourceCandidateError,
|
||||
SourceCandidateSelection,
|
||||
@@ -117,11 +112,7 @@ from one_memory.imports.connections import (
|
||||
ConnectionIntentReceipt,
|
||||
SourceConnectionStore,
|
||||
)
|
||||
from one_memory.imports.grants import (
|
||||
SourceAuthorizationState,
|
||||
SourceGrantReceipt,
|
||||
SourceGrantStore,
|
||||
)
|
||||
from one_memory.imports.grants import SourceGrantReceipt, SourceGrantStore
|
||||
from one_memory.imports.host_packages import CapabilityState, HostPackageCapabilities
|
||||
from one_memory.imports.jobs import (
|
||||
ImportIssueRecord,
|
||||
@@ -173,6 +164,7 @@ from one_memory.memory_library import (
|
||||
)
|
||||
from one_memory.memory_notes import MemoryNoteStore, memory_note_router
|
||||
from one_memory.memory_processing import (
|
||||
MemoryBridgeProtocol,
|
||||
MemoryProcessingSupervisor,
|
||||
_conversation_chunks,
|
||||
memory_task_timeout_seconds,
|
||||
@@ -186,8 +178,10 @@ from one_memory.memory_queue import (
|
||||
)
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
from one_memory.memory_tasks import MemoryTask, MemoryTaskStatus, MemoryTaskStore
|
||||
from one_memory.model_content_routes import model_content_router
|
||||
from one_memory.model_presets import seed_preset_profile
|
||||
from one_memory.model_probe import ModelProber, ModelProbeResult
|
||||
from one_memory.model_processing_permission_guard import ModelProcessingPermissionLease
|
||||
from one_memory.model_profiles import (
|
||||
ActiveModelProfileError,
|
||||
ModelCreationOperationConflictError,
|
||||
@@ -200,7 +194,6 @@ from one_memory.model_profiles import (
|
||||
ModelProfileStore,
|
||||
ModelProfileUnavailableError,
|
||||
)
|
||||
from one_memory.model_request_audit import AuditedModelTransport, ModelRequestAuditStore
|
||||
from one_memory.model_request_budget import BudgetedModelTransport, ModelRequestBudget
|
||||
from one_memory.onboarding import OnboardingState, OnboardingStore
|
||||
from one_memory.recall import GovernedRecallTarget, RecallReceipt, RecallRequest
|
||||
@@ -292,9 +285,6 @@ class SourceResponse(BaseModel):
|
||||
candidate_revision: int = 0
|
||||
candidate_change_allowed: bool = False
|
||||
candidate_change_reason: str | None = None
|
||||
#: Why `authorized` is false, so the UI can ask for the one thing that
|
||||
#: would fix it instead of offering every recovery at once.
|
||||
authorization_state: SourceAuthorizationState = SourceAuthorizationState.OK
|
||||
|
||||
|
||||
class PreviewSourceCandidateRequest(BaseModel):
|
||||
@@ -327,22 +317,6 @@ class SourceSyncState(BaseModel):
|
||||
sync_enabled: bool
|
||||
|
||||
|
||||
class SedimentationRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
enabled: StrictBool
|
||||
|
||||
|
||||
class SedimentationState(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
enabled: bool
|
||||
#: What the switch means for the model bill and for what still runs. Read
|
||||
#: by the app so the setting can say so without hardcoding the wording.
|
||||
pending_retained: bool = True
|
||||
imports_unaffected: bool = True
|
||||
|
||||
|
||||
class DetectionResponse(BaseModel):
|
||||
source_id: str
|
||||
detected: bool
|
||||
@@ -369,7 +343,7 @@ class ConnectSourcesRequest(BaseModel):
|
||||
import_upload_id: UUID | None = None
|
||||
|
||||
|
||||
ONBOARDING_LARGE_HISTORY_USER_TURNS = 200
|
||||
ONBOARDING_LARGE_HISTORY_SESSIONS = 200
|
||||
ONBOARDING_HISTORY_WINDOW = timedelta(days=3)
|
||||
|
||||
|
||||
@@ -844,13 +818,6 @@ def create_app(
|
||||
evidence=app.state.evidence_store,
|
||||
)
|
||||
app.state.memory_write_queue = memory_write_queue
|
||||
# Builds through 1.0.2 parked pre-invocation jobs behind the
|
||||
# retired model-content consent bit. The embedded SDK is trusted
|
||||
# now, so make those durable jobs claimable on the first startup.
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.readmit_parked,
|
||||
states=("waiting_for_permission",),
|
||||
)
|
||||
repaired_identity_failures = await asyncio.to_thread(
|
||||
memory_write_queue.repair_pre_send_identity_failures
|
||||
)
|
||||
@@ -916,7 +883,6 @@ def create_app(
|
||||
)
|
||||
app.state.import_job_store = ImportJobStore(database_path)
|
||||
app.state.onboarding_store = OnboardingStore(database_path)
|
||||
app.state.history_import_gate_released = False
|
||||
app.state.extraction_settings_store = ExtractionSettingsStore(database_path)
|
||||
app.state.source_grant_store = SourceGrantStore(
|
||||
database_path,
|
||||
@@ -939,7 +905,6 @@ def create_app(
|
||||
app.state.source_permission_locks = {
|
||||
source_id: asyncio.Lock() for source_id in app.state.source_catalog
|
||||
}
|
||||
app.state.sedimentation_lock = asyncio.Lock()
|
||||
_follow_workbuddy_home(app, runtime_settings.source_home)
|
||||
app.state.source_detection = {}
|
||||
app.state.secret_store = secret_store or KeyringSecretStore()
|
||||
@@ -995,7 +960,7 @@ def create_app(
|
||||
# Not fatal: without the tables every read checks exactly, as
|
||||
# it does today.
|
||||
app.state.memory_visibility_error = str(error)
|
||||
model_request_budget = (
|
||||
app.state.model_request_budget = (
|
||||
ModelRequestBudget(
|
||||
database_path,
|
||||
request_limit=runtime_settings.model_request_limit,
|
||||
@@ -1005,8 +970,6 @@ def create_app(
|
||||
if runtime_settings.model_request_limit is not None
|
||||
else None
|
||||
)
|
||||
app.state.model_request_budget = model_request_budget
|
||||
app.state.model_request_audit_store = ModelRequestAuditStore(database_path)
|
||||
app.state.memory_answerer = MemoryAnswerer(
|
||||
governance=app.state.memory_governance_store,
|
||||
filter_memories=app.state.source_read_access.memories,
|
||||
@@ -1045,23 +1008,6 @@ def create_app(
|
||||
governance=app.state.memory_governance_store,
|
||||
data_dir=runtime_settings.data_dir,
|
||||
extraction_settings=app.state.extraction_settings_store,
|
||||
bridge_factory=lambda profile, api_key, data_dir: HyMemoryBridge(
|
||||
profile=profile,
|
||||
api_key=api_key,
|
||||
data_dir=data_dir,
|
||||
http_transport_factory=lambda: AuditedModelTransport(
|
||||
app.state.model_request_audit_store,
|
||||
profile_id=profile.profile_id,
|
||||
profile_revision=profile.revision,
|
||||
provider_kind=str(profile.provider_kind),
|
||||
model=profile.model,
|
||||
inner=(
|
||||
BudgetedModelTransport(model_request_budget)
|
||||
if model_request_budget is not None
|
||||
else None
|
||||
),
|
||||
),
|
||||
),
|
||||
# `None` means "whatever the active profile allows". Pinned to
|
||||
# 1 in `a59f4b5` as temporary load-shedding for an acceptance
|
||||
# package, and left pinned after it: a hardcoded 1 is rule 9's
|
||||
@@ -1070,9 +1016,12 @@ def create_app(
|
||||
# profile concurrency 4, recall latency is unchanged while
|
||||
# precipitation goes 4x.
|
||||
max_concurrent_tasks=runtime_settings.precipitation_max_concurrent_tasks,
|
||||
model_content_config=app.state.config_store,
|
||||
permission_bridge_factory=lambda lease, secret, directory: _permission_context(
|
||||
app, lease, secret, directory
|
||||
),
|
||||
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
|
||||
service_levels=app.state.service_levels,
|
||||
sedimentation_enabled=lambda: _config_store(app).sedimentation_enabled(),
|
||||
)
|
||||
# C2 owns the single production budget (handle 5s + reserve 250ms,
|
||||
# total 5.25s, lease 10s, renew 1s, claim 250ms). The composition
|
||||
@@ -1090,27 +1039,18 @@ def create_app(
|
||||
job.job_id,
|
||||
)
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.retry_first_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(seconds=10),
|
||||
memory_write_queue.retry_timed_out_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(minutes=5),
|
||||
limit=32,
|
||||
)
|
||||
await asyncio.to_thread(
|
||||
app.state.memory_task_store.retry_first_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(seconds=10),
|
||||
limit=128,
|
||||
)
|
||||
|
||||
async def admit_hy_invocation(
|
||||
source_id: str,
|
||||
mark_invoked: Callable[[], Awaitable[bool]],
|
||||
) -> bool:
|
||||
"""Serialize pause with the last boundary before an HY call."""
|
||||
source_lock = app.state.source_permission_locks.setdefault(
|
||||
source_id, asyncio.Lock()
|
||||
)
|
||||
async with app.state.sedimentation_lock, source_lock:
|
||||
if not _config_store(app).sedimentation_enabled():
|
||||
raise RuntimeError("model_sedimentation_disabled")
|
||||
lock = app.state.source_permission_locks.setdefault(source_id, asyncio.Lock())
|
||||
async with lock:
|
||||
if not _source_writes_on(app, source_id):
|
||||
raise RuntimeError("model_content_source_disabled")
|
||||
return await mark_invoked()
|
||||
@@ -1148,18 +1088,8 @@ def create_app(
|
||||
+ 10
|
||||
),
|
||||
reconcile_unknown=reconcile_unknown_writes,
|
||||
source_writes_permitted=lambda source_id: _sedimentation_claim_enabled(
|
||||
app, source_id
|
||||
),
|
||||
# The claim gate has to be the same question the write gate
|
||||
# asks, not a second answer to it. It used to be
|
||||
# `sync_enabled`, the per-app toggle, and `one-memory-manual` is
|
||||
# not a connected app: it has no entry in the configuration
|
||||
# document, so that predicate answers False for it while the
|
||||
# write gate admits it by its grant. Every manual note's job was
|
||||
# therefore skipped on every drain tick — the note stayed
|
||||
# "saved, waiting" with no task behind it, forever.
|
||||
source_claim_enabled=lambda source_id: _sedimentation_claim_enabled(app, source_id),
|
||||
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
|
||||
source_claim_enabled=lambda source_id: _config_store(app).sync_enabled(source_id),
|
||||
)
|
||||
app.state.memory_write_drain = memory_write_drain
|
||||
app.state.memory_write_drain_task = watched_task(
|
||||
@@ -1188,7 +1118,10 @@ def create_app(
|
||||
for chunk in _conversation_chunks(conversation, max_turns=12, completed_only=True):
|
||||
if (
|
||||
record not in chunk
|
||||
or not any(item.role == "user" for item in chunk)
|
||||
or not any(
|
||||
item.role == "user" and item.written_by.value == "person"
|
||||
for item in chunk
|
||||
)
|
||||
or not any(item.role == "assistant" for item in chunk)
|
||||
):
|
||||
continue
|
||||
@@ -1201,6 +1134,7 @@ def create_app(
|
||||
app.state.matter_discovery.processor = app.state.memory_processing_supervisor
|
||||
app.state.matter_discovery.write_intake = app.state.memory_write_intake
|
||||
app.state.matter_discovery.write_queue = memory_write_queue
|
||||
app.state.processing_coordinator = None
|
||||
app.state.memory_note_store = MemoryNoteStore(
|
||||
app.state.evidence_store,
|
||||
app.state.memory_processing_supervisor,
|
||||
@@ -1512,6 +1446,7 @@ def create_app(
|
||||
app.include_router(matter_router(protected))
|
||||
app.include_router(memory_note_router(protected))
|
||||
app.include_router(file_import_router(protected))
|
||||
app.include_router(model_content_router(protected))
|
||||
app.include_router(context_router(protected))
|
||||
|
||||
def require_workbuddy_boundary(request: Request) -> None:
|
||||
@@ -1801,30 +1736,6 @@ def create_app(
|
||||
"events": events,
|
||||
}
|
||||
|
||||
@app.get("/api/v1/diagnostics/model-usage", dependencies=protected)
|
||||
def diagnostic_model_usage(
|
||||
since: datetime | None = Query(default=None), # noqa: B008
|
||||
request_id: str | None = Query( # noqa: B008
|
||||
default=None, min_length=1, max_length=512
|
||||
),
|
||||
limit: int = Query(default=200, ge=1, le=2_000), # noqa: B008
|
||||
) -> dict[str, object]:
|
||||
"""Return content-free precipitation usage for provider reconciliation."""
|
||||
|
||||
normalized_since = (
|
||||
(since if since.tzinfo is not None else since.replace(tzinfo=UTC))
|
||||
if since is not None
|
||||
else None
|
||||
)
|
||||
report = app.state.model_request_audit_store.report(
|
||||
since=normalized_since.astimezone(UTC).isoformat()
|
||||
if normalized_since is not None
|
||||
else None,
|
||||
request_id=request_id,
|
||||
limit=limit,
|
||||
)
|
||||
return {"schema_version": 1, **report}
|
||||
|
||||
@app.get(
|
||||
"/api/v1/runtime/identity",
|
||||
response_model=HealthResponse,
|
||||
@@ -1962,20 +1873,6 @@ def create_app(
|
||||
"""Record UI completion only; finishing or skipping is never model consent."""
|
||||
return await asyncio.to_thread(_onboarding_store(app).complete)
|
||||
|
||||
@app.post(
|
||||
"/api/v1/onboarding/reset",
|
||||
response_model=OnboardingState,
|
||||
dependencies=protected,
|
||||
)
|
||||
async def reset_onboarding() -> OnboardingState:
|
||||
"""Go back to the first-run screens. Only the completion flag moves.
|
||||
|
||||
Memories, sources and settings live in other tables and are untouched,
|
||||
which is the point: someone who wants to see the cold-start flow again
|
||||
should not have to delete anything to do it.
|
||||
"""
|
||||
return await asyncio.to_thread(_onboarding_store(app).reset)
|
||||
|
||||
@app.get(
|
||||
"/api/v1/sources",
|
||||
response_model=list[SourceResponse],
|
||||
@@ -2013,11 +1910,6 @@ def create_app(
|
||||
candidate_change_reason=app.state.source_candidate_store.change_reason(
|
||||
registration
|
||||
),
|
||||
authorization_state=grant_store.authorization_state(
|
||||
user_id="local",
|
||||
source_id=source_id,
|
||||
candidate=registration.candidate.path,
|
||||
),
|
||||
)
|
||||
for source_id, registration in _catalog(app).items()
|
||||
]
|
||||
@@ -2329,39 +2221,24 @@ def create_app(
|
||||
raise HTTPException(status_code=409, detail=error.code) from None
|
||||
if grant is None:
|
||||
raise HTTPException(status_code=409, detail="grant_required")
|
||||
try:
|
||||
await _prepare_source_resume(source_id)
|
||||
except BaseException as error:
|
||||
raise _sync_refused("sync_prepare_failed") from error
|
||||
await _prepare_source_resume(source_id)
|
||||
else:
|
||||
try:
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
except BaseException as error:
|
||||
raise _sync_refused("sync_prepare_failed") from error
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
try:
|
||||
await asyncio.to_thread(_config_store(app).set_sync, source_id, request.enabled)
|
||||
except BaseException as error:
|
||||
except BaseException:
|
||||
if request.enabled:
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
else:
|
||||
await _prepare_source_resume(source_id)
|
||||
await _activate_source_resume(source_id)
|
||||
await _refresh_live_watch(app)
|
||||
raise _sync_refused("sync_config_write_failed") from error
|
||||
raise
|
||||
try:
|
||||
if request.enabled:
|
||||
try:
|
||||
await _activate_source_resume(source_id)
|
||||
except BaseException as error:
|
||||
raise _sync_refused("sync_resume_failed") from error
|
||||
try:
|
||||
await _refresh_live_watch(app)
|
||||
except BaseException as error:
|
||||
raise _sync_refused("sync_watcher_failed") from error
|
||||
await _activate_source_resume(source_id)
|
||||
await _refresh_live_watch(app)
|
||||
except BaseException:
|
||||
# Rolls back the desired state and settles the watcher exactly as
|
||||
# before; the only change is that the raise above now carries a
|
||||
# code saying which stage failed.
|
||||
await asyncio.to_thread(_config_store(app).set_sync, source_id, previous)
|
||||
if request.enabled:
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
@@ -2451,7 +2328,7 @@ def create_app(
|
||||
await asyncio.to_thread(
|
||||
app.state.memory_task_store.park_source_for_permission,
|
||||
source_id=source_id,
|
||||
error_code="hy_memory_source_revoked",
|
||||
error_code="model_content_permission_required",
|
||||
)
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
_supervisor(app).unregister_adapter(source_id)
|
||||
@@ -2492,8 +2369,6 @@ def create_app(
|
||||
async def connect_sources(
|
||||
request: ConnectSourcesRequest,
|
||||
) -> ConnectSourcesResponse:
|
||||
if request.onboarding:
|
||||
app.state.history_import_gate_released = False
|
||||
source_ids = tuple(dict.fromkeys(request.source_ids))
|
||||
registrations = [_require_registration(app, source_id) for source_id in source_ids]
|
||||
if request.request_id is None:
|
||||
@@ -3659,48 +3534,6 @@ def create_app(
|
||||
def put_extraction_settings(request: ExtractionWindowUpdate) -> ExtractionSettings:
|
||||
return _memory_processing_supervisor(app).set_extraction_window(request.window_preset)
|
||||
|
||||
@app.get(
|
||||
"/api/v1/sedimentation",
|
||||
response_model=SedimentationState,
|
||||
dependencies=protected,
|
||||
)
|
||||
def get_sedimentation() -> SedimentationState:
|
||||
"""Whether the model may still turn conversations into memories."""
|
||||
return SedimentationState(enabled=_config_store(app).sedimentation_enabled())
|
||||
|
||||
@app.put(
|
||||
"/api/v1/sedimentation",
|
||||
response_model=SedimentationState,
|
||||
dependencies=protected,
|
||||
)
|
||||
async def put_sedimentation(request: SedimentationRequest) -> SedimentationState:
|
||||
"""Start or stop model-driven sedimentation, keeping the configuration.
|
||||
|
||||
Stopping is the user's cost control: nothing further goes to the model
|
||||
and already queued work waits rather than failing, so starting again
|
||||
resumes it. Importing conversations is not affected either way.
|
||||
"""
|
||||
store = _config_store(app)
|
||||
changed = False
|
||||
async with app.state.sedimentation_lock:
|
||||
previous = store.sedimentation_enabled()
|
||||
if previous is not request.enabled:
|
||||
await asyncio.to_thread(store.set_sedimentation_enabled, request.enabled)
|
||||
changed = True
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="memory.sedimentation",
|
||||
event="memory_sedimentation.toggled",
|
||||
outcome="succeeded",
|
||||
details={"enabled": request.enabled},
|
||||
)
|
||||
if changed and request.enabled:
|
||||
# Work parked by the stop is picked up again rather than waiting
|
||||
# for the next natural dispatch, but not on this request's time:
|
||||
# see _resume_sedimentation.
|
||||
_resume_sedimentation(app)
|
||||
return SedimentationState(enabled=store.sedimentation_enabled())
|
||||
|
||||
@app.post(
|
||||
"/api/v1/memory-processing/manual-extract",
|
||||
response_model=ManualExtractReceipt,
|
||||
@@ -4061,8 +3894,12 @@ async def _authorize_answer_generation(
|
||||
# The explicit question is this operation's input, not a standing permission for history.
|
||||
if not question.strip() or not grounding:
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
if not await asyncio.to_thread(_config_store(app).model_content_allowed):
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
|
||||
def check() -> None:
|
||||
if not _config_store(app).model_content_allowed():
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
if _model_profile_store(app).active_identity() != (
|
||||
profile.profile_id,
|
||||
profile.revision,
|
||||
@@ -4086,6 +3923,29 @@ async def _authorize_answer_generation(
|
||||
return recheck
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _permission_context(
|
||||
app: FastAPI,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
secret: str | None,
|
||||
directory: Path,
|
||||
) -> AsyncIterator[MemoryBridgeProtocol]:
|
||||
module = await asyncio.to_thread(import_module, "one_memory.hy_permission_context")
|
||||
budget = cast(ModelRequestBudget | None, app.state.model_request_budget)
|
||||
source_id = lease.records[0].source_id if lease.records else "manual"
|
||||
factory = module.permission_bridge_factory(
|
||||
lambda profile, api_key, data_dir: _memory_processing_supervisor(app)._bridge(
|
||||
profile,
|
||||
api_key,
|
||||
data_dir,
|
||||
scope_key=source_id,
|
||||
),
|
||||
**({"http_transport": BudgetedModelTransport(budget)} if budget else {}),
|
||||
)
|
||||
async with factory(lease, secret, directory) as bridge:
|
||||
yield cast(MemoryBridgeProtocol, bridge)
|
||||
|
||||
|
||||
class _LazyASGIApp:
|
||||
"""Compatibility ASGI app that is built only when a server imports it.
|
||||
|
||||
@@ -4117,16 +3977,6 @@ class _LazyASGIApp:
|
||||
app = _LazyASGIApp()
|
||||
|
||||
|
||||
def _sync_refused(code: str) -> HTTPException:
|
||||
"""A sync refusal that says which stage refused.
|
||||
|
||||
Turning the switch on takes four steps: authorize, prepare, write, resume.
|
||||
The UI used to be told only that the switch had sprung back, and every
|
||||
stage needs a different fix, so each one gets its own code.
|
||||
"""
|
||||
return HTTPException(status_code=409, detail=code)
|
||||
|
||||
|
||||
def _granted_roots(
|
||||
source_ids: Iterable[str],
|
||||
grant_store: SourceGrantStore,
|
||||
@@ -4178,7 +4028,6 @@ def _source_response(
|
||||
totals: tuple[int, int],
|
||||
*,
|
||||
candidate_change_reason: str | None = None,
|
||||
authorization_state: SourceAuthorizationState = SourceAuthorizationState.OK,
|
||||
) -> SourceResponse:
|
||||
location = inspect_source_location(
|
||||
registration.descriptor.source_id, registration.candidate.path
|
||||
@@ -4391,7 +4240,7 @@ async def _onboarding_history_after(
|
||||
home: Path,
|
||||
now: datetime | None = None,
|
||||
) -> datetime | None:
|
||||
"""Return one shared cutoff when any selected harness has large, old history.
|
||||
"""Return one shared cutoff when any selected harness has large history.
|
||||
|
||||
The cutoff is decided before any history job is created, then persisted on
|
||||
every accepted job. This is deliberately global: independently limiting
|
||||
@@ -4400,8 +4249,6 @@ async def _onboarding_history_after(
|
||||
"""
|
||||
if not request.onboarding:
|
||||
return None
|
||||
anchor = now or datetime.now(UTC)
|
||||
cutoff = anchor - ONBOARDING_HISTORY_WINDOW
|
||||
revisions = request.source_candidate_revisions or {}
|
||||
for registration in registrations:
|
||||
source_id = registration.descriptor.source_id
|
||||
@@ -4429,52 +4276,29 @@ async def _onboarding_history_after(
|
||||
home=home,
|
||||
)
|
||||
adapter = registration.create_adapter(grant)
|
||||
if await _has_large_old_history(adapter, cutoff=cutoff):
|
||||
return cutoff
|
||||
count_sessions = cast(
|
||||
Callable[..., Awaitable[int]] | None,
|
||||
getattr(adapter, "count_sessions", None),
|
||||
)
|
||||
if count_sessions is not None:
|
||||
session_count = await count_sessions(limit=ONBOARDING_LARGE_HISTORY_SESSIONS + 1)
|
||||
else:
|
||||
pending = _install_state(adapter)
|
||||
if pending is None:
|
||||
continue
|
||||
session_count = (await pending).session_count
|
||||
except (OSError, SourceAccessError):
|
||||
# A supported source whose metadata count cannot be completed is
|
||||
# treated conservatively. The later connection still owns the
|
||||
# precise authorization error, while onboarding never falls back
|
||||
# to an unbounded history import.
|
||||
return cutoff
|
||||
session_count = ONBOARDING_LARGE_HISTORY_SESSIONS + 1
|
||||
if session_count > ONBOARDING_LARGE_HISTORY_SESSIONS:
|
||||
anchor = now or datetime.now(UTC)
|
||||
return anchor - ONBOARDING_HISTORY_WINDOW
|
||||
return None
|
||||
|
||||
|
||||
async def _has_large_old_history(adapter: SourceAdapter, *, cutoff: datetime) -> bool:
|
||||
"""Probe one authorized Harness with the same parsers the import will use.
|
||||
|
||||
Stop as soon as both policy facts are proven: more than 200 genuine person
|
||||
turns and at least one record older than the three-day cutoff. Counting
|
||||
parsed Evidence instead of transcript files prevents one giant coding
|
||||
session from bypassing the limit, while reusing the adapters keeps host
|
||||
envelopes and tool-shaped user records out of the count.
|
||||
"""
|
||||
|
||||
user_turns = 0
|
||||
has_old_data = False
|
||||
async for event in adapter.scan(ScanOptions()):
|
||||
if isinstance(event, ScanFailure):
|
||||
# A partial probe must never turn into an unbounded first import.
|
||||
return True
|
||||
if not isinstance(event, ScannedEvidence):
|
||||
continue
|
||||
observed = event.evidence.created_at
|
||||
if observed is None and event.cursor.observed_modified_ns > 0:
|
||||
observed = datetime.fromtimestamp(
|
||||
event.cursor.observed_modified_ns / 1_000_000_000,
|
||||
tz=UTC,
|
||||
)
|
||||
if observed is not None:
|
||||
if observed.tzinfo is None:
|
||||
observed = observed.replace(tzinfo=UTC)
|
||||
has_old_data = has_old_data or observed.astimezone(UTC) < cutoff
|
||||
if event.evidence.role == "user":
|
||||
user_turns += 1
|
||||
if user_turns > ONBOARDING_LARGE_HISTORY_USER_TURNS and has_old_data:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _install_state(adapter: object) -> Awaitable[SourceInstallState] | None:
|
||||
"""Ask an adapter for the two-level answer when it has one.
|
||||
|
||||
@@ -4766,7 +4590,7 @@ async def _keep_loop_heartbeat(watchdog: LoopWatchdog) -> None:
|
||||
async def _emit_resource_snapshots(interval_seconds: float = 60.0) -> None:
|
||||
while True:
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="core.resources",
|
||||
event="core.resource_snapshot",
|
||||
outcome="observed",
|
||||
@@ -4791,33 +4615,6 @@ def _start_backfill(app: FastAPI) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _resume_sedimentation(app: FastAPI) -> None:
|
||||
"""Pick the work parked by a stop back up, off the caller's clock.
|
||||
|
||||
Turning the switch back on is a decision, not a job the caller waits for:
|
||||
the choice is already stored, and what follows it is maintenance over every
|
||||
unfinished task followed by a scan of the whole queue. On a real backlog
|
||||
that runs for many minutes, so awaiting it here left the PUT unanswered
|
||||
until the client gave up — the renderer showed "启用未能确认" for a switch
|
||||
that had in fact been saved, and the switch only appeared to turn on once
|
||||
the follow-up read came back. Measured: one toggle, twenty-plus minutes,
|
||||
no response, client timeout at thirty seconds.
|
||||
|
||||
Repeated toggles reuse the running walk; it reads the switch as it goes, so
|
||||
a stop part-way through is honoured on the next batch instead of needing a
|
||||
cancel.
|
||||
"""
|
||||
running = cast("asyncio.Task[None] | None", getattr(app.state, "sedimentation_resume", None))
|
||||
if running is not None and not running.done():
|
||||
return
|
||||
app.state.sedimentation_resume = watched_task(
|
||||
asyncio.get_running_loop(),
|
||||
lambda: _memory_processing_supervisor(app).resume_allowed(),
|
||||
component="memory.sedimentation",
|
||||
name="one-memory-sedimentation-resume",
|
||||
)
|
||||
|
||||
|
||||
def _start_visibility_recompute(app: FastAPI) -> None:
|
||||
"""Keep the stored visibility conclusions current, in the background.
|
||||
|
||||
@@ -4926,6 +4723,8 @@ async def _run_backfill(app: FastAPI) -> None:
|
||||
app.state.memory_backfill_error = None
|
||||
if not app.state.settings.automatic_memory_processing:
|
||||
return
|
||||
if not await asyncio.to_thread(_config_store(app).model_content_allowed):
|
||||
return
|
||||
try:
|
||||
queue = app.state.memory_write_queue
|
||||
intake = app.state.memory_write_intake
|
||||
@@ -5061,51 +4860,6 @@ def _source_writes_on(app: FastAPI, source_id: str) -> bool:
|
||||
return _source_processing_enabled(app, source_id)
|
||||
|
||||
|
||||
def _sedimentation_claim_enabled(app: FastAPI, source_id: str) -> bool:
|
||||
"""Claim queued model work only while both global and source gates allow it."""
|
||||
|
||||
return (
|
||||
_config_store(app).sedimentation_enabled()
|
||||
and _history_import_gate_released(app)
|
||||
and _source_writes_on(app, source_id)
|
||||
)
|
||||
|
||||
|
||||
def _history_import_gate_released(app: FastAPI) -> bool:
|
||||
"""Hold first sedimentation until every accepted history import is terminal."""
|
||||
|
||||
if getattr(app.state, "history_import_gate_released", False):
|
||||
return True
|
||||
onboarding = getattr(app.state, "onboarding_store", None)
|
||||
jobs = getattr(app.state, "import_job_store", None)
|
||||
if onboarding is None or jobs is None:
|
||||
return True
|
||||
state = onboarding.get()
|
||||
if state.status != "importing" or not state.selected_source_ids:
|
||||
app.state.history_import_gate_released = True
|
||||
return True
|
||||
selected = set(state.selected_source_ids)
|
||||
matching = [
|
||||
job
|
||||
for job in jobs.list_jobs()
|
||||
if job.operation == "history"
|
||||
and job.source_id in selected
|
||||
and (state.started_at is None or job.created_at >= state.started_at)
|
||||
]
|
||||
if {job.source_id for job in matching} != selected:
|
||||
return False
|
||||
active = {
|
||||
ImportStatus.PENDING,
|
||||
ImportStatus.RUNNING,
|
||||
ImportStatus.PAUSED,
|
||||
ImportStatus.INTERRUPTED,
|
||||
}
|
||||
released = all(job.status not in active for job in matching)
|
||||
if released:
|
||||
app.state.history_import_gate_released = True
|
||||
return released
|
||||
|
||||
|
||||
def _model_prober(app: FastAPI) -> ModelProber:
|
||||
return cast(ModelProber, app.state.model_prober)
|
||||
|
||||
|
||||
@@ -175,22 +175,6 @@ async def in_interactive_read[T](
|
||||
return await in_read_pool(interactive_read_pool(), work, *args, **kwargs)
|
||||
|
||||
|
||||
def _retrieve[T](future: "asyncio.Future[T]") -> None:
|
||||
"""Mark a shielded read's outcome as seen.
|
||||
|
||||
`in_read_pool` shields the wrapped worker, so a caller that gives up and
|
||||
cancels leaves the read running to completion. When that read then fails,
|
||||
nothing awaits the future any more: asyncio reports the orphan at GC as
|
||||
"Future exception was never retrieved", the loop handler turns it into a
|
||||
`core.loop.task_exception` error diagnostic, and a lane that degraded on
|
||||
its own budget -- correctly, and already reported -- looks like a crashed
|
||||
task on top of it. Retrieving here keeps the diagnostics honest. Awaiting
|
||||
still raises for callers that are still there, so no failure is hidden.
|
||||
"""
|
||||
if not future.cancelled():
|
||||
future.exception()
|
||||
|
||||
|
||||
async def in_read_pool[T](
|
||||
pool: ThreadPoolExecutor,
|
||||
work: Callable[..., T],
|
||||
@@ -209,9 +193,7 @@ async def in_read_pool[T](
|
||||
_interactive_reads -= 1
|
||||
|
||||
worker.add_done_callback(released)
|
||||
pending = asyncio.wrap_future(worker)
|
||||
pending.add_done_callback(_retrieve)
|
||||
return await asyncio.shield(pending)
|
||||
return await asyncio.shield(asyncio.wrap_future(worker))
|
||||
|
||||
|
||||
async def wait_for_interactive_reads() -> None:
|
||||
|
||||
@@ -266,13 +266,7 @@ def serve(*, maintenance: bool = False, reader: bool = False) -> None:
|
||||
)
|
||||
application = create_reader_app(reader_settings)
|
||||
print(f"reader_port:{port}", file=sys.stderr, flush=True)
|
||||
uvicorn.run(
|
||||
application,
|
||||
host=settings.host,
|
||||
port=port,
|
||||
reload=False,
|
||||
access_log=False,
|
||||
)
|
||||
uvicorn.run(application, host=settings.host, port=port, reload=False)
|
||||
return
|
||||
|
||||
from one_memory.api import create_app
|
||||
@@ -303,21 +297,9 @@ def serve(*, maintenance: bool = False, reader: bool = False) -> None:
|
||||
)
|
||||
if maintenance:
|
||||
with CoreRuntime(settings, publish_receipt=False):
|
||||
uvicorn.run(
|
||||
application,
|
||||
host=settings.host,
|
||||
port=settings.port,
|
||||
reload=False,
|
||||
access_log=False,
|
||||
)
|
||||
uvicorn.run(application, host=settings.host, port=settings.port, reload=False)
|
||||
else:
|
||||
uvicorn.run(
|
||||
application,
|
||||
host=settings.host,
|
||||
port=settings.port,
|
||||
reload=False,
|
||||
access_log=False,
|
||||
)
|
||||
uvicorn.run(application, host=settings.host, port=settings.port, reload=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -196,7 +196,7 @@ def read_raw_context(
|
||||
if _prefers_declared_facts(request.query)
|
||||
else "1"
|
||||
)
|
||||
person_authored = "CASE WHEN e.role='user' THEN 1 ELSE 0 END"
|
||||
person_authored = "CASE WHEN e.written_by='person' THEN 1 ELSE 0 END"
|
||||
query_echo = (
|
||||
"CASE WHEN trim(e.verbatim)=? OR trim(e.verbatim)=? THEN 0 ELSE 1 END"
|
||||
if _prefers_declared_facts(request.query)
|
||||
@@ -470,8 +470,9 @@ def revalidate_raw_context(
|
||||
)
|
||||
except (ValueError, TypeError):
|
||||
break
|
||||
if (row["role"], created) != (
|
||||
if (row["role"], row["written_by"], created) != (
|
||||
message.role,
|
||||
message.written_by,
|
||||
message.created_at,
|
||||
):
|
||||
break
|
||||
|
||||
@@ -66,12 +66,12 @@ def _field(text: str) -> ResumeField | None:
|
||||
return None
|
||||
|
||||
|
||||
def _raw_class(role: str, evidence_id: str, accepted: frozenset[str]) -> EvidenceClass:
|
||||
if role == "user":
|
||||
def _raw_class(written_by: str, evidence_id: str, accepted: frozenset[str]) -> EvidenceClass:
|
||||
if written_by == "person":
|
||||
return "person_statement"
|
||||
if role == "assistant":
|
||||
if written_by == "agent":
|
||||
return "agent_report"
|
||||
if role == "tool" and evidence_id in accepted:
|
||||
if written_by == "tool" and evidence_id in accepted:
|
||||
return "system_acceptance"
|
||||
return "tool_output"
|
||||
|
||||
@@ -118,7 +118,7 @@ def assemble_resume_packet(
|
||||
field=_field(message.text) or "unclassified",
|
||||
text=message.text,
|
||||
evidence_class=_raw_class(
|
||||
message.role,
|
||||
message.written_by,
|
||||
message.evidence_id,
|
||||
verified_system_evidence_ids,
|
||||
),
|
||||
|
||||
@@ -345,7 +345,7 @@ def _lexical_raw_rows(
|
||||
if context_raw._prefers_declared_facts(query.query)
|
||||
else "1"
|
||||
)
|
||||
person_authored = "CASE WHEN e.role='user' THEN 1 ELSE 0 END"
|
||||
person_authored = "CASE WHEN e.written_by='person' THEN 1 ELSE 0 END"
|
||||
query_echo = (
|
||||
"CASE WHEN trim(e.verbatim)=? OR trim(e.verbatim)=? THEN 0 ELSE 1 END"
|
||||
if context_raw._prefers_declared_facts(query.query)
|
||||
@@ -507,7 +507,7 @@ def _raw_item(row: sqlite3.Row, start1: int, matched_by: tuple[str, ...]) -> Raw
|
||||
"source_id": row["source_id"],
|
||||
"conversation_id": row["conversation_id"],
|
||||
"role": row["role"],
|
||||
"written_by": "person" if row["role"] == "user" else "agent",
|
||||
"written_by": "person" if row["written_by"] == "person" else "agent",
|
||||
"event_time": event_time.isoformat() if event_time else None,
|
||||
"spans": [{"start_cp": start, "end_cp": start + len(text), "text": text}],
|
||||
"matched_by": list(matched_by),
|
||||
|
||||
@@ -1039,7 +1039,7 @@ def close_main_databases() -> None:
|
||||
def log_main_database_stats(moment: str) -> None:
|
||||
"""One line in core.log: how many connections are open and who waited."""
|
||||
stats = main_database_stats()
|
||||
logger.debug(
|
||||
logger.info(
|
||||
"main_database_connections moment=%s %s",
|
||||
moment,
|
||||
" ".join(f"{key}={value}" for key, value in stats.items()),
|
||||
|
||||
@@ -19,7 +19,6 @@ from __future__ import annotations
|
||||
|
||||
import atexit
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
@@ -140,10 +139,6 @@ _DETAIL_KEYS = frozenset(
|
||||
"accounted_ms",
|
||||
"unaccounted_ms",
|
||||
"conversation_hash",
|
||||
# The sedimentation switch: a bounded boolean, the same shape as
|
||||
# `spawned` and `index_complete`. Left out of this list it did not
|
||||
# drop a log line, it turned the switch's own request into a 500.
|
||||
"enabled",
|
||||
}
|
||||
)
|
||||
_trace_id: ContextVar[str | None] = ContextVar("one_memory_trace_id", default=None)
|
||||
@@ -654,24 +649,12 @@ def diagnostic_event(**values: object) -> None:
|
||||
active = _active
|
||||
if active is None:
|
||||
return
|
||||
if values.get("level") == "debug" and not logging.getLogger().isEnabledFor(
|
||||
logging.DEBUG
|
||||
):
|
||||
return
|
||||
try:
|
||||
active.write(**values) # type: ignore[arg-type]
|
||||
except OSError:
|
||||
# Diagnostics must not become a new product outage when a disk is full
|
||||
# or the log directory is temporarily unavailable.
|
||||
return
|
||||
except ValueError:
|
||||
# Same promise, same reason, for a rejected payload instead of a failed
|
||||
# write. An unlisted `details` key once turned every external recall
|
||||
# into a 500 and then killed the process, and the sedimentation switch
|
||||
# into a permanent failure. The payload itself is still wrong and is
|
||||
# still caught in CI by tests/test_diagnostic_payload_keys.py; what
|
||||
# must not happen is the caller paying for it.
|
||||
return
|
||||
|
||||
|
||||
def current_session_id() -> str | None:
|
||||
|
||||
@@ -21,6 +21,7 @@ from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception, elapsed_ms
|
||||
from one_memory.hy_lifecycle import HyLifecycle, HyLifecycleUnreadable, read_hy_lifecycle
|
||||
from one_memory.hy_permission_store import native_store_read_slot
|
||||
from one_memory.hy_presence import HyPresence, confirm_deleted
|
||||
from one_memory.hy_runtime_config import (
|
||||
HY_MEMORY_MODE,
|
||||
@@ -31,9 +32,7 @@ from one_memory.hy_runtime_config import (
|
||||
diagnostic_log_level,
|
||||
)
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
from one_memory.hy_store_concurrency import native_store_read_slot
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind, attach_disable_thinking
|
||||
from one_memory.model_request_audit import audit_precipitation_request
|
||||
from one_memory.semantic_deadline import SemanticMemoryHit
|
||||
|
||||
#: D102: how relevant HY has to call a hit before it counts. Raised from 0.25
|
||||
@@ -144,22 +143,6 @@ class HyMemoryBridgeError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES = 512
|
||||
|
||||
|
||||
def coding_tool_result_max_bytes() -> int:
|
||||
"""Return the per-tool payload ceiling used by MemHub's HY client."""
|
||||
|
||||
raw = os.environ.get("MEMORY_CODING_TOOL_RESULT_MAX_BYTES")
|
||||
if raw is None:
|
||||
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError:
|
||||
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
return value if value > 0 else MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
|
||||
|
||||
def _exclusive[**P, T](
|
||||
method: Callable[P, Coroutine[Any, Any, T]],
|
||||
) -> Callable[P, Coroutine[Any, Any, T]]:
|
||||
@@ -393,13 +376,11 @@ class HyMemoryBridge:
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
client: HyMemoryClientProtocol | None = None,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
) -> None:
|
||||
self.profile = profile
|
||||
self.api_key = api_key
|
||||
self.data_dir = data_dir
|
||||
self._client = client
|
||||
self._http_transport_factory = http_transport_factory
|
||||
self._client_lock = asyncio.Lock()
|
||||
self._write_lock = asyncio.Lock()
|
||||
self._last_bindings: HyWriteBindings | None = None
|
||||
@@ -431,9 +412,8 @@ class HyMemoryBridge:
|
||||
|
||||
@_exclusive
|
||||
async def ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
|
||||
with audit_precipitation_request(request.request_id):
|
||||
async with self._write_lock:
|
||||
return await self._ingest(request)
|
||||
async with self._write_lock:
|
||||
return await self._ingest(request)
|
||||
|
||||
async def _ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
|
||||
started = monotonic()
|
||||
@@ -447,11 +427,6 @@ class HyMemoryBridge:
|
||||
)
|
||||
client = await self._get_client()
|
||||
guards = getattr(client, "_one_memory_write_guards", ())
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
reconcile_guards = tuple(guard for guard in guards if isinstance(guard, HyReconcileGuard))
|
||||
for guard in reconcile_guards:
|
||||
guard.begin_write()
|
||||
before = [len(guard.failures) for guard in guards]
|
||||
bindings = HyWriteBindings(request.source_context) if request.source_context else None
|
||||
if bindings:
|
||||
@@ -525,12 +500,6 @@ class HyMemoryBridge:
|
||||
if len(guard.failures) > count:
|
||||
raise HyMemoryBridgeError(guard.failures[count])
|
||||
result = await self._result(client, request, raw, expected_contents=None, bindings=bindings)
|
||||
result = await _with_native_mutation_delta(
|
||||
client,
|
||||
result,
|
||||
raw,
|
||||
reconcile_guards,
|
||||
)
|
||||
if bindings:
|
||||
outputs = []
|
||||
for memory in result.memories:
|
||||
@@ -560,9 +529,8 @@ class HyMemoryBridge:
|
||||
request: HyMemoryIngestRequest,
|
||||
contents: Sequence[str],
|
||||
) -> HyMemoryIngestResult:
|
||||
with audit_precipitation_request(request.request_id):
|
||||
async with self._write_lock:
|
||||
return await self._ingest_extracted(request, contents)
|
||||
async with self._write_lock:
|
||||
return await self._ingest_extracted(request, contents)
|
||||
|
||||
async def _ingest_extracted(
|
||||
self,
|
||||
@@ -585,8 +553,6 @@ class HyMemoryBridge:
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
report_guards = tuple(guard for guard in guards if isinstance(guard, HyReconcileGuard))
|
||||
for guard in report_guards:
|
||||
guard.begin_write()
|
||||
expected_contents = frozenset(content.strip() for content in contents if content.strip())
|
||||
for guard in report_guards:
|
||||
guard.imported_contents = expected_contents
|
||||
@@ -633,12 +599,6 @@ class HyMemoryBridge:
|
||||
raw,
|
||||
expected_contents=expected_contents,
|
||||
)
|
||||
result = await _with_native_mutation_delta(
|
||||
client,
|
||||
result,
|
||||
raw,
|
||||
report_guards,
|
||||
)
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="hy.bridge",
|
||||
@@ -1127,7 +1087,6 @@ class HyMemoryBridge:
|
||||
self.profile,
|
||||
api_key=self.api_key,
|
||||
data_dir=self.data_dir,
|
||||
http_transport_factory=self._http_transport_factory,
|
||||
)
|
||||
|
||||
await _joined_thread(initialize)
|
||||
@@ -1154,7 +1113,7 @@ class HyMemoryBridge:
|
||||
)
|
||||
raise HyMemoryBridgeError("hy_memory_init_failed") from error
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="hy.client",
|
||||
event="hy.client_initialization.completed",
|
||||
outcome="succeeded",
|
||||
@@ -1177,7 +1136,6 @@ def create_hy_memory_client(
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
) -> HyMemoryClientProtocol:
|
||||
configure_hy_memory_environment(data_dir)
|
||||
config = build_hy_memory_config(
|
||||
@@ -1185,7 +1143,6 @@ def create_hy_memory_client(
|
||||
api_key=api_key,
|
||||
data_dir=data_dir,
|
||||
)
|
||||
config.http_transport_factory = http_transport_factory
|
||||
reattach_legacy_vector_collection(profile, config)
|
||||
mismatch = inspect_graph_width_mismatch(config)
|
||||
if mismatch is not None:
|
||||
@@ -1330,72 +1287,6 @@ async def _coding_result(
|
||||
return HyMemoryIngestResult(success=True, memories=tuple(outputs))
|
||||
|
||||
|
||||
async def _with_native_mutation_delta(
|
||||
client: HyMemoryClientProtocol,
|
||||
result: HyMemoryIngestResult,
|
||||
raw: Mapping[str, object],
|
||||
reconcile_guards: Sequence[Any],
|
||||
) -> HyMemoryIngestResult:
|
||||
"""Map native SDK mutations back into MemHub governance.
|
||||
|
||||
Store permission wrappers used to discover this by intercepting every
|
||||
native CRUD call. The embedded SDK now owns unrestricted local CRUD, so
|
||||
chat targets come from the reconcile observer and coding targets come from
|
||||
the SDK's public result document.
|
||||
"""
|
||||
|
||||
retired = set(result.retired_memory_ids)
|
||||
superseded = {key: tuple(values) for key, values in result.superseded_memory_ids.items()}
|
||||
chat_targets = {
|
||||
key
|
||||
for guard in reconcile_guards
|
||||
for key in getattr(guard, "mutation_target_ids", ())
|
||||
if isinstance(key, str) and key
|
||||
}
|
||||
|
||||
operations = raw.get("ops")
|
||||
for item in operations if isinstance(operations, list) else ():
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
if item.get("action") == "DELETE":
|
||||
key = item.get("target_experience_id")
|
||||
if isinstance(key, str) and key:
|
||||
retired.add(key)
|
||||
|
||||
fact_operations = raw.get("fact_ops")
|
||||
for item in fact_operations if isinstance(fact_operations, list) else ():
|
||||
if not isinstance(item, Mapping) or item.get("action") != "SUPERSEDE":
|
||||
continue
|
||||
old, new = item.get("fact_id"), item.get("superseded_by")
|
||||
if isinstance(old, str) and old:
|
||||
retired.add(old)
|
||||
if isinstance(new, str) and new:
|
||||
superseded[old] = (new,)
|
||||
|
||||
for key in chat_targets:
|
||||
record = await _joined_thread(_read_hy_record, client, key)
|
||||
if record is None:
|
||||
retired.add(key)
|
||||
continue
|
||||
status = str(record.get("status", "")).lower()
|
||||
if status in {"deleted", "shadow", "superseded"}:
|
||||
retired.add(key)
|
||||
replacement = record.get("superseded_by")
|
||||
if isinstance(replacement, str) and replacement:
|
||||
superseded[key] = (replacement,)
|
||||
elif isinstance(replacement, list):
|
||||
values = tuple(value for value in replacement if isinstance(value, str) and value)
|
||||
if values:
|
||||
superseded[key] = values
|
||||
|
||||
return result.model_copy(
|
||||
update={
|
||||
"retired_memory_ids": tuple(sorted(retired)),
|
||||
"superseded_memory_ids": superseded,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _read_hy_record(
|
||||
client: HyMemoryClientProtocol,
|
||||
memory_id: str,
|
||||
@@ -1712,7 +1603,6 @@ def build_hy_memory_config(
|
||||
config.history.db_path = str(memory_root / "history.sqlite3")
|
||||
config.coding.enable = True
|
||||
config.coding.db_path = str(memory_root / "coding.sqlite3")
|
||||
config.coding.tool_result_max_bytes = coding_tool_result_max_bytes()
|
||||
config.recall.episode_recall_enabled = True
|
||||
config.recall.episode_summary_enabled = False
|
||||
config.recall.dynamic_episode_enabled = True
|
||||
|
||||
@@ -0,0 +1,607 @@
|
||||
"""Operation-bound read/write capability for the pinned native HY coding store.
|
||||
|
||||
HY's coding route writes two layers, experiences and working facts. Rows this
|
||||
lease cannot attest stay in the same backend but are invisible here, so they
|
||||
never reach a coding prompt and cannot be updated, superseded or linked.
|
||||
|
||||
HY's reconcilers catch store exceptions and carry on, so a denial latches and
|
||||
recheck() raises it again at finish; a swallowed denial still fails the task.
|
||||
Any native method not implemented here is denied rather than passed through.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from collections.abc import Iterator, Mapping, Sequence
|
||||
from copy import deepcopy
|
||||
from hashlib import sha256
|
||||
from types import MappingProxyType
|
||||
from typing import Any, NoReturn
|
||||
|
||||
from hy_memory.coding.types import CodingMemory, WorkingFact # type: ignore[import-untyped]
|
||||
|
||||
from one_memory.hy_permission_store import HyNodePermission
|
||||
from one_memory.model_processing_permission_guard import (
|
||||
ModelProcessingPermissionLease,
|
||||
permission_error,
|
||||
)
|
||||
|
||||
_NATIVE_WRITES = frozenset(
|
||||
{
|
||||
"insert",
|
||||
"update",
|
||||
"delete",
|
||||
"insert_fact",
|
||||
"supersede_fact",
|
||||
"set_fact_derived_experiences",
|
||||
"set_experience_source_facts",
|
||||
# Trace rows are raw conversation: writes must be gated like any other.
|
||||
"append_trace",
|
||||
}
|
||||
)
|
||||
|
||||
_TIMESTAMPS = frozenset({"created_at", "updated_at"})
|
||||
|
||||
# SDK 2.0.2 reads `config` off the coding store while routing a coding
|
||||
# turn. The wrapper must answer it, but the native `MemoryConfig` carries
|
||||
# credentials (llm/embedder api keys, graph and cache passwords) and is a live
|
||||
# handle into the backend, so it is never handed out. Only this redacted
|
||||
# projection is exposed: credential-bearing keys collapse to None.
|
||||
_CONFIG_REDACTED_KEYS = frozenset(
|
||||
{
|
||||
"api_key",
|
||||
"password",
|
||||
"username",
|
||||
"mysql_password",
|
||||
"mysql_user",
|
||||
"extra_headers",
|
||||
"extra_body",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class _RedactedConfig(Mapping[str, Any]):
|
||||
"""Read-only config view: readable by key and attribute, never the native one.
|
||||
|
||||
The SDK reaches config values both ways, so this keeps attribute access
|
||||
working. It is deliberately not the native MemoryConfig: it exposes only
|
||||
plain copied values, so no backend handle, connection or callable escapes.
|
||||
"""
|
||||
|
||||
__slots__ = ("_values",)
|
||||
|
||||
def __init__(self, values: Mapping[str, Any]) -> None:
|
||||
object.__setattr__(self, "_values", MappingProxyType(dict(values)))
|
||||
|
||||
def __getitem__(self, key: str) -> Any:
|
||||
return self._values[key]
|
||||
|
||||
def __iter__(self) -> Iterator[str]:
|
||||
return iter(self._values)
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self._values)
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
values = object.__getattribute__(self, "_values")
|
||||
try:
|
||||
return values[name]
|
||||
except KeyError:
|
||||
raise AttributeError(name) from None
|
||||
|
||||
def __setattr__(self, name: str, value: Any) -> NoReturn:
|
||||
raise AttributeError("redacted config is read-only")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"_RedactedConfig({sorted(self._values)})"
|
||||
|
||||
|
||||
def _redact_config(value: Any, depth: int = 0) -> Any:
|
||||
"""Copy config data into read-only views, collapsing credentials to None.
|
||||
|
||||
The SDK reads this both by attribute and by key, so each config section
|
||||
becomes a mapping-like view rather than a plain dict: attribute access stays
|
||||
available while the native object itself is never handed out.
|
||||
"""
|
||||
if depth > 4:
|
||||
return None
|
||||
if value is None or isinstance(value, (str, int, float, bool)):
|
||||
return value
|
||||
if isinstance(value, Mapping):
|
||||
return _RedactedConfig(
|
||||
{
|
||||
key: None if key in _CONFIG_REDACTED_KEYS else _redact_config(item, depth + 1)
|
||||
for key, item in value.items()
|
||||
}
|
||||
)
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [_redact_config(item, depth + 1) for item in value]
|
||||
if hasattr(value, "__dict__") and type(value).__module__.startswith("hy_memory"):
|
||||
return _redact_config(vars(value), depth + 1)
|
||||
return None
|
||||
|
||||
|
||||
def _config_projection(native: Any) -> Any:
|
||||
try:
|
||||
return _redact_config(native.config)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def hy_coding_fingerprint(row: CodingMemory | WorkingFact) -> str:
|
||||
"""Hash the complete native row as read back, including links and timestamps."""
|
||||
if type(row) is CodingMemory:
|
||||
kind = "experience"
|
||||
elif type(row) is WorkingFact:
|
||||
kind = "fact"
|
||||
else:
|
||||
raise permission_error("hy_coding_store_row_shape")
|
||||
payload = {"kind": kind, **row.to_dict()}
|
||||
return sha256(
|
||||
json.dumps(payload, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _row_id(row: CodingMemory | WorkingFact) -> str:
|
||||
return str(row.experience_id if type(row) is CodingMemory else row.fact_id)
|
||||
|
||||
|
||||
class HyPermissionCodingStore:
|
||||
"""Closed, task-exclusive adapter; initialize/close never own the native store.
|
||||
|
||||
node_permissions is the same attested ledger the chat vector store uses;
|
||||
coding rows are keyed by experience or fact id. written_nodes carries the
|
||||
whole lease footprint for each row written here, as the chat store does.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
native: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
*,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
node_permissions: Mapping[str, HyNodePermission] | None = None,
|
||||
) -> None:
|
||||
self._native = native
|
||||
self._lease = lease
|
||||
self._agent_id = agent_id
|
||||
self._session_id = session_id
|
||||
self._user_id = lease.user_id
|
||||
self._failure: str | None = None
|
||||
self._closed = False
|
||||
self._lock = asyncio.Lock()
|
||||
self._deleted: set[str] = set()
|
||||
self._written: dict[str, HyNodePermission] = {}
|
||||
self._superseded: dict[str, str] = {}
|
||||
self._evidence_ids = tuple(record.evidence_id for record in lease.records)
|
||||
self._permissions = {
|
||||
key: proof
|
||||
for key, proof in (node_permissions or {}).items()
|
||||
if set(proof.evidence_ids) <= set(self._evidence_ids)
|
||||
}
|
||||
self.recheck()
|
||||
if not agent_id or agent_id == "*" or not session_id or not self._evidence_ids:
|
||||
self._deny("scope")
|
||||
|
||||
def __getattr__(self, name: str) -> NoReturn:
|
||||
if name.startswith("__"):
|
||||
raise AttributeError(name)
|
||||
self._deny("unsupported_operation")
|
||||
|
||||
@property
|
||||
def written_nodes(self) -> Mapping[str, HyNodePermission]:
|
||||
self.recheck()
|
||||
return MappingProxyType(dict(self._written))
|
||||
|
||||
@property
|
||||
def config(self) -> Any:
|
||||
"""Read-only, redacted view of the native coding config.
|
||||
|
||||
SDK 2.0.2 reads `config` when it routes a coding turn. Answering
|
||||
it is safe only as a projection: the native MemoryConfig is a live
|
||||
backend handle and carries credentials, so callers get plain values
|
||||
with every credential-bearing key collapsed to None. This carries no
|
||||
identity, permission or tenant state and is not the native object.
|
||||
"""
|
||||
self._assert_open()
|
||||
return _config_projection(self._native)
|
||||
|
||||
@property
|
||||
def retired_ids(self) -> frozenset[str]:
|
||||
"""Experiences HY deleted in this operation."""
|
||||
self.recheck()
|
||||
return frozenset(self._deleted)
|
||||
|
||||
@property
|
||||
def superseded_ids(self) -> Mapping[str, tuple[str, ...]]:
|
||||
"""Facts HY superseded in this operation, with the fact that replaced each.
|
||||
|
||||
A successor linked before it existed must have been written here by
|
||||
now; otherwise the old fact points at a row nobody attested.
|
||||
"""
|
||||
self.recheck()
|
||||
if any(new not in self._written for new in self._superseded.values()):
|
||||
self._deny("link_denied")
|
||||
return MappingProxyType({old: (new,) for old, new in self._superseded.items()})
|
||||
|
||||
def recheck(self) -> None:
|
||||
self._assert_open()
|
||||
try:
|
||||
self._lease.recheck()
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
self._failure = (
|
||||
code
|
||||
if code.startswith("model_content_") and len(code) <= 100
|
||||
else "model_content_hy_coding_store_lease_failed"
|
||||
)
|
||||
raise permission_error(self._failure) from None
|
||||
|
||||
def _assert_open(self) -> None:
|
||||
if self._closed:
|
||||
self._deny("closed")
|
||||
if self._failure:
|
||||
raise permission_error(self._failure)
|
||||
|
||||
async def close(self) -> None:
|
||||
async with self._lock:
|
||||
self._closed = True
|
||||
self._permissions.clear()
|
||||
self._written.clear()
|
||||
|
||||
def _deny(self, suffix: str) -> NoReturn:
|
||||
self._failure = self._failure or f"model_content_hy_coding_store_{suffix}"
|
||||
raise permission_error(self._failure)
|
||||
|
||||
def _scope(self, user_id: str | None) -> None:
|
||||
if user_id not in (None, self._user_id):
|
||||
self._deny("scope")
|
||||
|
||||
async def _invoke(self, method: str, *args: Any, **kwargs: Any) -> Any:
|
||||
writing = method in _NATIVE_WRITES
|
||||
self.recheck() if writing else self._assert_open()
|
||||
try:
|
||||
result = await getattr(self._native, method)(*args, **kwargs)
|
||||
except Exception:
|
||||
self._deny("backend_failed")
|
||||
self.recheck() if writing else self._assert_open()
|
||||
return result
|
||||
|
||||
def _visible(self, row: Any, expected: Any) -> bool:
|
||||
if row is None:
|
||||
return False
|
||||
if type(row) is not expected:
|
||||
self._deny("row_shape")
|
||||
key = _row_id(row)
|
||||
proof = self._permissions.get(key)
|
||||
if proof is None or key in self._deleted or row.user_id != self._user_id:
|
||||
return False
|
||||
if hy_coding_fingerprint(row) != proof.fingerprint:
|
||||
self._deny("row_changed")
|
||||
return True
|
||||
|
||||
async def _experience(self, key: str) -> CodingMemory | None:
|
||||
if key not in self._permissions or key in self._deleted:
|
||||
return None
|
||||
row = await self._invoke("get_by_id", key, user_id=self._user_id)
|
||||
return deepcopy(row) if self._visible(row, CodingMemory) else None
|
||||
|
||||
async def _fact(self, key: str) -> WorkingFact | None:
|
||||
if key not in self._permissions or key in self._deleted:
|
||||
return None
|
||||
rows = await self._invoke("get_facts", [key], user_id=self._user_id)
|
||||
row = rows[0] if rows else None
|
||||
return deepcopy(row) if self._visible(row, WorkingFact) else None
|
||||
|
||||
def _new_row_in_scope(self, row: CodingMemory | WorkingFact) -> bool:
|
||||
return bool(
|
||||
row.user_id == self._user_id
|
||||
and row.agent_id == self._agent_id
|
||||
and row.session_id == self._session_id
|
||||
)
|
||||
|
||||
async def _capture(
|
||||
self, key: str, kind: Any, expected: CodingMemory | WorkingFact | None = None
|
||||
) -> None:
|
||||
row: Any
|
||||
if kind is CodingMemory:
|
||||
row = await self._invoke("get_by_id", key, user_id=self._user_id)
|
||||
else:
|
||||
rows = await self._invoke("get_facts", [key], user_id=self._user_id)
|
||||
row = rows[0] if rows else None
|
||||
if type(row) is not kind or row.user_id != self._user_id:
|
||||
self._deny("write_mismatch")
|
||||
if expected is not None:
|
||||
stored, planned = row.to_dict(), expected.to_dict()
|
||||
for field in _TIMESTAMPS:
|
||||
stored.pop(field, None)
|
||||
planned.pop(field, None)
|
||||
if stored != planned:
|
||||
self._deny("write_mismatch")
|
||||
proof = HyNodePermission(hy_coding_fingerprint(row), self._evidence_ids)
|
||||
self._permissions[key] = proof
|
||||
self._written[key] = proof
|
||||
|
||||
async def _plan_new(self, row: CodingMemory | WorkingFact, kind: Any) -> None:
|
||||
if type(row) is not kind or not self._new_row_in_scope(row):
|
||||
self._deny("scope")
|
||||
key = _row_id(row)
|
||||
if not key or key in self._permissions or key in self._deleted:
|
||||
self._deny("scope")
|
||||
if kind is CodingMemory:
|
||||
existing = await self._invoke("get_by_id", key)
|
||||
else:
|
||||
existing = next(iter(await self._invoke("get_facts", [key])), None)
|
||||
if existing is not None:
|
||||
self._deny("scope")
|
||||
|
||||
# Reads
|
||||
|
||||
async def get_by_id(
|
||||
self, experience_id: str, user_id: str | None = None
|
||||
) -> CodingMemory | None:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
return await self._experience(experience_id)
|
||||
|
||||
async def get_many(
|
||||
self, experience_ids: list[str], user_id: str | None = None
|
||||
) -> list[CodingMemory]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
wanted = [
|
||||
key
|
||||
for key in dict.fromkeys(experience_ids)
|
||||
if key in self._permissions and key not in self._deleted
|
||||
]
|
||||
if not wanted:
|
||||
return []
|
||||
rows = await self._invoke("get_many", wanted, user_id=self._user_id)
|
||||
return [deepcopy(row) for row in rows if self._visible(row, CodingMemory)]
|
||||
|
||||
async def search_by_query_embedding(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
user_id: str,
|
||||
*,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
top: int = 40,
|
||||
) -> list[dict[str, Any]]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
hits = await self._invoke(
|
||||
"search_by_query_embedding",
|
||||
deepcopy(query_embedding),
|
||||
self._user_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
top=top * 4,
|
||||
)
|
||||
result = []
|
||||
for hit in hits:
|
||||
key = hit.get("experience_id")
|
||||
if not isinstance(key, str) or await self._experience(key) is None:
|
||||
continue
|
||||
result.append(
|
||||
{
|
||||
"experience_id": key,
|
||||
"score": float(hit.get("score", 0.0)),
|
||||
"matched_key_kind": hit.get("matched_key_kind"),
|
||||
"matched_key_text": hit.get("matched_key_text"),
|
||||
}
|
||||
)
|
||||
return result[:top]
|
||||
|
||||
async def get_fact(self, fact_id: str, user_id: str | None = None) -> WorkingFact | None:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
return await self._fact(fact_id)
|
||||
|
||||
async def get_facts(self, fact_ids: list[str], user_id: str | None = None) -> list[WorkingFact]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
found = [await self._fact(key) for key in dict.fromkeys(fact_ids)]
|
||||
return [row for row in found if row is not None]
|
||||
|
||||
async def search_facts(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
user_id: str,
|
||||
*,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
top: int = 5,
|
||||
query_text: str | None = None,
|
||||
min_score: float = 0.0,
|
||||
) -> list[tuple[WorkingFact, float]]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
hits = await self._invoke(
|
||||
"search_facts",
|
||||
deepcopy(query_embedding),
|
||||
self._user_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
top=top * 4,
|
||||
query_text=query_text,
|
||||
min_score=min_score,
|
||||
)
|
||||
result = []
|
||||
for row, score in hits:
|
||||
if self._visible(row, WorkingFact):
|
||||
result.append((deepcopy(row), float(score)))
|
||||
return result[:top]
|
||||
|
||||
async def get_fact_ancestors(
|
||||
self, fact_id: str, *, user_id: str | None = None, max_depth: int = 8
|
||||
) -> list[WorkingFact]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
if await self._fact(fact_id) is None:
|
||||
return []
|
||||
rows = await self._invoke(
|
||||
"get_fact_ancestors", fact_id, user_id=self._user_id, max_depth=max_depth
|
||||
)
|
||||
# Oldest first; the chain is only as long as its nearest unapproved link.
|
||||
kept: list[WorkingFact] = []
|
||||
for row in reversed(rows):
|
||||
if not self._visible(row, WorkingFact):
|
||||
break
|
||||
kept.append(deepcopy(row))
|
||||
return list(reversed(kept))
|
||||
|
||||
# Writes
|
||||
|
||||
async def insert(self, memory: CodingMemory) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
await self._plan_new(memory, CodingMemory)
|
||||
await self._invoke("insert", memory)
|
||||
await self._capture(memory.experience_id, CodingMemory, memory)
|
||||
|
||||
async def update(self, memory: CodingMemory) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if type(memory) is not CodingMemory or not self._new_row_in_scope(memory):
|
||||
self._deny("scope")
|
||||
if await self._experience(memory.experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
await self._invoke("update", memory)
|
||||
await self._capture(memory.experience_id, CodingMemory, memory)
|
||||
|
||||
async def delete(self, experience_id: str, user_id: str) -> bool:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
if await self._experience(experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
if await self._invoke("delete", experience_id, user_id=self._user_id) is not True:
|
||||
self._deny("write_failed")
|
||||
if await self._invoke("get_by_id", experience_id) is not None:
|
||||
self._deny("write_mismatch")
|
||||
self._deleted.add(experience_id)
|
||||
self._permissions.pop(experience_id, None)
|
||||
self._written.pop(experience_id, None)
|
||||
return True
|
||||
|
||||
async def insert_fact(self, fact: WorkingFact) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
await self._plan_new(fact, WorkingFact)
|
||||
await self._invoke("insert_fact", fact)
|
||||
await self._capture(fact.fact_id, WorkingFact, fact)
|
||||
|
||||
async def supersede_fact(self, old_fact_id: str, new_fact_id: str) -> bool:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._fact(old_fact_id) is None or old_fact_id == new_fact_id:
|
||||
self._deny("target_denied")
|
||||
# HY links the old fact before it inserts the new one. A successor
|
||||
# that does not exist yet may only become a row this lease writes;
|
||||
# an existing row it cannot attest stays out of reach.
|
||||
if await self._fact(new_fact_id) is None:
|
||||
if not new_fact_id or new_fact_id in self._deleted:
|
||||
self._deny("target_denied")
|
||||
if next(iter(await self._invoke("get_facts", [new_fact_id])), None) is not None:
|
||||
self._deny("target_denied")
|
||||
changed = await self._invoke("supersede_fact", old_fact_id, new_fact_id)
|
||||
await self._capture(old_fact_id, WorkingFact)
|
||||
if changed:
|
||||
self._superseded[old_fact_id] = new_fact_id
|
||||
return bool(changed)
|
||||
|
||||
async def set_fact_derived_experiences(self, fact_id: str, experience_ids: list[str]) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._fact(fact_id) is None:
|
||||
self._deny("target_denied")
|
||||
for key in experience_ids:
|
||||
if await self._experience(key) is None:
|
||||
self._deny("link_denied")
|
||||
await self._invoke("set_fact_derived_experiences", fact_id, list(experience_ids))
|
||||
await self._capture(fact_id, WorkingFact)
|
||||
|
||||
async def set_experience_source_facts(self, experience_id: str, fact_ids: list[str]) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._experience(experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
for key in fact_ids:
|
||||
if await self._fact(key) is None:
|
||||
self._deny("link_denied")
|
||||
await self._invoke("set_experience_source_facts", experience_id, list(fact_ids))
|
||||
await self._capture(experience_id, CodingMemory)
|
||||
|
||||
async def get_recent_trace_messages(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
max_user_turns: int = 5,
|
||||
) -> list[Any]:
|
||||
"""Read recent raw trace rows, still bounded by this lease.
|
||||
|
||||
Public SDK 2.0.2 asks for recent history before a coding write. The native call is
|
||||
scoped to the lease's own user and agent, and rows this lease cannot
|
||||
attest are filtered out, so unapproved conversations never reach a
|
||||
coding prompt through this path.
|
||||
"""
|
||||
self._scope(user_id)
|
||||
if agent_id not in (self._agent_id, "*"):
|
||||
self._deny("scope")
|
||||
rows = await self._invoke(
|
||||
"get_recent_trace_messages",
|
||||
user_id=self._user_id,
|
||||
agent_id=self._agent_id,
|
||||
max_user_turns=max_user_turns,
|
||||
)
|
||||
return [row for row in (rows or []) if self._trace_visible(row)]
|
||||
|
||||
async def append_trace(
|
||||
self,
|
||||
messages: Sequence[Any],
|
||||
*,
|
||||
request_id: str,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
session_id: str | None = None,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
) -> None:
|
||||
"""Append raw trace rows for this lease only.
|
||||
|
||||
Trace rows are raw conversation, so this is a write path: the lease is
|
||||
rechecked around the call and every message must belong to the leased
|
||||
user and session. Nothing outside the lease may be recorded.
|
||||
"""
|
||||
self._scope(user_id)
|
||||
if agent_id not in (self._agent_id, "*"):
|
||||
self._deny("scope")
|
||||
if session_id not in (None, self._session_id):
|
||||
self._deny("scope")
|
||||
for row in messages or []:
|
||||
if not self._trace_visible(row):
|
||||
self._deny("scope")
|
||||
await self._invoke(
|
||||
"append_trace",
|
||||
list(messages or []),
|
||||
request_id=request_id,
|
||||
user_id=self._user_id,
|
||||
agent_id=self._agent_id,
|
||||
session_id=self._session_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
)
|
||||
|
||||
def _trace_visible(self, row: Any) -> bool:
|
||||
"""A trace row may be read or written only inside this lease's scope."""
|
||||
if row is None:
|
||||
return False
|
||||
owner = getattr(row, "user_id", None)
|
||||
session = getattr(row, "session_id", None)
|
||||
return not (
|
||||
(owner is not None and owner != self._user_id)
|
||||
or (session is not None and session != self._session_id)
|
||||
)
|
||||
@@ -0,0 +1,582 @@
|
||||
"""Task-exclusive HY wiring; provenance metadata stays beside the existing Core DB."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from collections.abc import AsyncIterator, Callable
|
||||
from contextlib import asynccontextmanager, nullcontext, suppress
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
import httpx
|
||||
|
||||
from one_memory.database import take_the_write_lock_first
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_memory_bridge import (
|
||||
HyMemoryBridge,
|
||||
HyMemoryBridgeError,
|
||||
HyMemoryIngestRequest,
|
||||
HyMemoryIngestResult,
|
||||
HyMemoryOutput,
|
||||
LocalHyEmbedService,
|
||||
_joined_thread,
|
||||
)
|
||||
from one_memory.hy_permission_coding_store import HyPermissionCodingStore
|
||||
from one_memory.hy_permission_store import (
|
||||
HyNodePermission,
|
||||
HyPermissionGraphStore,
|
||||
HyPermissionVectorStore,
|
||||
)
|
||||
from one_memory.hy_permission_transport import HyPermissionTransport, HyPermissionTransportError
|
||||
from one_memory.hy_response_guard import HyResponseGuard
|
||||
from one_memory.hy_step_timing import HyStepTimer
|
||||
from one_memory.memory_server.migrations import hy_node_permissions_ddl
|
||||
from one_memory.memory_tasks import evidence_ref
|
||||
from one_memory.model_processing_permission_guard import ModelProcessingPermissionLease
|
||||
|
||||
|
||||
class _NativeProofs:
|
||||
def __init__(self, database: Path) -> None:
|
||||
self.database = database
|
||||
self._loaded_states: dict[str, tuple[int, str]] | None = None
|
||||
with main_database(database).write() as db, db:
|
||||
# Schema owned by `memory_server.migrations` (migration 26).
|
||||
db.execute(hy_node_permissions_ddl("sqlite"))
|
||||
|
||||
def load(self, lease: ModelProcessingPermissionLease) -> dict[str, HyNodePermission]:
|
||||
allowed = {record.evidence_id: evidence_ref(record) for record in lease.records}
|
||||
with main_database(self.database).read() as db:
|
||||
rows = db.execute(
|
||||
"SELECT p.node_id,p.fingerprint,p.evidence_json,g.revision,g.status "
|
||||
"FROM hy_node_permissions p LEFT JOIN governed_memories g ON g.memory_id=p.node_id "
|
||||
# Superseded nodes stay attested: every later head links back to them.
|
||||
"WHERE g.status IS NULL OR g.status IN ('confirmed','superseded')"
|
||||
).fetchall()
|
||||
proofs = {}
|
||||
states = {}
|
||||
for identifier, fingerprint, evidence_json, revision, status in rows:
|
||||
refs = json.loads(evidence_json)
|
||||
if (
|
||||
not isinstance(refs, list)
|
||||
or not refs
|
||||
or not all(isinstance(ref, dict) for ref in refs)
|
||||
):
|
||||
continue
|
||||
if all(allowed.get(ref.get("evidence_id")) == ref for ref in refs):
|
||||
proofs[identifier] = HyNodePermission(
|
||||
fingerprint, tuple(ref["evidence_id"] for ref in refs)
|
||||
)
|
||||
if revision is not None:
|
||||
states[identifier] = (revision, status)
|
||||
self._loaded_states = states
|
||||
return proofs
|
||||
|
||||
def guard(self, values: dict[str, HyNodePermission]) -> Callable[[], None]:
|
||||
identifiers = tuple(values)
|
||||
|
||||
def state() -> dict[str, tuple[int, str]]:
|
||||
found: dict[str, tuple[int, str]] = {}
|
||||
with main_database(self.database).read() as db:
|
||||
for start in range(0, len(identifiers), 900):
|
||||
batch = identifiers[start : start + 900]
|
||||
placeholders = ",".join("?" for _ in batch)
|
||||
for identifier, revision, status in db.execute(
|
||||
"SELECT memory_id,revision,status FROM governed_memories "
|
||||
f"WHERE memory_id IN ({placeholders})",
|
||||
batch,
|
||||
):
|
||||
found[identifier] = (revision, status)
|
||||
return found
|
||||
|
||||
if self._loaded_states is None:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
before = dict(self._loaded_states)
|
||||
|
||||
def check() -> None:
|
||||
if state() != before:
|
||||
raise HyMemoryBridgeError("model_content_context_changed")
|
||||
|
||||
check()
|
||||
return check
|
||||
|
||||
def save(
|
||||
self, lease: ModelProcessingPermissionLease, values: dict[str, HyNodePermission]
|
||||
) -> None:
|
||||
refs = {record.evidence_id: evidence_ref(record) for record in lease.records}
|
||||
with main_database(self.database).write(autocommit=True) as db:
|
||||
take_the_write_lock_first(db)
|
||||
with db:
|
||||
lease.recheck()
|
||||
db.executemany(
|
||||
"INSERT INTO hy_node_permissions VALUES (?,?,?) ON CONFLICT(node_id) "
|
||||
"DO UPDATE SET fingerprint=excluded.fingerprint,"
|
||||
"evidence_json=excluded.evidence_json",
|
||||
[
|
||||
(
|
||||
identifier,
|
||||
proof.fingerprint,
|
||||
json.dumps([refs[key] for key in proof.evidence_ids]),
|
||||
)
|
||||
for identifier, proof in values.items()
|
||||
],
|
||||
)
|
||||
|
||||
def refresh(
|
||||
self, lease: ModelProcessingPermissionLease, node_id: str
|
||||
) -> HyNodePermission | None:
|
||||
del lease
|
||||
with main_database(self.database).read() as db:
|
||||
row = db.execute(
|
||||
"SELECT fingerprint,evidence_json FROM hy_node_permissions WHERE node_id=?",
|
||||
(node_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
fingerprint = str(row["fingerprint"])
|
||||
refs = json.loads(row["evidence_json"])
|
||||
if (
|
||||
len(fingerprint) != 64
|
||||
or any(value not in "0123456789abcdef" for value in fingerprint)
|
||||
or not isinstance(refs, list)
|
||||
or not refs
|
||||
or not all(isinstance(ref, dict) for ref in refs)
|
||||
):
|
||||
return None
|
||||
evidence_ids = tuple(ref.get("evidence_id") for ref in refs)
|
||||
if any(not isinstance(value, str) or not value for value in evidence_ids):
|
||||
return None
|
||||
return HyNodePermission(fingerprint, evidence_ids)
|
||||
|
||||
|
||||
class _BoundLease:
|
||||
def __init__(
|
||||
self,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proof_check: Callable[[], None],
|
||||
step_timer: HyStepTimer | None = None,
|
||||
) -> None:
|
||||
self.lease = lease
|
||||
self.step_timer = step_timer
|
||||
self.user_id = lease.user_id
|
||||
self.profile = lease.profile
|
||||
self.task_id = lease.task_id
|
||||
self.records = lease.records
|
||||
self.source_context = lease.source_context
|
||||
self.proof_check = proof_check
|
||||
self.vector: HyPermissionVectorStore | None = None
|
||||
self.coding: HyPermissionCodingStore | None = None
|
||||
|
||||
def recheck(self) -> None:
|
||||
timer = self.step_timer
|
||||
with timer.step("permission_check") if timer is not None else nullcontext():
|
||||
self.lease.recheck()
|
||||
self.proof_check()
|
||||
if self.vector is not None:
|
||||
self.vector.recheck()
|
||||
if self.coding is not None:
|
||||
self.coding.recheck()
|
||||
|
||||
|
||||
class _Wiring:
|
||||
def __init__(
|
||||
self,
|
||||
client: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proofs_store: _NativeProofs,
|
||||
proofs: dict[str, HyNodePermission],
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
transport: httpx.AsyncBaseTransport | None,
|
||||
proof_check: Callable[[], None],
|
||||
step_timer: HyStepTimer,
|
||||
) -> None:
|
||||
self.client, self.lease, self.proofs_store, self.proofs = (
|
||||
client,
|
||||
lease,
|
||||
proofs_store,
|
||||
proofs,
|
||||
)
|
||||
self.step_timer = step_timer
|
||||
self.agent_id, self.session_id, self.http_transport = agent_id, session_id, transport
|
||||
self.cross_evidence_scope = False
|
||||
self.proof_check = proof_check
|
||||
self.saved: list[tuple[Any, str, Any]] = []
|
||||
self.transport: HyPermissionTransport
|
||||
self.vector: HyPermissionVectorStore
|
||||
self.graph: HyPermissionGraphStore | None = None
|
||||
self.coding: HyPermissionCodingStore
|
||||
self.active = False
|
||||
|
||||
def replace(self, target: Any, name: str, value: Any) -> None:
|
||||
self.saved.append((target, name, getattr(target, name)))
|
||||
setattr(target, name, value)
|
||||
|
||||
async def enter(self) -> None:
|
||||
client, registry = self.client, self.client._registry
|
||||
config = client._config
|
||||
if config.llm.enable_summary or config.llm.enable_query_rewrite:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
bound = _BoundLease(self.lease, self.proof_check, self.step_timer)
|
||||
self.transport = await HyPermissionTransport(
|
||||
bound, http_transport=self.http_transport, step_timer=self.step_timer
|
||||
).__aenter__()
|
||||
self.active = True
|
||||
try:
|
||||
for guard in getattr(client, "_one_memory_write_guards", ()):
|
||||
if hasattr(type(guard), "step_timer"):
|
||||
self.replace(guard, "step_timer", self.step_timer)
|
||||
system2 = await registry.get_system2_writer()
|
||||
writers = (await registry.get_writer(), system2._writer)
|
||||
native_vector = client._vector_store
|
||||
store_lease = _BoundLease(self.lease, self.proof_check, self.step_timer)
|
||||
self.vector = HyPermissionVectorStore(
|
||||
native_vector,
|
||||
cast(ModelProcessingPermissionLease, store_lease),
|
||||
agent_id=self.agent_id,
|
||||
session_id=self.session_id,
|
||||
cross_evidence_scope=self.cross_evidence_scope,
|
||||
node_permissions=self.proofs,
|
||||
refresh_permission=lambda node_id: asyncio.to_thread(
|
||||
self.proofs_store.refresh, self.lease, node_id
|
||||
),
|
||||
step_timer=self.step_timer,
|
||||
)
|
||||
bound.vector = self.vector
|
||||
embed = client._embed_service
|
||||
if isinstance(embed, LocalHyEmbedService):
|
||||
self.transport.bind_local_embed_service(embed)
|
||||
else:
|
||||
self.transport.bind_embed_service(embed)
|
||||
for writer in writers:
|
||||
provider = writer.mem_agent.llm_provider
|
||||
self.transport.bind_provider(
|
||||
provider.provider if isinstance(provider, HyResponseGuard) else provider
|
||||
)
|
||||
reconciler = writer.reconciler
|
||||
provider = reconciler._get_llm()
|
||||
self.transport.bind_provider(
|
||||
provider.provider if isinstance(provider, HyResponseGuard) else provider
|
||||
)
|
||||
self.replace(writer, "_external_vector_store", self.vector)
|
||||
self.replace(writer, "_vector_store", self.vector)
|
||||
self.replace(writer, "_vector_store_initialized", True)
|
||||
self.replace(writer, "_cache", None)
|
||||
self.replace(client, "_vector_store", self.vector)
|
||||
native_graph = client._graph_store
|
||||
if self.cross_evidence_scope and native_graph is not None:
|
||||
self.graph = HyPermissionGraphStore(native_graph, self.proof_check)
|
||||
self.replace(client, "_graph_store", self.graph)
|
||||
else:
|
||||
self.replace(client, "_graph_store", None)
|
||||
self.replace(registry, "_shared_vector_store", self.vector)
|
||||
self.replace(
|
||||
registry,
|
||||
"_shared_graph_store",
|
||||
self.graph if self.graph is not None else None,
|
||||
)
|
||||
self.replace(registry, "_shared_cache", None)
|
||||
self.replace(system2, "_vector_store", self.vector)
|
||||
self.replace(
|
||||
system2,
|
||||
"_graph_store",
|
||||
self.graph if self.graph is not None else None,
|
||||
)
|
||||
self.replace(system2, "_cache", None)
|
||||
# Result fallback may call search; rebuild only its non-generative reader.
|
||||
self.replace(registry, "_reader", None)
|
||||
self.replace(registry, "_readers", {})
|
||||
await self._enter_coding(bound, embed)
|
||||
self.transport.assert_allowed()
|
||||
except BaseException:
|
||||
await self.exit()
|
||||
raise
|
||||
|
||||
async def _enter_coding(self, bound: _BoundLease, embed: Any) -> None:
|
||||
"""HY judges and writes the coding route itself; every seam it uses is bound here."""
|
||||
client = self.client
|
||||
# Created now, so the judge and every coding stage share one bound provider.
|
||||
writer = await client._get_coding_writer()
|
||||
stages = (writer, writer.extractor, writer.reconciler, writer.fact_reconciler)
|
||||
native_store = client._coding_store
|
||||
if any(
|
||||
stage.store is not native_store
|
||||
for stage in (writer, writer.reconciler, writer.fact_reconciler)
|
||||
) or any(
|
||||
item.embed_service is not embed
|
||||
for item in (native_store, writer, writer.reconciler, writer.fact_reconciler)
|
||||
):
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
providers = {
|
||||
id(p): p for p in (client._coding_llm_provider, *(s.llm_provider for s in stages))
|
||||
}
|
||||
for provider in providers.values():
|
||||
self.transport.bind_provider(provider)
|
||||
self.coding = HyPermissionCodingStore(
|
||||
native_store,
|
||||
cast(
|
||||
ModelProcessingPermissionLease,
|
||||
_BoundLease(self.lease, self.proof_check, self.step_timer),
|
||||
),
|
||||
agent_id=self.agent_id,
|
||||
session_id=self.session_id,
|
||||
node_permissions=self.proofs,
|
||||
)
|
||||
bound.coding = self.coding
|
||||
for stage in (writer, writer.reconciler, writer.fact_reconciler):
|
||||
self.replace(stage, "store", self.coding)
|
||||
self.replace(client, "_coding_store", self.coding)
|
||||
for stage in (writer, writer.extractor, writer.reconciler):
|
||||
self.replace(stage, "cache", None)
|
||||
# The judge logs its prompt through the client's own cache.
|
||||
self.replace(client, "_cache", None)
|
||||
|
||||
async def finish(self) -> dict[str, HyNodePermission]:
|
||||
self.transport.assert_allowed()
|
||||
self.vector.recheck()
|
||||
self.coding.recheck()
|
||||
return {**self.vector.written_nodes, **self.coding.written_nodes}
|
||||
|
||||
async def retired(self) -> tuple[tuple[str, ...], dict[str, tuple[str, ...]]]:
|
||||
self.transport.assert_allowed()
|
||||
return (
|
||||
tuple(sorted(self.vector.retired_node_ids | self.coding.retired_ids)),
|
||||
{**self.vector.superseded_nodes, **self.coding.superseded_ids},
|
||||
)
|
||||
|
||||
async def exit(self) -> None:
|
||||
if not self.active:
|
||||
return
|
||||
self.active = False
|
||||
try:
|
||||
await self.transport.__aexit__(None, None, None)
|
||||
finally:
|
||||
for target, name, value in reversed(self.saved):
|
||||
setattr(target, name, value)
|
||||
self.saved.clear()
|
||||
if hasattr(self, "vector"):
|
||||
await self.vector.close()
|
||||
if self.graph is not None:
|
||||
await self.graph.close()
|
||||
if hasattr(self, "coding"):
|
||||
await self.coding.close()
|
||||
|
||||
|
||||
class _LeasedBridge:
|
||||
def __init__(
|
||||
self,
|
||||
bridge: HyMemoryBridge,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proofs: _NativeProofs,
|
||||
transport: httpx.AsyncBaseTransport | None,
|
||||
) -> None:
|
||||
self.bridge, self.lease, self.proofs, self.transport = bridge, lease, proofs, transport
|
||||
self.wiring: _Wiring | None = None
|
||||
self.finished = False
|
||||
self.step_timer = HyStepTimer()
|
||||
self.request_id: str | None = None
|
||||
|
||||
def _step(self, name: str) -> Any:
|
||||
return self.step_timer.step(name)
|
||||
|
||||
async def _recheck(self) -> None:
|
||||
def recheck() -> None:
|
||||
with self._step("permission_check"):
|
||||
self.lease.recheck()
|
||||
|
||||
await _joined_thread(recheck)
|
||||
|
||||
async def _start(self, request: HyMemoryIngestRequest) -> None:
|
||||
if self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_reused")
|
||||
if request.user_id != self.lease.user_id or set(request.evidence_ids) - {
|
||||
item.evidence_id for item in self.lease.records
|
||||
}:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._start_operation(
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
request_id=request.request_id,
|
||||
session_id=request.session_id,
|
||||
)
|
||||
|
||||
async def _start_operation(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
request_id: str,
|
||||
session_id: str,
|
||||
cross_evidence_scope: bool = False,
|
||||
) -> None:
|
||||
if self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_reused")
|
||||
if user_id != self.lease.user_id:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
self.request_id = request_id
|
||||
await self._recheck()
|
||||
client = cast(Any, await self.bridge._get_client())
|
||||
with self._step("permission_store_read"):
|
||||
known = await asyncio.to_thread(self.proofs.load, self.lease)
|
||||
proof_check = await asyncio.to_thread(self.proofs.guard, known)
|
||||
self.wiring = _Wiring(
|
||||
client,
|
||||
self.lease,
|
||||
self.proofs,
|
||||
known,
|
||||
agent_id,
|
||||
session_id,
|
||||
self.transport,
|
||||
proof_check,
|
||||
self.step_timer,
|
||||
)
|
||||
self.wiring.cross_evidence_scope = cross_evidence_scope
|
||||
await _joined_thread(client._loop_thread.run, self.wiring.enter())
|
||||
|
||||
async def list_agent_ids(self, *, user_id: str, limit: int = 1_000) -> tuple[str, ...]:
|
||||
if user_id != self.lease.user_id or self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._recheck()
|
||||
result = await self.bridge.list_agent_ids(user_id=user_id, limit=limit)
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def digest(self, *, user_id: str, agent_id: str) -> dict[str, object]:
|
||||
await self._start_operation(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
request_id=f"digest:{self.lease.task_id}:{agent_id}",
|
||||
session_id=f"digest:{self.lease.task_id}",
|
||||
cross_evidence_scope=True,
|
||||
)
|
||||
try:
|
||||
result = dict(await self.bridge.digest(user_id=user_id, agent_id=agent_id))
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
proofs = await self._check()
|
||||
with self._step("permission_store_write"):
|
||||
await asyncio.to_thread(self.proofs.save, self.lease, proofs)
|
||||
await self.close()
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def graph_snapshot(self, *, user_id: str, agent_id: str, limit: int = 300) -> Any:
|
||||
await self._start_operation(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
request_id=f"digest-graph:{self.lease.task_id}:{agent_id}",
|
||||
session_id=f"digest:{self.lease.task_id}",
|
||||
cross_evidence_scope=True,
|
||||
)
|
||||
try:
|
||||
result = await self.bridge.graph_snapshot(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
limit=limit,
|
||||
)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
await self._check()
|
||||
await self.close()
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
|
||||
await self._start(request)
|
||||
try:
|
||||
result = await self.bridge.ingest(request)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
return await self._with_retired(result)
|
||||
|
||||
async def ingest_extracted(
|
||||
self, request: HyMemoryIngestRequest, contents: tuple[str, ...]
|
||||
) -> HyMemoryIngestResult:
|
||||
await self._start(request)
|
||||
try:
|
||||
result = await self.bridge.ingest_extracted(request, contents)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
return await self._with_retired(result)
|
||||
|
||||
async def _with_retired(self, result: HyMemoryIngestResult) -> HyMemoryIngestResult:
|
||||
await self._check()
|
||||
if self.wiring is None:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
try:
|
||||
retired, superseded = await _joined_thread(
|
||||
self.wiring.client._loop_thread.run, self.wiring.retired()
|
||||
)
|
||||
except HyPermissionTransportError as error:
|
||||
await self._recheck()
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
return result.model_copy(
|
||||
update={"retired_memory_ids": retired, "superseded_memory_ids": superseded}
|
||||
)
|
||||
|
||||
async def _check(self) -> dict[str, HyNodePermission]:
|
||||
if self.wiring is None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._recheck()
|
||||
try:
|
||||
return await _joined_thread(self.wiring.client._loop_thread.run, self.wiring.finish())
|
||||
except HyPermissionTransportError as error:
|
||||
await self._recheck()
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
|
||||
async def profile_memories(self, *, user_id: str, agent_id: str) -> tuple[HyMemoryOutput, ...]:
|
||||
if self.wiring is None or user_id != self.lease.user_id or agent_id != self.wiring.agent_id:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
result = await self.bridge.profile_memories(user_id=user_id, agent_id=agent_id)
|
||||
proofs = await self._check()
|
||||
with self._step("permission_store_write"):
|
||||
await asyncio.to_thread(self.proofs.save, self.lease, proofs)
|
||||
# SDK context closes before governance can update a previously known memory.
|
||||
await self.close()
|
||||
return result
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.wiring is not None and not self.finished:
|
||||
self.finished = True
|
||||
try:
|
||||
await _joined_thread(self.wiring.client._loop_thread.run, self.wiring.exit())
|
||||
finally:
|
||||
self.step_timer.emit(request_id=self.request_id, task_id=self.lease.task_id)
|
||||
|
||||
|
||||
def permission_bridge_factory(
|
||||
resolve_bridge: Callable[..., Any],
|
||||
*,
|
||||
http_transport: httpx.AsyncBaseTransport | None = None,
|
||||
) -> Callable[..., Any]:
|
||||
@asynccontextmanager
|
||||
async def factory(
|
||||
lease: ModelProcessingPermissionLease, secret: str | None, data_dir: Path
|
||||
) -> AsyncIterator[Any]:
|
||||
bridge = resolve_bridge(lease.profile, secret, data_dir)
|
||||
if not isinstance(bridge, HyMemoryBridge):
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
proofs = _NativeProofs(data_dir / "one-memory.sqlite3")
|
||||
async with bridge.exclusive_access():
|
||||
await bridge.prepare()
|
||||
bound = _LeasedBridge(bridge, lease, proofs, http_transport)
|
||||
try:
|
||||
yield bound
|
||||
except BaseException as error:
|
||||
with suppress(Exception):
|
||||
await bound.close()
|
||||
if isinstance(error, HyPermissionTransportError):
|
||||
await _joined_thread(lease.recheck)
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
raise
|
||||
else:
|
||||
try:
|
||||
await bound.close()
|
||||
except HyPermissionTransportError as error:
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
|
||||
return factory
|
||||
@@ -0,0 +1,880 @@
|
||||
"""Operation-bound read/write capability for the pinned native HY chat vector store.
|
||||
|
||||
No semantic storage or extraction lives here. Pre-existing nodes require a full
|
||||
native-node attestation with complete approved Evidence provenance; known content
|
||||
alone cannot authorize tags, custom fields, vectors or links. Unknown nodes stay
|
||||
in the same HY backend but are not exposed to this operation.
|
||||
|
||||
The caller must exclusively own the native backend's writes during execution:
|
||||
its API has no compare-and-swap, and a swallowed get_by_id error can look missing.
|
||||
This component cannot make that check/upsert interval atomic. Factory integration
|
||||
must replace client/registry/both writers/cached-reader references, bind physical
|
||||
LLM/embedding sends to the lease AND this denial latch, and guard publication.
|
||||
Coding rows go through hy_permission_coding_store instead; graph, entities,
|
||||
digest, sweeper and summary remain unsupported.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import math
|
||||
import struct
|
||||
import weakref
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable, Mapping
|
||||
from contextlib import asynccontextmanager, nullcontext
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from hashlib import sha256
|
||||
from types import MappingProxyType
|
||||
from typing import Any, NoReturn
|
||||
|
||||
from hy_memory.models.memory import ( # type: ignore[import-untyped]
|
||||
MemoryLayer,
|
||||
MemoryNode,
|
||||
MemoryStatus,
|
||||
)
|
||||
|
||||
from one_memory.hy_step_timing import HyStepTimer
|
||||
from one_memory.model_processing_permission_guard import (
|
||||
ModelProcessingPermissionLease,
|
||||
permission_error,
|
||||
)
|
||||
|
||||
_WRITE_LOCKS: weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
|
||||
weakref.WeakKeyDictionary()
|
||||
)
|
||||
|
||||
|
||||
def _native_write_lock() -> asyncio.Lock:
|
||||
loop = asyncio.get_running_loop()
|
||||
lock = _WRITE_LOCKS.get(loop)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
_WRITE_LOCKS[loop] = lock
|
||||
return lock
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def native_store_read_slot() -> AsyncIterator[bool]:
|
||||
"""Admit one native HY read only while no write is active.
|
||||
|
||||
Chroma's native reader may retain the GIL after an asyncio timeout. Running
|
||||
it concurrently with HY's multi-step store writes has deadlocked the Core
|
||||
under sustained imports. Reads therefore share the physical store lock,
|
||||
but never wait behind a writer: host recall can immediately use the local
|
||||
governance ledger instead.
|
||||
"""
|
||||
lock = _native_write_lock()
|
||||
if lock.locked():
|
||||
yield False
|
||||
return
|
||||
await lock.acquire()
|
||||
try:
|
||||
yield True
|
||||
finally:
|
||||
lock.release()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HyNodePermission:
|
||||
"""Trusted caller's full-node attestation, not permission inferred from an ID."""
|
||||
|
||||
fingerprint: str
|
||||
evidence_ids: tuple[str, ...]
|
||||
|
||||
|
||||
def hy_node_fingerprint(node: MemoryNode) -> str:
|
||||
"""Hash the complete native serialized node, including vectors and metadata.
|
||||
|
||||
Timestamp precision follows HY's to_dict contract (integer Unix seconds).
|
||||
Attest the node read from the backend, not an unpersisted input object.
|
||||
"""
|
||||
if type(node) is not MemoryNode:
|
||||
raise permission_error("hy_store_node_shape")
|
||||
return sha256(
|
||||
json.dumps(node.to_dict(), sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
_CHAT_LAYERS = frozenset(
|
||||
{
|
||||
MemoryLayer.L0_BASIC_INFO,
|
||||
MemoryLayer.L1_RAW,
|
||||
MemoryLayer.L2_FACT,
|
||||
MemoryLayer.L4_IDENTITY,
|
||||
MemoryLayer.L7_INTENTION,
|
||||
}
|
||||
)
|
||||
_UPDATE_FIELDS = frozenset(
|
||||
{
|
||||
"content",
|
||||
"embedding",
|
||||
"tags",
|
||||
"tag_ners",
|
||||
"layer",
|
||||
"memory_at",
|
||||
"gmt_modified",
|
||||
"is_latest",
|
||||
"status",
|
||||
"supersedes",
|
||||
"superseded_by",
|
||||
"source_raw_memory_id",
|
||||
# SDK 2.0.2 sends both of these on an in-place reconcile UPDATE.
|
||||
# They are metadata only and must never carry identity, permission or
|
||||
# tenant state: `owner` is a provenance label whose only values are
|
||||
# "user" / "agent" (who said this line), and `custom` is the SDK's own
|
||||
# free-form notes field. Neither may hold a user_id, scope, grant or
|
||||
# tenancy marker, so allowing the field names does not widen the
|
||||
# cross-user boundary that the store still enforces per node.
|
||||
"owner",
|
||||
"custom",
|
||||
}
|
||||
)
|
||||
_DEDUP_AUDIT_FIELDS = frozenset({"shadow_reason", "dedup_kept_id"})
|
||||
_DIGEST_COUNTER_FIELD = "s2_evidence_count"
|
||||
|
||||
|
||||
def _write_view(node: MemoryNode) -> dict[str, Any]:
|
||||
"""Only native Chroma's empty-string and float32 storage normalization."""
|
||||
payload: dict[str, Any] = node.to_dict()
|
||||
payload["source_session_id"] = payload["source_session_id"] or ""
|
||||
# The pinned public SDK serializes tag_ners as JSON, but its Chroma read path omits this
|
||||
# field from the JSON decode list. Decode only the exact list-of-strings
|
||||
# shape that the write path produced; malformed values still fail the
|
||||
# complete-node comparison below.
|
||||
tag_ners = payload.get("tag_ners")
|
||||
if (
|
||||
isinstance(tag_ners, list)
|
||||
and tag_ners
|
||||
and all(isinstance(value, str) and len(value) == 1 for value in tag_ners)
|
||||
and tag_ners[0] == "["
|
||||
and tag_ners[-1] == "]"
|
||||
):
|
||||
tag_ners = "".join(tag_ners)
|
||||
if isinstance(tag_ners, str):
|
||||
try:
|
||||
decoded = json.loads(tag_ners)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
decoded = None
|
||||
if isinstance(decoded, list) and all(isinstance(value, str) for value in decoded):
|
||||
payload["tag_ners"] = decoded
|
||||
if payload.get("tag_ners") in ([], ["[", "]"]):
|
||||
payload["tag_ners"] = []
|
||||
# HY Chroma encodes None as ""; these JSON fields keep "" when json.loads
|
||||
# fails (vector_store_chroma.py:741-788). Only an absent link is equivalent.
|
||||
for field in ("supersedes", "superseded_by"):
|
||||
if payload[field] == "":
|
||||
payload[field] = None
|
||||
if payload["embedding"] is not None:
|
||||
payload["embedding"] = [
|
||||
struct.unpack("f", struct.pack("f", n))[0] for n in payload["embedding"]
|
||||
]
|
||||
if not all(math.isfinite(n) for n in payload["embedding"]):
|
||||
raise ValueError("nonfinite_embedding")
|
||||
return payload
|
||||
|
||||
|
||||
def _same_written_vector(actual: list[float] | None, expected: list[float] | None) -> bool:
|
||||
if actual == expected:
|
||||
return True
|
||||
if actual is None or expected is None or len(actual) != len(expected):
|
||||
return False
|
||||
# Chroma's cosine index can renormalize float32 vectors on readback. Bound
|
||||
# only this write roundtrip; the saved full-node proof uses exact readback.
|
||||
return all(
|
||||
math.isclose(left, right, rel_tol=4 * 2**-23, abs_tol=0.0)
|
||||
for left, right in zip(actual, expected, strict=True)
|
||||
)
|
||||
|
||||
|
||||
_NATIVE_WRITES = frozenset(
|
||||
{"upsert", "upsert_batch", "update_payload", "update_embedding", "delete"}
|
||||
)
|
||||
|
||||
|
||||
class HyPermissionGraphStore:
|
||||
"""Lease-gated graph store for HY System 2's digest transaction.
|
||||
|
||||
The native graph is deliberately absent from ordinary ingest leases. Digest
|
||||
needs it to write L6 schemas, but every operation must remain inside the
|
||||
same frozen permission lease rather than exposing the raw store.
|
||||
"""
|
||||
|
||||
def __init__(self, native: Any, recheck: Any) -> None:
|
||||
self._native = native
|
||||
self._recheck = recheck
|
||||
self._closed = False
|
||||
|
||||
def recheck(self) -> None:
|
||||
if self._closed:
|
||||
raise permission_error("hy_graph_store_closed")
|
||||
self._recheck()
|
||||
|
||||
async def close(self) -> None:
|
||||
self._closed = True
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
if name.startswith("_") and name != "_run":
|
||||
raise AttributeError(name)
|
||||
attribute = getattr(self._native, name)
|
||||
if not callable(attribute):
|
||||
return attribute
|
||||
|
||||
async def guarded(*args: Any, **kwargs: Any) -> Any:
|
||||
self.recheck()
|
||||
result = await attribute(*args, **kwargs)
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
return guarded
|
||||
|
||||
|
||||
class HyPermissionVectorStore:
|
||||
"""Closed, task-exclusive adapter; initialize/close never own the native store.
|
||||
|
||||
node_permissions may attest additional native raw/profile nodes not represented
|
||||
in known_memories. Every attestation must cover all its contributors, and all
|
||||
Evidence IDs must already be in this lease. Known entries also constrain exact
|
||||
content/owner. Without a full attestation even a known entry is denied.
|
||||
|
||||
written_nodes records successful writes with the entire lease footprint, a
|
||||
conservative provenance bound, not a claim of semantic citation accuracy.
|
||||
These transient fingerprints must be persisted by the integrating factory if
|
||||
future operations are to reuse raw/profile nodes. No content is stored here.
|
||||
"""
|
||||
|
||||
__slots__ = (
|
||||
"__native",
|
||||
"__lease",
|
||||
"__agent_id",
|
||||
"__session_id",
|
||||
"__cross_evidence_scope",
|
||||
"__user_id",
|
||||
"__known",
|
||||
"__permissions",
|
||||
"__written",
|
||||
"__evidence_ids",
|
||||
"__deleted",
|
||||
"__shadowed",
|
||||
"__superseded",
|
||||
"__heads",
|
||||
"__lock",
|
||||
"__closed",
|
||||
"__failure",
|
||||
"__step_timer",
|
||||
"__refresh_permission",
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
native: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
*,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
cross_evidence_scope: bool = False,
|
||||
node_permissions: Mapping[str, HyNodePermission] | None = None,
|
||||
refresh_permission: Callable[[str], Awaitable[HyNodePermission | None]] | None = None,
|
||||
step_timer: HyStepTimer | None = None,
|
||||
) -> None:
|
||||
self.__step_timer = step_timer
|
||||
self.__native = native
|
||||
self.__lease = lease
|
||||
self.__agent_id = agent_id
|
||||
self.__session_id = session_id
|
||||
self.__cross_evidence_scope = cross_evidence_scope
|
||||
self.__user_id = lease.user_id
|
||||
self.__failure: str | None = None
|
||||
self.__closed = False
|
||||
self.__lock = asyncio.Lock()
|
||||
self.__deleted: set[str] = set()
|
||||
self.__written: dict[str, HyNodePermission] = {}
|
||||
self.__shadowed: set[str] = set()
|
||||
self.__superseded: dict[str, tuple[str, ...]] = {}
|
||||
self.__heads: dict[str, tuple[str, ...]] = {}
|
||||
self.__refresh_permission = refresh_permission
|
||||
self.recheck()
|
||||
self.__evidence_ids = tuple(record.evidence_id for record in lease.records)
|
||||
current = lease.source_context.current if lease.source_context else lease.records
|
||||
conversation_mismatch = any(r.conversation_id != session_id for r in current)
|
||||
if (
|
||||
not lease.task_id
|
||||
or not agent_id
|
||||
or agent_id == "*"
|
||||
or not session_id
|
||||
or not current
|
||||
or (conversation_mismatch and not cross_evidence_scope)
|
||||
or not self.__evidence_ids
|
||||
or len(set(self.__evidence_ids)) != len(lease.records)
|
||||
or any(r.user_id != self.__user_id for r in lease.records)
|
||||
):
|
||||
self.__deny("scope")
|
||||
known = lease.source_context.known_memories if lease.source_context else ()
|
||||
self.__known = {item.hy_memory_id: item for item in known}
|
||||
if len(self.__known) != len(known):
|
||||
self.__deny("provenance")
|
||||
self.__permissions = dict(node_permissions or {})
|
||||
for key, proof in self.__permissions.items():
|
||||
if (
|
||||
not key
|
||||
or not isinstance(proof, HyNodePermission)
|
||||
or len(proof.fingerprint) != 64
|
||||
or any(c not in "0123456789abcdef" for c in proof.fingerprint)
|
||||
or not isinstance(proof.evidence_ids, tuple)
|
||||
or not proof.evidence_ids
|
||||
or len(set(proof.evidence_ids)) != len(proof.evidence_ids)
|
||||
or not set(proof.evidence_ids) <= set(self.__evidence_ids)
|
||||
):
|
||||
self.__deny("provenance")
|
||||
for key, item in self.__known.items():
|
||||
if not item.evidence_ids or not set(item.evidence_ids) <= set(self.__evidence_ids):
|
||||
self.__deny("provenance")
|
||||
known_proof = self.__permissions.get(key)
|
||||
if known_proof and not set(item.evidence_ids) <= set(known_proof.evidence_ids):
|
||||
self.__deny("provenance")
|
||||
|
||||
@property
|
||||
def written_nodes(self) -> Mapping[str, HyNodePermission]:
|
||||
"""Latest verified write proofs; each access returns a new immutable snapshot.
|
||||
|
||||
Export after SDK writes have drained and before close/governance projection.
|
||||
Merge these rows into the lineage ledger, then remove deleted_node_ids;
|
||||
this write delta does not include untouched pre-existing permissions.
|
||||
"""
|
||||
self.recheck()
|
||||
return MappingProxyType(dict(self.__written))
|
||||
|
||||
@property
|
||||
def deleted_node_ids(self) -> frozenset[str]:
|
||||
"""Verified native deletions in this operation, for lineage-ledger cleanup."""
|
||||
self.recheck()
|
||||
return frozenset(self.__deleted)
|
||||
|
||||
@property
|
||||
def retired_node_ids(self) -> frozenset[str]:
|
||||
"""Nodes HY deleted or shadowed (its logical delete) in this operation."""
|
||||
self.recheck()
|
||||
return frozenset(self.__deleted | self.__shadowed)
|
||||
|
||||
@property
|
||||
def superseded_nodes(self) -> Mapping[str, tuple[str, ...]]:
|
||||
"""Nodes HY marked superseded in this operation, with what replaced them.
|
||||
|
||||
HY writes the two directions separately and may fail either one, so a
|
||||
replacement counts whether it was recorded on the old node or only on
|
||||
the new head.
|
||||
"""
|
||||
self.recheck()
|
||||
result = {}
|
||||
for key, by in self.__superseded.items():
|
||||
heads = set(by) | {head for head, olds in self.__heads.items() if key in olds}
|
||||
heads -= self.__deleted | {key}
|
||||
if key not in self.__deleted and heads:
|
||||
result[key] = tuple(sorted(heads))
|
||||
return MappingProxyType(result)
|
||||
|
||||
@property
|
||||
def supports_fulltext(self) -> bool:
|
||||
return False
|
||||
|
||||
def recheck(self) -> None:
|
||||
"""Must also gate transport/publication after HY swallows a store failure."""
|
||||
self.__assert_open()
|
||||
try:
|
||||
self.__lease.recheck()
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
self.__failure = (
|
||||
code
|
||||
if code.startswith("model_content_") and len(code) <= 100
|
||||
else "model_content_hy_store_lease_failed"
|
||||
)
|
||||
raise permission_error(self.__failure) from None
|
||||
|
||||
def __assert_open(self) -> None:
|
||||
if self.__closed:
|
||||
self.__deny("closed")
|
||||
if self.__failure:
|
||||
raise permission_error(self.__failure)
|
||||
|
||||
async def initialize(self) -> None:
|
||||
self.recheck()
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Revoke this wrapper locally without rechecking context or closing HY.
|
||||
|
||||
Cleanup must remain possible after governance projection changes the
|
||||
lease's known-memory revisions, or after permission has been revoked.
|
||||
"""
|
||||
async with self.__lock:
|
||||
self.__closed = True
|
||||
self.__permissions.clear()
|
||||
self.__written.clear()
|
||||
|
||||
async def has_tag_embedding(self, user_id: str, tag: str) -> bool:
|
||||
self.recheck()
|
||||
# Native chat calls this best-effort helper. NotImplemented prevents it
|
||||
# from embedding tags or falling through to an unguarded entity store.
|
||||
raise NotImplementedError("hy_permission_store_tag_index_unsupported")
|
||||
|
||||
def __deny(self, suffix: str) -> NoReturn:
|
||||
self.__failure = self.__failure or f"model_content_hy_store_{suffix}"
|
||||
raise permission_error(self.__failure)
|
||||
|
||||
async def __invoke(self, method: str, *args: Any, **kwargs: Any) -> Any:
|
||||
writing = method in _NATIVE_WRITES
|
||||
self.recheck() if writing else self.__assert_open()
|
||||
step = "vector_store_write" if writing else "vector_store_read"
|
||||
first = args[0] if args else None
|
||||
try:
|
||||
with (
|
||||
self.__step_timer.step(step, items=len(first) if isinstance(first, list) else 1)
|
||||
if self.__step_timer is not None
|
||||
else nullcontext()
|
||||
):
|
||||
result = await getattr(self.__native, method)(*args, **kwargs)
|
||||
except Exception:
|
||||
self.__deny("backend_failed")
|
||||
self.recheck() if writing else self.__assert_open()
|
||||
return result
|
||||
|
||||
async def __read_native(self, key: str) -> MemoryNode | None:
|
||||
value = await self.__invoke("get_by_id", key)
|
||||
if value is not None and (type(value) is not MemoryNode or value.node_id != key):
|
||||
self.__deny("node_shape")
|
||||
return deepcopy(value)
|
||||
|
||||
def __fingerprint(self, node: MemoryNode) -> str:
|
||||
try:
|
||||
return hy_node_fingerprint(node)
|
||||
except (TypeError, ValueError, AttributeError, OverflowError):
|
||||
self.__deny("node_shape")
|
||||
|
||||
def __write_view(self, node: MemoryNode) -> dict[str, Any]:
|
||||
self.__fingerprint(node)
|
||||
try:
|
||||
return _write_view(node)
|
||||
except (TypeError, ValueError, OverflowError, struct.error):
|
||||
self.__deny("node_shape")
|
||||
|
||||
def __in_scope(self, node: MemoryNode) -> bool:
|
||||
if node.user_id != self.__user_id or node.layer not in _CHAT_LAYERS:
|
||||
return False
|
||||
if node.layer is MemoryLayer.L0_BASIC_INFO:
|
||||
# Public SDK 2.0.2 migrates verified legacy agent-level profiles into one
|
||||
# user-level chain. Existing nodes still require a full proof in
|
||||
# __load; accepting their historical agent id does not authorize
|
||||
# unproven cross-agent memories.
|
||||
return True
|
||||
if self.__cross_evidence_scope:
|
||||
return bool(node.agent_id == self.__agent_id)
|
||||
return bool(node.agent_id == self.__agent_id and node.session_id == self.__session_id)
|
||||
|
||||
def __new_node_in_scope(self, node: MemoryNode) -> bool:
|
||||
if not self.__in_scope(node):
|
||||
return False
|
||||
if node.layer is MemoryLayer.L0_BASIC_INFO:
|
||||
return node.agent_id in ("", self.__agent_id) and node.session_id in (
|
||||
"",
|
||||
self.__session_id,
|
||||
)
|
||||
return True
|
||||
|
||||
def __references(self, node: MemoryNode) -> list[str]:
|
||||
refs = []
|
||||
for field in ("supersedes", "superseded_by", "evidence_chain"):
|
||||
values = getattr(node, field)
|
||||
if field != "evidence_chain" and values == "":
|
||||
continue
|
||||
if values is not None:
|
||||
if not isinstance(values, list) or any(
|
||||
not isinstance(v, str) or not v for v in values
|
||||
):
|
||||
self.__deny("node_shape")
|
||||
refs.extend(values)
|
||||
if node.source_raw_memory_id is not None:
|
||||
if not isinstance(node.source_raw_memory_id, str) or not node.source_raw_memory_id:
|
||||
self.__deny("node_shape")
|
||||
refs.append(node.source_raw_memory_id)
|
||||
return refs
|
||||
|
||||
async def __load(
|
||||
self,
|
||||
key: str,
|
||||
first: MemoryNode | None = None,
|
||||
*,
|
||||
allow_deleted_links: bool = False,
|
||||
) -> MemoryNode | None:
|
||||
pending = [key]
|
||||
checked: dict[str, MemoryNode] = {}
|
||||
while pending:
|
||||
current = pending.pop()
|
||||
if current in checked:
|
||||
continue
|
||||
# HY dedup deletes head then ancestors. Only this operation's
|
||||
# verified tombstones may terminate a closure during deletion.
|
||||
if allow_deleted_links and current != key and current in self.__deleted:
|
||||
continue
|
||||
proof = self.__permissions.get(current)
|
||||
if proof is None or current in self.__deleted:
|
||||
return None
|
||||
value = (
|
||||
first if current == key and first is not None else await self.__read_native(current)
|
||||
)
|
||||
if value is None or not self.__in_scope(value):
|
||||
return None
|
||||
if self.__fingerprint(value) != proof.fingerprint:
|
||||
refreshed = (
|
||||
await self.__refresh_permission(current)
|
||||
if self.__refresh_permission is not None
|
||||
else None
|
||||
)
|
||||
if refreshed is None or self.__fingerprint(value) != refreshed.fingerprint:
|
||||
self.__deny("node_changed")
|
||||
self.__permissions[current] = refreshed
|
||||
proof = refreshed
|
||||
known = self.__known.get(current) if current not in self.__written else None
|
||||
if known and (value.content != known.content or value.owner != known.owner):
|
||||
self.__deny("node_changed")
|
||||
checked[current] = value
|
||||
pending.extend(self.__references(value))
|
||||
return deepcopy(checked[key])
|
||||
|
||||
async def get_by_id(self, node_id: str) -> MemoryNode | None:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
result = await self.__load(node_id)
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
async def get_by_ids(self, node_ids: list[str]) -> list[MemoryNode]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
result = [
|
||||
n for key in dict.fromkeys(node_ids) if (n := await self.__load(key)) is not None
|
||||
]
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
async def get_embeddings(self, node_ids: list[str]) -> dict[str, list[float]]:
|
||||
return {
|
||||
n.node_id: list(n.embedding)
|
||||
for n in await self.get_by_ids(node_ids)
|
||||
if n.embedding is not None
|
||||
}
|
||||
|
||||
def __scope(self, user_id: str | None, agent_id: str | None) -> None:
|
||||
if user_id not in (None, self.__user_id) or agent_id not in (None, self.__agent_id):
|
||||
self.__deny("scope")
|
||||
|
||||
async def list_by_user(
|
||||
self,
|
||||
user_id: str | None = None,
|
||||
agent_id: str | None = None,
|
||||
limit: int = 10000,
|
||||
status_filter: list[Any] | None = None,
|
||||
layers: list[Any] | None = None,
|
||||
user_ids: list[str] | None = None,
|
||||
agent_ids: list[str] | None = None,
|
||||
only_latest: bool = False,
|
||||
with_vectors: bool = True,
|
||||
allow_raw_layer: bool = False,
|
||||
) -> list[MemoryNode]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
effective_users = user_ids or ([user_id] if user_id else [])
|
||||
effective_agents = agent_ids or ([agent_id] if agent_id else [])
|
||||
if effective_users != [self.__user_id] or any(
|
||||
value != self.__agent_id for value in effective_agents
|
||||
):
|
||||
self.__deny("scope")
|
||||
rows = await self.__invoke(
|
||||
"list_by_user",
|
||||
user_id=self.__user_id,
|
||||
agent_id=agent_id,
|
||||
limit=limit,
|
||||
status_filter=status_filter,
|
||||
layers=layers,
|
||||
user_ids=[self.__user_id],
|
||||
agent_ids=effective_agents or None,
|
||||
only_latest=only_latest,
|
||||
with_vectors=with_vectors,
|
||||
allow_raw_layer=allow_raw_layer,
|
||||
)
|
||||
result = []
|
||||
for row in rows:
|
||||
if type(row) is not MemoryNode:
|
||||
self.__deny("node_shape")
|
||||
value = await self.__load(row.node_id)
|
||||
if (
|
||||
value is not None
|
||||
and (not layers or value.layer in layers or value.layer.value in layers)
|
||||
and (
|
||||
not status_filter
|
||||
or value.status in status_filter
|
||||
or value.status.value in status_filter
|
||||
)
|
||||
):
|
||||
result.append(value)
|
||||
self.recheck()
|
||||
return result[:limit]
|
||||
|
||||
async def search(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
isolation_key: str = "",
|
||||
isolation_keys: list[str] | None = None,
|
||||
user_id: str | None = None,
|
||||
user_ids: list[str] | None = None,
|
||||
agent_ids: list[str] | None = None,
|
||||
layers: list[Any] | None = None,
|
||||
limit: int = 10,
|
||||
score_threshold: float = 0.0,
|
||||
status_filter: list[Any] | None = None,
|
||||
only_latest: bool = True,
|
||||
tags_match_any: list[str] | None = None,
|
||||
created_after: float | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
self.__scope(user_id, None)
|
||||
key = MemoryNode.build_isolation_key(self.__user_id, self.__agent_id, self.__session_id)
|
||||
if (
|
||||
any(uid != self.__user_id for uid in user_ids or [])
|
||||
or any(aid != self.__agent_id for aid in agent_ids or [])
|
||||
or isolation_key not in ("", key)
|
||||
or any(ik != key for ik in isolation_keys or [])
|
||||
):
|
||||
self.__deny("scope")
|
||||
hits = await self.__invoke(
|
||||
"search",
|
||||
query_embedding=deepcopy(query_embedding),
|
||||
isolation_key=isolation_key,
|
||||
isolation_keys=isolation_keys,
|
||||
user_id=self.__user_id,
|
||||
user_ids=[self.__user_id],
|
||||
agent_ids=[self.__agent_id],
|
||||
layers=layers,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold,
|
||||
status_filter=status_filter,
|
||||
only_latest=only_latest,
|
||||
tags_match_any=tags_match_any,
|
||||
created_after=created_after,
|
||||
)
|
||||
result = []
|
||||
for hit in hits:
|
||||
value = await self.__load(hit["node_id"])
|
||||
if value is None:
|
||||
continue
|
||||
score = float(hit["score"])
|
||||
if not math.isfinite(score):
|
||||
self.__deny("node_shape")
|
||||
# Do not forward arbitrary backend hit metadata. Rehydrate the
|
||||
# full fingerprinted node; native search often omits embedding.
|
||||
result.append({"node_id": value.node_id, "node": value, "score": score})
|
||||
self.recheck()
|
||||
return result[:limit]
|
||||
|
||||
async def __target(
|
||||
self,
|
||||
key: str,
|
||||
first: MemoryNode | None = None,
|
||||
*,
|
||||
allow_deleted_links: bool = False,
|
||||
) -> MemoryNode:
|
||||
value = await self.__load(key, first, allow_deleted_links=allow_deleted_links)
|
||||
if value is None:
|
||||
self.__deny("target_denied")
|
||||
return value
|
||||
|
||||
async def __check_links(self, value: MemoryNode) -> None:
|
||||
for ref in self.__references(value):
|
||||
if await self.__load(ref) is None:
|
||||
self.__deny("link_denied")
|
||||
|
||||
async def __plan_upsert(self, value: MemoryNode) -> MemoryNode:
|
||||
value = deepcopy(value)
|
||||
if type(value) is not MemoryNode or not value.node_id or not self.__in_scope(value):
|
||||
self.__deny("scope")
|
||||
existing = await self.__read_native(value.node_id)
|
||||
if existing is not None:
|
||||
await self.__target(value.node_id, existing)
|
||||
elif (
|
||||
not self.__new_node_in_scope(value)
|
||||
or value.node_id in self.__permissions
|
||||
or value.node_id in self.__known
|
||||
or value.node_id in self.__deleted
|
||||
):
|
||||
self.__deny("scope")
|
||||
await self.__check_links(value)
|
||||
self.__write_view(value)
|
||||
return value
|
||||
|
||||
async def __capture(
|
||||
self,
|
||||
expected: MemoryNode,
|
||||
*,
|
||||
allow_native_modified_time: bool = False,
|
||||
) -> None:
|
||||
actual = await self.__read_native(expected.node_id)
|
||||
if actual is None:
|
||||
self.__deny("write_mismatch")
|
||||
actual_view, expected_view = self.__write_view(actual), self.__write_view(expected)
|
||||
if allow_native_modified_time:
|
||||
if actual.gmt_modified < expected.gmt_modified:
|
||||
self.__deny("write_mismatch")
|
||||
actual_view["gmt_modified"] = expected_view["gmt_modified"]
|
||||
actual_vector = actual_view.pop("embedding")
|
||||
expected_vector = expected_view.pop("embedding")
|
||||
if actual_view != expected_view or not _same_written_vector(actual_vector, expected_vector):
|
||||
self.__deny("write_mismatch")
|
||||
proof = HyNodePermission(self.__fingerprint(actual), self.__evidence_ids)
|
||||
self.__permissions[actual.node_id] = proof
|
||||
self.__written[actual.node_id] = proof
|
||||
status = getattr(actual.status, "value", actual.status)
|
||||
if status == MemoryStatus.SHADOW.value:
|
||||
self.__shadowed.add(actual.node_id)
|
||||
else:
|
||||
self.__shadowed.discard(actual.node_id)
|
||||
if status == MemoryStatus.SUPERSEDED.value:
|
||||
by = actual.superseded_by
|
||||
self.__superseded[actual.node_id] = tuple(by) if isinstance(by, list) else ()
|
||||
else:
|
||||
self.__superseded.pop(actual.node_id, None)
|
||||
olds = actual.supersedes
|
||||
if isinstance(olds, list) and olds:
|
||||
self.__heads[actual.node_id] = tuple(olds)
|
||||
else:
|
||||
self.__heads.pop(actual.node_id, None)
|
||||
|
||||
async def upsert(self, node: MemoryNode) -> str:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
expected = await self.__plan_upsert(node)
|
||||
result = await self.__invoke("upsert", deepcopy(expected))
|
||||
if result != expected.node_id:
|
||||
self.__deny("write_mismatch")
|
||||
await self.__capture(expected)
|
||||
return str(result)
|
||||
|
||||
async def upsert_batch(self, nodes: list[MemoryNode]) -> list[str]:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
if len({n.node_id for n in nodes}) != len(nodes):
|
||||
self.__deny("node_shape")
|
||||
# No forward links to uncommitted batch members. Preflight every
|
||||
# existing target before the backend can make a partial write.
|
||||
planned = [await self.__plan_upsert(n) for n in nodes]
|
||||
ids = [n.node_id for n in planned]
|
||||
result = await self.__invoke("upsert_batch", deepcopy(planned))
|
||||
if result != ids:
|
||||
self.__deny("write_mismatch")
|
||||
for expected in planned:
|
||||
await self.__capture(expected)
|
||||
return ids
|
||||
|
||||
async def update_payload(self, node_id: str, updates: dict[str, Any]) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
current = await self.__target(node_id)
|
||||
updates = deepcopy(updates)
|
||||
dedup_audit = False
|
||||
if set(updates) & _DEDUP_AUDIT_FIELDS:
|
||||
if (
|
||||
not set(updates) <= (_UPDATE_FIELDS | _DEDUP_AUDIT_FIELDS)
|
||||
or updates.get("status") != MemoryStatus.SHADOW.value
|
||||
or updates.get("is_latest") is not False
|
||||
or not isinstance(updates.get("shadow_reason"), str)
|
||||
or not updates["shadow_reason"].startswith("dedup:")
|
||||
or not isinstance(updates.get("dedup_kept_id"), str)
|
||||
or not updates["dedup_kept_id"]
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
await self.__target(updates["dedup_kept_id"])
|
||||
custom = deepcopy(current.custom)
|
||||
custom["hy_dedup"] = {
|
||||
"reason": updates.pop("shadow_reason"),
|
||||
"kept_id": updates.pop("dedup_kept_id"),
|
||||
}
|
||||
updates["custom"] = custom
|
||||
dedup_audit = True
|
||||
allowed = _UPDATE_FIELDS | ({"custom"} if dedup_audit else set())
|
||||
if self.__cross_evidence_scope and set(updates) == {"custom"}:
|
||||
custom = updates.get("custom")
|
||||
if not isinstance(custom, dict):
|
||||
self.__deny("update_fields")
|
||||
current_custom = current.custom if isinstance(current.custom, dict) else {}
|
||||
previous = current_custom.get(_DIGEST_COUNTER_FIELD, 0)
|
||||
proposed = custom.get(_DIGEST_COUNTER_FIELD)
|
||||
if (
|
||||
not isinstance(previous, int)
|
||||
or isinstance(previous, bool)
|
||||
or not isinstance(proposed, int)
|
||||
or isinstance(proposed, bool)
|
||||
or proposed != previous + 1
|
||||
or set(custom) != set(current_custom) | {_DIGEST_COUNTER_FIELD}
|
||||
or any(custom[key] != value for key, value in current_custom.items())
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
allowed |= {"custom"}
|
||||
# `custom` is an allowed field name (SDK 2.0.2 sends it on a
|
||||
# reconcile UPDATE), but the System 2 evidence counter it can hold
|
||||
# stays digest-only: only the digest scope may advance it, and only
|
||||
# by one per write. Letting the name through must not let an
|
||||
# ordinary write move that counter.
|
||||
if not self.__cross_evidence_scope and _DIGEST_COUNTER_FIELD in (
|
||||
updates.get("custom") or {}
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
if not set(updates) <= allowed:
|
||||
self.__deny("update_fields")
|
||||
try:
|
||||
if "layer" in updates:
|
||||
MemoryLayer(updates["layer"])
|
||||
if "status" in updates:
|
||||
MemoryStatus(updates["status"])
|
||||
if "tag_ners" in updates and (
|
||||
not isinstance(updates["tag_ners"], list)
|
||||
or any(not isinstance(value, str) for value in updates["tag_ners"])
|
||||
):
|
||||
self.__deny("node_shape")
|
||||
expected = MemoryNode.from_dict({**current.to_dict(), **updates})
|
||||
self.__fingerprint(expected)
|
||||
except (TypeError, ValueError):
|
||||
self.__deny("node_shape")
|
||||
if not self.__in_scope(expected):
|
||||
self.__deny("scope")
|
||||
await self.__check_links(expected)
|
||||
self.__write_view(expected)
|
||||
if await self.__invoke("update_payload", node_id, updates) is not True:
|
||||
self.__deny("write_failed")
|
||||
await self.__capture(expected, allow_native_modified_time=True)
|
||||
return True
|
||||
|
||||
async def update_embedding(self, node_id: str, embedding: list[float]) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
expected = await self.__target(node_id)
|
||||
expected.embedding = deepcopy(embedding)
|
||||
self.__write_view(expected)
|
||||
if await self.__invoke("update_embedding", node_id, deepcopy(embedding)) is not True:
|
||||
self.__deny("write_failed")
|
||||
await self.__capture(expected, allow_native_modified_time=True)
|
||||
return True
|
||||
|
||||
async def delete(self, node_id: str) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
await self.__target(node_id, allow_deleted_links=True)
|
||||
if await self.__invoke("delete", node_id) is not True:
|
||||
self.__deny("write_failed")
|
||||
if await self.__read_native(node_id) is not None:
|
||||
self.__deny("write_mismatch")
|
||||
self.__deleted.add(node_id)
|
||||
self.__permissions.pop(node_id, None)
|
||||
self.__written.pop(node_id, None)
|
||||
self.__superseded.pop(node_id, None)
|
||||
self.__heads.pop(node_id, None)
|
||||
return True
|
||||
File diff suppressed because it is too large.
Load diff
@@ -144,10 +144,6 @@ class HyReconcileGuard:
|
||||
self.failures: list[str] = []
|
||||
self.bindings: HyWriteBindings | None = None
|
||||
self.imported_contents: frozenset[str] | None = None
|
||||
self.mutation_target_ids: set[str] = set()
|
||||
|
||||
def begin_write(self) -> None:
|
||||
self.mutation_target_ids.clear()
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(self.reconciler, name)
|
||||
@@ -214,15 +210,6 @@ class HyReconcileGuard:
|
||||
for operation in result.ops:
|
||||
raw_name = getattr(operation, "op", "")
|
||||
name = str(getattr(raw_name, "value", raw_name)).upper()
|
||||
if name in {"UPDATE", "SUPERSEDE", "DELETE", "SHADOW"}:
|
||||
target = getattr(operation, "memory_id", None)
|
||||
if isinstance(target, str) and target:
|
||||
self.mutation_target_ids.add(target)
|
||||
self.mutation_target_ids.update(
|
||||
key
|
||||
for key in getattr(operation, "memory_ids", ())
|
||||
if isinstance(key, str) and key
|
||||
)
|
||||
if name == "ADD":
|
||||
operation_counts["ADD"] += 1
|
||||
elif name in {"UPDATE", "MERGE", "NOOP"}:
|
||||
@@ -245,6 +232,20 @@ class HyReconcileGuard:
|
||||
"result_count": len(result.ops),
|
||||
},
|
||||
)
|
||||
timer = self.step_timer
|
||||
try:
|
||||
with (
|
||||
timer.step("write_verification", items=len(result.ops))
|
||||
if timer is not None
|
||||
else nullcontext()
|
||||
):
|
||||
await self._check_targets(result, kwargs)
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
if not code.startswith("hy_memory_write_verification_") or len(code) > 100:
|
||||
code = "hy_memory_write_verification_failed"
|
||||
self.failures.append(code)
|
||||
raise ValueError(code) from error
|
||||
return result
|
||||
|
||||
async def _check_targets(self, result: Any, kwargs: dict[str, Any]) -> None:
|
||||
|
||||
@@ -14,7 +14,7 @@ from pathlib import Path
|
||||
from typing import Literal, cast
|
||||
|
||||
HY_MEMORY_DISTRIBUTION = "hy-memory"
|
||||
HY_MEMORY_VERSION = "2.0.4"
|
||||
HY_MEMORY_VERSION = "2.0.2"
|
||||
HY_MEMORY_NAMESPACE = "governed-v2"
|
||||
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
|
||||
|
||||
@@ -31,12 +31,11 @@ _COLLECTION_LOGGERS = (
|
||||
"httpx",
|
||||
"httpcore",
|
||||
)
|
||||
_NOISY_LIBRARY_LOGGERS = ("httpx", "httpcore", "watchfiles", "jieba")
|
||||
|
||||
# One Memory deliberately stays on ultra because its existing, user-visible
|
||||
# profile/graph flow calls digest(), which the pinned public SDK exposes only
|
||||
# for ultra clients. Reader and non-reranker behavior remain pinned for
|
||||
# compatibility with public SDK 2.0.4.
|
||||
# compatibility with public SDK 2.0.2.
|
||||
HY_MEMORY_MODE = "ultra"
|
||||
# The SDK only recognises `legacy` and `hybrid` (see
|
||||
# hy_memory/pipelines/_retrieval/config.py: `ALL_READERS = (READER_LEGACY,
|
||||
@@ -140,18 +139,6 @@ def apply_diagnostic_log_level(level: DiagnosticLogLevel) -> None:
|
||||
for name in tuple(logging.Logger.manager.loggerDict):
|
||||
if any(name == root or name.startswith(f"{root}.") for root in _COLLECTION_LOGGERS):
|
||||
logging.getLogger(name).setLevel(selected)
|
||||
# Successful HTTP exchanges, file-watch notifications and tokenizer
|
||||
# initialization are useful only while actively debugging. At INFO they
|
||||
# dominated core.log and the in-app panel without describing product work.
|
||||
noisy_level = selected if selected == logging.DEBUG else max(selected, logging.WARNING)
|
||||
for root in _NOISY_LIBRARY_LOGGERS:
|
||||
logging.getLogger(root).setLevel(noisy_level)
|
||||
for name in tuple(logging.Logger.manager.loggerDict):
|
||||
if any(
|
||||
name == root or name.startswith(f"{root}.")
|
||||
for root in _NOISY_LIBRARY_LOGGERS
|
||||
):
|
||||
logging.getLogger(name).setLevel(noisy_level)
|
||||
|
||||
os.environ["MEMORY_LOG_LEVEL"] = logging.getLevelName(selected)
|
||||
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "true" if level == "debug" else "false"
|
||||
|
||||
@@ -7,7 +7,7 @@ from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord, WrittenBy
|
||||
from one_memory.input_sanitization import sanitize_model_input
|
||||
|
||||
|
||||
@@ -61,7 +61,17 @@ class HyWriteBindings:
|
||||
seed.workspace,
|
||||
)
|
||||
or record.content_kind is not EvidenceContentKind.RAW_CONVERSATION
|
||||
or record.role not in {"user", "assistant", "tool"}
|
||||
or (record.role, record.written_by)
|
||||
not in {
|
||||
# D98: the same change as the turn assembly. A sub agent's
|
||||
# question is a `user` message marked written by the
|
||||
# agent; excluding it here would make the source context
|
||||
# disagree with what was actually sent.
|
||||
("user", WrittenBy.PERSON),
|
||||
("user", WrittenBy.AGENT),
|
||||
("assistant", WrittenBy.AGENT),
|
||||
("tool", WrittenBy.TOOL),
|
||||
}
|
||||
):
|
||||
raise ValueError("hy_memory_write_verification_context_scope")
|
||||
self.all_sources: list[dict[str, Any]] = [
|
||||
|
||||
@@ -152,7 +152,7 @@ class HyStepTimer:
|
||||
if value is None:
|
||||
continue
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component=STEP_COMPONENTS[name],
|
||||
event="hy.step.completed",
|
||||
outcome="succeeded",
|
||||
@@ -168,7 +168,7 @@ class HyStepTimer:
|
||||
)
|
||||
model = measured.get("model")
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="hy.bridge",
|
||||
event="hy.steps.summary",
|
||||
outcome="succeeded",
|
||||
|
||||
@@ -1,41 +0,0 @@
|
||||
"""Process-local coordination for HY's native stores."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import weakref
|
||||
from collections.abc import AsyncIterator
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
_WRITE_LOCKS: weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
|
||||
weakref.WeakKeyDictionary()
|
||||
)
|
||||
|
||||
|
||||
def native_store_write_lock() -> asyncio.Lock:
|
||||
"""Return the write lock for the current event loop."""
|
||||
loop = asyncio.get_running_loop()
|
||||
lock = _WRITE_LOCKS.get(loop)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
_WRITE_LOCKS[loop] = lock
|
||||
return lock
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def native_store_read_slot() -> AsyncIterator[bool]:
|
||||
"""Admit a native HY read only while no write is active.
|
||||
|
||||
Chroma's native reader may retain the GIL after an asyncio timeout. Reads
|
||||
therefore share the physical-store lock, but never wait behind a writer:
|
||||
host recall can immediately fall back to the governance ledger.
|
||||
"""
|
||||
lock = native_store_write_lock()
|
||||
if lock.locked():
|
||||
yield False
|
||||
return
|
||||
await lock.acquire()
|
||||
try:
|
||||
yield True
|
||||
finally:
|
||||
lock.release()
|
||||
@@ -105,7 +105,7 @@ async def iterate_in_worker[T](
|
||||
# was unwinding with, and a diagnostic is not worth an import.
|
||||
try:
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="import.async_bridge",
|
||||
event="import.source_read_finished",
|
||||
outcome="succeeded",
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import sqlite3
|
||||
from contextlib import AbstractContextManager
|
||||
from datetime import UTC, datetime
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
@@ -21,23 +20,6 @@ CURSOR_MAX_SOURCE_FILE_BYTES = 64 * 1024 * 1024 * 1024
|
||||
GRANT_SCHEMA = source_grants_ddl("sqlite")
|
||||
|
||||
|
||||
class SourceAuthorizationState(StrEnum):
|
||||
"""Why a source cannot be read right now, or `ok` when it can.
|
||||
|
||||
The API used to answer `authorized=false` for all of these at once, which
|
||||
is how a revoked grant, a rebuilt history directory and a moved candidate
|
||||
all looked like the same bug and all got the same advice. The values carry
|
||||
no paths and no user content, because they are written to diagnostics.
|
||||
"""
|
||||
|
||||
OK = "ok"
|
||||
MISSING_GRANT = "missing_grant"
|
||||
REVOKED_GRANT = "revoked_grant"
|
||||
SOURCE_IDENTITY_CHANGED = "source_identity_changed"
|
||||
SOURCE_ROOT_UNAVAILABLE = "source_root_unavailable"
|
||||
CANDIDATE_CHANGED = "candidate_changed"
|
||||
|
||||
|
||||
class SourceGrantReceipt(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
@@ -164,46 +146,6 @@ class SourceGrantStore:
|
||||
raise SourceAccessError(SourceAccessErrorCode.SOURCE_IDENTITY_CHANGED)
|
||||
return authorized
|
||||
|
||||
def authorization_state(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
source_id: str,
|
||||
candidate: Path,
|
||||
) -> SourceAuthorizationState:
|
||||
"""Why reading this source would be refused.
|
||||
|
||||
Read-only and fails the same way `authorized_root` does: this names a
|
||||
state for the UI and for diagnostics, it does not widen access. The
|
||||
candidate is compared by path, so a grant that still points somewhere
|
||||
else reads as a moved candidate rather than as a rebuilt directory —
|
||||
the two need different fixes.
|
||||
"""
|
||||
with self._connect() as connection:
|
||||
row = connection.execute(
|
||||
"SELECT * FROM source_grants WHERE user_id=? AND source_id=?",
|
||||
(user_id, source_id),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return SourceAuthorizationState.MISSING_GRANT
|
||||
if row["revoked_at"] is not None or row["revision"] <= 0:
|
||||
return SourceAuthorizationState.REVOKED_GRANT
|
||||
granted = Path(row["requested_path"])
|
||||
if not granted.exists():
|
||||
return SourceAuthorizationState.SOURCE_ROOT_UNAVAILABLE
|
||||
try:
|
||||
if not root_identity_matches(granted, row["root_identity"]):
|
||||
return SourceAuthorizationState.SOURCE_IDENTITY_CHANGED
|
||||
except SourceAccessError:
|
||||
# Unreadable or gone between the two calls: there is no grant to
|
||||
# repair, only a directory to make available again.
|
||||
return SourceAuthorizationState.SOURCE_ROOT_UNAVAILABLE
|
||||
# Same directory, but the app now looks somewhere else. Re-authorizing
|
||||
# the old grant would not help; the candidate has to be granted.
|
||||
if granted != candidate:
|
||||
return SourceAuthorizationState.CANDIDATE_CHANGED
|
||||
return SourceAuthorizationState.OK
|
||||
|
||||
def revoke(self, *, user_id: str, source_id: str) -> SourceGrantReceipt:
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with self._connect(writing=True) as connection:
|
||||
|
||||
@@ -106,7 +106,7 @@ class PollingLiveSyncSupervisor:
|
||||
level=(
|
||||
"warn"
|
||||
if value.state in {LiveSyncState.DEGRADED, LiveSyncState.UNAVAILABLE}
|
||||
else "debug"
|
||||
else "info"
|
||||
),
|
||||
component="import.live_sync",
|
||||
event="live_sync.state_changed",
|
||||
|
||||
@@ -156,7 +156,7 @@ class FileWatchLiveSync:
|
||||
if source is not None
|
||||
):
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="import.live_watch",
|
||||
event="live_sync.hook_received",
|
||||
outcome="queued",
|
||||
@@ -190,7 +190,7 @@ class FileWatchLiveSync:
|
||||
try:
|
||||
await self._trigger_past(source_id, observed)
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="import.live_watch",
|
||||
event="live_sync.hook_completed",
|
||||
outcome="written",
|
||||
|
||||
@@ -79,8 +79,6 @@ class EvidenceInput(BaseModel):
|
||||
source_ordinal: str | None = Field(default=None, min_length=1, max_length=256)
|
||||
content_kind: EvidenceContentKind = EvidenceContentKind.RAW_CONVERSATION
|
||||
role: str = Field(min_length=1, max_length=128)
|
||||
# Legacy import/storage metadata only. Protocol `role` is authoritative;
|
||||
# this value must not affect identity, admission, scheduling or ownership.
|
||||
written_by: WrittenBy
|
||||
created_at: datetime | None = None
|
||||
verbatim: str = Field(min_length=1, max_length=MAX_VERBATIM_CHARACTERS)
|
||||
@@ -151,22 +149,17 @@ class EvidenceInput(BaseModel):
|
||||
)
|
||||
)
|
||||
else:
|
||||
# `role` is the protocol-level speaker identity. `written_by` was a
|
||||
# parser guess and differed between two imports of the same host
|
||||
# message (notably sub-agent/user and unknown/user), minting two
|
||||
# Evidence rows. Keep it out of every new identity.
|
||||
#
|
||||
# No workspace and no source_ordinal in the derived key: the hook
|
||||
# lane has neither (it reports no workspace and a constant ordinal),
|
||||
# so including either gave the same message a different key per lane
|
||||
# and stored it twice. Both are record metadata here, not identity.
|
||||
identity = "\0".join(
|
||||
(
|
||||
"derived-v4",
|
||||
"derived-v3",
|
||||
self.user_id,
|
||||
self.source_id,
|
||||
self.conversation_id,
|
||||
self.role,
|
||||
self.written_by,
|
||||
self.content_hash,
|
||||
)
|
||||
)
|
||||
@@ -175,54 +168,6 @@ class EvidenceInput(BaseModel):
|
||||
# two rows. The capture time is record metadata, not identity.
|
||||
return hashlib.sha256(identity.encode()).hexdigest()
|
||||
|
||||
@property
|
||||
def written_by_derived_v3_idempotency_key(self) -> str:
|
||||
"""The last identity that treated the parser's author guess as canonical.
|
||||
|
||||
New builds key by protocol role. During an upgrade this key lets a
|
||||
rescan find and reuse the row already stored by derived-v3 instead of
|
||||
inserting the same host message under a second Evidence id.
|
||||
"""
|
||||
identity = "\0".join(
|
||||
(
|
||||
"derived-v3",
|
||||
self.user_id,
|
||||
self.source_id,
|
||||
self.conversation_id,
|
||||
self.written_by,
|
||||
self.content_hash,
|
||||
)
|
||||
)
|
||||
return hashlib.sha256(identity.encode()).hexdigest()
|
||||
|
||||
@property
|
||||
def written_by_derived_v3_idempotency_keys(self) -> tuple[str, ...]:
|
||||
"""Every legacy author guess this protocol role may have carried."""
|
||||
guesses = {
|
||||
"user": (
|
||||
WrittenBy.PERSON,
|
||||
WrittenBy.AGENT,
|
||||
WrittenBy.UNKNOWN,
|
||||
),
|
||||
"assistant": (WrittenBy.AGENT, WrittenBy.UNKNOWN, WrittenBy.PERSON),
|
||||
"tool": (WrittenBy.TOOL, WrittenBy.UNKNOWN),
|
||||
}.get(self.role, tuple(WrittenBy))
|
||||
ordered = (self.written_by, *guesses)
|
||||
keys: list[str] = []
|
||||
for written_by in dict.fromkeys(ordered):
|
||||
identity = "\0".join(
|
||||
(
|
||||
"derived-v3",
|
||||
self.user_id,
|
||||
self.source_id,
|
||||
self.conversation_id,
|
||||
written_by,
|
||||
self.content_hash,
|
||||
)
|
||||
)
|
||||
keys.append(hashlib.sha256(identity.encode()).hexdigest())
|
||||
return tuple(keys)
|
||||
|
||||
@property
|
||||
def legacy_message_idempotency_key(self) -> str | None:
|
||||
"""The pre-workspace key for a message-shaped input, when one exists.
|
||||
@@ -267,13 +212,7 @@ class EvidenceInput(BaseModel):
|
||||
"""
|
||||
return self.derived_v2_idempotency_key("derived-v2", workspace_agnostic=False)
|
||||
|
||||
def derived_v2_idempotency_key(
|
||||
self,
|
||||
label: str,
|
||||
*,
|
||||
workspace_agnostic: bool,
|
||||
written_by: WrittenBy | None = None,
|
||||
) -> str:
|
||||
def derived_v2_idempotency_key(self, label: str, *, workspace_agnostic: bool) -> str:
|
||||
identity = "\0".join(
|
||||
(
|
||||
label,
|
||||
@@ -281,7 +220,7 @@ class EvidenceInput(BaseModel):
|
||||
self.source_id,
|
||||
*([] if workspace_agnostic else [self.workspace or "\uffffnull"]),
|
||||
self.conversation_id,
|
||||
written_by or self.written_by,
|
||||
self.written_by,
|
||||
self.created_at.isoformat() if self.created_at else "",
|
||||
self.source_ordinal or "",
|
||||
# v2 hashed the cleaned body; v1 predates raw_verbatim entirely,
|
||||
@@ -291,26 +230,6 @@ class EvidenceInput(BaseModel):
|
||||
)
|
||||
return hashlib.sha256(identity.encode()).hexdigest()
|
||||
|
||||
@property
|
||||
def older_written_by_idempotency_keys(self) -> tuple[str, ...]:
|
||||
keys: list[str] = []
|
||||
for written_by in WrittenBy:
|
||||
keys.append(
|
||||
self.derived_v2_idempotency_key(
|
||||
"derived-v1",
|
||||
workspace_agnostic=True,
|
||||
written_by=written_by,
|
||||
)
|
||||
)
|
||||
keys.append(
|
||||
self.derived_v2_idempotency_key(
|
||||
"derived-v2",
|
||||
workspace_agnostic=False,
|
||||
written_by=written_by,
|
||||
)
|
||||
)
|
||||
return tuple(keys)
|
||||
|
||||
@property
|
||||
def evidence_id(self) -> str:
|
||||
return str(uuid5(EVIDENCE_NAMESPACE, self.idempotency_key))
|
||||
@@ -330,7 +249,6 @@ class EvidenceRecord(BaseModel):
|
||||
source_ordinal: str | None
|
||||
content_kind: EvidenceContentKind
|
||||
role: str
|
||||
# Retained while old databases and export formats still carry the column.
|
||||
written_by: WrittenBy
|
||||
created_at: datetime | None
|
||||
verbatim: str
|
||||
|
||||
@@ -199,7 +199,7 @@ class ImportRunner:
|
||||
raise ValueError("Import job source does not match adapter source")
|
||||
job, runner_token = await in_the_background(self.job_store.claim, job_id)
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="import.parser",
|
||||
event="import.parse.started",
|
||||
outcome="started",
|
||||
@@ -310,7 +310,7 @@ class ImportRunner:
|
||||
if event.kind == "progress" and str(event.phase) != last_phase:
|
||||
phase = str(event.phase)
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="import.parser",
|
||||
event="import.phase_changed",
|
||||
outcome="changed",
|
||||
@@ -516,7 +516,7 @@ class ImportRunner:
|
||||
async def _emit_parse_result(self, job: ImportJob, started: float) -> None:
|
||||
succeeded = job.status is ImportStatus.COMPLETED
|
||||
diagnostic_event(
|
||||
level="debug" if succeeded else "warn",
|
||||
level="info" if succeeded else "warn",
|
||||
component="import.parser",
|
||||
event="import.parse.completed",
|
||||
outcome="succeeded" if succeeded else str(job.status),
|
||||
|
||||
@@ -141,18 +141,11 @@ def parse_codex_record(
|
||||
if role == "user":
|
||||
written_by = _user_author(record, payload, session)
|
||||
# Machine inputs may quote a whole transcript, including user_query tags.
|
||||
# Protocol role is authoritative; host author metadata does not change
|
||||
# how a user message is cleaned.
|
||||
if not (
|
||||
content.startswith(
|
||||
"The following is the Codex agent history whose request action you are assessing."
|
||||
)
|
||||
and ">>> TRANSCRIPT START" in content
|
||||
and ">>> TRANSCRIPT END" in content
|
||||
):
|
||||
# Session provenance is not proof of who wrote any particular future turn.
|
||||
if written_by is WrittenBy.PERSON:
|
||||
content = strip_host_wrappers(content)
|
||||
if not content:
|
||||
return ParsedSkip(ScanIssueCode.NO_TEXT)
|
||||
if not content:
|
||||
return ParsedSkip(ScanIssueCode.NO_TEXT)
|
||||
return ParsedMessage(
|
||||
conversation_id=conversation_from_relative_path(context.relative_path),
|
||||
message_id=first_string(payload.get("id"), record.get("id")),
|
||||
|
||||
@@ -355,7 +355,7 @@ class JsonlSourceAdapter:
|
||||
source_file, checkpoint, header = await asyncio.to_thread(
|
||||
self._prepare, listed, options
|
||||
)
|
||||
_log.debug(
|
||||
_log.info(
|
||||
"import_file_read source=%s file=%d/%d file_bytes=%d planned_bytes=%d",
|
||||
self.descriptor.source_id,
|
||||
index,
|
||||
@@ -376,7 +376,7 @@ class JsonlSourceAdapter:
|
||||
else DEFAULT_IMPORT_MAX_FILE_BYTES
|
||||
)
|
||||
if source_file.size > budget:
|
||||
_log.debug(
|
||||
_log.info(
|
||||
"import_file_budgeted source=%s file=%d/%d file_bytes=%d budget=%d path=%s",
|
||||
self.descriptor.source_id,
|
||||
index,
|
||||
|
||||
@@ -7,7 +7,7 @@ import sqlite3
|
||||
import stat
|
||||
from collections.abc import AsyncIterator, Iterator
|
||||
from dataclasses import dataclass, replace
|
||||
from datetime import UTC, datetime, timedelta, timezone
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from functools import partial
|
||||
|
||||
from pydantic import JsonValue
|
||||
@@ -82,11 +82,6 @@ _CURSOR_HOST_CONTEXT_START = re.compile(
|
||||
)
|
||||
_CURSOR_USER_QUERY = re.compile(r"<user_query\b")
|
||||
_CURSOR_TIMESTAMP_PREFIX = re.compile(r"<timestamp>[\s\S]*?</timestamp>\s*")
|
||||
_CURSOR_TIMESTAMP_VALUE = re.compile(
|
||||
r"<timestamp>\s*"
|
||||
r"([A-Za-z]+,\s+[A-Za-z]{3}\s+\d{1,2},\s+\d{4},\s+\d{1,2}:\d{2}\s+[AP]M)"
|
||||
r"\s+\(UTC([+-]\d{1,2})(?::(\d{2}))?\)\s*</timestamp>",
|
||||
)
|
||||
_CURSOR_IMAGE_FILES_PREFIX = re.compile(r"\[Image\]\s*<image_files>[\s\S]*?</image_files>\s*")
|
||||
_CURSOR_BARE_IMAGE_PREFIX = re.compile(r"(?:\[Image\]\s*)+")
|
||||
|
||||
@@ -128,24 +123,6 @@ def _strip_cursor_timestamp(content: str | None) -> str | None:
|
||||
return _CURSOR_TIMESTAMP_PREFIX.sub("", content, count=1) if content else None
|
||||
|
||||
|
||||
def _cursor_host_timestamp(content: str | None) -> datetime | None:
|
||||
"""Parse Cursor's user-visible timestamp envelope into UTC."""
|
||||
if not content:
|
||||
return None
|
||||
match = _CURSOR_TIMESTAMP_VALUE.search(content)
|
||||
if match is None:
|
||||
return None
|
||||
try:
|
||||
local = datetime.strptime(match[1], "%A, %b %d, %Y, %I:%M %p")
|
||||
hours = int(match[2])
|
||||
minutes = int(match[3] or "0")
|
||||
sign = -1 if hours < 0 else 1
|
||||
offset = timedelta(hours=hours, minutes=sign * minutes)
|
||||
return local.replace(tzinfo=timezone(offset)).astimezone(UTC)
|
||||
except (ValueError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def _strip_cursor_host_prefix(content: str | None) -> str | None:
|
||||
if not content:
|
||||
return content
|
||||
@@ -421,7 +398,10 @@ class CursorUnifiedSourceAdapter:
|
||||
if (
|
||||
isinstance(event, ScannedEvidence)
|
||||
and event.evidence.conversation_id in transcript_ids
|
||||
and not is_host_generated_user_turn(event.evidence.verbatim)
|
||||
and not (
|
||||
event.evidence.written_by is WrittenBy.AGENT
|
||||
and is_host_generated_user_turn(event.evidence.verbatim)
|
||||
)
|
||||
):
|
||||
yield ScanSkip(
|
||||
cursor=event.cursor,
|
||||
@@ -446,7 +426,10 @@ def _shadowed_by_transcript(event: ScanEvent, transcript_ids: set[str]) -> ScanE
|
||||
if (
|
||||
isinstance(event, ScannedEvidence)
|
||||
and event.evidence.conversation_id in transcript_ids
|
||||
and not is_host_generated_user_turn(event.evidence.verbatim)
|
||||
and not (
|
||||
event.evidence.written_by is WrittenBy.AGENT
|
||||
and is_host_generated_user_turn(event.evidence.verbatim)
|
||||
)
|
||||
):
|
||||
return ScanSkip(
|
||||
cursor=event.cursor,
|
||||
@@ -720,7 +703,6 @@ def _parse_transcript_record(
|
||||
project_name = path_parts[path_parts.index("projects") + 1] if "projects" in path_parts else ""
|
||||
session_name = context.relative_path.rsplit("/", 1)[-1].removesuffix(".jsonl")
|
||||
is_subagent = f"{project_name}\0{session_name}" in subagent_ids
|
||||
host_timestamp = _cursor_host_timestamp(content) if role == "user" else None
|
||||
created_at = _transcript_fallback_time(context)
|
||||
native_message_id = record.get("id") or message.get("id")
|
||||
message_id = (
|
||||
@@ -746,16 +728,7 @@ def _parse_transcript_record(
|
||||
created_at=created_at,
|
||||
workspace=workspace,
|
||||
record_type="cursor_transcript",
|
||||
origin_metadata=OriginMetadata(
|
||||
raw_meta={
|
||||
"time_basis": "file_mtime",
|
||||
**(
|
||||
{"turn_event_at": host_timestamp.isoformat()}
|
||||
if host_timestamp is not None
|
||||
else {}
|
||||
),
|
||||
}
|
||||
),
|
||||
origin_metadata=OriginMetadata(raw_meta={"time_basis": "file_mtime"}),
|
||||
),
|
||||
tool_blocks_evidence(message.get("content"), block_types=TOOL_BLOCKS),
|
||||
)
|
||||
@@ -846,13 +819,6 @@ def _read_store_messages(
|
||||
reason_code=ScanIssueCode.UNKNOWN_RECORD,
|
||||
)
|
||||
continue
|
||||
host_timestamp = (
|
||||
_cursor_host_timestamp(content) if role == "user" else None
|
||||
)
|
||||
fallback_timestamp = datetime.fromtimestamp(
|
||||
source_file.modified_ns / 1_000_000_000,
|
||||
tz=UTC,
|
||||
) + timedelta(microseconds=rowid)
|
||||
parsed = ParsedMessage(
|
||||
conversation_id=conversation_id,
|
||||
message_id=str(record.get("id") or row["id"]),
|
||||
@@ -863,19 +829,8 @@ def _read_store_messages(
|
||||
else written_by_user
|
||||
),
|
||||
verbatim=cleaned,
|
||||
created_at=fallback_timestamp,
|
||||
workspace=_store_workspace(source_file),
|
||||
record_type="cursor_store_blob",
|
||||
origin_metadata=OriginMetadata(
|
||||
raw_meta={
|
||||
"time_basis": "file_mtime",
|
||||
**(
|
||||
{"turn_event_at": host_timestamp.isoformat()}
|
||||
if host_timestamp is not None
|
||||
else {}
|
||||
),
|
||||
}
|
||||
),
|
||||
)
|
||||
yield ScannedEvidence(
|
||||
cursor=cursor,
|
||||
|
||||
@@ -57,8 +57,7 @@ _UNANSWERED_PREFIXES = (
|
||||
|
||||
|
||||
def _is_visible_user_row(role: str, written_by: str, verbatim: str) -> bool:
|
||||
del written_by # Legacy storage metadata; protocol role is authoritative.
|
||||
if role != "user":
|
||||
if role != "user" or written_by != WrittenBy.PERSON.value:
|
||||
return False
|
||||
return not verbatim.lstrip().lower().startswith(_UNANSWERED_PREFIXES)
|
||||
|
||||
@@ -469,54 +468,16 @@ class EvidenceStore:
|
||||
if anchored is not None:
|
||||
self._write_origins(connection, anchored, evidence.origin_meta, now)
|
||||
workspace, project = self._resolve_conversation_attribution(connection, evidence)
|
||||
existing = connection.execute(
|
||||
"SELECT role,revision FROM evidence WHERE evidence_id=?",
|
||||
(anchored,),
|
||||
).fetchone()
|
||||
if existing is None:
|
||||
raise RuntimeError("anchored evidence disappeared")
|
||||
if str(existing["role"]) != evidence.role:
|
||||
was_revised = True
|
||||
revision = int(existing["revision"]) + 1
|
||||
connection.execute(
|
||||
"""
|
||||
UPDATE evidence
|
||||
SET workspace=COALESCE(workspace,?),
|
||||
project=COALESCE(project,?),
|
||||
content_kind=?,role=?,created_at=COALESCE(?,created_at),
|
||||
verbatim=?,content_hash=?,revision=?,updated_at=?
|
||||
WHERE evidence_id=?
|
||||
""",
|
||||
(
|
||||
workspace,
|
||||
project,
|
||||
evidence.content_kind,
|
||||
evidence.role,
|
||||
evidence.created_at.isoformat() if evidence.created_at else None,
|
||||
evidence.verbatim,
|
||||
evidence.content_hash,
|
||||
revision,
|
||||
now,
|
||||
anchored,
|
||||
),
|
||||
)
|
||||
self._insert_current_version(
|
||||
connection,
|
||||
anchored,
|
||||
revision=revision,
|
||||
changed_at=now,
|
||||
)
|
||||
else:
|
||||
connection.execute(
|
||||
"""
|
||||
UPDATE evidence
|
||||
SET workspace=COALESCE(workspace,?),
|
||||
project=COALESCE(project,?),
|
||||
updated_at=?
|
||||
WHERE evidence_id=?
|
||||
""",
|
||||
(workspace, project, now, anchored),
|
||||
)
|
||||
connection.execute(
|
||||
"""
|
||||
UPDATE evidence
|
||||
SET workspace = COALESCE(workspace, ?),
|
||||
project = COALESCE(project, ?),
|
||||
updated_at = ?
|
||||
WHERE evidence_id = ?
|
||||
""",
|
||||
(workspace, project, now, anchored),
|
||||
)
|
||||
record = self._get_with_connection(connection, anchored)
|
||||
if record is None:
|
||||
raise RuntimeError("anchored evidence disappeared")
|
||||
@@ -524,11 +485,7 @@ class EvidenceStore:
|
||||
from one_memory import ingest_journal
|
||||
|
||||
ingest_journal.record(connection, anchored, cause="import")
|
||||
return PutEvidenceResult(
|
||||
record=record,
|
||||
was_created=False,
|
||||
was_revised=was_revised,
|
||||
)
|
||||
return PutEvidenceResult(record=record, was_created=False, was_revised=False)
|
||||
insert = connection.execute(
|
||||
"""
|
||||
INSERT INTO evidence (
|
||||
@@ -846,20 +803,27 @@ class EvidenceStore:
|
||||
and same_scope[0]["idempotency_key"] != evidence.idempotency_key
|
||||
):
|
||||
return str(same_scope[0]["evidence_id"])
|
||||
elif (
|
||||
legacy_id := next(
|
||||
(
|
||||
anchored
|
||||
for key in (
|
||||
*evidence.written_by_derived_v3_idempotency_keys,
|
||||
*evidence.older_written_by_idempotency_keys,
|
||||
)
|
||||
if (anchored := claim_legacy_key(key)) is not None
|
||||
),
|
||||
None,
|
||||
)
|
||||
) is not None:
|
||||
elif (legacy_id := claim_legacy_key(evidence.legacy_derived_idempotency_key)) is not None:
|
||||
return legacy_id
|
||||
if not evidence.message_id and not evidence.correlation_id:
|
||||
# BUG-03 moved the derived key off the capture time and onto the
|
||||
# pre-cleaning text. A replay of a row stored under the old scheme
|
||||
# has to be recognized as the same message.
|
||||
replayed = connection.execute(
|
||||
"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE idempotency_key = ? AND user_id = ? AND source_id = ?
|
||||
AND conversation_id = ?
|
||||
""",
|
||||
(
|
||||
evidence.prior_derived_idempotency_key,
|
||||
evidence.user_id,
|
||||
evidence.source_id,
|
||||
evidence.conversation_id,
|
||||
),
|
||||
).fetchone()
|
||||
if replayed is not None:
|
||||
return str(replayed["evidence_id"])
|
||||
if (
|
||||
EvidenceOrigin.HOOK in evidence.origins
|
||||
and evidence.message_id is None
|
||||
@@ -874,7 +838,7 @@ class EvidenceStore:
|
||||
ON hook.evidence_id=e.evidence_id AND hook.origin='hook'
|
||||
WHERE e.user_id=? AND e.source_id=? AND e.conversation_id=?
|
||||
AND (e.workspace IS ? OR e.workspace IS NULL OR ? IS NULL)
|
||||
AND e.role=? AND e.content_hash=?
|
||||
AND e.role=? AND e.written_by=? AND e.content_hash=?
|
||||
AND e.message_id IS NOT NULL AND hook.evidence_id IS NULL
|
||||
AND ABS((julianday(e.created_at)-julianday(?))*86400.0) <= 5.0
|
||||
ORDER BY ABS((julianday(e.created_at)-julianday(?))*86400.0), e.evidence_id
|
||||
@@ -887,6 +851,7 @@ class EvidenceStore:
|
||||
evidence.workspace,
|
||||
evidence.workspace,
|
||||
evidence.role,
|
||||
evidence.written_by,
|
||||
evidence.content_hash,
|
||||
evidence.created_at.isoformat(),
|
||||
evidence.created_at.isoformat(),
|
||||
@@ -907,7 +872,7 @@ class EvidenceStore:
|
||||
AND (e.workspace IS ? OR e.workspace IS NULL OR ? IS NULL)
|
||||
-- Unary plus keeps the coverage index from looking better than
|
||||
-- the message index, which narrows to this exact text.
|
||||
AND e.role = ? AND e.content_hash = ?
|
||||
AND e.role = ? AND +e.written_by = ? AND e.content_hash = ?
|
||||
AND e.message_id IS NULL
|
||||
""",
|
||||
(
|
||||
@@ -917,6 +882,7 @@ class EvidenceStore:
|
||||
evidence.workspace,
|
||||
evidence.workspace,
|
||||
evidence.role,
|
||||
evidence.written_by,
|
||||
evidence.content_hash,
|
||||
),
|
||||
).fetchall()
|
||||
@@ -1281,7 +1247,7 @@ class EvidenceStore:
|
||||
rows = connection.execute(
|
||||
"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE user_id = 'local' AND role = 'user'
|
||||
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
|
||||
ORDER BY imported_at, evidence_id
|
||||
LIMIT ?
|
||||
""",
|
||||
@@ -1335,7 +1301,7 @@ class EvidenceStore:
|
||||
rows = connection.execute(
|
||||
f"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE user_id = 'local' AND role = 'user'
|
||||
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
|
||||
AND ({predicates})
|
||||
ORDER BY imported_at, evidence_id
|
||||
LIMIT ?
|
||||
@@ -1367,7 +1333,7 @@ class EvidenceStore:
|
||||
rows = connection.execute(
|
||||
f"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE user_id = 'local' AND role = 'user'
|
||||
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
|
||||
AND workspace IS NOT NULL
|
||||
AND ({predicates})
|
||||
ORDER BY imported_at, evidence_id
|
||||
@@ -1510,7 +1476,7 @@ class EvidenceStore:
|
||||
)
|
||||
values: list[object] = [user_id, *(source_ids or ())]
|
||||
visible_user = """
|
||||
role='user' AND NOT (
|
||||
role='user' AND written_by='person' AND NOT (
|
||||
lower(ltrim(verbatim)) LIKE '<codex_internal_context%' OR
|
||||
lower(ltrim(verbatim)) LIKE '<environment_context>%' OR
|
||||
lower(ltrim(verbatim)) LIKE '<heartbeat>%' OR
|
||||
@@ -1519,7 +1485,7 @@ class EvidenceStore:
|
||||
)
|
||||
"""
|
||||
selected_visible_user = """
|
||||
e.role='user' AND NOT (
|
||||
e.role='user' AND e.written_by='person' AND NOT (
|
||||
lower(ltrim(e.verbatim)) LIKE '<codex_internal_context%' OR
|
||||
lower(ltrim(e.verbatim)) LIKE '<environment_context>%' OR
|
||||
lower(ltrim(e.verbatim)) LIKE '<heartbeat>%' OR
|
||||
@@ -1672,7 +1638,7 @@ class EvidenceStore:
|
||||
if seed is None:
|
||||
return ConversationTurnPage((), {}, 0, offset, limit)
|
||||
visible_user = """
|
||||
role='user' AND NOT (
|
||||
role='user' AND written_by='person' AND NOT (
|
||||
lower(ltrim(verbatim)) LIKE '<codex_internal_context%' OR
|
||||
lower(ltrim(verbatim)) LIKE '<environment_context>%' OR
|
||||
lower(ltrim(verbatim)) LIKE '<heartbeat>%' OR
|
||||
@@ -1765,7 +1731,7 @@ class EvidenceStore:
|
||||
AND first.conversation_id = evidence.conversation_id
|
||||
AND first.workspace IS evidence.workspace
|
||||
AND (
|
||||
first.role = 'user'
|
||||
(first.role = 'user' AND first.written_by = 'person')
|
||||
OR first.content_kind = 'memory_content'
|
||||
)
|
||||
ORDER BY julianday(COALESCE(first.created_at, first.imported_at)),
|
||||
@@ -2159,6 +2125,8 @@ class EvidenceStore:
|
||||
(
|
||||
existing["content_kind"] != evidence.content_kind,
|
||||
existing["role"] != evidence.role,
|
||||
evidence.written_by is not WrittenBy.UNKNOWN
|
||||
and existing["written_by"] != evidence.written_by,
|
||||
existing["content_hash"] != evidence.content_hash,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -111,7 +111,7 @@ class ImportSupervisor:
|
||||
)
|
||||
)
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="import.supervisor",
|
||||
event="import.queued",
|
||||
outcome="queued",
|
||||
@@ -198,7 +198,7 @@ class ImportSupervisor:
|
||||
started = monotonic()
|
||||
failure_type: str | None = None
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="import.supervisor",
|
||||
event="import.started",
|
||||
outcome="started",
|
||||
@@ -316,10 +316,6 @@ class ImportSupervisor:
|
||||
"source_id": current.source_id,
|
||||
"mode": current.operation,
|
||||
"count": current.stored,
|
||||
"stored_count": current.stored,
|
||||
"revised_count": current.revised,
|
||||
"skipped_count": current.skipped,
|
||||
"error_count": current.errors,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -85,14 +85,15 @@ def _state(connection: sqlite3.Connection, evidence_id: str) -> dict[str, str |
|
||||
value = evidence.model_dump(mode="json")
|
||||
# The stored normalized content_hash is insufficient for exact-content leases.
|
||||
content = _digest(value["verbatim"])
|
||||
material = _digest({key: value[key] for key in ("verbatim", "role", "content_kind")})
|
||||
material = _digest(
|
||||
{key: value[key] for key in ("verbatim", "role", "written_by", "content_kind")}
|
||||
)
|
||||
origins = _digest(value["origin_meta"])
|
||||
identity = _digest(
|
||||
{
|
||||
key: item
|
||||
for key, item in value.items()
|
||||
if key
|
||||
not in {"verbatim", "written_by", "origin_meta", "imported_at", "updated_at"}
|
||||
if key not in {"verbatim", "origin_meta", "imported_at", "updated_at"}
|
||||
}
|
||||
)
|
||||
return dict(
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
import json
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import TextIO, cast
|
||||
|
||||
@@ -105,13 +104,7 @@ def capture_hook(source_id: str, payload: object, database: Path) -> bool:
|
||||
# Cursor also runs ~/.claude/settings.json hooks; its own hook already files the turn.
|
||||
if source_id != "cursor" and "cursor_version" in payload:
|
||||
return False
|
||||
if not database.is_file():
|
||||
return False
|
||||
rejection = capture_rejection(database, source_id)
|
||||
if rejection is not None:
|
||||
# Silent, and the reason is counted rather than printed: a hook that
|
||||
# spoke would put this into the host's turn as text.
|
||||
record_capture_rejection(database, source_id, rejection)
|
||||
if not database.is_file() or not _sync_open(database, source_id):
|
||||
return False
|
||||
if source_id == "pi" and _is_automated_session(first_string(payload.get("cwd"))):
|
||||
return False
|
||||
@@ -146,79 +139,17 @@ def capture_hook(source_id: str, payload: object, database: Path) -> bool:
|
||||
|
||||
|
||||
def _sync_open(database: Path, source_id: str) -> bool:
|
||||
return capture_rejection(database, source_id) is None
|
||||
|
||||
|
||||
class CaptureRejection(StrEnum):
|
||||
"""Why a hook's turn was dropped, or nothing at all.
|
||||
|
||||
Capture fails open on purpose: a hook that raises or prints would be read
|
||||
back into the host's turn, so a refusal has to be silent and exit 0. It was
|
||||
also completely invisible, which is why "持续同步开着但一条都没进" could not be
|
||||
told apart from "capture 坏了". These values name the reason without any
|
||||
payload, path, session or message — the only part that may be recorded.
|
||||
"""
|
||||
|
||||
SYNC_DISABLED = "sync_disabled"
|
||||
GRANT_MISSING = "grant_missing"
|
||||
GRANT_REVOKED = "grant_revoked"
|
||||
|
||||
|
||||
def capture_rejection(database: Path, source_id: str) -> CaptureRejection | None:
|
||||
"""None when this turn may be captured, otherwise why it may not."""
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
|
||||
if not ConfigStore(database.parent).sync_enabled(source_id):
|
||||
return CaptureRejection.SYNC_DISABLED
|
||||
return False
|
||||
with main_database(database).read() as connection:
|
||||
row = connection.execute(
|
||||
"SELECT revoked_at FROM source_grants WHERE user_id='local' AND source_id=?",
|
||||
"SELECT 1 FROM source_grants WHERE user_id='local' AND source_id=? "
|
||||
"AND revoked_at IS NULL",
|
||||
(source_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return CaptureRejection.GRANT_MISSING
|
||||
# Indexed, not named: this connection has no row factory, and a hook is
|
||||
# not the place to add one.
|
||||
if row[0] is not None:
|
||||
return CaptureRejection.GRANT_REVOKED
|
||||
return None
|
||||
|
||||
|
||||
def record_capture_rejection(
|
||||
database: Path,
|
||||
source_id: str,
|
||||
reason: CaptureRejection,
|
||||
*,
|
||||
window_seconds: float = 600.0,
|
||||
) -> None:
|
||||
"""Note one dropped turn, at most once per window per source and reason.
|
||||
|
||||
Best effort by construction: a hook runs inside somebody's turn, so a sink
|
||||
that is slow, full or read-only must never surface here. The record is a
|
||||
count and nothing else — no payload, no path, no session or conversation
|
||||
id — because this file is meant to be handed to support as is.
|
||||
|
||||
One small file per source and reason, rather than a line per turn: capture
|
||||
runs on every prompt, and a log that grew with the conversation would be a
|
||||
privacy problem wearing a diagnosis as a disguise.
|
||||
"""
|
||||
try:
|
||||
directory = database.parent / "logs"
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
marker = directory / f"capture-rejection-{source_id}-{reason.value}.json"
|
||||
now = time.time()
|
||||
payload: dict[str, object] = {"source_id": source_id, "reason": reason.value}
|
||||
if marker.exists() and now - marker.stat().st_mtime < window_seconds:
|
||||
payload = json.loads(marker.read_text(encoding="utf8"))
|
||||
previous = payload.get("count", 1)
|
||||
payload["count"] = int(previous) + 1 if isinstance(previous, (str, int, float)) else 2
|
||||
else:
|
||||
payload["count"] = 1
|
||||
payload["window_started_at"] = datetime.fromtimestamp(now, UTC).isoformat()
|
||||
payload["last_at"] = payload["window_started_at"]
|
||||
marker.write_text(json.dumps(payload, sort_keys=True), encoding="utf8")
|
||||
except Exception:
|
||||
return
|
||||
return row is not None
|
||||
|
||||
|
||||
def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed: bool) -> None:
|
||||
@@ -236,7 +167,7 @@ def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed
|
||||
):
|
||||
if (
|
||||
record in chunk
|
||||
and any(item.role == "user" for item in chunk)
|
||||
and any(item.role == "user" and item.written_by is WrittenBy.PERSON for item in chunk)
|
||||
and any(item.role == "assistant" for item in chunk)
|
||||
):
|
||||
intake.submit_records(chunk, priority="live")
|
||||
|
||||
@@ -160,9 +160,7 @@ class MatterDiscovery:
|
||||
task_by_id: dict[str, MemoryTask] = {}
|
||||
for chunk in chunks:
|
||||
for record in chunk:
|
||||
task = await asyncio.to_thread(
|
||||
processor.tasks.task_for_evidence, record.evidence_id
|
||||
)
|
||||
task = await asyncio.to_thread(processor.tasks.task_for_evidence, record.evidence_id)
|
||||
if task is not None and task.status is MemoryTaskStatus.COMPLETED:
|
||||
task_by_id[task.task_id] = task
|
||||
tasks = tuple(sorted(task_by_id.values(), key=lambda task: (task.created_at, task.task_id)))
|
||||
|
||||
@@ -270,7 +270,7 @@ async def reconcile_current(
|
||||
(
|
||||
r
|
||||
for r in records
|
||||
if r.role == "user"
|
||||
if r.written_by == "person"
|
||||
and r.created_at
|
||||
and (r.source_id, r.conversation_id) in related
|
||||
),
|
||||
|
||||
@@ -51,19 +51,14 @@ def validate_claim(claim: Claim, records: Mapping[str, EvidenceRecord | SampleRe
|
||||
text = record.text if isinstance(record, SampleRecord) else record.verbatim
|
||||
if not citation.quote.strip() or citation.quote not in text:
|
||||
raise ValueError("citation quote is not in the source")
|
||||
author = {
|
||||
"user": "person",
|
||||
"assistant": "agent",
|
||||
"tool": "tool",
|
||||
}.get(record.role, "unknown")
|
||||
authors.add(author)
|
||||
authors.add(str(record.written_by))
|
||||
created = record.created_at
|
||||
citations.append(
|
||||
citation.model_copy(
|
||||
update={
|
||||
"revision": record.revision,
|
||||
"source_id": record.source_id,
|
||||
"written_by": author,
|
||||
"written_by": str(record.written_by),
|
||||
"created_at": created.isoformat() if isinstance(created, datetime) else created,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -290,7 +290,7 @@ def read_memory_library(
|
||||
COALESCE(
|
||||
MIN(
|
||||
CASE WHEN e.role = 'user'
|
||||
AND e.role = 'user'
|
||||
AND e.written_by IN ('person', 'user')
|
||||
THEN e.created_at END
|
||||
),
|
||||
MIN(e.created_at)
|
||||
|
||||
@@ -10,6 +10,7 @@ from fastapi.params import Depends
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.imports.models import (
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
@@ -185,10 +186,6 @@ class MemoryNoteStore:
|
||||
]
|
||||
if task is None or task.status == MemoryTaskStatus.WAITING_FOR_MODEL:
|
||||
state = "waiting_for_model"
|
||||
elif task.status == MemoryTaskStatus.WAITING_FOR_PERMISSION:
|
||||
# Upgrade compatibility: old rows are claimable and will leave
|
||||
# this state as soon as the direct SDK runner claims them.
|
||||
state = "waiting_for_permission"
|
||||
elif task.status == MemoryTaskStatus.COMPLETED:
|
||||
state = "ready" if memory_ids else "no_memories"
|
||||
elif task.status == MemoryTaskStatus.RUNNING:
|
||||
@@ -198,6 +195,22 @@ class MemoryNoteStore:
|
||||
else:
|
||||
state = "failed"
|
||||
error = task.last_error_code if task else None
|
||||
if self.processor.model_content_config is not None and state in {
|
||||
"waiting_for_model",
|
||||
"failed",
|
||||
}:
|
||||
try:
|
||||
active = self.processor.profiles.active(self.processor.secrets)
|
||||
if active is not None:
|
||||
self.processor._guard_records((record,), active)
|
||||
except HyMemoryBridgeError as failure:
|
||||
if str(failure).startswith("model_content_"):
|
||||
state = "waiting_for_permission"
|
||||
error = str(failure)
|
||||
else:
|
||||
raise
|
||||
except SecretStoreUnavailableError:
|
||||
error = "secure_storage_unavailable"
|
||||
return MemoryNoteReceipt(
|
||||
evidence_id=record.evidence_id,
|
||||
text=sanitize_model_input(record.verbatim),
|
||||
@@ -261,8 +274,11 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
# unchanged; replaying the same request returns the original
|
||||
# `entry:<evidence_id>` job instead of a second HY call.
|
||||
if readmit:
|
||||
# `/{evidence_id}/process` is the explicit retry signal.
|
||||
# It also revives pre-upgrade permission waiters.
|
||||
# `/{evidence_id}/process` is the explicit "the condition
|
||||
# that parked it may have changed" signal: the person just
|
||||
# granted consent or fixed the model. `submit_evidence`
|
||||
# alone would return an existing parked row unchanged, so
|
||||
# without this readmit the note stays parked forever.
|
||||
await asyncio.to_thread(notes.intake.queue.readmit_parked)
|
||||
submitted = await asyncio.to_thread(
|
||||
notes.intake.submit_evidence,
|
||||
@@ -280,6 +296,10 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
last_round=1,
|
||||
priority="live",
|
||||
)
|
||||
except HyMemoryBridgeError as error:
|
||||
if not str(error).startswith("model_content_"):
|
||||
raise
|
||||
return await asyncio.to_thread(notes.receipt, record.evidence_id)
|
||||
except SecretStoreUnavailableError:
|
||||
saved = await asyncio.to_thread(notes.receipt, record.evidence_id)
|
||||
return saved.model_copy(
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -29,13 +29,8 @@ class ConfigDocument(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
|
||||
|
||||
schema_version: int = Field(default=1, ge=1, le=1)
|
||||
model_content_allowed: StrictBool = False
|
||||
sources: dict[str, SourceConfig] = Field(default_factory=dict)
|
||||
#: Whether the configured model may turn imported conversations into
|
||||
#: memories. Defaulting to on keeps every existing install doing what it
|
||||
#: did before the switch existed; stopping it is the user's cost control
|
||||
#: and leaves the model configuration itself untouched, so starting again
|
||||
#: needs no re-entry of a key.
|
||||
sedimentation_enabled: StrictBool = True
|
||||
|
||||
@field_validator("sources")
|
||||
@classmethod
|
||||
@@ -69,18 +64,9 @@ class ConfigStore:
|
||||
source = self._document.sources.get(source_id)
|
||||
return bool(source and source.sync_enabled)
|
||||
|
||||
def sedimentation_enabled(self) -> bool:
|
||||
def model_content_allowed(self) -> bool:
|
||||
with self._lock:
|
||||
return bool(self._document.sedimentation_enabled)
|
||||
|
||||
def set_sedimentation_enabled(self, enabled: bool) -> ConfigDocument:
|
||||
if type(enabled) is not bool:
|
||||
raise ValueError("invalid_sedimentation_config")
|
||||
with self._lock:
|
||||
updated = self._document.model_copy(update={"sedimentation_enabled": enabled})
|
||||
self._write(updated)
|
||||
self._document = updated
|
||||
return updated
|
||||
return self._document.model_content_allowed
|
||||
|
||||
def set_sync(self, source_id: str, enabled: bool) -> ConfigDocument:
|
||||
return self.set_sync_many({source_id: enabled})
|
||||
@@ -100,17 +86,20 @@ class ConfigStore:
|
||||
self._document = updated
|
||||
return updated
|
||||
|
||||
def set_model_content_allowed(self, allowed: bool) -> ConfigDocument:
|
||||
if type(allowed) is not bool:
|
||||
raise ValueError("invalid_model_content_config")
|
||||
with self._lock:
|
||||
updated = self._document.model_copy(update={"model_content_allowed": allowed})
|
||||
self._write(updated)
|
||||
self._document = updated
|
||||
return updated
|
||||
|
||||
def _read(self) -> ConfigDocument:
|
||||
if not self.path.exists():
|
||||
return ConfigDocument()
|
||||
try:
|
||||
value = json.loads(self.path.read_text(encoding="utf-8"))
|
||||
if isinstance(value, dict):
|
||||
# Builds through 1.0.2 persisted this global consent bit.
|
||||
# The embedded SDK is now trusted to use all local memory, so
|
||||
# accept existing files while discarding the retired setting.
|
||||
value.pop("model_content_allowed", None)
|
||||
return ConfigDocument.model_validate(value)
|
||||
return ConfigDocument.model_validate_json(self.path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError) as error:
|
||||
raise ConfigFileError("config_invalid") from error
|
||||
|
||||
|
||||
@@ -23,13 +23,10 @@ import hashlib
|
||||
import hmac
|
||||
import os
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from enum import StrEnum
|
||||
from typing import Annotated, Final, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator
|
||||
|
||||
from one_memory.imports.models import EvidenceRecord
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
#: Bumped when a derivation rule changes, so old and new ids never compare equal.
|
||||
SCHEMA_VERSION: Final[str] = "ms1"
|
||||
@@ -169,6 +166,7 @@ class CanonicalMessage(BaseModel):
|
||||
message_id: str | None = Field(default=None, min_length=1, max_length=512)
|
||||
ordinal: int = Field(ge=0)
|
||||
role: str = Field(min_length=1, max_length=128)
|
||||
written_by: str = Field(min_length=1, max_length=64)
|
||||
content: str = Field(min_length=1)
|
||||
workspace: str | None = Field(default=None, max_length=1024)
|
||||
|
||||
@@ -215,13 +213,8 @@ class WriteRequest(BaseModel):
|
||||
source_id: str | None = Field(default=None, min_length=1, max_length=128)
|
||||
batch_identity: BatchIdentity
|
||||
evidence_ids: list[str] = Field(min_length=1)
|
||||
# New jobs freeze the complete input in their own payload. The default is
|
||||
# solely for reading pre-upgrade rows whose payload only carried ids.
|
||||
input_records: tuple[EvidenceRecord, ...] = ()
|
||||
idempotency_key: str = Field(min_length=1, max_length=256)
|
||||
priority: Literal["live", "normal"] = "normal"
|
||||
event_at: datetime | None = None
|
||||
time_quality: Literal["exact", "anchored", "session", "approximate"] = "approximate"
|
||||
|
||||
@field_validator("evidence_ids")
|
||||
@classmethod
|
||||
@@ -234,16 +227,6 @@ class WriteRequest(BaseModel):
|
||||
raise ValueError("evidence_ids must be unique")
|
||||
return value
|
||||
|
||||
@field_validator("input_records")
|
||||
@classmethod
|
||||
def _input_records_match_ids(
|
||||
cls, value: tuple[EvidenceRecord, ...], info: ValidationInfo
|
||||
) -> tuple[EvidenceRecord, ...]:
|
||||
evidence_ids = info.data.get("evidence_ids")
|
||||
if value and evidence_ids != [record.evidence_id for record in value]:
|
||||
raise ValueError("input_records must match evidence_ids in order")
|
||||
return value
|
||||
|
||||
@property
|
||||
def request_key(self) -> str:
|
||||
"""Stable dedup key. Same batch + same intent must not enqueue twice."""
|
||||
|
||||
@@ -30,7 +30,7 @@ from uuid import uuid4
|
||||
from one_memory.background_work import in_the_background
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.diagnostic_log import diagnostic_event
|
||||
from one_memory.imports.models import EvidenceOrigin, EvidenceRecord
|
||||
from one_memory.imports.models import EvidenceRecord
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_server.contracts import WriteRequest
|
||||
from one_memory.memory_server.data_safety import MIN_FREE_BYTES # noqa: F401 - re-exported
|
||||
@@ -249,7 +249,6 @@ MODEL_NOT_READY_CODES = frozenset(
|
||||
#: nothing was sent and retrying the same intent changes nothing — a terminal
|
||||
#: `failed`, not `unknown` (which would wrongly imply a possibly-applied
|
||||
#: write) and not a retry (which would fail identically).
|
||||
#:
|
||||
PRE_SEND_REFUSAL_CODES = frozenset(
|
||||
{
|
||||
"hy_memory_input_exceeds_window",
|
||||
@@ -283,6 +282,11 @@ _VALID_JOB_STATES = frozenset(
|
||||
#: A row in one of these states can never be moved back to a live one.
|
||||
_TERMINAL_WRITE_STATES = frozenset({COMPLETED, FAILED, DEAD, CANCELLED})
|
||||
|
||||
#: A job in one of these states can never run again, so its payload is dead
|
||||
#: weight. `unknown` is deliberately absent: it is non-terminal and still needs
|
||||
#: its intent for reconciliation. `running` is obviously still live.
|
||||
TERMINAL_PAYLOAD_STATES = (COMPLETED, FAILED, DEAD, CANCELLED)
|
||||
|
||||
#: Columns a current schema declares but an older `memory_write_jobs` may lack.
|
||||
#: `CREATE TABLE IF NOT EXISTS` is a no-op for an existing table, so without
|
||||
#: this an upgraded install would keep an older queue that every claim/lease
|
||||
@@ -300,12 +304,9 @@ LEGACY_MEMORY_WRITE_JOB_COLUMNS = (
|
||||
("lease_expires_at", "TEXT"),
|
||||
("hy_invoked", "INTEGER NOT NULL DEFAULT 0"),
|
||||
("last_error", "TEXT"),
|
||||
("event_at", "TEXT"),
|
||||
("time_quality", "TEXT NOT NULL DEFAULT 'approximate'"),
|
||||
)
|
||||
|
||||
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)
|
||||
EVENT_TIME_WINDOW = timedelta(minutes=10)
|
||||
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL, UNKNOWN)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -392,42 +393,6 @@ def _matches_payload_except_priority(stored: str, payload: str) -> bool:
|
||||
return old == new
|
||||
|
||||
|
||||
def _matches_legacy_reference_replay(stored: str, payload: str) -> bool:
|
||||
"""Match an old id-only caller to the same job's frozen payload."""
|
||||
try:
|
||||
old = json.loads(stored)
|
||||
new = json.loads(payload)
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
if not isinstance(old, dict) or not isinstance(new, dict):
|
||||
return False
|
||||
old_inputs = old.get("input_records")
|
||||
new_inputs = new.get("input_records")
|
||||
if bool(old_inputs) == bool(new_inputs):
|
||||
return False
|
||||
old.pop("input_records", None)
|
||||
new.pop("input_records", None)
|
||||
return old == new
|
||||
|
||||
|
||||
def _matches_reaped_legacy_payload(stored: str, payload: str) -> bool:
|
||||
"""Compare a current frozen request with a pre-snapshot reaped payload."""
|
||||
try:
|
||||
old = json.loads(stored)
|
||||
current = json.loads(payload)
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
if (
|
||||
not isinstance(old, dict)
|
||||
or not isinstance(current, dict)
|
||||
or not isinstance(old.get(REAPED_PAYLOAD_MARKER), str)
|
||||
):
|
||||
return False
|
||||
current.pop("input_records", None)
|
||||
legacy_payload = json.dumps(current, sort_keys=True)
|
||||
return bool(old[REAPED_PAYLOAD_MARKER] == _payload_digest(legacy_payload))
|
||||
|
||||
|
||||
class DurableQueueSchemaUnavailableError(RuntimeError):
|
||||
"""The durable queue schema has not been created yet.
|
||||
|
||||
@@ -480,7 +445,6 @@ class DurableWriteQueue:
|
||||
if not initialize:
|
||||
return
|
||||
with main_database(database_path).write() as connection, connection:
|
||||
connection.row_factory = sqlite3.Row
|
||||
# First create the base table if absent. Then widen an older table
|
||||
# before replaying indexes that reference the new columns; a bare
|
||||
# CREATE INDEX would fail on a pre-priority queue.
|
||||
@@ -491,7 +455,6 @@ class DurableWriteQueue:
|
||||
for statement in memory_write_jobs_sqlite_schema()[1:]:
|
||||
connection.execute(statement)
|
||||
self._backfill_legacy_references(connection)
|
||||
self._backfill_legacy_event_times(connection)
|
||||
# A process that died while holding a lease cannot run its own
|
||||
# recovery. Opening the queue is the first safe point to recover any
|
||||
# rows it left behind, before any new worker claims them.
|
||||
@@ -516,7 +479,8 @@ class DurableWriteQueue:
|
||||
"UPDATE memory_write_jobs SET source_key=session_key WHERE source_key=''"
|
||||
)
|
||||
legacy_rows = connection.execute(
|
||||
"SELECT job_id FROM memory_write_jobs WHERE queue_seq=0 ORDER BY created_at, rowid"
|
||||
"SELECT job_id FROM memory_write_jobs WHERE queue_seq=0 "
|
||||
"ORDER BY created_at, rowid"
|
||||
).fetchall()
|
||||
next_seq = int(
|
||||
connection.execute(
|
||||
@@ -550,49 +514,6 @@ class DurableWriteQueue:
|
||||
if columns and columns != ["state", "priority", "created_at"]:
|
||||
connection.execute("DROP INDEX idx_write_jobs_claim")
|
||||
|
||||
@staticmethod
|
||||
def _backfill_legacy_event_times(connection: sqlite3.Connection) -> None:
|
||||
if (
|
||||
connection.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='evidence'"
|
||||
).fetchone()
|
||||
is None
|
||||
):
|
||||
connection.execute(
|
||||
"UPDATE memory_write_jobs SET event_at=created_at WHERE event_at IS NULL"
|
||||
)
|
||||
return
|
||||
evidence_columns = {
|
||||
str(row[1]) for row in connection.execute("PRAGMA table_info(evidence)").fetchall()
|
||||
}
|
||||
if not {"created_at", "imported_at", "role"} <= evidence_columns:
|
||||
connection.execute(
|
||||
"UPDATE memory_write_jobs SET event_at=created_at WHERE event_at IS NULL"
|
||||
)
|
||||
return
|
||||
connection.execute(
|
||||
"""
|
||||
UPDATE memory_write_jobs
|
||||
SET event_at = COALESCE(
|
||||
(
|
||||
SELECT MAX(COALESCE(e.created_at, e.imported_at))
|
||||
FROM memory_write_job_evidence AS linked
|
||||
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
|
||||
WHERE linked.job_id = memory_write_jobs.job_id
|
||||
AND e.role = 'user'
|
||||
),
|
||||
(
|
||||
SELECT MIN(COALESCE(e.created_at, e.imported_at))
|
||||
FROM memory_write_job_evidence AS linked
|
||||
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
|
||||
WHERE linked.job_id = memory_write_jobs.job_id
|
||||
),
|
||||
created_at
|
||||
)
|
||||
WHERE event_at IS NULL
|
||||
"""
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _backfill_legacy_references(connection: sqlite3.Connection) -> None:
|
||||
"""Populate the reference table after its schema exists."""
|
||||
@@ -689,20 +610,8 @@ class DurableWriteQueue:
|
||||
replacing the earlier intent.
|
||||
"""
|
||||
self._refuse_if_below_reserve()
|
||||
request = self._freeze_request_inputs(request)
|
||||
payload = json.dumps(
|
||||
request.model_dump(
|
||||
mode="json",
|
||||
exclude={"event_at", "time_quality"},
|
||||
),
|
||||
sort_keys=True,
|
||||
)
|
||||
payload = json.dumps(request.model_dump(mode="json"), sort_keys=True)
|
||||
now = datetime.now(UTC).isoformat()
|
||||
resolved_event, resolved_quality = self._event_time_for_request(request)
|
||||
event = request.event_at or resolved_event or datetime.fromisoformat(now)
|
||||
if event.tzinfo is None:
|
||||
event = event.replace(tzinfo=UTC)
|
||||
event_at = event.astimezone(UTC).isoformat()
|
||||
key = request.request_key
|
||||
with (
|
||||
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
|
||||
@@ -715,22 +624,16 @@ class DurableWriteQueue:
|
||||
(key,),
|
||||
).fetchone()
|
||||
if existing is not None:
|
||||
# A replay of work that already finished is a hit, not a
|
||||
# conflict. Terminal rows store a reaped digest instead of the
|
||||
# original payload, and the caller's key is derived from the
|
||||
# evidence's revision, which can move on between replays while
|
||||
# the payload's evidence ids stay the same. Refusing that pair
|
||||
# turned every startup replay of one settled job into an
|
||||
# exception: 236 of them in a single afternoon, all one key.
|
||||
already_settled = (
|
||||
str(existing["state"]) == COMPLETED and int(existing["hy_invoked"]) == 1
|
||||
)
|
||||
stored_payload = str(existing["payload_json"])
|
||||
priority_replay = _matches_payload_except_priority(stored_payload, payload)
|
||||
legacy_reaped_match = bool(
|
||||
already_settled
|
||||
and _matches_reaped_legacy_payload(stored_payload, payload)
|
||||
)
|
||||
if (
|
||||
not _matches_stored_payload(stored_payload, payload)
|
||||
and not priority_replay
|
||||
and not _matches_legacy_reference_replay(stored_payload, payload)
|
||||
and not legacy_reaped_match
|
||||
):
|
||||
raise ValueError(f"idempotency key {key} already exists with different content")
|
||||
if already_settled:
|
||||
return DurableWriteRecord(
|
||||
job_id=str(existing["job_id"]),
|
||||
@@ -741,6 +644,14 @@ class DurableWriteQueue:
|
||||
updated_at=str(existing["updated_at"]),
|
||||
detail="already completed; replay treated as a hit",
|
||||
)
|
||||
priority_replay = _matches_payload_except_priority(
|
||||
str(existing["payload_json"]), payload
|
||||
)
|
||||
if (
|
||||
not _matches_stored_payload(str(existing["payload_json"]), payload)
|
||||
and not priority_replay
|
||||
):
|
||||
raise ValueError(f"idempotency key {key} already exists with different content")
|
||||
if priority_replay and request.priority == "live":
|
||||
connection.execute(
|
||||
"UPDATE memory_write_jobs SET priority='live', updated_at=? "
|
||||
@@ -800,13 +711,11 @@ class DurableWriteQueue:
|
||||
)
|
||||
connection.execute(
|
||||
"INSERT INTO memory_write_jobs ("
|
||||
" job_id, queue_seq, session_key, source_key, batch_id,"
|
||||
" idempotency_key, payload_json,"
|
||||
" job_id, queue_seq, session_key, source_key, batch_id, idempotency_key, payload_json,"
|
||||
" evidence_ids_json,"
|
||||
" state, priority, attempts, max_attempts, lease_owner,"
|
||||
" lease_expires_at, hy_invoked, last_error, event_at, time_quality,"
|
||||
" created_at, updated_at"
|
||||
") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
" lease_expires_at, hy_invoked, last_error, created_at, updated_at"
|
||||
") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(
|
||||
job_id,
|
||||
queue_seq,
|
||||
@@ -824,8 +733,6 @@ class DurableWriteQueue:
|
||||
None,
|
||||
0,
|
||||
None,
|
||||
event_at,
|
||||
request.time_quality if request.event_at is not None else resolved_quality,
|
||||
now,
|
||||
now,
|
||||
),
|
||||
@@ -834,18 +741,6 @@ class DurableWriteQueue:
|
||||
"INSERT INTO memory_write_job_evidence (job_id, evidence_id) VALUES (?,?)",
|
||||
((job_id, str(evidence_id)) for evidence_id in request.evidence_ids),
|
||||
)
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="memory.write_queue",
|
||||
event="memory_write_job.queued",
|
||||
outcome="queued",
|
||||
job_id=job_id,
|
||||
details={
|
||||
"source_id": request.source_id or request.internal_session_id,
|
||||
"mode": request.priority,
|
||||
"count": len(request.evidence_ids),
|
||||
},
|
||||
)
|
||||
return DurableWriteRecord(
|
||||
job_id=job_id,
|
||||
request_id=key,
|
||||
@@ -855,45 +750,6 @@ class DurableWriteQueue:
|
||||
updated_at=now,
|
||||
)
|
||||
|
||||
def _freeze_request_inputs(self, request: WriteRequest) -> WriteRequest:
|
||||
if request.input_records or self.evidence is None:
|
||||
return request
|
||||
records: list[EvidenceRecord] = []
|
||||
for evidence_id in request.evidence_ids:
|
||||
record = self.evidence.get(evidence_id)
|
||||
if record is None:
|
||||
return request
|
||||
records.append(record)
|
||||
return request.model_copy(update={"input_records": tuple(records)})
|
||||
|
||||
def _event_time_for_request(self, request: WriteRequest) -> tuple[datetime | None, str]:
|
||||
records = list(request.input_records)
|
||||
if not records and self.evidence is not None:
|
||||
records = [
|
||||
record
|
||||
for evidence_id in request.evidence_ids
|
||||
if (record := self.evidence.get(evidence_id)) is not None
|
||||
]
|
||||
if not records:
|
||||
return None, "approximate"
|
||||
opener = next(
|
||||
(record for record in records if record.role == "user"),
|
||||
records[0],
|
||||
)
|
||||
if opener.created_at is None:
|
||||
return opener.imported_at, "approximate"
|
||||
origin = opener.origin_meta.get(EvidenceOrigin.FILE)
|
||||
basis = origin.raw_meta.get("time_basis") if origin is not None else None
|
||||
turn_event_at = origin.raw_meta.get("turn_event_at") if origin is not None else None
|
||||
if isinstance(turn_event_at, str):
|
||||
try:
|
||||
return datetime.fromisoformat(turn_event_at), "exact"
|
||||
except ValueError:
|
||||
pass
|
||||
if basis == "file_mtime":
|
||||
return opener.created_at, "session"
|
||||
return opener.created_at, "exact"
|
||||
|
||||
def repair_pre_send_identity_failures(self, *, limit: int = 128) -> int:
|
||||
"""Refresh stale Evidence journals and requeue only provably unsent jobs."""
|
||||
if self.evidence is None:
|
||||
@@ -1003,55 +859,11 @@ class DurableWriteQueue:
|
||||
raise ReconcileSchemaUnavailableError(
|
||||
"evidence table is unavailable; run migrations before reconcile"
|
||||
)
|
||||
has_versions = (
|
||||
connection.execute(
|
||||
"SELECT 1 FROM sqlite_master "
|
||||
"WHERE type='table' AND name='evidence_versions'"
|
||||
).fetchone()
|
||||
is not None
|
||||
where = (
|
||||
"WHERE evidence.imported_at IS NOT NULL "
|
||||
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence ref "
|
||||
"WHERE ref.evidence_id=evidence.evidence_id)"
|
||||
)
|
||||
evidence_columns = {
|
||||
str(row[1])
|
||||
for row in connection.execute("PRAGMA table_info(evidence)").fetchall()
|
||||
}
|
||||
if {"revision", "content_hash", "updated_at"} <= evidence_columns:
|
||||
revision_time = (
|
||||
"COALESCE((SELECT version.changed_at FROM evidence_versions version "
|
||||
"WHERE version.evidence_id=evidence.evidence_id "
|
||||
"AND version.revision=evidence.revision),"
|
||||
"evidence.updated_at,evidence.imported_at)"
|
||||
if has_versions
|
||||
else "COALESCE(evidence.updated_at,evidence.imported_at)"
|
||||
)
|
||||
where = (
|
||||
"WHERE evidence.imported_at IS NOT NULL "
|
||||
"AND NOT EXISTS ("
|
||||
"SELECT 1 FROM memory_write_job_evidence linked "
|
||||
"JOIN memory_write_jobs job ON job.job_id=linked.job_id, "
|
||||
"json_each(job.payload_json,'$.input_records') frozen "
|
||||
"WHERE linked.evidence_id=evidence.evidence_id "
|
||||
"AND json_extract(frozen.value,'$.evidence_id')=evidence.evidence_id "
|
||||
"AND json_extract(frozen.value,'$.revision')=evidence.revision "
|
||||
"AND json_extract(frozen.value,'$.content_hash')=evidence.content_hash"
|
||||
") "
|
||||
# Pre-snapshot builds only recorded an Evidence id. Treat
|
||||
# that as coverage only when its Job was created after this
|
||||
# exact revision.
|
||||
"AND NOT EXISTS ("
|
||||
"SELECT 1 FROM memory_write_job_evidence legacy "
|
||||
"JOIN memory_write_jobs legacy_job ON legacy_job.job_id=legacy.job_id "
|
||||
"WHERE legacy.evidence_id=evidence.evidence_id "
|
||||
"AND COALESCE(json_array_length("
|
||||
"json_extract(legacy_job.payload_json,'$.input_records')),0)=0 "
|
||||
f"AND legacy_job.created_at>=({revision_time}))"
|
||||
)
|
||||
else:
|
||||
# A genuinely old/minimal schema has no revision identity.
|
||||
where = (
|
||||
"WHERE evidence.imported_at IS NOT NULL "
|
||||
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence legacy "
|
||||
"WHERE legacy.evidence_id=evidence.evidence_id)"
|
||||
)
|
||||
params: tuple[Any, ...] = ()
|
||||
cursor_clause = ""
|
||||
if after_cursor is not None:
|
||||
@@ -1283,7 +1095,6 @@ class DurableWriteQueue:
|
||||
with main_database(self.database_path).write() as connection, connection:
|
||||
updated = connection.execute(
|
||||
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,"
|
||||
"attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,"
|
||||
"lease_expires_at=NULL,updated_at=? WHERE " + where,
|
||||
(PENDING, now, *params),
|
||||
)
|
||||
@@ -1351,10 +1162,45 @@ class DurableWriteQueue:
|
||||
return state
|
||||
|
||||
def clear_terminal_payloads(self, *, limit: int = 512) -> int:
|
||||
"""Deprecated compatibility hook; durable inputs are immutable audit data."""
|
||||
"""Replace terminal rows' request bodies with a content digest.
|
||||
|
||||
A completed/failed/dead/cancelled row can never run again, so the full
|
||||
serialized `WriteRequest` is unbounded dead weight. A digest keeps the
|
||||
idempotency contract intact: a replay of the same intent still resolves
|
||||
to the same row, while the same key with different content still
|
||||
conflicts. `unknown` is left alone because it is non-terminal and its
|
||||
intent is needed for reconciliation.
|
||||
|
||||
Bounded by `limit` so a large backlog drains over several calls instead
|
||||
of one long write on the single writer.
|
||||
"""
|
||||
if type(limit) is not int or limit < 1:
|
||||
raise ValueError("limit must be positive")
|
||||
return 0
|
||||
placeholders = ",".join("?" for _ in TERMINAL_PAYLOAD_STATES)
|
||||
with (
|
||||
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
|
||||
connection,
|
||||
):
|
||||
rows = connection.execute(
|
||||
"SELECT job_id, payload_json FROM memory_write_jobs "
|
||||
f"WHERE state IN ({placeholders}) "
|
||||
"AND instr(payload_json, ?) = 0 "
|
||||
"LIMIT ?",
|
||||
(*TERMINAL_PAYLOAD_STATES, REAPED_PAYLOAD_MARKER, limit),
|
||||
).fetchall()
|
||||
if not rows:
|
||||
return 0
|
||||
for row in rows:
|
||||
connection.execute(
|
||||
"UPDATE memory_write_jobs SET payload_json=? "
|
||||
f"WHERE job_id=? AND state IN ({placeholders})",
|
||||
(
|
||||
_reaped_payload(_payload_digest(str(row["payload_json"]))),
|
||||
str(row["job_id"]),
|
||||
*TERMINAL_PAYLOAD_STATES,
|
||||
),
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
def readmit_parked(self, *, states: tuple[str, ...] = PARKED_STATES) -> int:
|
||||
"""Move parked jobs back to `pending` for a fresh claim.
|
||||
@@ -1556,13 +1402,7 @@ class DurableWriteQueue:
|
||||
cross back to pending, so this cannot duplicate a possibly-sent write.
|
||||
"""
|
||||
now = datetime.now(UTC).isoformat()
|
||||
payload = json.dumps(
|
||||
request.model_dump(
|
||||
mode="json",
|
||||
exclude={"event_at", "time_quality"},
|
||||
),
|
||||
sort_keys=True,
|
||||
)
|
||||
payload = json.dumps(request.model_dump(mode="json"), sort_keys=True)
|
||||
with (
|
||||
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
|
||||
connection,
|
||||
@@ -1652,59 +1492,6 @@ class DurableWriteQueue:
|
||||
)
|
||||
return len(job_ids)
|
||||
|
||||
def retry_first_unknown_after_reconcile(
|
||||
self,
|
||||
*,
|
||||
older_than: datetime,
|
||||
limit: int = 32,
|
||||
) -> int:
|
||||
"""Retry one unresolved unknown outcome once, after reconciliation.
|
||||
|
||||
The caller first asks HY and checks linked governed output. Rows still
|
||||
unknown may have lost the response after the SDK call, so one bounded
|
||||
retry is useful. `attempts=1` is the guard: the retry claim increments
|
||||
it to two, and a second unknown is left for explicit user recovery.
|
||||
"""
|
||||
if type(limit) is not int or limit < 1:
|
||||
raise ValueError("limit must be positive")
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with (
|
||||
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
|
||||
connection,
|
||||
):
|
||||
rows = connection.execute(
|
||||
"SELECT job_id FROM memory_write_jobs "
|
||||
"WHERE state=? AND attempts=1 AND attempts < max_attempts "
|
||||
"AND updated_at <= ? ORDER BY updated_at, job_id LIMIT ?",
|
||||
(UNKNOWN, older_than.isoformat(), limit),
|
||||
).fetchall()
|
||||
job_ids = tuple(str(row["job_id"]) for row in rows)
|
||||
if not job_ids:
|
||||
return 0
|
||||
placeholders = ",".join("?" for _ in job_ids)
|
||||
connection.execute(
|
||||
f"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
|
||||
f"lease_expires_at=NULL, hy_invoked=0, "
|
||||
f"last_error='model_content_unknown_auto_retry', updated_at=? "
|
||||
f"WHERE state=? AND attempts=1 AND job_id IN ({placeholders})",
|
||||
(PENDING, now, UNKNOWN, *job_ids),
|
||||
)
|
||||
if connection.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_tasks'"
|
||||
).fetchone():
|
||||
connection.execute(
|
||||
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
|
||||
f"last_error_code='model_content_unknown_auto_retry', updated_at=? "
|
||||
f"WHERE status=? AND task_id IN ("
|
||||
f"SELECT inputs.task_id FROM memory_task_inputs AS inputs "
|
||||
f"JOIN memory_write_job_evidence AS linked ON EXISTS ("
|
||||
f"SELECT 1 FROM json_each(inputs.refs_json) AS ref "
|
||||
f"WHERE json_extract(ref.value, '$.evidence_id')=linked.evidence_id) "
|
||||
f"WHERE linked.job_id IN ({placeholders}))",
|
||||
(MemoryTaskStatus.PENDING, now, MemoryTaskStatus.UNKNOWN, *job_ids),
|
||||
)
|
||||
return len(job_ids)
|
||||
|
||||
def resolve_unknown_from_linked_output(self, job_id: str) -> bool:
|
||||
"""Close an unknown durable row when local governance output proves success."""
|
||||
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
|
||||
@@ -1901,12 +1688,9 @@ class DurableWriteRegistry:
|
||||
def evidence_for_request(store: EvidenceStore, request: WriteRequest) -> tuple[EvidenceRecord, ...]:
|
||||
"""The Evidence a stored write points at, in the request's own order.
|
||||
|
||||
Current jobs are self-contained: their immutable records are the model
|
||||
input even if a Harness later rewinds or edits its live transcript. The
|
||||
store lookup remains only for payloads written by pre-upgrade builds.
|
||||
A write may only point at Evidence that still exists; a missing id is a
|
||||
rejection rather than a silent partial send.
|
||||
"""
|
||||
if request.input_records:
|
||||
return request.input_records
|
||||
records = []
|
||||
for evidence_id in request.evidence_ids:
|
||||
record = store.get(evidence_id)
|
||||
@@ -2013,14 +1797,7 @@ async def execute_job(
|
||||
# Never persist the raw message: it may be a provider response or the
|
||||
# user's own text, and `last_error` is shown over HTTP as `detail`.
|
||||
code = safe_error_code(error)
|
||||
if (
|
||||
code
|
||||
in {
|
||||
"model_content_source_disabled",
|
||||
"model_sedimentation_disabled",
|
||||
}
|
||||
and not hy_invoked
|
||||
):
|
||||
if code == "model_content_source_disabled" and not hy_invoked:
|
||||
await in_the_background(queue.release_unstarted, job_id, owner=owner)
|
||||
return
|
||||
# The task layer classifies these before any bridge is opened, so HY
|
||||
@@ -2031,11 +1808,7 @@ async def execute_job(
|
||||
# - transient model/provider trouble -> waiting_for_model (parked).
|
||||
# Everything else may have crossed into HY, and HY has no idempotency
|
||||
# key, so it is `unknown` rather than a blind retry.
|
||||
if code in PRE_SEND_REFUSAL_CODES:
|
||||
# The frozen input window is checked before the payload reaches
|
||||
# HY. This one refusal therefore has a proven pre-send boundary.
|
||||
state, invoked_marker = FAILED, False
|
||||
elif hy_invoked:
|
||||
if hy_invoked:
|
||||
# Error names do not prove execution phase: permission and
|
||||
# evidence rechecks also run after HY returns. Once the durable
|
||||
# marker is set, every failure is possibly-sent and must stay out
|
||||
@@ -2047,6 +1820,10 @@ async def execute_job(
|
||||
state, invoked_marker = WAITING_FOR_PERMISSION, False
|
||||
elif code in MODEL_NOT_READY_CODES:
|
||||
state, invoked_marker = WAITING_FOR_MODEL, False
|
||||
elif code in PRE_SEND_REFUSAL_CODES:
|
||||
# Provably before the bridge: the frozen input is stale or too
|
||||
# large. Nothing was sent, so this is terminal rather than unknown.
|
||||
state, invoked_marker = FAILED, False
|
||||
elif code == "model_content_transport_request_failed":
|
||||
# The generation or embedding HTTP exchange can fail after HY has
|
||||
# already persisted a node. Treating that as a pre-send failure
|
||||
@@ -2119,13 +1896,6 @@ class DurableWriteDrain:
|
||||
#: Reaping is a write; it runs at most once per this interval on the
|
||||
#: idle path so an empty queue does not issue one write per poll.
|
||||
self._reap_interval = reap_interval
|
||||
# Unknown recovery is user-visible and lightweight enough to check
|
||||
# sooner than general database housekeeping. Tests that set the common
|
||||
# interval to zero keep immediate maintenance.
|
||||
self._unknown_reconcile_interval = min(
|
||||
reap_interval,
|
||||
timedelta(seconds=10),
|
||||
)
|
||||
#: The maintenance writes below are recoverability/retention work, not
|
||||
#: idle housekeeping, so each has its own interval and runs at the start
|
||||
#: of every drain tick. Sharing one timestamp would let a busy tick
|
||||
@@ -2135,6 +1905,7 @@ class DurableWriteDrain:
|
||||
#: sustained backlog must not keep finished rows' request bodies in the
|
||||
#: business database indefinitely. Its own gate so neither interval can
|
||||
#: starve the other.
|
||||
self._last_terminal_reap: datetime | None = None
|
||||
#: `waiting_for_model` re-admission is also time-based recoverability:
|
||||
#: a model outage clears on its own, and a queue that never goes idle
|
||||
#: must still re-admit the write once the wait has elapsed.
|
||||
@@ -2185,7 +1956,7 @@ class DurableWriteDrain:
|
||||
now: datetime | None = None,
|
||||
budget_ms: int | None = None,
|
||||
) -> tuple[str, ...]:
|
||||
"""Claim event-time heads, with one running job per enabled Harness."""
|
||||
"""Claim at most one FIFO head per enabled Harness."""
|
||||
if type(limit) is not int or limit < 1:
|
||||
raise ValueError("limit must be positive")
|
||||
now = now or datetime.now(UTC)
|
||||
@@ -2227,7 +1998,9 @@ class DurableWriteDrain:
|
||||
(PENDING,),
|
||||
).fetchall()
|
||||
enabled_sources = {
|
||||
str(row[0]) for row in source_rows if self.source_writes_permitted(str(row[0]))
|
||||
str(row[0])
|
||||
for row in source_rows
|
||||
if self.source_writes_permitted(str(row[0]))
|
||||
}
|
||||
if not enabled_sources:
|
||||
return ()
|
||||
@@ -2246,142 +2019,38 @@ class DurableWriteDrain:
|
||||
placeholders = ",".join("?" for _ in enabled_sources)
|
||||
source_clause = f"AND candidate.source_key IN ({placeholders}) "
|
||||
source_params = tuple(sorted(enabled_sources))
|
||||
window = connection.execute(
|
||||
"SELECT window_anchor,window_end FROM memory_write_schedule_state WHERE singleton=1"
|
||||
).fetchone()
|
||||
if window is not None:
|
||||
anchor_text, end_text = str(window[0]), str(window[1])
|
||||
active_source_clause = ""
|
||||
active_source_params: tuple[Any, ...] = ()
|
||||
if enabled_sources is not None:
|
||||
placeholders = ",".join("?" for _ in enabled_sources)
|
||||
active_source_clause = f"AND active.source_key IN ({placeholders}) "
|
||||
active_source_params = tuple(sorted(enabled_sources))
|
||||
# The contract retires a window when every *eligible*
|
||||
# pending/running job in it has settled or parked. A pending row
|
||||
# that scheduling already holds back (an unexpired lease, a
|
||||
# busy Harness, or an earlier event in its Session) is not eligible, so it
|
||||
# must not keep the anchor alive: those blockers usually sit
|
||||
# outside this ten-minute range and can never be claimed from
|
||||
# inside it, which pinned the anchor and starved the whole
|
||||
# queue. Reuse the claim's own predicate so "the window is
|
||||
# busy" and "the window can still be worked" agree.
|
||||
active = connection.execute(
|
||||
"SELECT 1 FROM memory_write_jobs AS active "
|
||||
"WHERE active.event_at>=? AND active.event_at<? AND ("
|
||||
"active.state=? OR (active.state=? "
|
||||
"AND (active.lease_expires_at IS NULL "
|
||||
"OR active.lease_expires_at < ?) " + active_source_clause + "AND NOT EXISTS ("
|
||||
"SELECT 1 FROM memory_write_jobs AS held "
|
||||
"WHERE held.source_key=active.source_key AND held.state=? "
|
||||
"AND held.lease_owner IS NOT NULL "
|
||||
"AND held.lease_expires_at>=?) "
|
||||
"AND NOT EXISTS ("
|
||||
"SELECT 1 FROM memory_write_jobs AS earlier "
|
||||
"WHERE earlier.session_key=active.session_key "
|
||||
f"AND earlier.state IN ({blocking}) "
|
||||
"AND (earlier.event_at<active.event_at OR "
|
||||
"(earlier.event_at=active.event_at "
|
||||
"AND earlier.queue_seq<active.queue_seq))) "
|
||||
")) LIMIT 1",
|
||||
(
|
||||
anchor_text,
|
||||
end_text,
|
||||
RUNNING,
|
||||
PENDING,
|
||||
now.isoformat(),
|
||||
*active_source_params,
|
||||
RUNNING,
|
||||
now.isoformat(),
|
||||
*_FIFO_BLOCKING_STATES,
|
||||
),
|
||||
).fetchone()
|
||||
if active is None:
|
||||
connection.execute("DELETE FROM memory_write_schedule_state WHERE singleton=1")
|
||||
window = None
|
||||
if window is None:
|
||||
anchor = connection.execute(
|
||||
"SELECT candidate.event_at "
|
||||
"FROM memory_write_jobs AS candidate "
|
||||
"WHERE candidate.state=? "
|
||||
"AND (candidate.lease_expires_at IS NULL "
|
||||
"OR candidate.lease_expires_at < ?) " + source_clause + "AND NOT EXISTS ("
|
||||
"SELECT 1 FROM memory_write_jobs AS held "
|
||||
"WHERE held.source_key=candidate.source_key AND held.state=? "
|
||||
"AND held.lease_owner IS NOT NULL "
|
||||
"AND held.lease_expires_at>=?) "
|
||||
"AND NOT EXISTS ("
|
||||
"SELECT 1 FROM memory_write_jobs AS earlier "
|
||||
"WHERE earlier.session_key=candidate.session_key "
|
||||
f"AND earlier.state IN ({blocking}) "
|
||||
"AND (earlier.event_at<candidate.event_at OR "
|
||||
"(earlier.event_at=candidate.event_at "
|
||||
"AND earlier.queue_seq<candidate.queue_seq))) "
|
||||
"ORDER BY candidate.event_at,candidate.queue_seq LIMIT 1",
|
||||
(
|
||||
PENDING,
|
||||
now.isoformat(),
|
||||
*source_params,
|
||||
RUNNING,
|
||||
now.isoformat(),
|
||||
*_FIFO_BLOCKING_STATES,
|
||||
),
|
||||
).fetchone()
|
||||
if anchor is None:
|
||||
return ()
|
||||
anchor_at = datetime.fromisoformat(str(anchor[0]))
|
||||
if anchor_at.tzinfo is None:
|
||||
anchor_at = anchor_at.replace(tzinfo=UTC)
|
||||
anchor_text = anchor_at.astimezone(UTC).isoformat()
|
||||
end_text = (anchor_at + EVENT_TIME_WINDOW).astimezone(UTC).isoformat()
|
||||
connection.execute(
|
||||
"INSERT INTO memory_write_schedule_state "
|
||||
"(singleton,window_anchor,window_end,updated_at) "
|
||||
"VALUES (1,?,?,?)",
|
||||
(anchor_text, end_text, now.isoformat()),
|
||||
)
|
||||
rows = connection.execute(
|
||||
"SELECT candidate.job_id,candidate.source_key "
|
||||
"FROM memory_write_jobs AS candidate "
|
||||
"WHERE candidate.state=? "
|
||||
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
|
||||
+ source_clause
|
||||
+ "AND candidate.event_at>=? AND candidate.event_at<? "
|
||||
+ source_clause +
|
||||
"AND NOT EXISTS ("
|
||||
"SELECT 1 FROM memory_write_jobs AS held "
|
||||
"WHERE held.source_key=candidate.source_key AND held.state=? "
|
||||
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
|
||||
"AND NOT EXISTS ("
|
||||
"SELECT 1 FROM memory_write_jobs AS earlier "
|
||||
"WHERE earlier.session_key=candidate.session_key "
|
||||
"WHERE earlier.source_key=candidate.source_key "
|
||||
f"AND earlier.state IN ({blocking}) "
|
||||
"AND (earlier.event_at<candidate.event_at OR "
|
||||
"(earlier.event_at=candidate.event_at "
|
||||
"AND earlier.queue_seq<candidate.queue_seq))) "
|
||||
"ORDER BY candidate.event_at,candidate.queue_seq",
|
||||
"AND earlier.queue_seq<candidate.queue_seq) "
|
||||
"ORDER BY candidate.queue_seq",
|
||||
(
|
||||
PENDING,
|
||||
now.isoformat(),
|
||||
*source_params,
|
||||
anchor_text,
|
||||
end_text,
|
||||
RUNNING,
|
||||
now.isoformat(),
|
||||
*_FIFO_BLOCKING_STATES,
|
||||
),
|
||||
).fetchall()
|
||||
selected: list[str] = []
|
||||
selected_sources: set[str] = set()
|
||||
for row in rows:
|
||||
source_key = str(row["source_key"])
|
||||
if source_key in selected_sources:
|
||||
continue
|
||||
if self.source_claim_enabled is not None and not self.source_claim_enabled(
|
||||
source_key
|
||||
str(row["source_key"])
|
||||
):
|
||||
continue
|
||||
selected.append(str(row["job_id"]))
|
||||
selected_sources.add(source_key)
|
||||
if len(selected) == min(limit, self.max_concurrent_sources):
|
||||
break
|
||||
if not selected:
|
||||
@@ -2610,6 +2279,19 @@ class DurableWriteDrain:
|
||||
self.queue.reclaim_expired,
|
||||
operation_name="reclaim_expired",
|
||||
)
|
||||
# Terminal-payload reaping is a bounded retention write: a
|
||||
# finished row's full request body must not sit in the business
|
||||
# database forever just because the queue never went idle. Its
|
||||
# own interval keeps it off every poll.
|
||||
if (
|
||||
self._last_terminal_reap is None
|
||||
or tick_started - self._last_terminal_reap >= self._reap_interval
|
||||
):
|
||||
self._last_terminal_reap = tick_started
|
||||
await self._run_idle_maintenance(
|
||||
self.queue.clear_terminal_payloads,
|
||||
operation_name="clear_terminal_payloads",
|
||||
)
|
||||
# A model outage clears on its own, so `waiting_for_model` jobs
|
||||
# are re-admitted on a timer. That must not depend on idleness: a
|
||||
# queue busy for longer than the wait window would otherwise keep
|
||||
@@ -2626,8 +2308,7 @@ class DurableWriteDrain:
|
||||
)
|
||||
if self.reconcile_unknown is not None and (
|
||||
self._last_unknown_reconcile is None
|
||||
or tick_started - self._last_unknown_reconcile
|
||||
>= self._unknown_reconcile_interval
|
||||
or tick_started - self._last_unknown_reconcile >= self._reap_interval
|
||||
):
|
||||
self._last_unknown_reconcile = tick_started
|
||||
try:
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
The result decides whether the `unknown` job state can ever be retired, so it
|
||||
is probed against the installed SDK rather than written down from memory.
|
||||
|
||||
What was verified against `hy-memory==2.0.4`:
|
||||
What was verified against `hy-memory==2.0.2`:
|
||||
|
||||
- `HyMemoryClient.add(..., request_id=...)` accepts a caller-supplied id and
|
||||
echoes it back. It is described as a request/receipt handle, and the SDK
|
||||
|
||||
@@ -151,8 +151,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
|
||||
lease_expires_at TEXT,
|
||||
hy_invoked INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT,
|
||||
event_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
time_quality TEXT NOT NULL DEFAULT 'approximate',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
@@ -166,8 +164,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
|
||||
ON memory_write_jobs(source_key, state, queue_seq);
|
||||
CREATE INDEX IF NOT EXISTS idx_write_jobs_fifo
|
||||
ON memory_write_jobs(queue_seq);
|
||||
CREATE INDEX IF NOT EXISTS idx_write_jobs_event_claim
|
||||
ON memory_write_jobs(state, event_at, queue_seq);
|
||||
CREATE INDEX IF NOT EXISTS idx_write_jobs_source_recent
|
||||
ON memory_write_jobs(source_key, updated_at DESC);
|
||||
CREATE TABLE IF NOT EXISTS memory_write_job_evidence (
|
||||
@@ -176,13 +172,7 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
|
||||
PRIMARY KEY (job_id, evidence_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_write_job_evidence_evidence
|
||||
ON memory_write_job_evidence(evidence_id);
|
||||
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
|
||||
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
|
||||
window_anchor TEXT NOT NULL,
|
||||
window_end TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)
|
||||
ON memory_write_job_evidence(evidence_id)
|
||||
"""
|
||||
return """
|
||||
CREATE TABLE IF NOT EXISTS memory_write_jobs (
|
||||
@@ -202,8 +192,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
|
||||
lease_expires_at DATETIME(6) NULL,
|
||||
hy_invoked TINYINT(1) NOT NULL DEFAULT 0,
|
||||
last_error TEXT NULL,
|
||||
event_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
|
||||
time_quality VARCHAR(32) NOT NULL DEFAULT 'approximate',
|
||||
created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
|
||||
updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6)
|
||||
ON UPDATE CURRENT_TIMESTAMP(6),
|
||||
@@ -213,7 +201,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
|
||||
KEY idx_write_jobs_source_claim (state, source_key, priority, created_at),
|
||||
KEY idx_write_jobs_source_fifo (source_key, state, queue_seq),
|
||||
KEY idx_write_jobs_fifo (queue_seq),
|
||||
KEY idx_write_jobs_event_claim (state, event_at, queue_seq),
|
||||
KEY idx_write_jobs_source_recent (source_key, updated_at)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
|
||||
@@ -222,13 +209,6 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
|
||||
evidence_id VARCHAR(128) NOT NULL,
|
||||
PRIMARY KEY (job_id, evidence_id),
|
||||
KEY idx_write_job_evidence_evidence (evidence_id)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
|
||||
singleton INT PRIMARY KEY CHECK (singleton = 1),
|
||||
window_anchor DATETIME(6) NOT NULL,
|
||||
window_end DATETIME(6) NOT NULL,
|
||||
updated_at DATETIME(6) NOT NULL
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
|
||||
"""
|
||||
|
||||
@@ -1251,68 +1231,6 @@ CREATE TABLE IF NOT EXISTS model_request_budget_events (
|
||||
"""
|
||||
|
||||
|
||||
def model_request_audit_ddl(dialect: str = "mysql") -> str:
|
||||
"""Content-free accounting for physical precipitation model requests."""
|
||||
if dialect == "sqlite":
|
||||
return """
|
||||
CREATE TABLE IF NOT EXISTS model_request_audit (
|
||||
audit_id TEXT PRIMARY KEY,
|
||||
request_id TEXT NOT NULL,
|
||||
profile_id TEXT NOT NULL,
|
||||
profile_revision INTEGER NOT NULL,
|
||||
provider_kind TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
request_kind TEXT NOT NULL,
|
||||
prompt_stage TEXT NOT NULL DEFAULT 'other',
|
||||
started_at TEXT NOT NULL,
|
||||
completed_at TEXT,
|
||||
outcome TEXT NOT NULL CHECK(outcome IN ('in_flight','succeeded','failed')),
|
||||
http_status INTEGER,
|
||||
error_type TEXT,
|
||||
provider_request_id TEXT,
|
||||
prompt_tokens INTEGER,
|
||||
completion_tokens INTEGER,
|
||||
total_tokens INTEGER,
|
||||
cached_prompt_tokens INTEGER,
|
||||
reasoning_tokens INTEGER
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS model_request_audit_request
|
||||
ON model_request_audit(request_id, started_at);
|
||||
CREATE INDEX IF NOT EXISTS model_request_audit_started
|
||||
ON model_request_audit(started_at);
|
||||
"""
|
||||
return """
|
||||
CREATE TABLE IF NOT EXISTS model_request_audit (
|
||||
audit_id VARCHAR(128) PRIMARY KEY,
|
||||
request_id VARCHAR(512) NOT NULL,
|
||||
profile_id VARCHAR(128) NOT NULL,
|
||||
profile_revision INT NOT NULL,
|
||||
provider_kind VARCHAR(64) NOT NULL,
|
||||
model VARCHAR(512) NOT NULL,
|
||||
request_kind VARCHAR(64) NOT NULL,
|
||||
prompt_stage VARCHAR(128) NOT NULL DEFAULT 'other',
|
||||
started_at DATETIME(6) NOT NULL,
|
||||
completed_at DATETIME(6) NULL,
|
||||
outcome VARCHAR(32) NOT NULL CHECK(outcome IN ('in_flight','succeeded','failed')),
|
||||
http_status INT NULL,
|
||||
error_type VARCHAR(128) NULL,
|
||||
provider_request_id VARCHAR(512) NULL,
|
||||
prompt_tokens BIGINT NULL,
|
||||
completion_tokens BIGINT NULL,
|
||||
total_tokens BIGINT NULL,
|
||||
cached_prompt_tokens BIGINT NULL,
|
||||
reasoning_tokens BIGINT NULL,
|
||||
KEY model_request_audit_request (request_id, started_at),
|
||||
KEY model_request_audit_started (started_at)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
|
||||
"""
|
||||
|
||||
|
||||
async def _052_model_request_audit(cursor: Any) -> None:
|
||||
for statement in _split_statements(model_request_audit_ddl(_dialect(cursor))):
|
||||
await cursor.execute(statement)
|
||||
|
||||
|
||||
def model_probe_operations_ddl(dialect: str = "mysql") -> str:
|
||||
"""Probe operations; `state` is CHECKed to a known set.
|
||||
|
||||
@@ -1486,105 +1404,7 @@ async def _024_matter_model_cache(cursor: Any) -> None:
|
||||
|
||||
|
||||
async def _026_hy_node_permissions(cursor: Any) -> None:
|
||||
# Historical migration retained as a ledger slot. Fresh databases no
|
||||
# longer create this retired proof table; migration 51 removes it from
|
||||
# databases that ran the original v26 body.
|
||||
return None
|
||||
|
||||
|
||||
async def _051_drop_hy_node_permissions(cursor: Any) -> None:
|
||||
await cursor.execute("DROP TABLE IF EXISTS hy_node_permissions")
|
||||
|
||||
|
||||
async def _053_memory_write_event_time(cursor: Any) -> None:
|
||||
"""Add event-time scheduling without re-keying any existing write."""
|
||||
|
||||
dialect = _dialect(cursor)
|
||||
columns = await cursor.columns("memory_write_jobs")
|
||||
added_event_at = "event_at" not in columns
|
||||
if added_event_at:
|
||||
await cursor.execute(
|
||||
"ALTER TABLE memory_write_jobs ADD COLUMN event_at "
|
||||
+ ("TEXT" if dialect == "sqlite" else "DATETIME(6) NULL")
|
||||
)
|
||||
if "time_quality" not in columns:
|
||||
await cursor.execute(
|
||||
"ALTER TABLE memory_write_jobs ADD COLUMN time_quality "
|
||||
+ (
|
||||
"TEXT NOT NULL DEFAULT 'approximate'"
|
||||
if dialect == "sqlite"
|
||||
else "VARCHAR(32) NOT NULL DEFAULT 'approximate'"
|
||||
)
|
||||
)
|
||||
await cursor.execute(
|
||||
"""
|
||||
UPDATE memory_write_jobs
|
||||
SET event_at = COALESCE(
|
||||
(
|
||||
SELECT MAX(COALESCE(e.created_at, e.imported_at))
|
||||
FROM memory_write_job_evidence AS linked
|
||||
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
|
||||
WHERE linked.job_id = memory_write_jobs.job_id
|
||||
AND e.role = 'user'
|
||||
),
|
||||
(
|
||||
SELECT MIN(COALESCE(e.created_at, e.imported_at))
|
||||
FROM memory_write_job_evidence AS linked
|
||||
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
|
||||
WHERE linked.job_id = memory_write_jobs.job_id
|
||||
),
|
||||
created_at
|
||||
)
|
||||
WHERE event_at IS NULL
|
||||
"""
|
||||
)
|
||||
if dialect == "sqlite":
|
||||
await cursor.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_write_jobs_event_claim "
|
||||
"ON memory_write_jobs(state, event_at, queue_seq)"
|
||||
)
|
||||
await cursor.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
|
||||
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
|
||||
window_anchor TEXT NOT NULL,
|
||||
window_end TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)
|
||||
"""
|
||||
)
|
||||
else:
|
||||
if added_event_at:
|
||||
await cursor.execute(
|
||||
"CREATE INDEX idx_write_jobs_event_claim "
|
||||
"ON memory_write_jobs(state, event_at, queue_seq)"
|
||||
)
|
||||
await cursor.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
|
||||
singleton INT PRIMARY KEY CHECK (singleton = 1),
|
||||
window_anchor DATETIME(6) NOT NULL,
|
||||
window_end DATETIME(6) NOT NULL,
|
||||
updated_at DATETIME(6) NOT NULL
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
async def _054_legacy_possibly_sent_tasks(cursor: Any) -> None:
|
||||
"""Preserve the old sent marker before removing its retired permission table."""
|
||||
columns = await cursor.columns("memory_task_continuous_permissions")
|
||||
if "task_id" not in columns:
|
||||
return
|
||||
await cursor.execute(
|
||||
"""
|
||||
UPDATE memory_tasks
|
||||
SET hy_invocation_count=CASE
|
||||
WHEN hy_invocation_count<1 THEN 1 ELSE hy_invocation_count END
|
||||
WHERE task_id IN (SELECT task_id FROM memory_task_continuous_permissions)
|
||||
"""
|
||||
)
|
||||
await cursor.execute("DROP TABLE memory_task_continuous_permissions")
|
||||
await cursor.execute(hy_node_permissions_ddl(_dialect(cursor)))
|
||||
|
||||
|
||||
def model_profiles_ddl(dialect: str = "mysql") -> str:
|
||||
@@ -2853,11 +2673,7 @@ MIGRATIONS: tuple[Migration, ...] = (
|
||||
2,
|
||||
"memory_write_jobs",
|
||||
_002_write_jobs,
|
||||
(
|
||||
"memory_write_jobs",
|
||||
"memory_write_job_evidence",
|
||||
"memory_write_schedule_state",
|
||||
),
|
||||
("memory_write_jobs", "memory_write_job_evidence"),
|
||||
),
|
||||
Migration(3, "source_file_cursors", _003_scanner_cursors, ("source_file_cursors",)),
|
||||
Migration(4, "change_outbox", _004_change_outbox, ("change_outbox",)),
|
||||
@@ -2940,7 +2756,7 @@ MIGRATIONS: tuple[Migration, ...] = (
|
||||
),
|
||||
Migration(23, "source_candidates", _023_source_candidates, ("source_candidates",)),
|
||||
Migration(24, "matter_model_cache", _024_matter_model_cache, ("matter_model_cache",)),
|
||||
Migration(26, "hy_node_permissions", _026_hy_node_permissions),
|
||||
Migration(26, "hy_node_permissions", _026_hy_node_permissions, ("hy_node_permissions",)),
|
||||
Migration(
|
||||
27,
|
||||
"model_profiles",
|
||||
@@ -3025,22 +2841,4 @@ MIGRATIONS: tuple[Migration, ...] = (
|
||||
"model_probe_operations_reserved_at",
|
||||
_050_model_probe_operations_reserved_at,
|
||||
),
|
||||
Migration(51, "drop_hy_node_permissions", _051_drop_hy_node_permissions),
|
||||
Migration(
|
||||
52,
|
||||
"model_request_audit",
|
||||
_052_model_request_audit,
|
||||
("model_request_audit",),
|
||||
),
|
||||
Migration(
|
||||
53,
|
||||
"memory_write_event_time",
|
||||
_053_memory_write_event_time,
|
||||
("memory_write_schedule_state",),
|
||||
),
|
||||
Migration(
|
||||
54,
|
||||
"legacy_possibly_sent_tasks",
|
||||
_054_legacy_possibly_sent_tasks,
|
||||
),
|
||||
)
|
||||
@@ -107,8 +107,6 @@ def build_request(
|
||||
first_round: int,
|
||||
last_round: int,
|
||||
evidence_ids: list[str],
|
||||
input_records: tuple[EvidenceRecord, ...] = (),
|
||||
source_id: str | None = None,
|
||||
priority: Literal["live", "normal"] = "normal",
|
||||
idempotency_key: str | None = None,
|
||||
) -> WriteRequest:
|
||||
@@ -127,8 +125,6 @@ def build_request(
|
||||
internal_session_id=internal_session_id,
|
||||
batch_identity=batch,
|
||||
evidence_ids=evidence_ids,
|
||||
input_records=input_records,
|
||||
source_id=source_id,
|
||||
idempotency_key=idempotency_key or batch.batch_id,
|
||||
priority=priority,
|
||||
)
|
||||
@@ -275,8 +271,6 @@ def build_request_from_evidence(
|
||||
first_round=first_round,
|
||||
last_round=last_round,
|
||||
evidence_ids=list(ordered),
|
||||
input_records=tuple(ordered.values()),
|
||||
source_id=seed.source_id,
|
||||
priority=priority,
|
||||
idempotency_key=idempotency_key,
|
||||
)
|
||||
|
||||
@@ -18,11 +18,10 @@ from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from hashlib import sha256
|
||||
from typing import Literal
|
||||
|
||||
from one_memory.imports.models import EvidenceOrigin, EvidenceRecord
|
||||
from one_memory.imports.models import EvidenceRecord
|
||||
from one_memory.memory_server.contracts import (
|
||||
BatchIdentity,
|
||||
WriteRequest,
|
||||
@@ -125,12 +124,6 @@ class UnifiedWriteIntake:
|
||||
priority: WritePriority,
|
||||
) -> WriteRequest:
|
||||
session_id = internal_session_id_for(record.conversation_id, record.source_id)
|
||||
event_at, time_quality = _job_time((record,))
|
||||
request_key = (
|
||||
f"entry:{record.evidence_id}"
|
||||
if record.revision == 1
|
||||
else (f"entry:v2:{record.evidence_id}:{record.revision}:{record.content_hash}")
|
||||
)
|
||||
return WriteRequest(
|
||||
internal_session_id=session_id,
|
||||
source_id=record.source_id,
|
||||
@@ -140,11 +133,8 @@ class UnifiedWriteIntake:
|
||||
last_round=last_round,
|
||||
),
|
||||
evidence_ids=[record.evidence_id],
|
||||
input_records=(record,),
|
||||
idempotency_key=request_key,
|
||||
idempotency_key=f"entry:{record.evidence_id}",
|
||||
priority=priority,
|
||||
event_at=event_at,
|
||||
time_quality=time_quality,
|
||||
)
|
||||
|
||||
def submit_records(
|
||||
@@ -171,7 +161,6 @@ class UnifiedWriteIntake:
|
||||
)
|
||||
request_id = f"turn:v1:{sha256(material.encode()).hexdigest()}"
|
||||
session_id = internal_session_id_for(first.conversation_id, first.source_id)
|
||||
event_at, time_quality = _job_time(records)
|
||||
durable = self.queue.submit(
|
||||
WriteRequest(
|
||||
internal_session_id=session_id,
|
||||
@@ -182,11 +171,8 @@ class UnifiedWriteIntake:
|
||||
last_round=1,
|
||||
),
|
||||
evidence_ids=[record.evidence_id for record in records],
|
||||
input_records=records,
|
||||
idempotency_key=request_id,
|
||||
priority=priority,
|
||||
event_at=event_at,
|
||||
time_quality=time_quality,
|
||||
)
|
||||
)
|
||||
return UnifiedWriteReceipt(
|
||||
@@ -203,29 +189,3 @@ class SourceWriteDisabledError(RuntimeError):
|
||||
def __init__(self, evidence_id: str) -> None:
|
||||
super().__init__("model_content_source_disabled")
|
||||
self.evidence_id = evidence_id
|
||||
|
||||
|
||||
def _job_time(
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
) -> tuple[datetime, Literal["exact", "anchored", "session", "approximate"]]:
|
||||
"""Use the turn opener's wall clock, with an explicit fallback quality."""
|
||||
|
||||
opener = next(
|
||||
(record for record in records if record.role == "user"),
|
||||
records[0],
|
||||
)
|
||||
if opener.created_at is None:
|
||||
return opener.imported_at, "approximate"
|
||||
file_origin = opener.origin_meta.get(EvidenceOrigin.FILE)
|
||||
basis = file_origin.raw_meta.get("time_basis") if file_origin is not None else None
|
||||
turn_event_at = file_origin.raw_meta.get("turn_event_at") if file_origin is not None else None
|
||||
if isinstance(turn_event_at, str):
|
||||
try:
|
||||
return datetime.fromisoformat(turn_event_at), "exact"
|
||||
except ValueError:
|
||||
pass
|
||||
if basis == "file_mtime":
|
||||
return opener.created_at, "session"
|
||||
if EvidenceOrigin.HOOK in opener.origins:
|
||||
return opener.created_at, "exact"
|
||||
return opener.created_at, "exact"
|
||||
@@ -17,7 +17,7 @@ from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.database import take_the_write_lock_first
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.imports.models import EvidenceRecord, WrittenBy
|
||||
from one_memory.imports.models import EvidenceRecord
|
||||
from one_memory.memory_server.migrations import memory_tasks_ddl
|
||||
from one_memory.model_profiles import ModelProfile, ModelProfileSnapshot
|
||||
|
||||
@@ -174,60 +174,29 @@ def evidence_ref(record: EvidenceRecord) -> TaskEvidenceRef:
|
||||
"segment_end": int(marker["end"]),
|
||||
"segment_hash": str(marker["hash"]),
|
||||
}
|
||||
identity = record.model_dump(mode="json", exclude={"origin_meta", "imported_at", "updated_at"})
|
||||
return {
|
||||
"evidence_id": record.evidence_id,
|
||||
"revision": record.revision,
|
||||
"content_hash": record.content_hash,
|
||||
"identity_hash": _evidence_identity_hash(record, include_written_by=False),
|
||||
"identity_hash": sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest(),
|
||||
}
|
||||
|
||||
|
||||
def _evidence_identity_hash(record: EvidenceRecord, *, include_written_by: bool) -> str:
|
||||
excluded = {"origin_meta", "imported_at", "updated_at"}
|
||||
if not include_written_by:
|
||||
excluded.add("written_by")
|
||||
identity = record.model_dump(mode="json", exclude=excluded)
|
||||
return sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest()
|
||||
|
||||
|
||||
def _identity_hash_matches(record: EvidenceRecord, expected: object) -> bool:
|
||||
candidates = {_evidence_identity_hash(record, include_written_by=False)}
|
||||
# Tasks created by pre-role-only builds keep their original proof. The
|
||||
# legacy metadata could also have changed in place without a revision, so
|
||||
# accept the old hash under every historical author guess.
|
||||
for written_by in WrittenBy:
|
||||
candidates.add(
|
||||
_evidence_identity_hash(
|
||||
record.model_copy(update={"written_by": written_by}),
|
||||
include_written_by=True,
|
||||
)
|
||||
)
|
||||
return expected in candidates
|
||||
|
||||
|
||||
def matches_evidence_ref(record: EvidenceRecord, ref: TaskEvidenceRef) -> bool:
|
||||
if "segment_hash" in ref:
|
||||
base = evidence_ref(record)
|
||||
if any(
|
||||
base.get(key) != ref.get(key)
|
||||
for key in ("evidence_id", "revision", "content_hash")
|
||||
for key in ("evidence_id", "revision", "content_hash", "identity_hash")
|
||||
):
|
||||
return False
|
||||
if not _identity_hash_matches(record, ref.get("identity_hash")):
|
||||
return False
|
||||
start, end = ref.get("segment_start"), ref.get("segment_end")
|
||||
if not isinstance(start, int) or not isinstance(end, int) or not (0 <= start < end):
|
||||
return False
|
||||
return sha256(record.verbatim[start:end].encode()).hexdigest() == ref["segment_hash"]
|
||||
current = evidence_ref(record)
|
||||
return all(
|
||||
(
|
||||
_identity_hash_matches(record, value)
|
||||
if key == "identity_hash"
|
||||
else current.get(key) == value
|
||||
)
|
||||
for key, value in ref.items()
|
||||
)
|
||||
return all(current.get(key) == value for key, value in ref.items())
|
||||
|
||||
|
||||
def segment_evidence_record(
|
||||
@@ -298,13 +267,15 @@ class MemoryTaskOutput(BaseModel):
|
||||
MEMORY_TASK_SCHEMA = memory_tasks_ddl("sqlite")
|
||||
|
||||
|
||||
#: Legacy transport refusals raised before the physical send, so provably
|
||||
#: nothing reached HY: nothing can have been written and retrying cannot
|
||||
#: duplicate. Kept to recover rows written by builds through 1.0.2.
|
||||
#: Transport refusals raised before the physical send, so provably nothing
|
||||
#: reached HY: nothing can have been written and retrying cannot duplicate.
|
||||
#:
|
||||
#: The retired request hook denied every name below before opening a socket.
|
||||
#: This stays explicit: other old transport codes may have happened after HY
|
||||
#: persisted a node and must not be blindly requeued.
|
||||
#: Every name here is denied by `HyPermissionTransport._check_request` (the
|
||||
#: httpx request hook) or by `_PhysicalTransport.handle_async_request` before
|
||||
#: it counts the request — both run inside httpx, before the socket. Explicit
|
||||
#: on purpose (D88): the transport also denies `budget_exceeded`, `redirect`,
|
||||
#: `lease_*` and `request_failed`, all of which can arrive after HY persisted
|
||||
#: a node, so a prefix rule would requeue work that may already exist.
|
||||
_PRE_SEND_REFUSALS = (
|
||||
"model_content_transport_headers_mismatch",
|
||||
"model_content_transport_body_mismatch",
|
||||
@@ -1008,44 +979,6 @@ class MemoryTaskStore:
|
||||
error_code=error_code,
|
||||
)
|
||||
|
||||
def release_unstarted(
|
||||
self,
|
||||
task_id: str,
|
||||
*,
|
||||
runner_token: str,
|
||||
error_code: str,
|
||||
) -> MemoryTask:
|
||||
"""Return a claimed task to pending when a user pause wins before HY.
|
||||
|
||||
Claim increments ``attempts``. A pause is not an execution failure, so
|
||||
the attempt is returned as well; repeatedly pausing and resuming can
|
||||
never exhaust an otherwise healthy task.
|
||||
"""
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with self._connect(writing=True) as connection:
|
||||
changed = connection.execute(
|
||||
"""
|
||||
UPDATE memory_tasks
|
||||
SET status=?, runner_token=NULL,
|
||||
attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,
|
||||
last_error_code=?, duration_seconds=NULL, updated_at=?
|
||||
WHERE task_id=? AND status=? AND runner_token=?
|
||||
""",
|
||||
(
|
||||
MemoryTaskStatus.PENDING,
|
||||
error_code,
|
||||
now,
|
||||
task_id,
|
||||
MemoryTaskStatus.RUNNING,
|
||||
runner_token,
|
||||
),
|
||||
)
|
||||
if changed.rowcount != 1:
|
||||
raise MemoryTaskOwnershipError(
|
||||
"Memory task runner no longer owns this task"
|
||||
)
|
||||
return self.require(task_id)
|
||||
|
||||
def park_for_permission(self, task_id: str, *, error_code: str) -> bool:
|
||||
"""Park unstarted work until a later permission snapshot admits it."""
|
||||
now = datetime.now(UTC).isoformat()
|
||||
@@ -2230,43 +2163,6 @@ class MemoryTaskStore:
|
||||
)
|
||||
return changed.rowcount
|
||||
|
||||
def retry_first_unknown_after_reconcile(
|
||||
self,
|
||||
*,
|
||||
older_than: datetime,
|
||||
limit: int = 128,
|
||||
) -> int:
|
||||
"""Automatically retry a task-only unknown exactly once.
|
||||
|
||||
Reconciliation runs before this transition. Attempts are preserved, so
|
||||
the retry claim moves `attempts=1` to two and another unknown is left
|
||||
for explicit user recovery.
|
||||
"""
|
||||
if type(limit) is not int or limit < 1:
|
||||
raise ValueError("limit must be positive")
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with self._connect(writing=True) as connection:
|
||||
changed = connection.execute(
|
||||
"""
|
||||
UPDATE memory_tasks
|
||||
SET status=?, runner_token=NULL,
|
||||
last_error_code='model_content_unknown_auto_retry', updated_at=?
|
||||
WHERE task_id IN (
|
||||
SELECT task_id FROM memory_tasks
|
||||
WHERE status=? AND attempts=1 AND updated_at <= ?
|
||||
ORDER BY updated_at, task_id LIMIT ?
|
||||
)
|
||||
""",
|
||||
(
|
||||
MemoryTaskStatus.PENDING,
|
||||
now,
|
||||
MemoryTaskStatus.UNKNOWN,
|
||||
older_than.isoformat(),
|
||||
limit,
|
||||
),
|
||||
)
|
||||
return changed.rowcount
|
||||
|
||||
def recover_running(self) -> int:
|
||||
"""Return tasks orphaned by a previous process to the pending queue."""
|
||||
return self.recover_orphan_running()
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
"""One global final-state permission for remote model content egress."""
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Sequence
|
||||
from typing import Annotated, cast
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.params import Depends
|
||||
from pydantic import BaseModel, ConfigDict, StrictBool
|
||||
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
|
||||
|
||||
class ModelContentPermissionState(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
allowed: bool
|
||||
|
||||
|
||||
class SetModelContentPermission(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
allowed: StrictBool
|
||||
|
||||
|
||||
def model_content_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
router = APIRouter(prefix="/api/v1/model-content-permission", dependencies=dependencies)
|
||||
|
||||
def config(request: Request) -> ConfigStore:
|
||||
return cast(ConfigStore, request.app.state.config_store)
|
||||
|
||||
Config = Annotated[ConfigStore, Depends(config)]
|
||||
|
||||
@router.get("", response_model=ModelContentPermissionState)
|
||||
async def current(store: Config) -> ModelContentPermissionState:
|
||||
return ModelContentPermissionState(allowed=store.model_content_allowed())
|
||||
|
||||
@router.post("", response_model=ModelContentPermissionState)
|
||||
async def set_permission(
|
||||
request: Request,
|
||||
value: SetModelContentPermission,
|
||||
store: Config,
|
||||
) -> ModelContentPermissionState:
|
||||
await asyncio.to_thread(store.set_model_content_allowed, value.allowed)
|
||||
if value.allowed:
|
||||
request.app.state.start_memory_backfill()
|
||||
return ModelContentPermissionState(allowed=value.allowed)
|
||||
|
||||
return router
|
||||
@@ -0,0 +1,180 @@
|
||||
"""Live global permission checks for remote model-content egress.
|
||||
|
||||
``config.json`` is the only durable permission state. Tasks do not carry a
|
||||
permission receipt, revision, source scope, or model pin. Every remote send
|
||||
and every publication boundary comes back through the lease and reads the
|
||||
current final-state boolean.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from one_memory.hy_source_context import HySourceContext
|
||||
from one_memory.imports.models import EvidenceRecord
|
||||
from one_memory.memory_governance import MemoryGovernanceStore, MemoryStatus
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
from one_memory.memory_tasks import MemoryTask, MemoryTaskStatus
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ModelProfileStore
|
||||
|
||||
|
||||
def permission_error(code: str) -> RuntimeError:
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
|
||||
return HyMemoryBridgeError(
|
||||
code if code.startswith("model_content_") else f"model_content_{code}"
|
||||
)
|
||||
|
||||
|
||||
def error_is_policy_stop(code: str) -> bool:
|
||||
"""A disabled final-state permission is an intentional stop."""
|
||||
|
||||
return code in {
|
||||
"model_content_source_disabled",
|
||||
"hy_memory_evidence_changed",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelProcessingPermissionLease:
|
||||
task_id: str
|
||||
user_id: str
|
||||
profile: ModelProfileSnapshot
|
||||
records: tuple[EvidenceRecord, ...]
|
||||
source_context: HySourceContext | None
|
||||
_guard: ModelProcessingPermissionGuard = field(repr=False, compare=False)
|
||||
|
||||
def recheck(self) -> None:
|
||||
"""Read current config/profile immediately before an egress boundary."""
|
||||
|
||||
self.recheck_permission()
|
||||
self._guard.check_context(self.records, self.source_context)
|
||||
|
||||
def recheck_permission(self) -> None:
|
||||
self._guard.check(self.profile, self.records)
|
||||
|
||||
|
||||
class ModelProcessingPermissionGuard:
|
||||
def __init__(
|
||||
self,
|
||||
config: ConfigStore,
|
||||
governance: MemoryGovernanceStore,
|
||||
profiles: ModelProfileStore,
|
||||
) -> None:
|
||||
self.config = config
|
||||
self.governance = governance
|
||||
self.profiles = profiles
|
||||
|
||||
def require_allowed(self) -> None:
|
||||
if not self.config.model_content_allowed():
|
||||
raise permission_error("permission_required")
|
||||
|
||||
def check(
|
||||
self,
|
||||
profile: ModelProfileSnapshot,
|
||||
records: tuple[EvidenceRecord, ...] = (),
|
||||
) -> None:
|
||||
"""Validate only live global consent and the currently active model."""
|
||||
|
||||
self.require_allowed()
|
||||
if self.profiles.active_identity() != (profile.profile_id, profile.revision):
|
||||
raise permission_error("profile_changed")
|
||||
if records and any(record.user_id != records[0].user_id for record in records):
|
||||
raise permission_error("context_scope_invalid")
|
||||
|
||||
def lease(
|
||||
self,
|
||||
task: MemoryTask,
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
context: HySourceContext | None,
|
||||
) -> ModelProcessingPermissionLease:
|
||||
combined = self.context_records(records, context)
|
||||
lease = ModelProcessingPermissionLease(
|
||||
task_id=task.task_id,
|
||||
user_id=task.user_id,
|
||||
profile=task.profile_snapshot,
|
||||
records=combined,
|
||||
source_context=context,
|
||||
_guard=self,
|
||||
)
|
||||
lease.recheck()
|
||||
return lease
|
||||
|
||||
@staticmethod
|
||||
def context_records(
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
context: HySourceContext | None,
|
||||
) -> tuple[EvidenceRecord, ...]:
|
||||
by_id = {record.evidence_id: record for record in records}
|
||||
if len(by_id) != len(records):
|
||||
raise permission_error("context_scope_invalid")
|
||||
if context is not None:
|
||||
for record in (*context.current, *context.history, *context.supporting):
|
||||
previous = by_id.get(record.evidence_id)
|
||||
if previous is not None and previous != record:
|
||||
raise permission_error("evidence_changed")
|
||||
by_id[record.evidence_id] = record
|
||||
return tuple(by_id.values())
|
||||
|
||||
def check_context(
|
||||
self,
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
context: HySourceContext | None,
|
||||
) -> None:
|
||||
if context is None:
|
||||
return
|
||||
ids = {record.evidence_id for record in records}
|
||||
if not context.current:
|
||||
raise permission_error("context_scope_invalid")
|
||||
context_records = (*context.current, *context.history, *context.supporting)
|
||||
if len({record.evidence_id for record in context_records}) != len(context_records):
|
||||
raise permission_error("context_scope_invalid")
|
||||
for known in context.known_memories:
|
||||
if not known.evidence_ids or not set(known.evidence_ids) <= ids:
|
||||
raise permission_error("context_support_missing")
|
||||
memory = self.governance.get(
|
||||
known.governed_memory_id,
|
||||
user_id=context.current[0].user_id,
|
||||
)
|
||||
if (
|
||||
memory is None
|
||||
or memory.revision != known.revision
|
||||
or memory.content != known.content
|
||||
or set(memory.evidence_ids) != set(known.evidence_ids)
|
||||
or memory.status is not MemoryStatus.CONFIRMED
|
||||
):
|
||||
raise permission_error("context_changed")
|
||||
|
||||
def reject_unstarted(self, task_id: str, code: str) -> None:
|
||||
stopped = error_is_policy_stop(code)
|
||||
with self._task_write() as tasks:
|
||||
tasks.execute(
|
||||
"UPDATE memory_tasks SET status=?, last_error_code=?, updated_at=? "
|
||||
"WHERE task_id=? AND status IN (?,?,?,?)",
|
||||
(
|
||||
MemoryTaskStatus.CANCELLED if stopped else MemoryTaskStatus.FAILED,
|
||||
str(permission_error(code)),
|
||||
datetime.now(UTC).isoformat(),
|
||||
task_id,
|
||||
MemoryTaskStatus.PENDING,
|
||||
MemoryTaskStatus.WAITING_FOR_PERMISSION,
|
||||
MemoryTaskStatus.WAITING_FOR_MODEL,
|
||||
MemoryTaskStatus.DEFERRED_OUTSIDE_WINDOW,
|
||||
),
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def _task_write(self) -> Iterator[object]:
|
||||
from one_memory.database import take_the_write_lock_first
|
||||
from one_memory.database_access import main_database
|
||||
|
||||
with main_database(self.governance.database_path).write(
|
||||
autocommit=True,
|
||||
foreign_keys=True,
|
||||
) as connection:
|
||||
take_the_write_lock_first(connection)
|
||||
with connection:
|
||||
yield connection
|
||||
@@ -1,404 +0,0 @@
|
||||
"""Persistent, content-free accounting for precipitation model requests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
from typing import TypedDict
|
||||
from uuid import uuid4
|
||||
|
||||
import httpx
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.memory_server.migrations import model_request_audit_ddl
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelRequestAuditContext:
|
||||
request_id: str
|
||||
|
||||
|
||||
_audit_context: ContextVar[ModelRequestAuditContext | None] = ContextVar(
|
||||
"one_memory_model_request_audit_context",
|
||||
default=None,
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def audit_precipitation_request(request_id: str) -> Iterator[None]:
|
||||
"""Enable physical-request accounting for one memory precipitation."""
|
||||
|
||||
token = _audit_context.set(ModelRequestAuditContext(request_id=request_id))
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_audit_context.reset(token)
|
||||
|
||||
|
||||
def _current_prompt_stage() -> str:
|
||||
try:
|
||||
resolver = getattr( # noqa: B009 - optional SDK version boundary
|
||||
import_module("hy_memory.utils.model_call_context"),
|
||||
"current_model_call_stage",
|
||||
)
|
||||
except (ImportError, AttributeError):
|
||||
return "other"
|
||||
return str(resolver())
|
||||
|
||||
|
||||
class ModelRequestAuditStore:
|
||||
def __init__(self, database: Path) -> None:
|
||||
self.database = database
|
||||
with main_database(database).write() as connection, connection:
|
||||
for statement in model_request_audit_ddl("sqlite").split(";"):
|
||||
if statement.strip():
|
||||
connection.execute(statement)
|
||||
|
||||
def start(
|
||||
self,
|
||||
*,
|
||||
request_id: str,
|
||||
profile_id: str,
|
||||
profile_revision: int,
|
||||
provider_kind: str,
|
||||
model: str,
|
||||
request_kind: str,
|
||||
prompt_stage: str,
|
||||
) -> str:
|
||||
audit_id = str(uuid4())
|
||||
with main_database(self.database).write() as connection, connection:
|
||||
connection.execute(
|
||||
"""
|
||||
INSERT INTO model_request_audit (
|
||||
audit_id,request_id,profile_id,profile_revision,provider_kind,
|
||||
model,request_kind,prompt_stage,started_at,outcome
|
||||
) VALUES (?,?,?,?,?,?,?,?,?,?)
|
||||
""",
|
||||
(
|
||||
audit_id,
|
||||
request_id,
|
||||
profile_id,
|
||||
profile_revision,
|
||||
provider_kind,
|
||||
model,
|
||||
request_kind,
|
||||
prompt_stage,
|
||||
datetime.now(UTC).isoformat(),
|
||||
"in_flight",
|
||||
),
|
||||
)
|
||||
return audit_id
|
||||
|
||||
def finish(
|
||||
self,
|
||||
audit_id: str,
|
||||
*,
|
||||
outcome: str,
|
||||
http_status: int | None,
|
||||
error_type: str | None = None,
|
||||
provider_request_id: str | None = None,
|
||||
prompt_tokens: int | None = None,
|
||||
completion_tokens: int | None = None,
|
||||
total_tokens: int | None = None,
|
||||
cached_prompt_tokens: int | None = None,
|
||||
reasoning_tokens: int | None = None,
|
||||
) -> None:
|
||||
with main_database(self.database).write() as connection, connection:
|
||||
connection.execute(
|
||||
"""
|
||||
UPDATE model_request_audit
|
||||
SET completed_at=?,outcome=?,http_status=?,error_type=?,
|
||||
provider_request_id=?,prompt_tokens=?,completion_tokens=?,
|
||||
total_tokens=?,cached_prompt_tokens=?,reasoning_tokens=?
|
||||
WHERE audit_id=?
|
||||
""",
|
||||
(
|
||||
datetime.now(UTC).isoformat(),
|
||||
outcome,
|
||||
http_status,
|
||||
error_type,
|
||||
provider_request_id,
|
||||
prompt_tokens,
|
||||
completion_tokens,
|
||||
total_tokens,
|
||||
cached_prompt_tokens,
|
||||
reasoning_tokens,
|
||||
audit_id,
|
||||
),
|
||||
)
|
||||
|
||||
def report(
|
||||
self,
|
||||
*,
|
||||
since: str | None = None,
|
||||
request_id: str | None = None,
|
||||
limit: int = 200,
|
||||
) -> dict[str, object]:
|
||||
clauses: list[str] = []
|
||||
params: list[object] = []
|
||||
if since is not None:
|
||||
clauses.append("started_at>=?")
|
||||
params.append(since)
|
||||
if request_id is not None:
|
||||
clauses.append("request_id=?")
|
||||
params.append(request_id)
|
||||
where = f"WHERE {' AND '.join(clauses)}" if clauses else ""
|
||||
with main_database(self.database).read() as connection:
|
||||
summary = connection.execute(
|
||||
f"""
|
||||
SELECT COUNT(*) AS request_count,
|
||||
SUM(CASE WHEN outcome='succeeded' THEN 1 ELSE 0 END) AS succeeded,
|
||||
SUM(CASE WHEN outcome='failed' THEN 1 ELSE 0 END) AS failed,
|
||||
SUM(CASE WHEN outcome='in_flight' THEN 1 ELSE 0 END) AS in_flight,
|
||||
SUM(prompt_tokens) AS prompt_tokens,
|
||||
SUM(completion_tokens) AS completion_tokens,
|
||||
SUM(total_tokens) AS total_tokens,
|
||||
SUM(cached_prompt_tokens) AS cached_prompt_tokens,
|
||||
SUM(reasoning_tokens) AS reasoning_tokens,
|
||||
AVG(prompt_tokens) AS avg_prompt_tokens,
|
||||
AVG(completion_tokens) AS avg_completion_tokens
|
||||
FROM model_request_audit {where}
|
||||
""",
|
||||
tuple(params),
|
||||
).fetchone()
|
||||
rows = connection.execute(
|
||||
f"""
|
||||
SELECT audit_id,request_id,profile_id,profile_revision,provider_kind,
|
||||
model,request_kind,prompt_stage,started_at,completed_at,outcome,http_status,
|
||||
error_type,provider_request_id,prompt_tokens,completion_tokens,
|
||||
total_tokens,cached_prompt_tokens,reasoning_tokens
|
||||
FROM model_request_audit {where}
|
||||
ORDER BY started_at DESC,audit_id DESC LIMIT ?
|
||||
""",
|
||||
(*params, limit),
|
||||
).fetchall()
|
||||
minute_rows = connection.execute(
|
||||
f"""
|
||||
SELECT substr(started_at,1,16) || ':00Z' AS minute,
|
||||
prompt_stage,
|
||||
COUNT(*) AS request_count,
|
||||
SUM(prompt_tokens) AS prompt_tokens,
|
||||
SUM(completion_tokens) AS completion_tokens,
|
||||
SUM(total_tokens) AS total_tokens,
|
||||
AVG(prompt_tokens) AS avg_prompt_tokens,
|
||||
AVG(completion_tokens) AS avg_completion_tokens
|
||||
FROM model_request_audit {where}
|
||||
GROUP BY substr(started_at,1,16),prompt_stage
|
||||
ORDER BY minute,prompt_stage
|
||||
""",
|
||||
tuple(params),
|
||||
).fetchall()
|
||||
keys = (
|
||||
"audit_id",
|
||||
"request_id",
|
||||
"profile_id",
|
||||
"profile_revision",
|
||||
"provider_kind",
|
||||
"model",
|
||||
"request_kind",
|
||||
"prompt_stage",
|
||||
"started_at",
|
||||
"completed_at",
|
||||
"outcome",
|
||||
"http_status",
|
||||
"error_type",
|
||||
"provider_request_id",
|
||||
"prompt_tokens",
|
||||
"completion_tokens",
|
||||
"total_tokens",
|
||||
"cached_prompt_tokens",
|
||||
"reasoning_tokens",
|
||||
)
|
||||
summary_keys = (
|
||||
"request_count",
|
||||
"succeeded",
|
||||
"failed",
|
||||
"in_flight",
|
||||
"prompt_tokens",
|
||||
"completion_tokens",
|
||||
"total_tokens",
|
||||
"cached_prompt_tokens",
|
||||
"reasoning_tokens",
|
||||
"avg_prompt_tokens",
|
||||
"avg_completion_tokens",
|
||||
)
|
||||
return {
|
||||
"summary": {
|
||||
key: (
|
||||
round(float(value), 2)
|
||||
if key.startswith("avg_") and value is not None
|
||||
else int(value)
|
||||
if value is not None
|
||||
else 0
|
||||
)
|
||||
for key, value in zip(summary_keys, summary, strict=True)
|
||||
},
|
||||
"requests": [
|
||||
dict(zip(keys, tuple(row), strict=True))
|
||||
for row in rows
|
||||
],
|
||||
"by_minute": [
|
||||
{
|
||||
"minute": str(row[0]),
|
||||
"prompt_stage": str(row[1]),
|
||||
"request_count": int(row[2]),
|
||||
"prompt_tokens": int(row[3] or 0),
|
||||
"completion_tokens": int(row[4] or 0),
|
||||
"total_tokens": int(row[5] or 0),
|
||||
"avg_prompt_tokens": round(float(row[6] or 0), 2),
|
||||
"avg_completion_tokens": round(float(row[7] or 0), 2),
|
||||
}
|
||||
for row in minute_rows
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _non_negative_int(value: object) -> int | None:
|
||||
return value if type(value) is int and value >= 0 else None
|
||||
|
||||
|
||||
class _ResponseUsage(TypedDict):
|
||||
provider_request_id: str | None
|
||||
prompt_tokens: int | None
|
||||
completion_tokens: int | None
|
||||
total_tokens: int | None
|
||||
cached_prompt_tokens: int | None
|
||||
reasoning_tokens: int | None
|
||||
|
||||
|
||||
def _empty_usage() -> _ResponseUsage:
|
||||
return {
|
||||
"provider_request_id": None,
|
||||
"prompt_tokens": None,
|
||||
"completion_tokens": None,
|
||||
"total_tokens": None,
|
||||
"cached_prompt_tokens": None,
|
||||
"reasoning_tokens": None,
|
||||
}
|
||||
|
||||
|
||||
def _response_usage(content: bytes) -> _ResponseUsage:
|
||||
try:
|
||||
body = json.loads(content)
|
||||
except (json.JSONDecodeError, UnicodeDecodeError):
|
||||
return _empty_usage()
|
||||
if not isinstance(body, dict):
|
||||
return _empty_usage()
|
||||
usage = body.get("usage")
|
||||
if not isinstance(usage, dict):
|
||||
usage = {}
|
||||
prompt_details = usage.get("prompt_tokens_details")
|
||||
completion_details = usage.get("completion_tokens_details")
|
||||
cached = (
|
||||
prompt_details.get("cached_tokens")
|
||||
if isinstance(prompt_details, dict)
|
||||
else usage.get("prompt_cache_hit_tokens")
|
||||
)
|
||||
reasoning = (
|
||||
completion_details.get("reasoning_tokens")
|
||||
if isinstance(completion_details, dict)
|
||||
else None
|
||||
)
|
||||
return {
|
||||
"provider_request_id": body.get("id") if isinstance(body.get("id"), str) else None,
|
||||
"prompt_tokens": _non_negative_int(usage.get("prompt_tokens")),
|
||||
"completion_tokens": _non_negative_int(usage.get("completion_tokens")),
|
||||
"total_tokens": _non_negative_int(usage.get("total_tokens")),
|
||||
"cached_prompt_tokens": _non_negative_int(cached),
|
||||
"reasoning_tokens": _non_negative_int(reasoning),
|
||||
}
|
||||
|
||||
|
||||
def _request_kind(request: httpx.Request) -> str:
|
||||
path = request.url.path.rstrip("/")
|
||||
if path.endswith("/chat/completions"):
|
||||
return "chat_completion"
|
||||
if path.endswith("/embeddings"):
|
||||
return "embedding"
|
||||
return "model_http"
|
||||
|
||||
|
||||
def _request_model(content: bytes, fallback: str) -> str:
|
||||
try:
|
||||
body = json.loads(content)
|
||||
except (json.JSONDecodeError, UnicodeDecodeError):
|
||||
return fallback
|
||||
model = body.get("model") if isinstance(body, dict) else None
|
||||
return model if isinstance(model, str) and model else fallback
|
||||
|
||||
|
||||
class AuditedModelTransport(httpx.AsyncBaseTransport):
|
||||
"""Record one row for each physical precipitation HTTP attempt."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
store: ModelRequestAuditStore,
|
||||
*,
|
||||
profile_id: str,
|
||||
profile_revision: int,
|
||||
provider_kind: str,
|
||||
model: str,
|
||||
inner: httpx.AsyncBaseTransport | None = None,
|
||||
) -> None:
|
||||
self.store = store
|
||||
self.profile_id = profile_id
|
||||
self.profile_revision = profile_revision
|
||||
self.provider_kind = provider_kind
|
||||
self.model = model
|
||||
self.inner = inner or httpx.AsyncHTTPTransport(retries=0, trust_env=False)
|
||||
|
||||
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
|
||||
context = _audit_context.get()
|
||||
if context is None:
|
||||
return await self.inner.handle_async_request(request)
|
||||
content = await request.aread()
|
||||
audit_id = await asyncio.to_thread(
|
||||
self.store.start,
|
||||
request_id=context.request_id,
|
||||
profile_id=self.profile_id,
|
||||
profile_revision=self.profile_revision,
|
||||
provider_kind=self.provider_kind,
|
||||
model=_request_model(content, self.model),
|
||||
request_kind=_request_kind(request),
|
||||
prompt_stage=_current_prompt_stage(),
|
||||
)
|
||||
try:
|
||||
response = await self.inner.handle_async_request(request)
|
||||
except BaseException as error:
|
||||
await asyncio.to_thread(
|
||||
self.store.finish,
|
||||
audit_id,
|
||||
outcome="failed",
|
||||
http_status=None,
|
||||
error_type=type(error).__name__,
|
||||
)
|
||||
raise
|
||||
content = await response.aread()
|
||||
usage = _response_usage(content)
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
self.store.finish,
|
||||
audit_id,
|
||||
outcome="succeeded" if response.is_success else "failed",
|
||||
http_status=response.status_code,
|
||||
error_type=None if response.is_success else f"http_{response.status_code}",
|
||||
**usage,
|
||||
)
|
||||
except Exception:
|
||||
# The durable in_flight row is more truthful than hiding the model
|
||||
# response after it has already incurred cost.
|
||||
logger.exception("model_request_audit_finish_failed audit_id=%s", audit_id)
|
||||
return response
|
||||
|
||||
async def aclose(self) -> None:
|
||||
await self.inner.aclose()
|
||||
@@ -68,16 +68,6 @@ COMPLETE_INSERT_MYSQL = (
|
||||
"singleton, status, selected_source_ids_json, started_at, completed_at, updated_at"
|
||||
") VALUES (1, 'completed', '[]', NULL, %s, %s)"
|
||||
)
|
||||
RESET_UPDATE_SQLITE = (
|
||||
"UPDATE onboarding_state SET status = 'not_started', "
|
||||
"selected_source_ids_json = '[]', started_at = NULL, completed_at = NULL, "
|
||||
"updated_at = ? WHERE singleton = 1"
|
||||
)
|
||||
RESET_UPDATE_MYSQL = (
|
||||
"UPDATE onboarding_state SET status = 'not_started', "
|
||||
"selected_source_ids_json = '[]', started_at = NULL, completed_at = NULL, "
|
||||
"updated_at = %s WHERE singleton = 1"
|
||||
)
|
||||
|
||||
|
||||
class OnboardingState(BaseModel):
|
||||
@@ -150,30 +140,6 @@ class OnboardingStore:
|
||||
)
|
||||
return self.get()
|
||||
|
||||
def reset(self) -> OnboardingState:
|
||||
"""Return onboarding to its first-run state, touching nothing else.
|
||||
|
||||
The completion flag is stored beside the memories on purpose: a fresh
|
||||
install of the app must not greet someone who already has memories as a
|
||||
first-time user. That makes the way back to the cold-start screens an
|
||||
explicit action rather than a side effect of reinstalling, and it is
|
||||
the whole reason this exists: QA and long-time users need to see those
|
||||
screens again without giving up a single memory.
|
||||
|
||||
Sources, memories and settings are separate tables and are not read or
|
||||
written here.
|
||||
"""
|
||||
now = datetime.now(UTC).isoformat()
|
||||
mysql = is_mysql(self._store)
|
||||
with self._connect(writing=True) as connection:
|
||||
execute(
|
||||
connection,
|
||||
mysql,
|
||||
RESET_UPDATE_MYSQL if mysql else RESET_UPDATE_SQLITE,
|
||||
(now,),
|
||||
)
|
||||
return self.get()
|
||||
|
||||
@contextmanager
|
||||
def _connect(
|
||||
self,
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Continuous recovery monitor for model processing.
|
||||
|
||||
The monitor is intentionally retained. It observes the global config final
|
||||
state plus ``memory_tasks`` and asks the supervisor to recover work; it owns no
|
||||
second request/receipt state machine.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from collections.abc import Hashable
|
||||
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
|
||||
|
||||
class ProcessingCoordinator:
|
||||
RESCAN_SECONDS = 60.0
|
||||
POLL_SECONDS = 2.0
|
||||
|
||||
def __init__(self, config: ConfigStore, supervisor: MemoryProcessingSupervisor):
|
||||
self.config = config
|
||||
self.supervisor = supervisor
|
||||
self._stopping = False
|
||||
self._monitor: asyncio.Task[None] | None = None
|
||||
self._error: str | None = None
|
||||
self._observed: tuple[Hashable, ...] | None = None
|
||||
self._last_scan = 0.0
|
||||
self._clock = time.monotonic
|
||||
|
||||
async def recover(self) -> None:
|
||||
"""Start the one process-owned monitor, whether permission is on or off."""
|
||||
|
||||
if self._stopping:
|
||||
return
|
||||
if self._monitor is None or self._monitor.done():
|
||||
self._monitor = asyncio.create_task(
|
||||
self._monitor_loop(),
|
||||
name="one-memory-processing-monitor",
|
||||
)
|
||||
|
||||
def _identity(self) -> tuple[Hashable, ...]:
|
||||
counts = self.supervisor.tasks.counts()
|
||||
return (
|
||||
self.config.model_content_allowed(),
|
||||
tuple(sorted((str(status), count) for status, count in counts.items())),
|
||||
)
|
||||
|
||||
async def _tick(self) -> None:
|
||||
identity = await asyncio.to_thread(self._identity)
|
||||
now = self._clock()
|
||||
if identity[0] and (
|
||||
identity != self._observed or now - self._last_scan >= self.RESCAN_SECONDS
|
||||
):
|
||||
await self.supervisor.resume_allowed()
|
||||
self._last_scan = now
|
||||
identity = await asyncio.to_thread(self._identity)
|
||||
self._observed = identity
|
||||
self._error = None
|
||||
|
||||
async def _monitor_loop(self) -> None:
|
||||
while not self._stopping:
|
||||
try:
|
||||
await self._tick()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
self._error = "model_content_recovery_blocked"
|
||||
await asyncio.sleep(self.POLL_SECONDS)
|
||||
|
||||
def status(self) -> dict[str, object]:
|
||||
return {
|
||||
"monitoring": self._monitor is not None and not self._monitor.done(),
|
||||
"allowed": self._observed[0] if self._observed else None,
|
||||
"error": self._error,
|
||||
}
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
self._stopping = True
|
||||
if self._monitor is not None:
|
||||
self._monitor.cancel()
|
||||
await asyncio.gather(self._monitor, return_exceptions=True)
|
||||
@@ -87,7 +87,7 @@ def build_acceptance_pack(
|
||||
"""
|
||||
SELECT DISTINCT e.evidence_id
|
||||
FROM evidence e
|
||||
WHERE e.role = 'user'
|
||||
WHERE e.role = 'user' AND e.written_by = 'person'
|
||||
AND LENGTH(e.verbatim) <= 300
|
||||
AND (
|
||||
e.verbatim LIKE '%我是一名%' OR e.verbatim LIKE '%我是一个%'
|
||||
@@ -118,7 +118,7 @@ def build_acceptance_pack(
|
||||
for row in connection.execute(
|
||||
"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE role != 'user'
|
||||
WHERE role = 'user' AND written_by != 'person'
|
||||
"""
|
||||
):
|
||||
strata["attribution_edge"].add(row["evidence_id"])
|
||||
@@ -138,7 +138,11 @@ def build_acceptance_pack(
|
||||
quota,
|
||||
)
|
||||
if len(selected) < sample_size:
|
||||
eligible = (evidence_id for evidence_id, row in evidence.items() if row["role"] == "user")
|
||||
eligible = (
|
||||
evidence_id
|
||||
for evidence_id, row in evidence.items()
|
||||
if row["role"] == "user" and row["written_by"] == "person"
|
||||
)
|
||||
_extend_unique(selected, _stable_order(eligible, seed=f"{seed}:fill"), sample_size)
|
||||
selected = selected[:sample_size]
|
||||
if len(selected) < sample_size:
|
||||
@@ -159,11 +163,7 @@ def build_acceptance_pack(
|
||||
row_id=f"{index:03d}",
|
||||
evidence_id=evidence_id,
|
||||
source_id=source["source_id"],
|
||||
written_by=(
|
||||
"person"
|
||||
if source["role"] == "user"
|
||||
else ("agent" if source["role"] == "assistant" else source["role"])
|
||||
),
|
||||
written_by=source["written_by"],
|
||||
character_count=len(source["verbatim"]),
|
||||
content_hash=source["content_hash"],
|
||||
strata=tuple(
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord, WrittenBy
|
||||
from one_memory.imports.sources.parsing import is_host_generated_user_turn, strip_host_wrappers
|
||||
from one_memory.input_sanitization import sanitize_model_input
|
||||
from one_memory.memory_processing import _is_hy_input_turn
|
||||
@@ -15,7 +15,10 @@ def identity_content_ready(record: EvidenceRecord) -> bool:
|
||||
# cleaning requires a source map and remains explicitly uncovered for now.
|
||||
return (
|
||||
record.content_kind is EvidenceContentKind.RAW_CONVERSATION
|
||||
and record.role in {"user", "assistant"}
|
||||
and (
|
||||
(record.role == "user" and record.written_by is WrittenBy.PERSON)
|
||||
or (record.role == "assistant" and record.written_by is WrittenBy.AGENT)
|
||||
)
|
||||
and _is_hy_input_turn(record)
|
||||
and not is_host_generated_user_turn(record.verbatim)
|
||||
and strip_host_wrappers(record.verbatim) == record.verbatim.strip()
|
||||
|
||||
@@ -10,6 +10,7 @@ from one_memory.imports.models import (
|
||||
MAX_VERBATIM_CHARACTERS,
|
||||
EvidenceContentKind,
|
||||
EvidenceRecord,
|
||||
WrittenBy,
|
||||
)
|
||||
from one_memory.imports.sources.parsing import (
|
||||
is_host_generated_user_turn,
|
||||
@@ -163,10 +164,10 @@ def prepare_mapped_content(record: EvidenceRecord) -> PreparedMappedContent:
|
||||
The returned text is transient; only metadata is suitable for an outer manifest.
|
||||
That manifest must independently bind Evidence identity/revision/hash and grants.
|
||||
"""
|
||||
if record.content_kind is not EvidenceContentKind.RAW_CONVERSATION or record.role not in {
|
||||
"user",
|
||||
"assistant",
|
||||
}:
|
||||
if record.content_kind is not EvidenceContentKind.RAW_CONVERSATION or not (
|
||||
(record.role == "user" and record.written_by is WrittenBy.PERSON)
|
||||
or (record.role == "assistant" and record.written_by is WrittenBy.AGENT)
|
||||
):
|
||||
raise MappedContentError("mapped_content_ineligible")
|
||||
original = record.verbatim
|
||||
if len(original) > MAX_VERBATIM_CHARACTERS:
|
||||
|
||||
@@ -11,7 +11,7 @@ from datetime import UTC
|
||||
|
||||
from one_memory import ingest_journal as journal
|
||||
from one_memory.context_raw import _check, _connection
|
||||
from one_memory.imports.models import EvidenceRecord
|
||||
from one_memory.imports.models import EvidenceRecord, WrittenBy
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.retrieval.index_store import RetrievalIndexStore
|
||||
from one_memory.retrieval.models import EvidenceSpan, RawEvidence
|
||||
@@ -166,7 +166,7 @@ def materialize_raw_candidates(
|
||||
source_id=record.source_id,
|
||||
conversation_id=record.conversation_id,
|
||||
role=record.role,
|
||||
written_by="person" if record.role == "user" else "agent",
|
||||
written_by="person" if record.written_by is WrittenBy.PERSON else "agent",
|
||||
event_time=event_time,
|
||||
spans=tuple(spans),
|
||||
matched_by=("semantic",),
|
||||
|
||||
@@ -264,7 +264,7 @@ class ServiceLevelScheduler:
|
||||
self._quiet_admission_logged_at[key] = now
|
||||
if last is not None:
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="service_level.scheduler",
|
||||
event="service_level.summary",
|
||||
outcome="succeeded",
|
||||
@@ -276,7 +276,7 @@ class ServiceLevelScheduler:
|
||||
)
|
||||
return
|
||||
diagnostic_event(
|
||||
level="debug",
|
||||
level="info",
|
||||
component="service_level.scheduler",
|
||||
event="service_level.admitted",
|
||||
outcome="succeeded",
|
||||
|
||||
@@ -56,10 +56,10 @@ def build_source_import_receipt(
|
||||
"SELECT count(*) AS messages, "
|
||||
"count(DISTINCT conversation_id || char(0) || COALESCE(workspace, '')) "
|
||||
"AS conversations, "
|
||||
"coalesce(sum(role='user'),0) AS person_messages, "
|
||||
"coalesce(sum(role='assistant'),0) AS agent_messages, "
|
||||
"coalesce(sum(role='tool'),0) AS tool_messages, "
|
||||
"coalesce(sum(role NOT IN ('user','assistant','tool')),0) "
|
||||
"coalesce(sum(written_by='person'),0) AS person_messages, "
|
||||
"coalesce(sum(written_by='agent'),0) AS agent_messages, "
|
||||
"coalesce(sum(written_by='tool'),0) AS tool_messages, "
|
||||
"coalesce(sum(written_by NOT IN ('person','agent','tool')),0) "
|
||||
"AS unknown_author_messages, "
|
||||
"coalesce(sum(created_at IS NULL),0) AS unknown_time_messages "
|
||||
"FROM evidence WHERE user_id=? AND source_id=?",
|
||||
@@ -68,9 +68,7 @@ def build_source_import_receipt(
|
||||
# Bound text reads before materializing snippets. These are samples, not a
|
||||
# recency ranking; sorting every large original body is unnecessary here.
|
||||
examples = db.execute(
|
||||
"SELECT evidence_id, role, CASE role "
|
||||
"WHEN 'user' THEN 'person' WHEN 'assistant' THEN 'agent' "
|
||||
"WHEN 'tool' THEN 'tool' ELSE 'unknown' END AS written_by, created_at, "
|
||||
"SELECT evidence_id, role, written_by, created_at, "
|
||||
"substr(verbatim,1,600) AS excerpt, length(verbatim)>600 AS truncated "
|
||||
"FROM evidence WHERE evidence_id IN ("
|
||||
"SELECT evidence_id FROM evidence WHERE user_id=? AND source_id=? LIMIT 3)",
|
||||
@@ -82,13 +80,10 @@ def build_source_import_receipt(
|
||||
# Qualify before LIMIT: agent/system rows must not consume the three samples.
|
||||
query_filter = " AND evidence_id=?" if query_evidence_id is not None else ""
|
||||
queries = db.execute(
|
||||
f"""SELECT evidence_id, revision AS evidence_revision, role,
|
||||
CASE role WHEN 'user' THEN 'person' WHEN 'assistant' THEN 'agent'
|
||||
WHEN 'tool' THEN 'tool' ELSE 'unknown' END AS written_by,
|
||||
created_at,
|
||||
CASE WHEN user_id=? AND source_id=? AND role='user'
|
||||
f"""SELECT evidence_id, revision AS evidence_revision, role, written_by, created_at,
|
||||
CASE WHEN user_id=? AND source_id=? AND role='user' AND written_by='person'
|
||||
AND length(verbatim)<=65536
|
||||
THEN display_query_projection(verbatim,role,'person')
|
||||
THEN display_query_projection(verbatim,role,written_by)
|
||||
ELSE NULL END AS projection
|
||||
FROM evidence WHERE user_id=? AND source_id=?{query_filter}
|
||||
AND projection IS NOT NULL
|
||||
|
||||
@@ -13,7 +13,10 @@
|
||||
| `test_archive_extract.py` | `contract` | Unpacking an archive a stranger sent, onto this user's disk. |
|
||||
| `test_ask_workspace_path.py` | `contract` | Ask must honour the same workspace_path expansion recall uses. |
|
||||
| `test_async_bridge.py` | `contract` | 覆盖 reader rejects parallel workers that could close during next、cancelling a read stops the worker before the batch finishes、starting a read does not stop the app answering 等 10 个场景。 |
|
||||
| `test_async_dispatch_checks.py` | `contract` | 覆盖 cancelling quiet timer does not start any permission reader、dispatch check does not queue work after shutdown、dispatch permission check denies before queue after classification 等 6 个场景。 |
|
||||
| `test_auth.py` | `integration` | 覆盖 local api requires bearer token、browser origin is checked in addition to token 场景。 |
|
||||
| `test_backfill_acceptance_lifecycle.py` | `contract` | A processing acknowledgement belongs to one consent revision and live worker. |
|
||||
| `test_backfill_conversation_isolation.py` | `contract` | One stale conversation cannot invalidate an entire backfill scan. |
|
||||
| `test_background_work.py` | `contract` | 覆盖 a request does not wait behind a catch up、bulk work is kept to a few threads at a time、the pool outlives one hand off 等 4 个场景。 |
|
||||
| `test_backup_destroy_restore_reconciliation.py` | `contract` | Backup -> destroy -> restore, reconciled row for row. |
|
||||
| `test_backup_endpoint.py` | `integration` | The 'POST /api/v1/backup' endpoint the CLI's 'backup create' calls. |
|
||||
@@ -61,6 +64,7 @@
|
||||
| `test_cli_lifecycle.py` | `integration` | 覆盖 cli starts and stops without residual runtime 场景。 |
|
||||
| `test_cli_restore.py` | `contract` | 覆盖 restore requires a source、restore refuses while the runtime is running、restore previews without confirmation 等 4 个场景。 |
|
||||
| `test_codex_header_provenance.py` | `contract` | 覆盖 header workspace versions and native ids reach file origin、new adapter rereads header after a real checkpoint、machine and unknown session sources do not establish human or agent sender 等 14 个场景。 |
|
||||
| `test_compact_model_content.py` | `contract` | Server-held previews preserve exact grants without round-tripping all refs. |
|
||||
| `test_connection_lifetime.py` | `integration` | Every SQLite connection a request opens must be closed when the request ends. |
|
||||
| `test_context_access.py` | `contract` | 覆盖 memory only never reads raw authority、raw permission does not grant memory and removes revoked sources、raw requires all apps and source 等 13 个场景。 |
|
||||
| `test_context_api.py` | `integration` | 覆盖 no model local query returns real raw and governed items、existing single lane authorization never runs denied reader、authentication and origin remain required 等 14 个场景。 |
|
||||
@@ -72,6 +76,12 @@
|
||||
| `test_context_search.py` | `contract` | 覆盖 authorization and requested paths never invoke other lane、empty partial zero unavailable and safe failures、parallel common deadline and finalizers wait for both reads 等 24 个场景。 |
|
||||
| `test_context_search_both_paths.py` | `integration` | The objective names 'recall' and requires BOTH paths: raw and memory. |
|
||||
| `test_context_v2_contract_parity.py` | `unit` | Mechanical parity checks for the shared v2 coverage receipt contract. |
|
||||
| `test_continuous_completed_input_review.py` | `contract` | Independent review of completed-input reuse; real stores and controlled HY only. |
|
||||
| `test_continuous_coordinator.py` | `contract` | 覆盖 activation hook and startup recovery start single monitor then stop、failed preparation does not advance observed watermark、new journal event and source regrant wake replay without legacy request 等 6 个场景。 |
|
||||
| `test_continuous_outcome_unknown.py` | `contract` | 覆盖 a timed out continuous task becomes unknown not pending、a timed out non continuous task also becomes unknown、an unsettled continuous send becomes unknown not failed 等 8 个场景。 |
|
||||
| `test_continuous_policy.py` | `contract` | 覆盖 backlog snapshot over one hundred thousand is compact、caller rollback includes policy and backlog、enable idempotence rejects changed payload and superseded operation 等 19 个场景。 |
|
||||
| `test_continuous_processing_bridge.py` | `contract` | 覆盖 bound source regrant refuses old task、continuous inactive never falls back to legacy、continuous prepare lease ingest has no legacy receipt or old context 等 17 个场景。 |
|
||||
| `test_continuous_processing_integration.py` | `contract` | Real temporary Core/stores with controlled probe and HY boundaries, never real data. |
|
||||
| `test_contract_parity.py` | `contract` | Python 的枚举和 TypeScript 里那份手写的联合类型必须一致。 |
|
||||
| `test_conversation_search.py` | `integration` | Searching imported conversations before any generation model is configured. |
|
||||
| `test_conversation_seed_ids.py` | `contract` | 覆盖 seed ids read metadata only without hydrating、seed eligibility keeps person and memory content only、latest activity includes nonseed turns and import time fallback 等 13 个场景。 |
|
||||
@@ -87,6 +97,7 @@
|
||||
| `test_database_contention.py` | `contract` | Writers meet in this one file, and the loser used to lose data. |
|
||||
| `test_database_file_permissions.py` | `contract` | 覆盖 a new main database is readable by its owner alone、the wal sidecars are not left world readable、a database from an older build is tightened on open 场景。 |
|
||||
| `test_db_session_utc.py` | `contract` | The MySQL session must be pinned to UTC. |
|
||||
| `test_delete_no_conversation_freeze.py` | `contract` | 删掉一条记忆,不得让同一对话之后的新 turn 停止沉淀(RC10)。 |
|
||||
| `test_delivery_board.py` | `contract` | 覆盖 parses only the marked structured record、verified requires evidence with a positive control、rejects unknown dependency 等 5 个场景。 |
|
||||
| `test_derived_echo.py` | `contract` | A host that saves our injection back must not mint a new user decision. |
|
||||
| `test_detailed_logging.py` | `contract` | 覆盖 detailed pipeline logging defaults off and survives restart、live log level applies to full hy and provider call chain、info collects hy info without enabling debug pipeline content 等 7 个场景。 |
|
||||
@@ -100,6 +111,7 @@
|
||||
| `test_dual_engine_helper.py` | `contract` | Tests for the dual-engine harness itself. |
|
||||
| `test_event_loop_blocking.py` | `contract` | Proof, as a test, that blocking work on the loop freezes it — and the fix works. |
|
||||
| `test_evidence_reconcile_primitive.py` | `integration` | A bounded, read-only discovery primitive for the H23 Evidence reconcile. |
|
||||
| `test_evidence_revision_read_review.py` | `contract` | Read invalidation review: real guard/provenance, controlled HY, no model/network. |
|
||||
| `test_evidence_store.py` | `contract` | 覆盖 same source message merges origins without duplication、identical text from different sources is not deduplicated、rescanning an unchanged file does not write the message again 等 26 个场景。 |
|
||||
| `test_export_probe.py` | `contract` | What a probe may say about a stranger's chat history, and what it may not. |
|
||||
| `test_external_endpoint_scopes.py` | `integration` | Core boundary tests using the broker's envelope, never a host's claimed identity. |
|
||||
@@ -113,9 +125,10 @@
|
||||
| `test_file_import_operations.py` | `integration` | Bounded synthetic uploads; operation identity is not a content dedup key. |
|
||||
| `test_file_import_operations_review.py` | `contract` | Upload refusal/legacy compatibility review with bounded synthetic files only. |
|
||||
| `test_file_publish_job_boundary.py` | `contract` | Staged uploads cannot widen an earlier directory import's uncaptured input. |
|
||||
| `test_four_app_live_ingest.py` | `contract` | Actual source readers + OS watch + Core routes; no model or native host calls. |
|
||||
| `test_four_app_live_ingest.py` | `integration` | Actual source readers + OS watch + Core routes; no model or native host calls. |
|
||||
| `test_frozen_core_layout.py` | `contract` | 覆盖 single file layout remains available without private preset、folder layout keeps same core and all native dependencies、unknown layout does not silently build another package 等 5 个场景。 |
|
||||
| `test_golden_corpus.py` | `contract` | A7-01: the golden corpus comparison. |
|
||||
| `test_governance_authority_review.py` | `contract` | D04 authority and workspace scope through the real continuous permission guard. |
|
||||
| `test_governance_permission_commit.py` | `contract` | 覆盖 permission guard is rechecked inside write transaction and rolls back、guard context does not leak into subsequent manual actions、guard check and commit hold database write authority 场景。 |
|
||||
| `test_governance_read_path_does_not_take_the_writer.py` | `contract` | The recall read path must not acquire the single global writer. |
|
||||
| `test_governance_store_root.py` | `contract` | 生产代码里治理 store 只在 Core 启动的组合根建一次。 |
|
||||
@@ -131,19 +144,32 @@
|
||||
| `test_hourly_c1_peak_replay_contract.py` | `contract` | Guard the package peak-replay probe's error accounting. |
|
||||
| `test_hourly_c1_recall_stage_probe_contract.py` | `contract` | Guard the packaged recall stage probe's command and error semantics. |
|
||||
| `test_hy_baseline_source.py` | `contract` | 覆盖 first import replay copies raw prefix without selecting message ids、frozen transcript keeps middle user message and excludes later records 场景。 |
|
||||
| `test_hy_coding_lease.py` | `contract` | HY's own coding/working route under the permission lease, on the pinned SDK. |
|
||||
| `test_hy_coding_reply.py` | `contract` | R-07: HY's coding-shaped replies read as "no memories at all". |
|
||||
| `test_hy_lease_responsiveness.py` | `contract` | Blocked lease checks, local failing doubles only; no network or credentials. |
|
||||
| `test_hy_lifecycle_sync.py` | `contract` | HY decides deletions; the ledger mirrors them. Pinned SDK, scripted HTTP. |
|
||||
| `test_hy_log_privacy.py` | `contract` | HY SDK records are not rewritten by One Memory's logging setup. |
|
||||
| `test_hy_memory_bridge.py` | `contract` | 覆盖 hy client initialization is serial across bridges、native hy clients share one runtime per data directory、ollama profile ignores ambient remote storage and models 等 41 个场景。 |
|
||||
| `test_hy_metrics_lifecycle.py` | `contract` | What the pinned public SDK's MetricsCollector does, and what that costs us. |
|
||||
| `test_hy_native_write_chain.py` | `contract` | HY writes with its own prompts and judgment; the product adds no audit of its own. |
|
||||
| `test_hy_network_egress.py` | `contract` | Where a real HY client is allowed to send bytes, and where it is not. |
|
||||
| `test_hy_permission_context.py` | `contract` | Whole native HY client with synthetic content and HTTP MockTransport, never cloud. |
|
||||
| `test_hy_permission_context_races.py` | `contract` | Bounded factory races with a temporary native HY client and no model calls. |
|
||||
| `test_hy_permission_raw_context.py` | `contract` | Real pinned HY SDK/Chroma and guards; scripted HTTP is not model acceptance. |
|
||||
| `test_hy_permission_store.py` | `contract` | Native HY node/algorithm tests with synthetic data and an in-memory backend only. |
|
||||
| `test_hy_permission_transport.py` | `contract` | Native HY entry points, synthetic inputs and MockTransport only. |
|
||||
| `test_hy_response_guard.py` | `contract` | 覆盖 invalid response retries same hy provider without changing prompt、repeated empty response fails instead of becoming no op、failure receipt survives a later successful model call 等 23 个场景。 |
|
||||
| `test_hy_response_shapes.py` | `contract` | HY response shape selection only; no provider, write routing or model calls. |
|
||||
| `test_hy_runtime_config.py` | `contract` | MEMORY.md is the audited record of what we send HY; keep it answerable. |
|
||||
| `test_hy_sdk_idempotency_probe.py` | `contract` | A0-03: probe whether the HY SDK can make a write idempotent. |
|
||||
| `test_hy_source_context.py` | `contract` | 覆盖 bound context rejects a same named conversation in another workspace、bound context rejects a duplicated evidence id、bound context rejects a same conversation from another source 等 9 个场景。 |
|
||||
| `test_hy_stage_trace.py` | `contract` | 覆盖 observer returns native results without editing or requesting models、observer redacts content without recording prompts or exception details 场景。 |
|
||||
| `test_hy_state_source.py` | `contract` | RC4:记忆生命周期(有效 / 被取代 / 去重隐藏 / 过期)直接读 HY,产品账本不再逐类镜像。 |
|
||||
| `test_hy_step_timing.py` | `contract` | Where an HY write spends the time that is not a model call. |
|
||||
| `test_hy_step_trace_switch.py` | `contract` | Rule 15: the per-step HY trace is off unless the debug tier asks for it. |
|
||||
| `test_hy_supersede_sync.py` | `contract` | HY decides supersessions; the ledger mirrors them. Pinned SDK, scripted HTTP. |
|
||||
| `test_hy_trial_budget.py` | `contract` | Offline proofs for the real-model acceptance budget (no network, no Keychain). |
|
||||
| `test_hy_turn_input.py` | `contract` | HY gets one turn at a time, exactly as it happened, and picks its own route. |
|
||||
| `test_hy_writeback.py` | `integration` | HY 的结果只有一扇门进治理表,而且每次进门都带着调用开始时的证据快照。 |
|
||||
| `test_import_api.py` | `integration` | 覆盖 resume during cleanup returns conflict without internal error、authorize detect import and revoke api flow、protected api rejects missing token and foreign origin 等 5 个场景。 |
|
||||
| `test_import_control_races_review.py` | `contract` | Read-only review repro: real runner/store, synthetic delayed adapter cleanup. |
|
||||
@@ -155,6 +181,7 @@
|
||||
| `test_import_three_scenarios.py` | `contract` | The three import scenarios a user actually meets. |
|
||||
| `test_import_three_scenarios_real.py` | `contract` | The three import scenarios, on a real Core at a real volume. |
|
||||
| `test_imported_index_background.py` | `contract` | The ordering index is built off the startup path. |
|
||||
| `test_incremental_memory_backfill.py` | `contract` | 覆盖 accept backfill returns after first batch while later history is still read、backfill budget yields between rounds and resumes same cursor、backfill holds an unanswered raw turn until its reply arrives 等 18 个场景。 |
|
||||
| `test_ingest_journal.py` | `contract` | 覆盖 baseline and empty initialize are explicit transactional and idempotent、old date new import and revise noop and rollbacks、import metadata changes invalidate without authorizing content 等 12 个场景。 |
|
||||
| `test_input_sanitization.py` | `unit` | 覆盖 redacts cli and assignment secrets without removing context、redacts bearer known tokens urls and private keys、redacts quoted json assignments and basic authorization 等 8 个场景。 |
|
||||
| `test_installer.py` | `contract` | 覆盖 installer create verify and rollback is transactional、installer updates owned file and restores backup、installer refuses unowned or changed target 场景。 |
|
||||
@@ -182,6 +209,7 @@
|
||||
| `test_main_database_fairness.py` | `contract` | Waiting for the one database has to be a queue, not a scramble. |
|
||||
| `test_main_database_isolation.py` | `contract` | What one checkout of the main database can do, and what it leaves for the next. |
|
||||
| `test_managed_restore.py` | `contract` | 覆盖 restore rejects a source outside the managed backup directory、restore rejects a symlinked source、restore rejects a non one memory database 等 9 个场景。 |
|
||||
| `test_manual_memory_via_hy.py` | `contract` | T04: what the person explicitly asks to remember is written through HY's import path. |
|
||||
| `test_manual_transcript_import.py` | `integration` | 覆盖 manual json keeps authors unknown time and exact text、marked text keeps code fences and never invents timestamps、ambiguous partial or invalid manual data never leaves partial import 等 4 个场景。 |
|
||||
| `test_matter_collection.py` | `integration` | 覆盖 hy windows cover every turn including leading and boundary assistants、oversize turn is split without silently dropping content、partial timestamps preserve source sequence 等 8 个场景。 |
|
||||
| `test_matter_discovery.py` | `contract` | 覆盖 synthesis materializes only known verbatim citations、duplicate claims keep all governed refs and distinct states、synthesis cannot echo the original sixty claim list 等 27 个场景。 |
|
||||
@@ -239,19 +267,30 @@
|
||||
| `test_memory_write_registry.py` | `contract` | A3-01: acceptance and outcome recording for the unified write path. |
|
||||
| `test_memory_write_routes.py` | `integration` | A3-01: the HTTP write entry, tested against a real FastAPI app. |
|
||||
| `test_migrate_off_sqlite.py` | `contract` | A1-06: the migration is only done when reconciliation agrees. |
|
||||
| `test_model_content_bulk.py` | `contract` | Bulk consent reads preserve exact identities and SQLite snapshot boundaries. |
|
||||
| `test_model_content_parse_cost.py` | `contract` | Memoize parsing only; every permission decision still reads current SQLite state. |
|
||||
| `test_model_content_permissions.py` | `contract` | Content permission tests use only a temporary database and synthetic sources. |
|
||||
| `test_model_content_preview_races.py` | `contract` | Confirmation must tolerate append-only import without broadening consent. |
|
||||
| `test_model_content_routes.py` | `integration` | 覆盖 permission route reads and writes only config final state、core composition starts with config permission and durable drain 场景。 |
|
||||
| `test_model_content_scope_seals.py` | `contract` | Indexed scope invalidation retains complete-snapshot semantics without rereading it. |
|
||||
| `test_model_content_snapshot_cost.py` | `contract` | Reuse derived fingerprints only after reading every raw row in this transaction. |
|
||||
| `test_model_creation_api.py` | `integration` | Normal protected model creation routes with temporary data and in-memory secrets. |
|
||||
| `test_model_creation_operations.py` | `contract` | 覆盖 lookup without table is read only and legacy create does not initialize、operation commits with profile and version without autoactivation、any replay conflicts before secret put 等 21 个场景。 |
|
||||
| `test_model_creation_validation_boundary.py` | `integration` | Real ASGI validation must precede creation reservation and secret writes. |
|
||||
| `test_model_policy_activation.py` | `contract` | 覆盖 activation helper is transactional and restores row factory、backlog scan does not hold the single writer、cloud credential is checked without keychain write 等 21 个场景。 |
|
||||
| `test_model_policy_routes.py` | `contract` | 覆盖 all routes require auth、backlog is required not defaulted、enable and transition safe shape and callback once after commit 等 10 个场景。 |
|
||||
| `test_model_probe.py` | `contract` | 覆盖 probe still measures embedding when structured output fails、probe records the real embedding width on the profile、openai compatible probe checks model and structured output 等 23 个场景。 |
|
||||
| `test_model_probe_input_revision.py` | `contract` | 覆盖 input revision precedes measured width update、concurrent update still rejects width or capability cas、early returns record the actual starting revision 等 6 个场景。 |
|
||||
| `test_model_probe_operations.py` | `integration` | Persisted probe identity, real SQLite/API and synthetic transport only. |
|
||||
| `test_model_processing_permissions.py` | `contract` | 覆盖 lease reads current allowed before each boundary、lease refuses a task profile that is no longer active 场景。 |
|
||||
| `test_model_profile_api.py` | `integration` | 覆盖 model profile api never returns key and enforces revision、model profile probe route returns only machine results、model profile validation never echoes api key 等 4 个场景。 |
|
||||
| `test_model_profile_context_reset.py` | `integration` | 覆盖 create context default and inclusive bounds、explicit null clears override and preserves history and secret、omitted or same value is noop with unchanged revision and metadata 等 11 个场景。 |
|
||||
| `test_model_profile_identity.py` | `contract` | Stable model editing and configured-vs-available credentials; no Keychain/network. |
|
||||
| `test_model_profile_queue_handover.py` | `contract` | Queued work when the model changes or its configuration is deleted (D75). |
|
||||
| `test_model_profiles.py` | `contract` | 覆盖 profile uses one model and keeps key out of database、api key updates reuse one stable secret reference、startup cleanup removes only secrets not referenced by current profiles 等 15 个场景。 |
|
||||
| `test_model_request_audit.py` | `contract` | 覆盖 failure is recorded without error message、missing usage stays unknown in detail and zero in summary、records one physical request with provider usage and no content 等 5 个场景。 |
|
||||
| `test_model_request_budget.py` | `contract` | 覆盖 persists failed request reservations without content、parallel leases cannot exceed request limit、unknown or excessive output and body are denied 等 6 个场景。 |
|
||||
| `test_model_request_budget_integration.py` | `contract` | Real HY leaf retries plus durable budget accounting; no network or provider. |
|
||||
| `test_model_scope_lifecycle.py` | `contract` | Bound derived/temporary data without changing consent or original history. |
|
||||
| `test_mysql_lifecycle.py` | `contract` | A1-07: the decisions Runtime makes around the bundled MySQL. |
|
||||
| `test_mysql_platform.py` | `contract` | A1-07: the machine-facing half of managing the bundled MySQL. |
|
||||
| `test_obs03_health_probe.py` | `contract` | OBS-03: '/health' answered with the web app instead of a health payload. |
|
||||
@@ -260,11 +299,14 @@
|
||||
| `test_oversized_file_guard.py` | `contract` | One pass over a source file is bounded by a byte budget, not skipped. |
|
||||
| `test_parent_watchdog.py` | `integration` | A Core that outlives its launcher makes the next launch impossible. |
|
||||
| `test_policy_api_wiring.py` | `integration` | 覆盖 global model permission off does not backfill、backfill puts history on write jobs without scheduling memory tasks、source pause skips history until the source is reenabled 场景。 |
|
||||
| `test_policy_operation_lookup.py` | `contract` | 覆盖 corrupt current row cannot make history or missing operation look safe、current operation uses safe receipt without callback or writes、lookup connection rejects writes and preserves database 等 12 个场景。 |
|
||||
| `test_policy_production_entry.py` | `contract` | 覆盖 production serve passes only supported app options、maintenance serve does not own the runtime 场景。 |
|
||||
| `test_precipitation_concurrency_setting.py` | `contract` | Rule 2/9: precipitation concurrency is configurable, not hardcoded to 1. |
|
||||
| `test_primary_source_adapters.py` | `contract` | 覆盖 pipeline json is host generated even when truncated、workbuddy record parsing never blocks the event loop、cursor store user info envelope is not attributed to the person 等 34 个场景。 |
|
||||
| `test_probe_code_contract_parity.py` | `contract` | 探测码在三处必须同时存在:python 枚举、TS 契约、界面文案表。 |
|
||||
| `test_probe_payment_required.py` | `contract` | 覆盖 http 402 without a useful body is still understood as unpaid、a wrong address is still a missing model not an unpaid one 场景。 |
|
||||
| `test_processing_coordinator.py` | `contract` | 覆盖 monitor is retained and observes config plus tasks 场景。 |
|
||||
| `test_processing_merged_context.py` | `contract` | 覆盖 new chunk omits known memory with changed support but keeps original history 场景。 |
|
||||
| `test_processing_requests.py` | `contract` | 覆盖 concurrent claim only one snapshot wins、concurrent create same grant has one id and profile、concurrent different profile never reuses another scope 等 14 个场景。 |
|
||||
| `test_quality_acceptance.py` | `contract` | 覆盖 acceptance pack is deterministic and scores personal boundary、acceptance score fails missing personal memory 场景。 |
|
||||
| `test_raw_search_without_model.py` | `integration` | Search imported conversations with no generative model configured at all. |
|
||||
@@ -328,17 +370,17 @@
|
||||
| `test_secret_store_error_codes.py` | `integration` | 3925a kernel item 3: keychain failure classification and backoff that heals. |
|
||||
| `test_secure_storage.py` | `integration` | 覆盖 isolated keychain service namespace、custom keychain service is refused outside isolated tests、packaged keychain budget covers loaded process startup 等 20 个场景。 |
|
||||
| `test_sedimentation_concurrency.py` | `contract` | Sedimentation runs several harnesses at once, but one harness at a time. |
|
||||
| `test_sedimentation_switch.py` | `contract` | Stopping model-driven sedimentation without losing the configuration. |
|
||||
| `test_service_level_pressure_acceptance.py` | `contract` | 覆盖 workbuddy recall remains available during 100k import 场景。 |
|
||||
| `test_service_levels.py` | `contract` | 覆盖 background never consumes the realtime reservation、realtime is next for the same source after background yields、import history gets one slice after nine realtime admissions 等 8 个场景。 |
|
||||
| `test_session_inbox.py` | `contract` | A session inbox is not a shared mailbox and a pull is not adoption. |
|
||||
| `test_session_lease.py` | `contract` | A session file on disk is not proof that the session is online. |
|
||||
| `test_session_registry.py` | `contract` | Trusted session identity is not a workspace path. |
|
||||
| `test_settings_policy_lifecycle.py` | `contract` | Settings data-plane lifecycle: real temporary Core, fake probe/HY, no UI or package. |
|
||||
| `test_simplified_model_permission_migrations.py` | `contract` | 覆盖 fresh schema omits removed permission state machines 场景。 |
|
||||
| `test_snapshot_adapter.py` | `contract` | Selection-only binding reopens checksum-verified frozen copies. |
|
||||
| `test_snapshot_pool.py` | `contract` | A whole-directory snapshot must not hold a thread everything else is waiting for. |
|
||||
| `test_soak_probe_effectiveness.py` | `integration` | The soak probe must observe the load it reports on. |
|
||||
| `test_source_access.py` | `contract` | 覆盖 source root requires explicit user grant、sensitive home directory is rejected without requesting os access、authorized file cannot escape root through a symlink 等 24 个场景。 |
|
||||
| `test_source_authorization_state.py` | `unit` | Why a source cannot be read, named separately from whether it can. |
|
||||
| `test_source_candidate_api.py` | `integration` | Candidate location changes must not implicitly grant or import anything. |
|
||||
| `test_source_candidate_connections.py` | `contract` | 覆盖 empty candidate revisions preserve default fingerprint、persisted candidate requires exact revision、connection resolves candidate in transaction and prior replay never regrants 等 7 个场景。 |
|
||||
| `test_source_candidates.py` | `contract` | 覆盖 preview is metadata only and cancel changes nothing、select persists only candidate and restarts with revision、compare and swap and aba cannot reuse old preview 等 9 个场景。 |
|
||||
@@ -350,7 +392,7 @@
|
||||
| `test_source_parsing.py` | `contract` | 覆盖 reminders are removed only at message edges、original supported outer context and query envelopes、attributed reminder is opaque before finding the outer query 等 11 个场景。 |
|
||||
| `test_source_permission_races.py` | `integration` | Permission execution regressions using temporary stores and Core ASGI instances. |
|
||||
| `test_source_query_preview.py` | `contract` | 覆盖 preserves exact user text and span、supported outer wrapper is stripped without paraphrasing、cursor host prefixes are not shown as the user query 等 8 个场景。 |
|
||||
| `test_source_read_revocation.py` | `contract` | 覆盖 revocation during model wait prevents late answer、source revocation stops every outward read but keeps other sources 场景。 |
|
||||
| `test_source_read_revocation.py` | `integration` | 覆盖 source revocation stops every outward read but keeps other sources、revocation during model wait prevents late answer 场景。 |
|
||||
| `test_source_readers.py` | `contract` | 覆盖 jsonl is resumable and does not advance over an incomplete tail、sqlite reader uses monotonic row checkpoint、zstd jsonl reader resumes by completed record 等 7 个场景。 |
|
||||
| `test_source_registry_and_grants.py` | `contract` | 覆盖 catalog resolves defaults and marks environment overrides、recall hosts that can append join the live sync poller、declarations with no reader are named rather than mistaken for behaviour 等 15 个场景。 |
|
||||
| `test_source_revocation_parking.py` | `integration` | Revoking a source parks its unfinished extraction instead of failing it (D73). |
|
||||
@@ -360,7 +402,7 @@
|
||||
| `test_source_sync_idempotent.py` | `integration` | Final-state Harness sync is idempotent and resumes missed source work. |
|
||||
| `test_source_sync_permissions.py` | `integration` | S1.3 source permission tests use temporary files only, never real host history. |
|
||||
| `test_source_totals.py` | `contract` | 覆盖 the first ask is answered from the rows、same named sessions in two workspaces count as two conversations、insert during count is not marked as already counted 等 7 个场景。 |
|
||||
| `test_source_upload_intents.py` | `contract` | Upload identity binds retries without replacing file receipts or directory consent. |
|
||||
| `test_source_upload_intents.py` | `integration` | Upload identity binds retries without replacing file receipts or directory consent. |
|
||||
| `test_source_write_switch.py` | `contract` | A frozen Harness produces no extraction tasks. |
|
||||
| `test_sql_compat.py` | `contract` | The shared compatibility layer, especially 'rowcount'. |
|
||||
| `test_sqlite_semantics_gate.py` | `contract` | D56: prevent SQLite implementation details spreading outside the data layer. |
|
||||
@@ -369,10 +411,13 @@
|
||||
| `test_store_binding.py` | `contract` | A2: which store is in force, and honest reporting of it. |
|
||||
| `test_store_binding_ratchet.py` | `contract` | D56: ratchet the retired dual-engine 'store_for' seam out of the Core. |
|
||||
| `test_store_port.py` | `contract` | A1-04/A2: the store port, and that real implementations satisfy it. |
|
||||
| `test_subagent_sessions.py` | `contract` | D98: a sub agent's transcript is its own conversation, not part of the main one. |
|
||||
| `test_subagent_turns.py` | `contract` | D98: a sub agent's turn is ordinary user-assistant data. |
|
||||
| `test_subscription_visibility.py` | `unit` | A shared directory is not a read grant. |
|
||||
| `test_sync_facade.py` | `contract` | A2: the sync facade must keep the sync shape call sites already use. |
|
||||
| `test_tail_metadata_lane.py` | `contract` | Post-response metadata retries must be bounded and never block the route. |
|
||||
| `test_terminal_payload_reaping.py` | `contract` | Rule 14: dead-end payloads are actually reaped, not just reapable. |
|
||||
| `test_tool_evidence_hy_turns.py` | `contract` | Every adapter's tool records reach HY inside the turn they happened in. |
|
||||
| `test_two_level_discovery.py` | `contract` | 3925a kernel item 1: two-level discovery. |
|
||||
| `test_ui_session.py` | `contract` | 覆盖 exchange is single use and creates a session、expired exchange cannot create a session、exchange is bound to the core instance 等 5 个场景。 |
|
||||
| `test_ui_session_api.py` | `integration` | 覆盖 ui session exchange sets httponly strict cookie、ui session requires bearer for exchange、ui session reports missing web assets 场景。 |
|
||||
@@ -384,8 +429,8 @@
|
||||
| `test_windows_process_control.py` | `contract` | R-29:Windows 上 'os.kill' 不是发信号,是无条件杀进程。 |
|
||||
| `test_workbuddy_boundary.py` | `contract` | WorkBuddy only injects on SessionStart and UserPromptSubmit. |
|
||||
| `test_workbuddy_parsing.py` | `contract` | 覆盖 automation preserves query without claiming fresh human authorship、automation context attributes do not depend on order or quote style、quoted full automation envelope is not scheduled provenance 等 17 个场景。 |
|
||||
| `test_workbuddy_prompt_backfill.py` | `contract` | When the prompt hook's capture fails, the transcript file is what brings the words in. |
|
||||
| `test_workbuddy_prompt_backfill.py` | `integration` | When the prompt hook's capture fails, the transcript file is what brings the words in. |
|
||||
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
|
||||
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
|
||||
|
||||
共 383 个测试文件。
|
||||
共 428 个测试文件。
|
||||
@@ -14,8 +14,8 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
|
||||
EXPECTED_TEST_FILE_COUNT = 383
|
||||
EXPECTED_TEST_FILES_SHA256 = "3c1a3d3febaf478e1ba005fbcd22ad23d76fc5925b32b8d8740e4e65c554b922"
|
||||
EXPECTED_TEST_FILE_COUNT = 428
|
||||
EXPECTED_TEST_FILES_SHA256 = "bd3bf4fc9cda2a8e4427d106283c40d22dad026fd452db954046b6f28d21b896"
|
||||
|
||||
# Pure transformations, validation models, parity checks and state machines.
|
||||
# This list is intentionally small: uncertain files default to contract.
|
||||
@@ -35,7 +35,6 @@ UNIT_FILES = frozenset(
|
||||
"test_memory_server_job_state.py",
|
||||
"test_scan_roots.py",
|
||||
"test_scanner_classifier.py",
|
||||
"test_source_authorization_state.py",
|
||||
"test_scanner_sessionizer.py",
|
||||
"test_source_envelope_positions.py",
|
||||
"test_source_lifecycle.py",
|
||||
@@ -69,6 +68,7 @@ INTEGRATION_FILES = frozenset(
|
||||
"test_external_installation_permissions.py",
|
||||
"test_fault_matrix.py",
|
||||
"test_file_import_operations.py",
|
||||
"test_four_app_live_ingest.py",
|
||||
"test_graph_width_endpoint.py",
|
||||
"test_home_summary.py",
|
||||
"test_hy_writeback.py",
|
||||
@@ -100,6 +100,7 @@ INTEGRATION_FILES = frozenset(
|
||||
"test_memory_supersede.py",
|
||||
"test_memory_visibility.py",
|
||||
"test_memory_write_routes.py",
|
||||
"test_model_content_routes.py",
|
||||
"test_model_creation_api.py",
|
||||
"test_model_creation_validation_boundary.py",
|
||||
"test_model_probe_operations.py",
|
||||
@@ -123,14 +124,17 @@ INTEGRATION_FILES = frozenset(
|
||||
"test_source_connection_intents.py",
|
||||
"test_source_import_receipts.py",
|
||||
"test_source_permission_races.py",
|
||||
"test_source_read_revocation.py",
|
||||
"test_source_revocation_parking.py",
|
||||
"test_source_session_count_api.py",
|
||||
"test_source_status_latency.py",
|
||||
"test_source_sync_idempotent.py",
|
||||
"test_source_sync_permissions.py",
|
||||
"test_source_upload_intents.py",
|
||||
"test_ui_session_api.py",
|
||||
"test_unified_write_entry_wiring.py",
|
||||
"test_visibility_background.py",
|
||||
"test_workbuddy_prompt_backfill.py",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@@ -26,6 +26,67 @@ async def thaw_source(client, source_id, receipt):
|
||||
return response.json()
|
||||
|
||||
|
||||
async def confirm_model_content(
|
||||
client,
|
||||
*,
|
||||
source_ids,
|
||||
evidence_ids=(),
|
||||
all_imported=None,
|
||||
future_additions=True,
|
||||
process=True,
|
||||
):
|
||||
"""Grant model processing the same way the product does: preview, then confirm.
|
||||
|
||||
Live suites need this because importing Evidence is not consent to send it
|
||||
to a model; without it the processing guard refuses, which is the correct
|
||||
product behaviour and not something a test may bypass by writing the store.
|
||||
"""
|
||||
profiles = await client.get("/api/v1/model-profiles")
|
||||
profiles.raise_for_status()
|
||||
active = next(item for item in profiles.json() if item["is_active"])
|
||||
preview = await client.post(
|
||||
"/api/v1/model-content-permission/preview",
|
||||
json={
|
||||
"profile_id": active["profile_id"],
|
||||
"profile_revision": active["revision"],
|
||||
"source_ids": list(source_ids),
|
||||
"evidence_ids": list(evidence_ids),
|
||||
# The manual source only accepts named originals, so "everything
|
||||
# from these sources" is not a default the caller can ignore.
|
||||
"all_imported_from_selected_sources": (
|
||||
not evidence_ids if all_imported is None else all_imported
|
||||
),
|
||||
"include_future_additions": future_additions,
|
||||
},
|
||||
)
|
||||
# Report the refusal code, not just the status: every content refusal is a
|
||||
# named product decision and a live run that hides it cannot be diagnosed.
|
||||
assert preview.status_code == 200, preview.text
|
||||
current = await client.get("/api/v1/model-content-permission")
|
||||
current.raise_for_status()
|
||||
existing = current.json()
|
||||
confirmed = await client.post(
|
||||
"/api/v1/model-content-permission",
|
||||
json={
|
||||
"snapshot": preview.json(),
|
||||
"expected_revision": existing["revision"] if existing else 0,
|
||||
"user_confirmed": True,
|
||||
},
|
||||
)
|
||||
confirmed.raise_for_status()
|
||||
receipt = confirmed.json()
|
||||
if process:
|
||||
# What the product does after the user confirms: take the already
|
||||
# imported Evidence through the pipeline instead of waiting for the
|
||||
# next import to trigger a debounce.
|
||||
accepted = await client.post(
|
||||
"/api/v1/model-content-permission/process",
|
||||
json={"expected_revision": receipt["revision"], "user_confirmed": True},
|
||||
)
|
||||
accepted.raise_for_status()
|
||||
return receipt
|
||||
|
||||
|
||||
async def connection_request(client, source_ids, **options):
|
||||
roots = {}
|
||||
for source_id in source_ids:
|
||||
|
||||
Loaded 100 of 316 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user