Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d0d8b4e163 | ||
|
|
20b92d8a89 | ||
|
|
2a4c5e38e7 | ||
|
|
c891a1fd1d | ||
|
|
7741e17f09 | ||
|
|
2a4dc2ffa2 | ||
|
|
61b5ef30a9 | ||
|
|
f79770cdb0 |
No files matched your search
@@ -122,19 +122,25 @@ jobs:
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
shopt -s globstar nullglob
|
||||
version=$(node -p 'require("./package.json").version')
|
||||
node scripts/verify-release-version.mjs \
|
||||
--channel stable \
|
||||
--version "$version" \
|
||||
--build-number "$(node -p 'require("./package.json").release.build_number')" \
|
||||
--tag "$GITHUB_REF_NAME"
|
||||
mkdir -p publish-assets
|
||||
for arch in x64 arm64; do
|
||||
archive="release-assets/memhub-runtime-v${version}-linux-${arch}.tar.gz"
|
||||
test -f "$archive"
|
||||
test -f "$archive.sha256"
|
||||
name="memhub-runtime-v${version}-linux-${arch}.tar.gz"
|
||||
archives=(release-assets/**/"$name")
|
||||
checksums=(release-assets/**/"$name.sha256")
|
||||
test "${#archives[@]}" -eq 1
|
||||
test "${#checksums[@]}" -eq 1
|
||||
cp -- "${archives[0]}" "publish-assets/$name"
|
||||
cp -- "${checksums[0]}" "publish-assets/$name.sha256"
|
||||
(
|
||||
cd release-assets
|
||||
sha256sum -c "$(basename "$archive.sha256")"
|
||||
cd publish-assets
|
||||
sha256sum -c "$name.sha256"
|
||||
)
|
||||
done
|
||||
- name: Upload Linux assets to a draft GitHub Release
|
||||
@@ -160,7 +166,7 @@ jobs:
|
||||
--title "$GITHUB_REF_NAME"
|
||||
fi
|
||||
gh release upload "$GITHUB_REF_NAME" \
|
||||
release-assets/memhub-runtime-v*-linux-*.tar.gz \
|
||||
release-assets/memhub-runtime-v*-linux-*.tar.gz.sha256 \
|
||||
publish-assets/memhub-runtime-v*-linux-*.tar.gz \
|
||||
publish-assets/memhub-runtime-v*-linux-*.tar.gz.sha256 \
|
||||
--repo "$GITHUB_REPOSITORY" \
|
||||
--clobber
|
||||
+35
-1
@@ -2,6 +2,26 @@
|
||||
|
||||
Hy Memory and MemHub are versioned and released independently.
|
||||
|
||||
## Hy Memory 2.0.3 — 2026-10-17
|
||||
|
||||
- Added an optional runtime HTTP transport factory so embedded products can
|
||||
enforce request budgets without wrapping or restricting SDK local-store
|
||||
access.
|
||||
- Published
|
||||
[`hy-memory==2.0.3`](https://pypi.org/project/hy-memory/2.0.3/) with
|
||||
unchanged memory extraction, reconciliation, storage, and Coding/Working
|
||||
behavior.
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Removed MemHub's task-scoped HyPermission store wrappers and global
|
||||
model-content consent switch. The trusted embedded Hy Memory SDK now uses
|
||||
its native local stores directly.
|
||||
- Existing installations ignore the retired config flag, readmit previously
|
||||
parked pre-invocation work, and drop only the obsolete
|
||||
`hy_node_permissions` proof table. Evidence, governed memories, model
|
||||
profiles, SDK stores, and source/host grants are preserved.
|
||||
|
||||
## Hy Memory 2.0.2 — 2026-10-17
|
||||
|
||||
- Published
|
||||
@@ -16,7 +36,21 @@ Hy Memory and MemHub are versioned and released independently.
|
||||
Because PyPI release metadata is immutable, the next SDK patch release must
|
||||
carry the updated license metadata; `2.0.1` retains its published metadata.
|
||||
|
||||
## MemHub 1.0.2 — Unreleased
|
||||
## MemHub 1.0.3 — Unreleased
|
||||
|
||||
- Updated the source version to `1.0.3` (build 7).
|
||||
- Pinned the public SDK to `hy-memory==2.0.3`.
|
||||
- Removed the HyPermission store wrappers and global model-content consent
|
||||
switch; the trusted embedded SDK now uses native local stores.
|
||||
- Prevented uncertain writes from blocking later source work and added one
|
||||
reconciled automatic retry after ten seconds.
|
||||
- Added product-facing memory pipeline counters and the log troubleshooting
|
||||
runbook.
|
||||
- Allowed a verified packaged Desktop Runtime from a different commit to take
|
||||
over an equal-build install atomically while preserving downgrade and
|
||||
tamper refusal.
|
||||
|
||||
## MemHub 1.0.2 — 2026-10-17
|
||||
|
||||
- Updated the source version to `1.0.2` (build 6).
|
||||
- Pinned the public SDK to `hy-memory==2.0.2`.
|
||||
|
||||
@@ -32,7 +32,7 @@ English | [中文](README.zh.md)
|
||||
|
||||
## Latest News
|
||||
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.2`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.2/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.3`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.3/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** on Hugging Face along with its [technical report](https://arxiv.org/abs/2610.22196) on attributed reinforcement learning for long-term memory writing!
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **Released** **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)** (submitted to **ICLR 2027**).
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **Accepted** at **EMNLP 2026**: **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)**!
|
||||
@@ -63,13 +63,13 @@ Welcome to the **Hy Memory Research Family**, covering cognitive chat memory, lo
|
||||
### 1. Installation
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.2 # core (includes ChromaDB and Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.2" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.2" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.2" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.2" # + MySQL cache
|
||||
pip install "hy-memory[rerank]==2.0.2" # + local multilingual cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.2" # everything
|
||||
pip install hy-memory==2.0.3 # core (includes ChromaDB and Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.3" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.3" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.3" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.3" # + MySQL cache
|
||||
pip install "hy-memory[rerank]==2.0.3" # + local multilingual cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.3" # everything
|
||||
```
|
||||
|
||||
### 2. Configuration
|
||||
|
||||
+8
-8
@@ -32,7 +32,7 @@
|
||||
|
||||
## 最新动态
|
||||
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.2`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.2/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.3`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.3/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** 长期记忆写入模型及[技术报告](https://arxiv.org/abs/2610.22196)!
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **基准与方法发布** 推出长周期与交错式编程记忆评测基准 **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)**(已投稿 **ICLR 2027**)。
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **论文录用** 双系统认知记忆架构论文 **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)** 被 **EMNLP 2026** 接收!
|
||||
@@ -63,13 +63,13 @@
|
||||
### 1. 安装
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.2 # 核心(含 ChromaDB 与 Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.2" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.2" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.2" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.2" # + MySQL 缓存
|
||||
pip install "hy-memory[rerank]==2.0.2" # + 本地多语言 cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.2" # 全部
|
||||
pip install hy-memory==2.0.3 # 核心(含 ChromaDB 与 Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.3" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.3" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.3" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.3" # + MySQL 缓存
|
||||
pip install "hy-memory[rerank]==2.0.3" # + 本地多语言 cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.3" # 全部
|
||||
```
|
||||
|
||||
### 2. 配置
|
||||
|
||||
+1
-1
@@ -37,7 +37,7 @@ do not belong in the public source tree.
|
||||
- Root GitHub workflows run SDK and MemHub checks from their respective working
|
||||
directories. Native lifecycle workflows produce reviewable Actions artifacts;
|
||||
tag releases remain fail-closed on public identity, signing and notarization.
|
||||
- Hy Memory is published as `hy-memory==2.0.2`. MemHub pins that public
|
||||
- Hy Memory is published as `hy-memory==2.0.3`. MemHub pins that public
|
||||
distribution; its selected npm package is `memhub-cli`, which is not yet
|
||||
published.
|
||||
- Hy-Memory source uses Apache-2.0 at the repository root. Third-party
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@
|
||||
## 安装与快速开始
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.2
|
||||
pip install hy-memory==2.0.3
|
||||
|
||||
# 可选后端
|
||||
pip install 'hy-memory[qdrant]==2.0.1'
|
||||
|
||||
@@ -496,11 +496,21 @@ class LLMProvider:
|
||||
clients = self._clients.setdefault(loop, {})
|
||||
client = clients.get(client_key)
|
||||
if client is None:
|
||||
client_options: Dict[str, Any] = {
|
||||
"api_key": base_api_key,
|
||||
"base_url": base_url,
|
||||
"timeout": self._llm_config.timeout,
|
||||
"max_retries": 0,
|
||||
}
|
||||
transport_factory = getattr(self.config, "http_transport_factory", None)
|
||||
if callable(transport_factory):
|
||||
import httpx
|
||||
|
||||
client_options["http_client"] = httpx.AsyncClient(
|
||||
transport=transport_factory()
|
||||
)
|
||||
client = AsyncOpenAI(
|
||||
api_key=base_api_key,
|
||||
base_url=base_url,
|
||||
timeout=self._llm_config.timeout,
|
||||
max_retries=0,
|
||||
**client_options,
|
||||
)
|
||||
clients[client_key] = client
|
||||
|
||||
|
||||
+8
-1
@@ -6,7 +6,7 @@ Agent Memory - 配置管理
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional, Dict, Any, List
|
||||
from typing import Optional, Dict, Any, Callable, List
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -1252,6 +1252,13 @@ class MemoryConfig:
|
||||
enable_agent: bool = False
|
||||
debug: bool = False
|
||||
metrics_enabled: bool = True
|
||||
# Runtime-only HTTP transport factory. MemHub uses this for request-cost
|
||||
# budgeting without restricting the embedded SDK's local store access.
|
||||
http_transport_factory: Optional[Callable[[], Any]] = field(
|
||||
default=None,
|
||||
repr=False,
|
||||
compare=False,
|
||||
)
|
||||
|
||||
def __post_init__(self):
|
||||
# enable_graph 跟随 mode 走:mode=ultra 时自动启用,无需独立环境变量
|
||||
|
||||
@@ -356,11 +356,21 @@ class EmbedService:
|
||||
key = self._client_cache_key(base_url, api_key, timeout)
|
||||
client = self._client_cache.get(key)
|
||||
if client is None:
|
||||
client_options = {
|
||||
"api_key": api_key,
|
||||
"base_url": base_url or None,
|
||||
"timeout": timeout,
|
||||
"max_retries": 0,
|
||||
}
|
||||
transport_factory = getattr(self.config, "http_transport_factory", None)
|
||||
if callable(transport_factory):
|
||||
import httpx
|
||||
|
||||
client_options["http_client"] = httpx.AsyncClient(
|
||||
transport=transport_factory()
|
||||
)
|
||||
client = AsyncOpenAI(
|
||||
api_key=api_key,
|
||||
base_url=base_url or None,
|
||||
timeout=timeout,
|
||||
max_retries=0,
|
||||
**client_options,
|
||||
)
|
||||
self._client_cache[key] = client
|
||||
return client
|
||||
|
||||
+10
-7
@@ -15,12 +15,12 @@ write、search,并把搜索结果格式化为可交给模型的 memory text ch
|
||||
|
||||
```text
|
||||
distribution: hy-memory
|
||||
version: 2.0.2
|
||||
version: 2.0.3
|
||||
mode: 按运行时配置选择
|
||||
reader: hybrid
|
||||
```
|
||||
|
||||
仓库 pin、lockfile 和运行时版本守卫均固定到公开的 `hy-memory==2.0.2`。
|
||||
仓库 pin、lockfile 和运行时版本守卫均固定到公开的 `hy-memory==2.0.3`。
|
||||
MemHub 在本地 Runtime 内嵌入 SDK,但不绕过自身的来源授权、模型授权、Evidence、
|
||||
Governance 和撤权边界。
|
||||
|
||||
@@ -305,9 +305,9 @@ Runtime 应长期复用 client,不为每次请求重新初始化数据库和
|
||||
|
||||
```text
|
||||
Evidence
|
||||
-> ModelProcessingPermissionLease
|
||||
-> HyPermissionTransport
|
||||
-> HyMemoryBridge.ingest / ingest_extracted / search_ids
|
||||
-> HyMemoryBridge
|
||||
-> embedded Hy Memory SDK
|
||||
-> native Vector / Coding / Graph stores
|
||||
-> write result verification
|
||||
-> Governance
|
||||
-> Recall Broker / UI
|
||||
@@ -316,8 +316,11 @@ Evidence
|
||||
不要在新业务模块里直接创建 `HyMemoryClient`,否则会绕过:
|
||||
|
||||
- 来源读取授权;
|
||||
- 模型内容处理授权;
|
||||
- 发送前与响应后的撤权复核;
|
||||
- 共享 Runtime、模型配置、API Key 安全存储与请求预算;
|
||||
- Evidence 引用与作者归属;
|
||||
- 写后回读验证;
|
||||
- Governance 状态、版本、更正和删除级联。
|
||||
|
||||
Hy Memory SDK 是 MemHub 的内置可信组件,可自由访问和管理本机的全部
|
||||
Hy Memory 数据。来源授权仍决定哪些宿主数据可以进入本机;宿主 Recall Grant
|
||||
仍决定哪些外部 AI 应用可以读取治理后的记忆。这两类边界不限制内置 SDK 的本地 CRUD。
|
||||
+6
-4
@@ -6,13 +6,13 @@ SDK 和记忆引擎。One Memory / `one-memory` 是历史模块、数据目录
|
||||
|
||||
## 发行与平台合同
|
||||
|
||||
公共 Python SDK 为 `hy-memory==2.0.2`。MemHub 源码版本为 `1.0.2`
|
||||
(build 6),GitHub 仓库为
|
||||
公共 Python SDK 为 `hy-memory==2.0.3`。MemHub 源码版本为 `1.0.3`
|
||||
(build 7),GitHub 仓库为
|
||||
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
|
||||
`memhub-cli`。当前代码架构见
|
||||
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
|
||||
|
||||
`v1.0.2` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
|
||||
`v1.0.3` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
|
||||
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
|
||||
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
|
||||
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
|
||||
@@ -42,6 +42,8 @@ Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命
|
||||
|
||||
架构详情见 [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) 和
|
||||
[`docs/contracts/runtime-unification.md`](docs/contracts/runtime-unification.md)。
|
||||
产品、测试和一线支持排查导入与记忆沉淀时,使用
|
||||
[`docs/PRODUCT_LOG_TROUBLESHOOTING.md`](docs/PRODUCT_LOG_TROUBLESHOOTING.md)。
|
||||
|
||||
## 开发环境
|
||||
|
||||
@@ -99,7 +101,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
|
||||
|
||||
## Hy Memory
|
||||
|
||||
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.2`。SDK 配置、权限边界及 search
|
||||
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.3`。SDK 配置、权限边界及 search
|
||||
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
|
||||
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@
|
||||
"dataLicense": "CC0-1.0",
|
||||
"SPDXID": "SPDXRef-DOCUMENT",
|
||||
"name": "one-memory",
|
||||
"documentNamespace": "https://one-memory.local/spdx/2026-10-15T17:26:47Z",
|
||||
"documentNamespace": "https://one-memory.local/spdx/2026-10-17T11:07:57Z",
|
||||
"creationInfo": {
|
||||
"created": "2026-10-15T17:26:47Z",
|
||||
"created": "2026-10-17T11:07:57Z",
|
||||
"creators": [
|
||||
"Tool: one-memory-generate-sbom"
|
||||
]
|
||||
@@ -2335,7 +2335,7 @@
|
||||
{
|
||||
"SPDXID": "SPDXRef-Package-hy-memory",
|
||||
"name": "hy-memory",
|
||||
"versionInfo": "2.0.2",
|
||||
"versionInfo": "2.0.3",
|
||||
"downloadLocation": "NOASSERTION",
|
||||
"filesAnalyzed": false,
|
||||
"licenseConcluded": "Apache-2.0",
|
||||
|
||||
@@ -2,14 +2,14 @@
|
||||
|
||||
MemHub itself is licensed under Apache-2.0 (see `LICENSE`).
|
||||
It bundles the runtime dependencies listed below.
|
||||
Generated from the installed dependency tree on 2026-10-15T17:26:47Z
|
||||
Generated from the installed dependency tree on 2026-10-17T11:07:57Z
|
||||
by `python3 scripts/generate_sbom.py`. Do not edit by hand.
|
||||
|
||||
Per-package license texts are not reproduced here yet; the license
|
||||
column is filled in as part of release review, and the SBOM records
|
||||
`NOASSERTION` until then rather than guessing.
|
||||
|
||||
## hy-memory 2.0.2
|
||||
## hy-memory 2.0.3
|
||||
|
||||
License: Apache-2.0
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@one-memory/cli",
|
||||
"version": "1.0.2",
|
||||
"version": "1.0.3",
|
||||
"description": "MemHub bootstrap CLI for installing and managing the local-first shared memory runtime",
|
||||
"license": "Apache-2.0",
|
||||
"repository": {
|
||||
|
||||
@@ -32,7 +32,6 @@ const REQUIRED_COMMAND_SURFACE = [
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"models",
|
||||
@@ -77,7 +76,7 @@ const REQUIRED_MARKERS = [
|
||||
|
||||
/**
|
||||
* Sub-commands the UI-write parity work added. A package can declare the
|
||||
* parent command (`permission`, `models`) while still missing the new verb, so
|
||||
* parent command (`models`) while still missing the new verb, so
|
||||
* checking command names alone would pass a stale bundle. The marker is the
|
||||
* exact literal the catalog usage string contains.
|
||||
*/
|
||||
@@ -88,9 +87,6 @@ const REQUIRED_SUBCOMMANDS = [
|
||||
{ command: "import-file", marker: "memhub import-file discard" },
|
||||
{ command: "notes", marker: "memhub notes add" },
|
||||
{ command: "notes", marker: "memhub notes process" },
|
||||
{ command: "permission", marker: "memhub permission status" },
|
||||
{ command: "permission", marker: "memhub permission allow" },
|
||||
{ command: "permission", marker: "memhub permission deny" },
|
||||
{ command: "extraction", marker: "memhub extraction set" },
|
||||
{ command: "graph", marker: "memhub graph width" },
|
||||
{ command: "models", marker: "memhub models edit" },
|
||||
@@ -153,8 +149,6 @@ const STOPPED_DESTRUCTIVE_PROBES = [
|
||||
"--confirm",
|
||||
],
|
||||
},
|
||||
{ name: "permission_allow", args: ["permission", "allow"] },
|
||||
{ name: "permission_deny", args: ["permission", "deny"] },
|
||||
{
|
||||
name: "extraction_set_apply",
|
||||
args: ["extraction", "set", "--window", "6m", "--confirm"],
|
||||
|
||||
+31
-71
@@ -81,6 +81,7 @@ import {
|
||||
readMemoryWrite,
|
||||
readScanRoots,
|
||||
removeScanRoot,
|
||||
resetOnboarding,
|
||||
submitBackup,
|
||||
submitMemoryWrite,
|
||||
submitScan,
|
||||
@@ -190,13 +191,6 @@ import {
|
||||
renderMemoryDetailText,
|
||||
renderMemoryPageText,
|
||||
} from "./memory-report.js";
|
||||
import {
|
||||
type ModelContentPermissionDocument,
|
||||
modelContentPermissionExitCode,
|
||||
readModelContentPermission,
|
||||
renderModelContentPermissionText,
|
||||
setModelContentPermission,
|
||||
} from "./model-content-permission-flow.js";
|
||||
import {
|
||||
executeModelProbe,
|
||||
inspectModelProbePlan,
|
||||
@@ -217,12 +211,10 @@ import {
|
||||
executeModelProfileActivate,
|
||||
executeModelProfileEdit,
|
||||
type ModelWriteDocument,
|
||||
type ModelWritePlan,
|
||||
modelWriteExitCode,
|
||||
planModelProfileActivate,
|
||||
planModelProfileEdit,
|
||||
renderModelWriteText,
|
||||
validateModelWritePlan,
|
||||
} from "./model-write-flows.js";
|
||||
import { claimPlanAttempt } from "./operation-journal.js";
|
||||
import {
|
||||
@@ -697,6 +689,36 @@ async function runCommand(
|
||||
// `path list` and `adapter list` are read-only: they have no plan to confirm,
|
||||
// so they are answered here rather than falling into the two-phase branch
|
||||
// (which would reject them for having nothing to apply).
|
||||
// Back to the cold-start screens on purpose. Deleting the app cannot do this
|
||||
// (the completion flag lives with the memories, by design), and deleting the
|
||||
// data to get there would cost the user everything they have stored.
|
||||
if (named === "onboarding" && arguments_.includes("reset")) {
|
||||
if (
|
||||
arguments_.filter((item) => item === "--json").length > 1 ||
|
||||
arguments_.filter((item) => item !== "--json").join(" ") !==
|
||||
"onboarding reset"
|
||||
) {
|
||||
return usage();
|
||||
}
|
||||
const attachment = await attachToRuntime(
|
||||
resolveRuntimeCliConfig(environment, process.argv[1]).dataDir,
|
||||
);
|
||||
if (attachment === null) {
|
||||
return runtimeUnavailable(descriptor, asJson);
|
||||
}
|
||||
const state = await resetOnboarding(attachment);
|
||||
if (state === null) {
|
||||
return reject(descriptor, asJson, "引导状态端点不可用。");
|
||||
}
|
||||
if (asJson) {
|
||||
process.stdout.write(`${JSON.stringify(state)}\n`);
|
||||
return EXIT_OK;
|
||||
}
|
||||
process.stdout.write(
|
||||
`引导流程已回到首次运行(status=${state.status}),记忆数据未改动。\n`,
|
||||
);
|
||||
return EXIT_OK;
|
||||
}
|
||||
if (named === "path" && arguments_.includes("list")) {
|
||||
if (
|
||||
arguments_.filter((item) => item === "--json").length > 1 ||
|
||||
@@ -2462,45 +2484,6 @@ async function runCommand(
|
||||
);
|
||||
return uiWriteExitCode(document);
|
||||
}
|
||||
if (named === "permission") {
|
||||
const parsed = parsePermissionCommand(arguments_.slice(1));
|
||||
if (!parsed) return usage();
|
||||
const config = resolveRuntimeCliConfig(environment, process.argv[1]);
|
||||
const runtime = await inspectRuntimeStatus({
|
||||
dataDir: config.dataDir,
|
||||
probeIdentity,
|
||||
requireServiceReceipt: true,
|
||||
});
|
||||
const attached =
|
||||
runtime.state === "running"
|
||||
? await attachClients(config.dataDir)
|
||||
: { reader: null, operations: null, failure: runtime.code };
|
||||
if (
|
||||
runtime.state !== "running" ||
|
||||
(parsed === "status" ? !attached.reader : !attached.operations)
|
||||
) {
|
||||
const document: ModelContentPermissionDocument = {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
state: "unknown",
|
||||
code:
|
||||
runtime.state !== "running"
|
||||
? runtime.code
|
||||
: (attached.failure ?? "runtime_attach_failed"),
|
||||
};
|
||||
emitModelContentPermissionDocument(document, asJson);
|
||||
return runtime.state === "stopped" ? 3 : 1;
|
||||
}
|
||||
const document =
|
||||
parsed === "status"
|
||||
? await readModelContentPermission(attached.reader!)
|
||||
: await setModelContentPermission(
|
||||
attached.operations!,
|
||||
parsed === "allow",
|
||||
);
|
||||
emitModelContentPermissionDocument(document, asJson);
|
||||
return modelContentPermissionExitCode(document);
|
||||
}
|
||||
if (named === "notes" || named === "extraction" || named === "graph") {
|
||||
const parsed = parseExtendedWriteCommand(
|
||||
named as "notes" | "extraction" | "graph",
|
||||
@@ -5124,18 +5107,6 @@ function parseHeadlessCommand(
|
||||
: null;
|
||||
}
|
||||
|
||||
function parsePermissionCommand(
|
||||
rest: readonly string[],
|
||||
): "status" | "allow" | "deny" | null {
|
||||
if (rest.filter((item) => item === "--json").length > 1) return null;
|
||||
const tokens = rest.filter((item) => item !== "--json");
|
||||
const action = tokens[0];
|
||||
return tokens.length === 1 &&
|
||||
(action === "status" || action === "allow" || action === "deny")
|
||||
? action
|
||||
: null;
|
||||
}
|
||||
|
||||
function unwrapPlan(value: unknown): unknown {
|
||||
if (
|
||||
value &&
|
||||
@@ -5323,17 +5294,6 @@ function emitModelReportDocument(
|
||||
);
|
||||
}
|
||||
|
||||
function emitModelContentPermissionDocument(
|
||||
document: ModelContentPermissionDocument,
|
||||
asJson: boolean,
|
||||
): void {
|
||||
process.stdout.write(
|
||||
asJson
|
||||
? `${JSON.stringify(document)}\n`
|
||||
: renderModelContentPermissionText(document),
|
||||
);
|
||||
}
|
||||
|
||||
function emitSourceSyncDocument(
|
||||
document: SourceSyncFlowDocument,
|
||||
asJson: boolean,
|
||||
|
||||
@@ -493,14 +493,6 @@ export const COMMANDS: readonly CommandDescriptor[] = [
|
||||
],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
|
||||
},
|
||||
{
|
||||
name: "permission",
|
||||
summary: "读取或直接设置全局模型内容外发许可。",
|
||||
usage:
|
||||
"memhub permission status [--json]\n memhub permission allow [--json]\n memhub permission deny [--json]",
|
||||
options: [JSON_OPTION],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
|
||||
},
|
||||
{
|
||||
name: "extraction",
|
||||
summary: "设置提取窗口;默认只出计划,--confirm 后才写入。",
|
||||
@@ -659,6 +651,14 @@ export const COMMANDS: readonly CommandDescriptor[] = [
|
||||
],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
|
||||
},
|
||||
{
|
||||
name: "onboarding",
|
||||
summary:
|
||||
"onboarding reset 回到首次运行的引导流程。只清除引导完成状态:不删记忆数据,也不需要卸载重装。",
|
||||
usage: "memhub onboarding reset [--json]",
|
||||
options: [JSON_OPTION],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE],
|
||||
},
|
||||
{
|
||||
name: "help",
|
||||
summary: "显示总帮助或单个命令的用法。",
|
||||
|
||||
@@ -81,7 +81,6 @@ const COMMANDS = [
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"host",
|
||||
@@ -93,6 +92,7 @@ const COMMANDS = [
|
||||
"uninstall",
|
||||
"upgrade",
|
||||
"delete-data",
|
||||
"onboarding",
|
||||
"help",
|
||||
"version",
|
||||
// A6 product commands. Appended, not inserted: the catalog is a stable
|
||||
@@ -259,7 +259,14 @@ describe("CLI-22 shared output contract", () => {
|
||||
|
||||
describe("argument validation", () => {
|
||||
it("fails unknown commands without printing data or running anything", async () => {
|
||||
for (const argv of [["unknown"], ["unknown", "--help"], ["--typo"]]) {
|
||||
for (const argv of [
|
||||
["unknown"],
|
||||
["unknown", "--help"],
|
||||
["--typo"],
|
||||
["permission", "status"],
|
||||
["permission", "allow"],
|
||||
["permission", "deny"],
|
||||
]) {
|
||||
const streams = captureStreams();
|
||||
try {
|
||||
expect(await run(argv, {})).toBe(2);
|
||||
|
||||
@@ -8,7 +8,6 @@ import axios from "axios";
|
||||
*/
|
||||
const READ_ONLY_PATHS = [
|
||||
"/api/v1/call-events",
|
||||
"/api/v1/model-content-permission",
|
||||
"/api/v1/home-summary",
|
||||
"/api/v1/import-jobs",
|
||||
"/api/v1/live-sync",
|
||||
@@ -398,6 +397,41 @@ export async function readMemoryWrite(
|
||||
}
|
||||
}
|
||||
|
||||
export interface OnboardingState {
|
||||
status: string;
|
||||
selected_source_ids: string[];
|
||||
started_at: string | null;
|
||||
completed_at: string | null;
|
||||
}
|
||||
|
||||
/** Send onboarding back to its first-run state.
|
||||
*
|
||||
* Returns null when the endpoint is not there, so an older runtime degrades to
|
||||
* "not available" instead of failing the command. Only the completion flag
|
||||
* moves: memories, sources and settings live in other tables, which is what
|
||||
* makes this the honest way to see the cold-start screens again.
|
||||
*/
|
||||
export async function resetOnboarding(
|
||||
attachment: RuntimeAttachmentLike,
|
||||
): Promise<OnboardingState | null> {
|
||||
try {
|
||||
const response = await axios.post<OnboardingState>(
|
||||
`${baseUrlOf(attachment)}/api/v1/onboarding/reset`,
|
||||
undefined,
|
||||
{
|
||||
headers: { Authorization: `Bearer ${attachment.apiToken}` },
|
||||
timeout: 15_000,
|
||||
proxy: false,
|
||||
maxRedirects: 0,
|
||||
maxContentLength: MAX_RESPONSE_BYTES,
|
||||
},
|
||||
);
|
||||
return response.data;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function readScanRoots(
|
||||
attachment: RuntimeAttachmentLike,
|
||||
): Promise<{
|
||||
|
||||
@@ -30,7 +30,6 @@ const IMPORT_FILE =
|
||||
const MEMORY_NOTES_PATH = "/api/v1/memory-notes";
|
||||
const MEMORY_NOTE_PROCESS =
|
||||
/^\/api\/v1\/memory-notes\/[A-Za-z0-9][\w.:-]{0,127}\/process$/;
|
||||
const MODEL_CONTENT_PERMISSION_PATH = "/api/v1/model-content-permission";
|
||||
const EXTRACTION_SETTINGS_PATH = "/api/v1/extraction-settings";
|
||||
const GRAPH_WIDTH_PATH = "/api/v1/graph-width";
|
||||
const MANUAL_EXTRACT_PATH = "/api/v1/memory-processing/manual-extract";
|
||||
@@ -134,7 +133,6 @@ export function createCoreOperations(attachment: {
|
||||
!IMPORT_FILE_COMMIT.test(path) &&
|
||||
path !== MEMORY_NOTES_PATH &&
|
||||
!MEMORY_NOTE_PROCESS.test(path) &&
|
||||
path !== MODEL_CONTENT_PERMISSION_PATH &&
|
||||
path !== GRAPH_WIDTH_PATH &&
|
||||
path !== MANUAL_EXTRACT_PATH &&
|
||||
path !== ONBOARDING_COMPLETE_PATH &&
|
||||
|
||||
@@ -259,7 +259,7 @@ describe("A2 dual-mode init", () => {
|
||||
expect(streams.value()).toContain("处理:已开启");
|
||||
});
|
||||
|
||||
it("continues the TTY wizard through model probe and content permission", async () => {
|
||||
it("continues the TTY wizard through model setup and probe", async () => {
|
||||
const profileId = "22222222-2222-4222-8222-222222222222";
|
||||
const profile = {
|
||||
profile_id: profileId,
|
||||
@@ -361,12 +361,6 @@ describe("A2 dual-mode init", () => {
|
||||
};
|
||||
}
|
||||
if (url.endsWith("/activate")) return { status: 200, data: profile };
|
||||
if (url === "/api/v1/model-content-permission") {
|
||||
return {
|
||||
status: 200,
|
||||
data: { allowed: body.allowed },
|
||||
};
|
||||
}
|
||||
if (url === "/api/v1/onboarding/complete") {
|
||||
return {
|
||||
status: 200,
|
||||
@@ -383,7 +377,7 @@ describe("A2 dual-mode init", () => {
|
||||
},
|
||||
);
|
||||
const prompt = extendedPrompter(
|
||||
["", "", "", "", "", "", "", "", "n"],
|
||||
["", "", "", "", "", "", "", "n"],
|
||||
"sk-never-print-this",
|
||||
);
|
||||
const streams = captureStdout();
|
||||
@@ -402,7 +396,6 @@ describe("A2 dual-mode init", () => {
|
||||
streams.restore();
|
||||
}
|
||||
expect(streams.value()).toContain("已接入来源:codex");
|
||||
expect(prompt.written.join("\n")).toContain("模型内容外发:已允许");
|
||||
expect(streams.value()).toContain("首次设置完成。");
|
||||
expect(`${streams.value()}\n${prompt.written.join("\n")}`).not.toContain(
|
||||
"sk-never-print-this",
|
||||
@@ -412,10 +405,10 @@ describe("A2 dual-mode init", () => {
|
||||
expect.objectContaining({ api_key: "sk-never-print-this" }),
|
||||
undefined,
|
||||
);
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
expect(transport.post).not.toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
{ allowed: true },
|
||||
undefined,
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v1/onboarding/complete",
|
||||
|
||||
@@ -132,12 +132,6 @@ function clients(options: { failedProbes?: number } = {}) {
|
||||
if (path.endsWith("/activate")) {
|
||||
return { state: "committed" as const, data: profile(true) };
|
||||
}
|
||||
if (path === "/api/v1/model-content-permission") {
|
||||
return {
|
||||
state: "committed" as const,
|
||||
data: { allowed: body.allowed },
|
||||
};
|
||||
}
|
||||
return {
|
||||
state: "rejected" as const,
|
||||
code: "unexpected_path",
|
||||
@@ -160,8 +154,8 @@ function clients(options: { failedProbes?: number } = {}) {
|
||||
}
|
||||
|
||||
describe("interactive model setup", () => {
|
||||
it("uses App defaults, keeps the key out of output, and allows model content", async () => {
|
||||
const prompt = prompter(["", "", "", "", "", ""]);
|
||||
it("uses App defaults and keeps the key out of output", async () => {
|
||||
const prompt = prompter(["", "", ""]);
|
||||
const api = clients();
|
||||
|
||||
const result = await runInteractiveModelSetup(prompt.value, api, {
|
||||
@@ -170,10 +164,7 @@ describe("interactive model setup", () => {
|
||||
installHosts: false,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: true,
|
||||
});
|
||||
expect(result).toMatchObject({ state: "configured" });
|
||||
expect(api.post).toHaveBeenCalledWith(
|
||||
"/api/v1/model-profiles",
|
||||
expect.objectContaining({
|
||||
@@ -184,9 +175,10 @@ describe("interactive model setup", () => {
|
||||
embedding_provider_kind: "local_onnx",
|
||||
}),
|
||||
);
|
||||
expect(api.post).toHaveBeenCalledWith("/api/v1/model-content-permission", {
|
||||
allowed: true,
|
||||
});
|
||||
expect(api.post).not.toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
expect.anything(),
|
||||
);
|
||||
expect(prompt.written.join("\n")).not.toContain("sk-do-not-print");
|
||||
});
|
||||
|
||||
@@ -205,7 +197,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("removes an unverified profile before skipping after a failed probe", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "0"]);
|
||||
const prompt = prompter(["", "", "", "0"]);
|
||||
const api = clients();
|
||||
api.post.mockImplementation(
|
||||
async (path: string, body: Record<string, unknown>) => {
|
||||
@@ -280,7 +272,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("cleans a failed probe and lets the user re-enter the model settings", async () => {
|
||||
const prompt = prompter(["", "", "", "", "", "", "", "", "", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", "", "", ""]);
|
||||
const api = clients({ failedProbes: 1 });
|
||||
|
||||
const result = await runInteractiveModelSetup(prompt.value, api, {
|
||||
@@ -289,10 +281,7 @@ describe("interactive model setup", () => {
|
||||
installHosts: false,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: true,
|
||||
});
|
||||
expect(result).toMatchObject({ state: "configured" });
|
||||
expect(api.remove).toHaveBeenCalledTimes(1);
|
||||
expect(
|
||||
api.post.mock.calls.filter(([path]) => path === "/api/v1/model-profiles"),
|
||||
@@ -305,7 +294,7 @@ describe("interactive model setup", () => {
|
||||
const prompt = prompter(["", "", ""]);
|
||||
prompt.value.ask = async () => {
|
||||
asked += 1;
|
||||
if (asked === 4) throw new Error("interrupted");
|
||||
if (asked === 3) throw new Error("interrupted");
|
||||
return "";
|
||||
};
|
||||
const api = clients();
|
||||
@@ -322,7 +311,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("installs selected detected hosts through preview and fingerprint apply", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -360,7 +349,6 @@ describe("interactive model setup", () => {
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: ["codex"],
|
||||
});
|
||||
expect(read).toHaveBeenNthCalledWith(
|
||||
@@ -373,7 +361,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("lists and installs only hosts that are actually present", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -420,7 +408,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("completes the wizard and keeps onboarding reachable when one host install fails", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -469,7 +457,6 @@ describe("interactive model setup", () => {
|
||||
// calling the install a failure.
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: ["codex"],
|
||||
});
|
||||
const summary = prompt.written.join("\n");
|
||||
@@ -479,7 +466,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("tells the user to install App-only hosts in the App instead of calling them failed", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -506,7 +493,6 @@ describe("interactive model setup", () => {
|
||||
// only the App can install, so nothing counts as "installed".
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: [],
|
||||
});
|
||||
const summary = prompt.written.join("\n");
|
||||
|
||||
@@ -11,7 +11,6 @@ import {
|
||||
} from "./headless-flows.js";
|
||||
import { approvalFingerprint, listHosts, readHost } from "./host-flow.js";
|
||||
import { parseSelection, type WizardPrompter } from "./init-wizard.js";
|
||||
import { setModelContentPermission } from "./model-content-permission-flow.js";
|
||||
import { executeModelProbe, planModelProbe } from "./model-probe-flow.js";
|
||||
import {
|
||||
executeModelProfileActivate,
|
||||
@@ -122,7 +121,6 @@ export type InteractiveModelSetupState =
|
||||
| {
|
||||
state: "configured";
|
||||
profile: ModelProfile;
|
||||
model_content_allowed: boolean;
|
||||
installed_hosts: string[];
|
||||
}
|
||||
| { state: "skipped" }
|
||||
@@ -289,15 +287,11 @@ async function installHosts(
|
||||
function writeSummary(
|
||||
prompter: WizardPrompter,
|
||||
profile: ModelProfile,
|
||||
modelContentAllowed: boolean,
|
||||
installedHosts: readonly string[],
|
||||
failedHosts: readonly { hostId: string; label: string }[],
|
||||
): void {
|
||||
prompter.write("配置摘要:");
|
||||
prompter.write(` 模型:${profile.name} / ${profile.model}`);
|
||||
prompter.write(
|
||||
` 模型内容外发:${modelContentAllowed ? "已允许" : "已禁止"}`,
|
||||
);
|
||||
prompter.write(
|
||||
` 应用接入:${installedHosts.length > 0 ? installedHosts.join("、") : "已跳过"}`,
|
||||
);
|
||||
@@ -353,14 +347,6 @@ export async function runInteractiveModelSetup(
|
||||
prompter.write("未填写 API Key。输入 back 返回,或重新选择模型服务。");
|
||||
continue;
|
||||
}
|
||||
const permissionAnswer = (
|
||||
await prompter.ask("允许将对话内容发送给模型以整理记忆?[Y/n]: ")
|
||||
)
|
||||
.trim()
|
||||
.toLowerCase();
|
||||
const modelContentAllowed = !["n", "no", "否", "0"].includes(
|
||||
permissionAnswer,
|
||||
);
|
||||
const hostApi = options.hostApi ?? { list: listHosts, read: readHost };
|
||||
const selectedHosts = options.installHosts
|
||||
? await chooseDetectedHosts(prompter, options.hostHome, hostApi)
|
||||
@@ -452,13 +438,6 @@ export async function runInteractiveModelSetup(
|
||||
return { state: "failed", code: "model_activate_failed" };
|
||||
}
|
||||
|
||||
const permission = await setModelContentPermission(
|
||||
clients.operations,
|
||||
modelContentAllowed,
|
||||
);
|
||||
if (permission.state !== "ok") {
|
||||
return { state: "failed", code: "model_content_permission_set_failed" };
|
||||
}
|
||||
const { installed, failed } = await installHosts(
|
||||
options.dataDir,
|
||||
selectedHosts,
|
||||
@@ -467,7 +446,6 @@ export async function runInteractiveModelSetup(
|
||||
writeSummary(
|
||||
prompter,
|
||||
profile,
|
||||
modelContentAllowed,
|
||||
installed,
|
||||
failed.map((hostId) => ({
|
||||
hostId,
|
||||
@@ -477,7 +455,6 @@ export async function runInteractiveModelSetup(
|
||||
return {
|
||||
state: "configured",
|
||||
profile,
|
||||
model_content_allowed: modelContentAllowed,
|
||||
installed_hosts: installed,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -180,5 +180,8 @@ describe("import-job write allow-list", () => {
|
||||
await expect(
|
||||
operations.post("/api/v1/import-jobs/job-1/delete", {}),
|
||||
).rejects.toThrow("core_operation_path_not_allowed");
|
||||
await expect(
|
||||
operations.post("/api/v1/model-content-permission", { allowed: true }),
|
||||
).rejects.toThrow("core_operation_path_not_allowed");
|
||||
});
|
||||
});
|
||||
@@ -1,154 +0,0 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { run } from "./cli.js";
|
||||
|
||||
const transport = vi.hoisted(() => ({
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
put: vi.fn(),
|
||||
patch: vi.fn(),
|
||||
delete: vi.fn(),
|
||||
}));
|
||||
vi.mock("axios", () => ({
|
||||
default: { create: vi.fn(() => transport), get: vi.fn() },
|
||||
}));
|
||||
|
||||
const runtime = vi.hoisted(() => ({
|
||||
inspectRuntimeStatus: vi.fn(),
|
||||
discoverRuntimeAttachment: vi.fn(),
|
||||
}));
|
||||
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
|
||||
...runtime,
|
||||
}));
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
runtime.inspectRuntimeStatus.mockResolvedValue({
|
||||
state: "running",
|
||||
code: "runtime_verified",
|
||||
instanceId: "instance-1",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
version: "1.2.3",
|
||||
});
|
||||
runtime.discoverRuntimeAttachment.mockResolvedValue({
|
||||
instanceId: "instance-1",
|
||||
apiToken: "synthetic-runtime-token",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
servicePid: 4242,
|
||||
health: {
|
||||
status: "ok",
|
||||
service: "one-memory-core",
|
||||
version: "1.2.3",
|
||||
instance_id: "instance-1",
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
describe("model content permission CLI", () => {
|
||||
it("documents only final-state permission and manual extraction commands", async () => {
|
||||
const permission = await capturedRun(["help", "permission"]);
|
||||
expect(permission.code).toBe(0);
|
||||
expect(permission.stdout).toContain("memhub permission allow");
|
||||
expect(permission.stdout).toContain("memhub permission deny");
|
||||
expect(permission.stdout).not.toMatch(
|
||||
/\b(preview|confirm|process|revoke)\b/,
|
||||
);
|
||||
expect(permission.stdout).not.toContain("revision");
|
||||
expect(permission.stdout).not.toContain("receipt");
|
||||
|
||||
const processing = await capturedRun(["help", "processing"]);
|
||||
expect(processing.code).toBe(0);
|
||||
expect(processing.stdout).toContain("memhub processing extract");
|
||||
expect(processing.stdout).not.toMatch(/\b(plan|apply|receipt)\b/);
|
||||
expect(processing.stdout).not.toContain("backlog");
|
||||
});
|
||||
|
||||
it("reads the final state", async () => {
|
||||
transport.get.mockResolvedValue({ status: 200, data: { allowed: true } });
|
||||
|
||||
const result = await capturedRun(["permission", "status", "--json"]);
|
||||
|
||||
expect(result.code).toBe(0);
|
||||
expect(JSON.parse(result.stdout)).toMatchObject({
|
||||
state: "ok",
|
||||
allowed: true,
|
||||
});
|
||||
expect(transport.get).toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["allow", true],
|
||||
["deny", false],
|
||||
] as const)("sets %s directly with only allowed", async (verb, allowed) => {
|
||||
transport.post.mockResolvedValue({ status: 200, data: { allowed } });
|
||||
|
||||
const result = await capturedRun(["permission", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(0);
|
||||
expect(JSON.parse(result.stdout)).toMatchObject({ state: "ok", allowed });
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
{ allowed },
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
|
||||
it.each(["preview", "confirm", "process", "revoke"])(
|
||||
"rejects removed %s commands before runtime access",
|
||||
async (verb) => {
|
||||
const result = await capturedRun(["permission", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(2);
|
||||
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
|
||||
expect(transport.get).not.toHaveBeenCalled();
|
||||
expect(transport.post).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["plan", "apply", "receipt"])(
|
||||
"rejects removed processing %s commands before runtime access",
|
||||
async (verb) => {
|
||||
const result = await capturedRun(["processing", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(2);
|
||||
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
|
||||
expect(transport.get).not.toHaveBeenCalled();
|
||||
expect(transport.post).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
async function capturedRun(arguments_: string[]) {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const stdoutSpy = vi
|
||||
.spyOn(process.stdout, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stdout += String(chunk);
|
||||
return true;
|
||||
});
|
||||
const stderrSpy = vi
|
||||
.spyOn(process.stderr, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stderr += String(chunk);
|
||||
return true;
|
||||
});
|
||||
try {
|
||||
return {
|
||||
code: await run(arguments_, {
|
||||
ONE_MEMORY_DATA_DIR: "/synthetic/data",
|
||||
}),
|
||||
stdout,
|
||||
stderr,
|
||||
};
|
||||
} finally {
|
||||
stdoutSpy.mockRestore();
|
||||
stderrSpy.mockRestore();
|
||||
}
|
||||
}
|
||||
@@ -1,66 +0,0 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import type { CoreReader } from "./core-client.js";
|
||||
import type { CoreOperations } from "./core-operations.js";
|
||||
import {
|
||||
readModelContentPermission,
|
||||
setModelContentPermission,
|
||||
} from "./model-content-permission-flow.js";
|
||||
|
||||
describe("model content permission flow", () => {
|
||||
it("reads the single final-state boolean", async () => {
|
||||
const reader = vi.fn().mockResolvedValue({
|
||||
state: "ok",
|
||||
data: { allowed: true },
|
||||
});
|
||||
|
||||
await expect(
|
||||
readModelContentPermission(reader as CoreReader),
|
||||
).resolves.toMatchObject({ state: "ok", allowed: true });
|
||||
expect(reader).toHaveBeenCalledWith("/api/v1/model-content-permission");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["allow", true],
|
||||
["deny", false],
|
||||
] as const)(
|
||||
"sets %s directly without revision or receipt fields",
|
||||
async (_, allowed) => {
|
||||
const post = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed },
|
||||
});
|
||||
|
||||
await expect(
|
||||
setModelContentPermission({ post } as CoreOperations, allowed),
|
||||
).resolves.toMatchObject({ state: "ok", allowed });
|
||||
expect(post).toHaveBeenCalledWith("/api/v1/model-content-permission", {
|
||||
allowed,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("does not accept a malformed or mismatched success response", async () => {
|
||||
const malformed = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed: true, revision: 1 },
|
||||
});
|
||||
await expect(
|
||||
setModelContentPermission({ post: malformed } as CoreOperations, true),
|
||||
).resolves.toMatchObject({
|
||||
state: "unknown",
|
||||
code: "model_content_permission_response_invalid",
|
||||
});
|
||||
|
||||
const mismatched = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed: false },
|
||||
});
|
||||
await expect(
|
||||
setModelContentPermission({ post: mismatched } as CoreOperations, true),
|
||||
).resolves.toMatchObject({
|
||||
state: "unknown",
|
||||
code: "model_content_permission_state_mismatch",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,96 +0,0 @@
|
||||
import type { CoreReader } from "./core-client.js";
|
||||
import type { CoreOperations } from "./core-operations.js";
|
||||
import { PRODUCT } from "./version.js";
|
||||
|
||||
export type ModelContentPermissionDocument =
|
||||
| {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
state: "ok";
|
||||
allowed: boolean;
|
||||
}
|
||||
| {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
state: "rejected" | "unknown";
|
||||
code: string;
|
||||
};
|
||||
|
||||
export async function readModelContentPermission(
|
||||
reader: CoreReader,
|
||||
): Promise<ModelContentPermissionDocument> {
|
||||
const outcome = await reader<unknown>("/api/v1/model-content-permission");
|
||||
if (outcome.state !== "ok") {
|
||||
return failure("unknown", outcome.code);
|
||||
}
|
||||
return permissionState(
|
||||
outcome.data,
|
||||
"model_content_permission_response_invalid",
|
||||
);
|
||||
}
|
||||
|
||||
export async function setModelContentPermission(
|
||||
operations: CoreOperations,
|
||||
allowed: boolean,
|
||||
): Promise<ModelContentPermissionDocument> {
|
||||
const outcome = await operations.post<unknown>(
|
||||
"/api/v1/model-content-permission",
|
||||
{ allowed },
|
||||
);
|
||||
if (outcome.state === "rejected") {
|
||||
return failure("rejected", outcome.code);
|
||||
}
|
||||
if (outcome.state === "unknown") {
|
||||
return failure("unknown", outcome.code);
|
||||
}
|
||||
const document = permissionState(
|
||||
outcome.data,
|
||||
"model_content_permission_response_invalid",
|
||||
);
|
||||
return document.state === "ok" && document.allowed !== allowed
|
||||
? failure("unknown", "model_content_permission_state_mismatch")
|
||||
: document;
|
||||
}
|
||||
|
||||
export function modelContentPermissionExitCode(
|
||||
document: ModelContentPermissionDocument,
|
||||
): number {
|
||||
return document.state === "ok" ? 0 : 1;
|
||||
}
|
||||
|
||||
export function renderModelContentPermissionText(
|
||||
document: ModelContentPermissionDocument,
|
||||
): string {
|
||||
if (document.state === "ok") {
|
||||
return `模型内容外发:${document.allowed ? "允许" : "禁止"}\n`;
|
||||
}
|
||||
return `模型内容权限未更新:${document.state}(${document.code})\n`;
|
||||
}
|
||||
|
||||
function permissionState(
|
||||
value: unknown,
|
||||
invalidCode: string,
|
||||
): ModelContentPermissionDocument {
|
||||
if (
|
||||
value === null ||
|
||||
typeof value !== "object" ||
|
||||
Array.isArray(value) ||
|
||||
Object.keys(value).length !== 1 ||
|
||||
typeof (value as Record<string, unknown>).allowed !== "boolean"
|
||||
) {
|
||||
return failure("unknown", invalidCode);
|
||||
}
|
||||
return {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
state: "ok",
|
||||
allowed: (value as { allowed: boolean }).allowed,
|
||||
};
|
||||
}
|
||||
|
||||
function failure(
|
||||
state: "rejected" | "unknown",
|
||||
code: string,
|
||||
): ModelContentPermissionDocument {
|
||||
return { schema_version: 1, product: PRODUCT, state, code };
|
||||
}
|
||||
@@ -137,8 +137,8 @@ describe("package matrix harness", () => {
|
||||
};
|
||||
expect(document.state).toBe("verified");
|
||||
const results = document.commands.stopped_destructive.results;
|
||||
// 13 destructive commands, each unknown/runtime_not_running with exit 3.
|
||||
expect(Object.keys(results)).toHaveLength(13);
|
||||
// 11 destructive commands, each unknown/runtime_not_running with exit 3.
|
||||
expect(Object.keys(results)).toHaveLength(11);
|
||||
for (const [name, result] of Object.entries(results)) {
|
||||
expect(result, `${name} did not fail closed`).toEqual({
|
||||
exit: 3,
|
||||
@@ -162,7 +162,7 @@ describe("package matrix harness", () => {
|
||||
const shim = [
|
||||
"#!/usr/bin/env node",
|
||||
"const a=process.argv.slice(2);",
|
||||
"if(['calls','import-file','notes','permission','extraction','graph','models'].includes(a[0])||(a[0]==='processing'&&a[1]==='extract')){",
|
||||
"if(['calls','import-file','notes','extraction','graph','models'].includes(a[0])||(a[0]==='processing'&&a[1]==='extract')){",
|
||||
' process.stdout.write(\'{"schema_version":1,"product":"memhub","state":"committed","operation_id":"x","receipt":{}}\\n\');',
|
||||
" process.exit(0);",
|
||||
"}",
|
||||
@@ -188,7 +188,7 @@ describe("package matrix harness", () => {
|
||||
expect(failure?.state).toBe("failed");
|
||||
expect(
|
||||
failure?.failures.filter((item) => item.includes("did not fail closed")),
|
||||
).toHaveLength(13);
|
||||
).toHaveLength(11);
|
||||
});
|
||||
|
||||
it("can require an expected bundle SHA and reports bundle metadata", {
|
||||
@@ -280,9 +280,8 @@ describe("package matrix harness", () => {
|
||||
});
|
||||
|
||||
it("rejects a catalog that declares the parent command but not the new verb", async () => {
|
||||
// A bundle can still declare `permission`/`models` while missing
|
||||
// `permission allow` or `models activate`. Names alone would pass it; the
|
||||
// usage marker must not.
|
||||
// A bundle can still declare `models` while missing `models activate`.
|
||||
// Names alone would pass it; the usage marker must not.
|
||||
const declared = [
|
||||
"install",
|
||||
"init",
|
||||
@@ -292,7 +291,6 @@ describe("package matrix harness", () => {
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"models",
|
||||
@@ -342,9 +340,6 @@ describe("package matrix harness", () => {
|
||||
},
|
||||
};
|
||||
const failures = harness.validateCatalog(catalog);
|
||||
expect(failures).toContain(
|
||||
'command "permission" is missing usage "memhub permission allow"',
|
||||
);
|
||||
expect(failures).toContain(
|
||||
'command "models" is missing usage "memhub models activate"',
|
||||
);
|
||||
|
||||
@@ -151,9 +151,6 @@ describe("memhub short-command contract", () => {
|
||||
"memhub import-file discard",
|
||||
"memhub notes add",
|
||||
"memhub notes process",
|
||||
"memhub permission status",
|
||||
"memhub permission allow",
|
||||
"memhub permission deny",
|
||||
"memhub models setup",
|
||||
"memhub models activate",
|
||||
"memhub models edit",
|
||||
@@ -163,6 +160,9 @@ describe("memhub short-command contract", () => {
|
||||
]) {
|
||||
expect(flattened).toContain(shortForm);
|
||||
}
|
||||
expect(COMMANDS.some((command) => command.name === "permission")).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("ships a memhub bin entry pointing at a real bundle path", () => {
|
||||
|
||||
@@ -840,6 +840,7 @@ describe("CLI-21 memory reads", () => {
|
||||
expect(isReadOnlyPath("/api/v1/memories/graph")).toBe(false);
|
||||
expect(isReadOnlyPath("/api/v1/memories/ask")).toBe(false);
|
||||
expect(isReadOnlyPath("/api/v1/memories/m1")).toBe(true);
|
||||
expect(isReadOnlyPath("/api/v1/model-content-permission")).toBe(false);
|
||||
expect(
|
||||
isReadOnlyPath(
|
||||
"/api/v1/memory-mutations/123e4567-e89b-12d3-a456-426614174000",
|
||||
|
||||
@@ -11,7 +11,7 @@ folder_layout = layout == 'folder'
|
||||
datas = [('src/one_memory/embedding_assets', 'one_memory/embedding_assets')]
|
||||
binaries = []
|
||||
# Loaded lazily by the API; static dependency analysis cannot discover it.
|
||||
hiddenimports = ['one_memory.hy_permission_context']
|
||||
hiddenimports = []
|
||||
tmp_ret = collect_all('hy_memory')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('chromadb')
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "uv_build"
|
||||
|
||||
[project]
|
||||
name = "one-memory"
|
||||
version = "1.0.2"
|
||||
version = "1.0.3"
|
||||
description = "MemHub local-first memory core"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
@@ -12,7 +12,7 @@ dependencies = [
|
||||
"fastapi>=0.141.1",
|
||||
"filelock>=3.32.4",
|
||||
# Public SDK version validated against MemHub's permission and recall contracts.
|
||||
"hy-memory==2.0.2",
|
||||
"hy-memory==2.0.3",
|
||||
# Retrieval's derived vector index directly consumes these locked versions.
|
||||
"chromadb==1.5.9",
|
||||
"numpy==2.5.2",
|
||||
|
||||
@@ -20,7 +20,6 @@ from copy import copy
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from enum import StrEnum
|
||||
from functools import partial
|
||||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, cast
|
||||
from uuid import UUID
|
||||
@@ -164,7 +163,6 @@ from one_memory.memory_library import (
|
||||
)
|
||||
from one_memory.memory_notes import MemoryNoteStore, memory_note_router
|
||||
from one_memory.memory_processing import (
|
||||
MemoryBridgeProtocol,
|
||||
MemoryProcessingSupervisor,
|
||||
_conversation_chunks,
|
||||
memory_task_timeout_seconds,
|
||||
@@ -178,10 +176,8 @@ from one_memory.memory_queue import (
|
||||
)
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
from one_memory.memory_tasks import MemoryTask, MemoryTaskStatus, MemoryTaskStore
|
||||
from one_memory.model_content_routes import model_content_router
|
||||
from one_memory.model_presets import seed_preset_profile
|
||||
from one_memory.model_probe import ModelProber, ModelProbeResult
|
||||
from one_memory.model_processing_permission_guard import ModelProcessingPermissionLease
|
||||
from one_memory.model_profiles import (
|
||||
ActiveModelProfileError,
|
||||
ModelCreationOperationConflictError,
|
||||
@@ -818,6 +814,13 @@ def create_app(
|
||||
evidence=app.state.evidence_store,
|
||||
)
|
||||
app.state.memory_write_queue = memory_write_queue
|
||||
# Builds through 1.0.2 parked pre-invocation jobs behind the
|
||||
# retired model-content consent bit. The embedded SDK is trusted
|
||||
# now, so make those durable jobs claimable on the first startup.
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.readmit_parked,
|
||||
states=("waiting_for_permission",),
|
||||
)
|
||||
repaired_identity_failures = await asyncio.to_thread(
|
||||
memory_write_queue.repair_pre_send_identity_failures
|
||||
)
|
||||
@@ -960,7 +963,7 @@ def create_app(
|
||||
# Not fatal: without the tables every read checks exactly, as
|
||||
# it does today.
|
||||
app.state.memory_visibility_error = str(error)
|
||||
app.state.model_request_budget = (
|
||||
model_request_budget = (
|
||||
ModelRequestBudget(
|
||||
database_path,
|
||||
request_limit=runtime_settings.model_request_limit,
|
||||
@@ -970,6 +973,7 @@ def create_app(
|
||||
if runtime_settings.model_request_limit is not None
|
||||
else None
|
||||
)
|
||||
app.state.model_request_budget = model_request_budget
|
||||
app.state.memory_answerer = MemoryAnswerer(
|
||||
governance=app.state.memory_governance_store,
|
||||
filter_memories=app.state.source_read_access.memories,
|
||||
@@ -1008,6 +1012,16 @@ def create_app(
|
||||
governance=app.state.memory_governance_store,
|
||||
data_dir=runtime_settings.data_dir,
|
||||
extraction_settings=app.state.extraction_settings_store,
|
||||
bridge_factory=lambda profile, api_key, data_dir: HyMemoryBridge(
|
||||
profile=profile,
|
||||
api_key=api_key,
|
||||
data_dir=data_dir,
|
||||
http_transport_factory=(
|
||||
(lambda: BudgetedModelTransport(model_request_budget))
|
||||
if model_request_budget is not None
|
||||
else None
|
||||
),
|
||||
),
|
||||
# `None` means "whatever the active profile allows". Pinned to
|
||||
# 1 in `a59f4b5` as temporary load-shedding for an acceptance
|
||||
# package, and left pinned after it: a hardcoded 1 is rule 9's
|
||||
@@ -1016,10 +1030,6 @@ def create_app(
|
||||
# profile concurrency 4, recall latency is unchanged while
|
||||
# precipitation goes 4x.
|
||||
max_concurrent_tasks=runtime_settings.precipitation_max_concurrent_tasks,
|
||||
model_content_config=app.state.config_store,
|
||||
permission_bridge_factory=lambda lease, secret, directory: _permission_context(
|
||||
app, lease, secret, directory
|
||||
),
|
||||
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
|
||||
service_levels=app.state.service_levels,
|
||||
)
|
||||
@@ -1039,10 +1049,15 @@ def create_app(
|
||||
job.job_id,
|
||||
)
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.retry_timed_out_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(minutes=5),
|
||||
memory_write_queue.retry_first_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(seconds=10),
|
||||
limit=32,
|
||||
)
|
||||
await asyncio.to_thread(
|
||||
app.state.memory_task_store.retry_first_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(seconds=10),
|
||||
limit=128,
|
||||
)
|
||||
|
||||
async def admit_hy_invocation(
|
||||
source_id: str,
|
||||
@@ -1134,7 +1149,6 @@ def create_app(
|
||||
app.state.matter_discovery.processor = app.state.memory_processing_supervisor
|
||||
app.state.matter_discovery.write_intake = app.state.memory_write_intake
|
||||
app.state.matter_discovery.write_queue = memory_write_queue
|
||||
app.state.processing_coordinator = None
|
||||
app.state.memory_note_store = MemoryNoteStore(
|
||||
app.state.evidence_store,
|
||||
app.state.memory_processing_supervisor,
|
||||
@@ -1446,7 +1460,6 @@ def create_app(
|
||||
app.include_router(matter_router(protected))
|
||||
app.include_router(memory_note_router(protected))
|
||||
app.include_router(file_import_router(protected))
|
||||
app.include_router(model_content_router(protected))
|
||||
app.include_router(context_router(protected))
|
||||
|
||||
def require_workbuddy_boundary(request: Request) -> None:
|
||||
@@ -1873,6 +1886,20 @@ def create_app(
|
||||
"""Record UI completion only; finishing or skipping is never model consent."""
|
||||
return await asyncio.to_thread(_onboarding_store(app).complete)
|
||||
|
||||
@app.post(
|
||||
"/api/v1/onboarding/reset",
|
||||
response_model=OnboardingState,
|
||||
dependencies=protected,
|
||||
)
|
||||
async def reset_onboarding() -> OnboardingState:
|
||||
"""Go back to the first-run screens. Only the completion flag moves.
|
||||
|
||||
Memories, sources and settings live in other tables and are untouched,
|
||||
which is the point: someone who wants to see the cold-start flow again
|
||||
should not have to delete anything to do it.
|
||||
"""
|
||||
return await asyncio.to_thread(_onboarding_store(app).reset)
|
||||
|
||||
@app.get(
|
||||
"/api/v1/sources",
|
||||
response_model=list[SourceResponse],
|
||||
@@ -2328,7 +2355,7 @@ def create_app(
|
||||
await asyncio.to_thread(
|
||||
app.state.memory_task_store.park_source_for_permission,
|
||||
source_id=source_id,
|
||||
error_code="model_content_permission_required",
|
||||
error_code="hy_memory_source_revoked",
|
||||
)
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
_supervisor(app).unregister_adapter(source_id)
|
||||
@@ -3894,12 +3921,7 @@ async def _authorize_answer_generation(
|
||||
# The explicit question is this operation's input, not a standing permission for history.
|
||||
if not question.strip() or not grounding:
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
if not await asyncio.to_thread(_config_store(app).model_content_allowed):
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
|
||||
def check() -> None:
|
||||
if not _config_store(app).model_content_allowed():
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
if _model_profile_store(app).active_identity() != (
|
||||
profile.profile_id,
|
||||
profile.revision,
|
||||
@@ -3922,30 +3944,6 @@ async def _authorize_answer_generation(
|
||||
await recheck()
|
||||
return recheck
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _permission_context(
|
||||
app: FastAPI,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
secret: str | None,
|
||||
directory: Path,
|
||||
) -> AsyncIterator[MemoryBridgeProtocol]:
|
||||
module = await asyncio.to_thread(import_module, "one_memory.hy_permission_context")
|
||||
budget = cast(ModelRequestBudget | None, app.state.model_request_budget)
|
||||
source_id = lease.records[0].source_id if lease.records else "manual"
|
||||
factory = module.permission_bridge_factory(
|
||||
lambda profile, api_key, data_dir: _memory_processing_supervisor(app)._bridge(
|
||||
profile,
|
||||
api_key,
|
||||
data_dir,
|
||||
scope_key=source_id,
|
||||
),
|
||||
**({"http_transport": BudgetedModelTransport(budget)} if budget else {}),
|
||||
)
|
||||
async with factory(lease, secret, directory) as bridge:
|
||||
yield cast(MemoryBridgeProtocol, bridge)
|
||||
|
||||
|
||||
class _LazyASGIApp:
|
||||
"""Compatibility ASGI app that is built only when a server imports it.
|
||||
|
||||
@@ -4723,8 +4721,6 @@ async def _run_backfill(app: FastAPI) -> None:
|
||||
app.state.memory_backfill_error = None
|
||||
if not app.state.settings.automatic_memory_processing:
|
||||
return
|
||||
if not await asyncio.to_thread(_config_store(app).model_content_allowed):
|
||||
return
|
||||
try:
|
||||
queue = app.state.memory_write_queue
|
||||
intake = app.state.memory_write_intake
|
||||
|
||||
@@ -21,7 +21,6 @@ from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception, elapsed_ms
|
||||
from one_memory.hy_lifecycle import HyLifecycle, HyLifecycleUnreadable, read_hy_lifecycle
|
||||
from one_memory.hy_permission_store import native_store_read_slot
|
||||
from one_memory.hy_presence import HyPresence, confirm_deleted
|
||||
from one_memory.hy_runtime_config import (
|
||||
HY_MEMORY_MODE,
|
||||
@@ -32,6 +31,7 @@ from one_memory.hy_runtime_config import (
|
||||
diagnostic_log_level,
|
||||
)
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
from one_memory.hy_store_concurrency import native_store_read_slot
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind, attach_disable_thinking
|
||||
from one_memory.semantic_deadline import SemanticMemoryHit
|
||||
|
||||
@@ -376,11 +376,13 @@ class HyMemoryBridge:
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
client: HyMemoryClientProtocol | None = None,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
) -> None:
|
||||
self.profile = profile
|
||||
self.api_key = api_key
|
||||
self.data_dir = data_dir
|
||||
self._client = client
|
||||
self._http_transport_factory = http_transport_factory
|
||||
self._client_lock = asyncio.Lock()
|
||||
self._write_lock = asyncio.Lock()
|
||||
self._last_bindings: HyWriteBindings | None = None
|
||||
@@ -427,6 +429,13 @@ class HyMemoryBridge:
|
||||
)
|
||||
client = await self._get_client()
|
||||
guards = getattr(client, "_one_memory_write_guards", ())
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
reconcile_guards = tuple(
|
||||
guard for guard in guards if isinstance(guard, HyReconcileGuard)
|
||||
)
|
||||
for guard in reconcile_guards:
|
||||
guard.begin_write()
|
||||
before = [len(guard.failures) for guard in guards]
|
||||
bindings = HyWriteBindings(request.source_context) if request.source_context else None
|
||||
if bindings:
|
||||
@@ -500,6 +509,12 @@ class HyMemoryBridge:
|
||||
if len(guard.failures) > count:
|
||||
raise HyMemoryBridgeError(guard.failures[count])
|
||||
result = await self._result(client, request, raw, expected_contents=None, bindings=bindings)
|
||||
result = await _with_native_mutation_delta(
|
||||
client,
|
||||
result,
|
||||
raw,
|
||||
reconcile_guards,
|
||||
)
|
||||
if bindings:
|
||||
outputs = []
|
||||
for memory in result.memories:
|
||||
@@ -553,6 +568,8 @@ class HyMemoryBridge:
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
report_guards = tuple(guard for guard in guards if isinstance(guard, HyReconcileGuard))
|
||||
for guard in report_guards:
|
||||
guard.begin_write()
|
||||
expected_contents = frozenset(content.strip() for content in contents if content.strip())
|
||||
for guard in report_guards:
|
||||
guard.imported_contents = expected_contents
|
||||
@@ -599,6 +616,12 @@ class HyMemoryBridge:
|
||||
raw,
|
||||
expected_contents=expected_contents,
|
||||
)
|
||||
result = await _with_native_mutation_delta(
|
||||
client,
|
||||
result,
|
||||
raw,
|
||||
report_guards,
|
||||
)
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="hy.bridge",
|
||||
@@ -1087,6 +1110,7 @@ class HyMemoryBridge:
|
||||
self.profile,
|
||||
api_key=self.api_key,
|
||||
data_dir=self.data_dir,
|
||||
http_transport_factory=self._http_transport_factory,
|
||||
)
|
||||
|
||||
await _joined_thread(initialize)
|
||||
@@ -1136,6 +1160,7 @@ def create_hy_memory_client(
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
) -> HyMemoryClientProtocol:
|
||||
configure_hy_memory_environment(data_dir)
|
||||
config = build_hy_memory_config(
|
||||
@@ -1143,6 +1168,7 @@ def create_hy_memory_client(
|
||||
api_key=api_key,
|
||||
data_dir=data_dir,
|
||||
)
|
||||
config.http_transport_factory = http_transport_factory
|
||||
reattach_legacy_vector_collection(profile, config)
|
||||
mismatch = inspect_graph_width_mismatch(config)
|
||||
if mismatch is not None:
|
||||
@@ -1287,6 +1313,74 @@ async def _coding_result(
|
||||
return HyMemoryIngestResult(success=True, memories=tuple(outputs))
|
||||
|
||||
|
||||
async def _with_native_mutation_delta(
|
||||
client: HyMemoryClientProtocol,
|
||||
result: HyMemoryIngestResult,
|
||||
raw: Mapping[str, object],
|
||||
reconcile_guards: Sequence[Any],
|
||||
) -> HyMemoryIngestResult:
|
||||
"""Map native SDK mutations back into MemHub governance.
|
||||
|
||||
Store permission wrappers used to discover this by intercepting every
|
||||
native CRUD call. The embedded SDK now owns unrestricted local CRUD, so
|
||||
chat targets come from the reconcile observer and coding targets come from
|
||||
the SDK's public result document.
|
||||
"""
|
||||
|
||||
retired = set(result.retired_memory_ids)
|
||||
superseded = {
|
||||
key: tuple(values) for key, values in result.superseded_memory_ids.items()
|
||||
}
|
||||
chat_targets = {
|
||||
key
|
||||
for guard in reconcile_guards
|
||||
for key in getattr(guard, "mutation_target_ids", ())
|
||||
if isinstance(key, str) and key
|
||||
}
|
||||
|
||||
for item in raw.get("ops", ()) if isinstance(raw.get("ops"), list) else ():
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
if item.get("action") == "DELETE":
|
||||
key = item.get("target_experience_id")
|
||||
if isinstance(key, str) and key:
|
||||
retired.add(key)
|
||||
|
||||
for item in raw.get("fact_ops", ()) if isinstance(raw.get("fact_ops"), list) else ():
|
||||
if not isinstance(item, Mapping) or item.get("action") != "SUPERSEDE":
|
||||
continue
|
||||
old, new = item.get("fact_id"), item.get("superseded_by")
|
||||
if isinstance(old, str) and old:
|
||||
retired.add(old)
|
||||
if isinstance(new, str) and new:
|
||||
superseded[old] = (new,)
|
||||
|
||||
for key in chat_targets:
|
||||
record = await _joined_thread(_read_hy_record, client, key)
|
||||
if record is None:
|
||||
retired.add(key)
|
||||
continue
|
||||
status = str(record.get("status", "")).lower()
|
||||
if status in {"deleted", "shadow", "superseded"}:
|
||||
retired.add(key)
|
||||
replacement = record.get("superseded_by")
|
||||
if isinstance(replacement, str) and replacement:
|
||||
superseded[key] = (replacement,)
|
||||
elif isinstance(replacement, list):
|
||||
values = tuple(
|
||||
value for value in replacement if isinstance(value, str) and value
|
||||
)
|
||||
if values:
|
||||
superseded[key] = values
|
||||
|
||||
return result.model_copy(
|
||||
update={
|
||||
"retired_memory_ids": tuple(sorted(retired)),
|
||||
"superseded_memory_ids": superseded,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _read_hy_record(
|
||||
client: HyMemoryClientProtocol,
|
||||
memory_id: str,
|
||||
|
||||
@@ -1,607 +0,0 @@
|
||||
"""Operation-bound read/write capability for the pinned native HY coding store.
|
||||
|
||||
HY's coding route writes two layers, experiences and working facts. Rows this
|
||||
lease cannot attest stay in the same backend but are invisible here, so they
|
||||
never reach a coding prompt and cannot be updated, superseded or linked.
|
||||
|
||||
HY's reconcilers catch store exceptions and carry on, so a denial latches and
|
||||
recheck() raises it again at finish; a swallowed denial still fails the task.
|
||||
Any native method not implemented here is denied rather than passed through.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from collections.abc import Iterator, Mapping, Sequence
|
||||
from copy import deepcopy
|
||||
from hashlib import sha256
|
||||
from types import MappingProxyType
|
||||
from typing import Any, NoReturn
|
||||
|
||||
from hy_memory.coding.types import CodingMemory, WorkingFact # type: ignore[import-untyped]
|
||||
|
||||
from one_memory.hy_permission_store import HyNodePermission
|
||||
from one_memory.model_processing_permission_guard import (
|
||||
ModelProcessingPermissionLease,
|
||||
permission_error,
|
||||
)
|
||||
|
||||
_NATIVE_WRITES = frozenset(
|
||||
{
|
||||
"insert",
|
||||
"update",
|
||||
"delete",
|
||||
"insert_fact",
|
||||
"supersede_fact",
|
||||
"set_fact_derived_experiences",
|
||||
"set_experience_source_facts",
|
||||
# Trace rows are raw conversation: writes must be gated like any other.
|
||||
"append_trace",
|
||||
}
|
||||
)
|
||||
|
||||
_TIMESTAMPS = frozenset({"created_at", "updated_at"})
|
||||
|
||||
# SDK 2.0.2 reads `config` off the coding store while routing a coding
|
||||
# turn. The wrapper must answer it, but the native `MemoryConfig` carries
|
||||
# credentials (llm/embedder api keys, graph and cache passwords) and is a live
|
||||
# handle into the backend, so it is never handed out. Only this redacted
|
||||
# projection is exposed: credential-bearing keys collapse to None.
|
||||
_CONFIG_REDACTED_KEYS = frozenset(
|
||||
{
|
||||
"api_key",
|
||||
"password",
|
||||
"username",
|
||||
"mysql_password",
|
||||
"mysql_user",
|
||||
"extra_headers",
|
||||
"extra_body",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class _RedactedConfig(Mapping[str, Any]):
|
||||
"""Read-only config view: readable by key and attribute, never the native one.
|
||||
|
||||
The SDK reaches config values both ways, so this keeps attribute access
|
||||
working. It is deliberately not the native MemoryConfig: it exposes only
|
||||
plain copied values, so no backend handle, connection or callable escapes.
|
||||
"""
|
||||
|
||||
__slots__ = ("_values",)
|
||||
|
||||
def __init__(self, values: Mapping[str, Any]) -> None:
|
||||
object.__setattr__(self, "_values", MappingProxyType(dict(values)))
|
||||
|
||||
def __getitem__(self, key: str) -> Any:
|
||||
return self._values[key]
|
||||
|
||||
def __iter__(self) -> Iterator[str]:
|
||||
return iter(self._values)
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self._values)
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
values = object.__getattribute__(self, "_values")
|
||||
try:
|
||||
return values[name]
|
||||
except KeyError:
|
||||
raise AttributeError(name) from None
|
||||
|
||||
def __setattr__(self, name: str, value: Any) -> NoReturn:
|
||||
raise AttributeError("redacted config is read-only")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"_RedactedConfig({sorted(self._values)})"
|
||||
|
||||
|
||||
def _redact_config(value: Any, depth: int = 0) -> Any:
|
||||
"""Copy config data into read-only views, collapsing credentials to None.
|
||||
|
||||
The SDK reads this both by attribute and by key, so each config section
|
||||
becomes a mapping-like view rather than a plain dict: attribute access stays
|
||||
available while the native object itself is never handed out.
|
||||
"""
|
||||
if depth > 4:
|
||||
return None
|
||||
if value is None or isinstance(value, (str, int, float, bool)):
|
||||
return value
|
||||
if isinstance(value, Mapping):
|
||||
return _RedactedConfig(
|
||||
{
|
||||
key: None if key in _CONFIG_REDACTED_KEYS else _redact_config(item, depth + 1)
|
||||
for key, item in value.items()
|
||||
}
|
||||
)
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [_redact_config(item, depth + 1) for item in value]
|
||||
if hasattr(value, "__dict__") and type(value).__module__.startswith("hy_memory"):
|
||||
return _redact_config(vars(value), depth + 1)
|
||||
return None
|
||||
|
||||
|
||||
def _config_projection(native: Any) -> Any:
|
||||
try:
|
||||
return _redact_config(native.config)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def hy_coding_fingerprint(row: CodingMemory | WorkingFact) -> str:
|
||||
"""Hash the complete native row as read back, including links and timestamps."""
|
||||
if type(row) is CodingMemory:
|
||||
kind = "experience"
|
||||
elif type(row) is WorkingFact:
|
||||
kind = "fact"
|
||||
else:
|
||||
raise permission_error("hy_coding_store_row_shape")
|
||||
payload = {"kind": kind, **row.to_dict()}
|
||||
return sha256(
|
||||
json.dumps(payload, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _row_id(row: CodingMemory | WorkingFact) -> str:
|
||||
return str(row.experience_id if type(row) is CodingMemory else row.fact_id)
|
||||
|
||||
|
||||
class HyPermissionCodingStore:
|
||||
"""Closed, task-exclusive adapter; initialize/close never own the native store.
|
||||
|
||||
node_permissions is the same attested ledger the chat vector store uses;
|
||||
coding rows are keyed by experience or fact id. written_nodes carries the
|
||||
whole lease footprint for each row written here, as the chat store does.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
native: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
*,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
node_permissions: Mapping[str, HyNodePermission] | None = None,
|
||||
) -> None:
|
||||
self._native = native
|
||||
self._lease = lease
|
||||
self._agent_id = agent_id
|
||||
self._session_id = session_id
|
||||
self._user_id = lease.user_id
|
||||
self._failure: str | None = None
|
||||
self._closed = False
|
||||
self._lock = asyncio.Lock()
|
||||
self._deleted: set[str] = set()
|
||||
self._written: dict[str, HyNodePermission] = {}
|
||||
self._superseded: dict[str, str] = {}
|
||||
self._evidence_ids = tuple(record.evidence_id for record in lease.records)
|
||||
self._permissions = {
|
||||
key: proof
|
||||
for key, proof in (node_permissions or {}).items()
|
||||
if set(proof.evidence_ids) <= set(self._evidence_ids)
|
||||
}
|
||||
self.recheck()
|
||||
if not agent_id or agent_id == "*" or not session_id or not self._evidence_ids:
|
||||
self._deny("scope")
|
||||
|
||||
def __getattr__(self, name: str) -> NoReturn:
|
||||
if name.startswith("__"):
|
||||
raise AttributeError(name)
|
||||
self._deny("unsupported_operation")
|
||||
|
||||
@property
|
||||
def written_nodes(self) -> Mapping[str, HyNodePermission]:
|
||||
self.recheck()
|
||||
return MappingProxyType(dict(self._written))
|
||||
|
||||
@property
|
||||
def config(self) -> Any:
|
||||
"""Read-only, redacted view of the native coding config.
|
||||
|
||||
SDK 2.0.2 reads `config` when it routes a coding turn. Answering
|
||||
it is safe only as a projection: the native MemoryConfig is a live
|
||||
backend handle and carries credentials, so callers get plain values
|
||||
with every credential-bearing key collapsed to None. This carries no
|
||||
identity, permission or tenant state and is not the native object.
|
||||
"""
|
||||
self._assert_open()
|
||||
return _config_projection(self._native)
|
||||
|
||||
@property
|
||||
def retired_ids(self) -> frozenset[str]:
|
||||
"""Experiences HY deleted in this operation."""
|
||||
self.recheck()
|
||||
return frozenset(self._deleted)
|
||||
|
||||
@property
|
||||
def superseded_ids(self) -> Mapping[str, tuple[str, ...]]:
|
||||
"""Facts HY superseded in this operation, with the fact that replaced each.
|
||||
|
||||
A successor linked before it existed must have been written here by
|
||||
now; otherwise the old fact points at a row nobody attested.
|
||||
"""
|
||||
self.recheck()
|
||||
if any(new not in self._written for new in self._superseded.values()):
|
||||
self._deny("link_denied")
|
||||
return MappingProxyType({old: (new,) for old, new in self._superseded.items()})
|
||||
|
||||
def recheck(self) -> None:
|
||||
self._assert_open()
|
||||
try:
|
||||
self._lease.recheck()
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
self._failure = (
|
||||
code
|
||||
if code.startswith("model_content_") and len(code) <= 100
|
||||
else "model_content_hy_coding_store_lease_failed"
|
||||
)
|
||||
raise permission_error(self._failure) from None
|
||||
|
||||
def _assert_open(self) -> None:
|
||||
if self._closed:
|
||||
self._deny("closed")
|
||||
if self._failure:
|
||||
raise permission_error(self._failure)
|
||||
|
||||
async def close(self) -> None:
|
||||
async with self._lock:
|
||||
self._closed = True
|
||||
self._permissions.clear()
|
||||
self._written.clear()
|
||||
|
||||
def _deny(self, suffix: str) -> NoReturn:
|
||||
self._failure = self._failure or f"model_content_hy_coding_store_{suffix}"
|
||||
raise permission_error(self._failure)
|
||||
|
||||
def _scope(self, user_id: str | None) -> None:
|
||||
if user_id not in (None, self._user_id):
|
||||
self._deny("scope")
|
||||
|
||||
async def _invoke(self, method: str, *args: Any, **kwargs: Any) -> Any:
|
||||
writing = method in _NATIVE_WRITES
|
||||
self.recheck() if writing else self._assert_open()
|
||||
try:
|
||||
result = await getattr(self._native, method)(*args, **kwargs)
|
||||
except Exception:
|
||||
self._deny("backend_failed")
|
||||
self.recheck() if writing else self._assert_open()
|
||||
return result
|
||||
|
||||
def _visible(self, row: Any, expected: Any) -> bool:
|
||||
if row is None:
|
||||
return False
|
||||
if type(row) is not expected:
|
||||
self._deny("row_shape")
|
||||
key = _row_id(row)
|
||||
proof = self._permissions.get(key)
|
||||
if proof is None or key in self._deleted or row.user_id != self._user_id:
|
||||
return False
|
||||
if hy_coding_fingerprint(row) != proof.fingerprint:
|
||||
self._deny("row_changed")
|
||||
return True
|
||||
|
||||
async def _experience(self, key: str) -> CodingMemory | None:
|
||||
if key not in self._permissions or key in self._deleted:
|
||||
return None
|
||||
row = await self._invoke("get_by_id", key, user_id=self._user_id)
|
||||
return deepcopy(row) if self._visible(row, CodingMemory) else None
|
||||
|
||||
async def _fact(self, key: str) -> WorkingFact | None:
|
||||
if key not in self._permissions or key in self._deleted:
|
||||
return None
|
||||
rows = await self._invoke("get_facts", [key], user_id=self._user_id)
|
||||
row = rows[0] if rows else None
|
||||
return deepcopy(row) if self._visible(row, WorkingFact) else None
|
||||
|
||||
def _new_row_in_scope(self, row: CodingMemory | WorkingFact) -> bool:
|
||||
return bool(
|
||||
row.user_id == self._user_id
|
||||
and row.agent_id == self._agent_id
|
||||
and row.session_id == self._session_id
|
||||
)
|
||||
|
||||
async def _capture(
|
||||
self, key: str, kind: Any, expected: CodingMemory | WorkingFact | None = None
|
||||
) -> None:
|
||||
row: Any
|
||||
if kind is CodingMemory:
|
||||
row = await self._invoke("get_by_id", key, user_id=self._user_id)
|
||||
else:
|
||||
rows = await self._invoke("get_facts", [key], user_id=self._user_id)
|
||||
row = rows[0] if rows else None
|
||||
if type(row) is not kind or row.user_id != self._user_id:
|
||||
self._deny("write_mismatch")
|
||||
if expected is not None:
|
||||
stored, planned = row.to_dict(), expected.to_dict()
|
||||
for field in _TIMESTAMPS:
|
||||
stored.pop(field, None)
|
||||
planned.pop(field, None)
|
||||
if stored != planned:
|
||||
self._deny("write_mismatch")
|
||||
proof = HyNodePermission(hy_coding_fingerprint(row), self._evidence_ids)
|
||||
self._permissions[key] = proof
|
||||
self._written[key] = proof
|
||||
|
||||
async def _plan_new(self, row: CodingMemory | WorkingFact, kind: Any) -> None:
|
||||
if type(row) is not kind or not self._new_row_in_scope(row):
|
||||
self._deny("scope")
|
||||
key = _row_id(row)
|
||||
if not key or key in self._permissions or key in self._deleted:
|
||||
self._deny("scope")
|
||||
if kind is CodingMemory:
|
||||
existing = await self._invoke("get_by_id", key)
|
||||
else:
|
||||
existing = next(iter(await self._invoke("get_facts", [key])), None)
|
||||
if existing is not None:
|
||||
self._deny("scope")
|
||||
|
||||
# Reads
|
||||
|
||||
async def get_by_id(
|
||||
self, experience_id: str, user_id: str | None = None
|
||||
) -> CodingMemory | None:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
return await self._experience(experience_id)
|
||||
|
||||
async def get_many(
|
||||
self, experience_ids: list[str], user_id: str | None = None
|
||||
) -> list[CodingMemory]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
wanted = [
|
||||
key
|
||||
for key in dict.fromkeys(experience_ids)
|
||||
if key in self._permissions and key not in self._deleted
|
||||
]
|
||||
if not wanted:
|
||||
return []
|
||||
rows = await self._invoke("get_many", wanted, user_id=self._user_id)
|
||||
return [deepcopy(row) for row in rows if self._visible(row, CodingMemory)]
|
||||
|
||||
async def search_by_query_embedding(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
user_id: str,
|
||||
*,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
top: int = 40,
|
||||
) -> list[dict[str, Any]]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
hits = await self._invoke(
|
||||
"search_by_query_embedding",
|
||||
deepcopy(query_embedding),
|
||||
self._user_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
top=top * 4,
|
||||
)
|
||||
result = []
|
||||
for hit in hits:
|
||||
key = hit.get("experience_id")
|
||||
if not isinstance(key, str) or await self._experience(key) is None:
|
||||
continue
|
||||
result.append(
|
||||
{
|
||||
"experience_id": key,
|
||||
"score": float(hit.get("score", 0.0)),
|
||||
"matched_key_kind": hit.get("matched_key_kind"),
|
||||
"matched_key_text": hit.get("matched_key_text"),
|
||||
}
|
||||
)
|
||||
return result[:top]
|
||||
|
||||
async def get_fact(self, fact_id: str, user_id: str | None = None) -> WorkingFact | None:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
return await self._fact(fact_id)
|
||||
|
||||
async def get_facts(self, fact_ids: list[str], user_id: str | None = None) -> list[WorkingFact]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
found = [await self._fact(key) for key in dict.fromkeys(fact_ids)]
|
||||
return [row for row in found if row is not None]
|
||||
|
||||
async def search_facts(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
user_id: str,
|
||||
*,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
top: int = 5,
|
||||
query_text: str | None = None,
|
||||
min_score: float = 0.0,
|
||||
) -> list[tuple[WorkingFact, float]]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
hits = await self._invoke(
|
||||
"search_facts",
|
||||
deepcopy(query_embedding),
|
||||
self._user_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
top=top * 4,
|
||||
query_text=query_text,
|
||||
min_score=min_score,
|
||||
)
|
||||
result = []
|
||||
for row, score in hits:
|
||||
if self._visible(row, WorkingFact):
|
||||
result.append((deepcopy(row), float(score)))
|
||||
return result[:top]
|
||||
|
||||
async def get_fact_ancestors(
|
||||
self, fact_id: str, *, user_id: str | None = None, max_depth: int = 8
|
||||
) -> list[WorkingFact]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
if await self._fact(fact_id) is None:
|
||||
return []
|
||||
rows = await self._invoke(
|
||||
"get_fact_ancestors", fact_id, user_id=self._user_id, max_depth=max_depth
|
||||
)
|
||||
# Oldest first; the chain is only as long as its nearest unapproved link.
|
||||
kept: list[WorkingFact] = []
|
||||
for row in reversed(rows):
|
||||
if not self._visible(row, WorkingFact):
|
||||
break
|
||||
kept.append(deepcopy(row))
|
||||
return list(reversed(kept))
|
||||
|
||||
# Writes
|
||||
|
||||
async def insert(self, memory: CodingMemory) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
await self._plan_new(memory, CodingMemory)
|
||||
await self._invoke("insert", memory)
|
||||
await self._capture(memory.experience_id, CodingMemory, memory)
|
||||
|
||||
async def update(self, memory: CodingMemory) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if type(memory) is not CodingMemory or not self._new_row_in_scope(memory):
|
||||
self._deny("scope")
|
||||
if await self._experience(memory.experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
await self._invoke("update", memory)
|
||||
await self._capture(memory.experience_id, CodingMemory, memory)
|
||||
|
||||
async def delete(self, experience_id: str, user_id: str) -> bool:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
if await self._experience(experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
if await self._invoke("delete", experience_id, user_id=self._user_id) is not True:
|
||||
self._deny("write_failed")
|
||||
if await self._invoke("get_by_id", experience_id) is not None:
|
||||
self._deny("write_mismatch")
|
||||
self._deleted.add(experience_id)
|
||||
self._permissions.pop(experience_id, None)
|
||||
self._written.pop(experience_id, None)
|
||||
return True
|
||||
|
||||
async def insert_fact(self, fact: WorkingFact) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
await self._plan_new(fact, WorkingFact)
|
||||
await self._invoke("insert_fact", fact)
|
||||
await self._capture(fact.fact_id, WorkingFact, fact)
|
||||
|
||||
async def supersede_fact(self, old_fact_id: str, new_fact_id: str) -> bool:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._fact(old_fact_id) is None or old_fact_id == new_fact_id:
|
||||
self._deny("target_denied")
|
||||
# HY links the old fact before it inserts the new one. A successor
|
||||
# that does not exist yet may only become a row this lease writes;
|
||||
# an existing row it cannot attest stays out of reach.
|
||||
if await self._fact(new_fact_id) is None:
|
||||
if not new_fact_id or new_fact_id in self._deleted:
|
||||
self._deny("target_denied")
|
||||
if next(iter(await self._invoke("get_facts", [new_fact_id])), None) is not None:
|
||||
self._deny("target_denied")
|
||||
changed = await self._invoke("supersede_fact", old_fact_id, new_fact_id)
|
||||
await self._capture(old_fact_id, WorkingFact)
|
||||
if changed:
|
||||
self._superseded[old_fact_id] = new_fact_id
|
||||
return bool(changed)
|
||||
|
||||
async def set_fact_derived_experiences(self, fact_id: str, experience_ids: list[str]) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._fact(fact_id) is None:
|
||||
self._deny("target_denied")
|
||||
for key in experience_ids:
|
||||
if await self._experience(key) is None:
|
||||
self._deny("link_denied")
|
||||
await self._invoke("set_fact_derived_experiences", fact_id, list(experience_ids))
|
||||
await self._capture(fact_id, WorkingFact)
|
||||
|
||||
async def set_experience_source_facts(self, experience_id: str, fact_ids: list[str]) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._experience(experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
for key in fact_ids:
|
||||
if await self._fact(key) is None:
|
||||
self._deny("link_denied")
|
||||
await self._invoke("set_experience_source_facts", experience_id, list(fact_ids))
|
||||
await self._capture(experience_id, CodingMemory)
|
||||
|
||||
async def get_recent_trace_messages(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
max_user_turns: int = 5,
|
||||
) -> list[Any]:
|
||||
"""Read recent raw trace rows, still bounded by this lease.
|
||||
|
||||
Public SDK 2.0.2 asks for recent history before a coding write. The native call is
|
||||
scoped to the lease's own user and agent, and rows this lease cannot
|
||||
attest are filtered out, so unapproved conversations never reach a
|
||||
coding prompt through this path.
|
||||
"""
|
||||
self._scope(user_id)
|
||||
if agent_id not in (self._agent_id, "*"):
|
||||
self._deny("scope")
|
||||
rows = await self._invoke(
|
||||
"get_recent_trace_messages",
|
||||
user_id=self._user_id,
|
||||
agent_id=self._agent_id,
|
||||
max_user_turns=max_user_turns,
|
||||
)
|
||||
return [row for row in (rows or []) if self._trace_visible(row)]
|
||||
|
||||
async def append_trace(
|
||||
self,
|
||||
messages: Sequence[Any],
|
||||
*,
|
||||
request_id: str,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
session_id: str | None = None,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
) -> None:
|
||||
"""Append raw trace rows for this lease only.
|
||||
|
||||
Trace rows are raw conversation, so this is a write path: the lease is
|
||||
rechecked around the call and every message must belong to the leased
|
||||
user and session. Nothing outside the lease may be recorded.
|
||||
"""
|
||||
self._scope(user_id)
|
||||
if agent_id not in (self._agent_id, "*"):
|
||||
self._deny("scope")
|
||||
if session_id not in (None, self._session_id):
|
||||
self._deny("scope")
|
||||
for row in messages or []:
|
||||
if not self._trace_visible(row):
|
||||
self._deny("scope")
|
||||
await self._invoke(
|
||||
"append_trace",
|
||||
list(messages or []),
|
||||
request_id=request_id,
|
||||
user_id=self._user_id,
|
||||
agent_id=self._agent_id,
|
||||
session_id=self._session_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
)
|
||||
|
||||
def _trace_visible(self, row: Any) -> bool:
|
||||
"""A trace row may be read or written only inside this lease's scope."""
|
||||
if row is None:
|
||||
return False
|
||||
owner = getattr(row, "user_id", None)
|
||||
session = getattr(row, "session_id", None)
|
||||
return not (
|
||||
(owner is not None and owner != self._user_id)
|
||||
or (session is not None and session != self._session_id)
|
||||
)
|
||||
@@ -1,582 +0,0 @@
|
||||
"""Task-exclusive HY wiring; provenance metadata stays beside the existing Core DB."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from collections.abc import AsyncIterator, Callable
|
||||
from contextlib import asynccontextmanager, nullcontext, suppress
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
import httpx
|
||||
|
||||
from one_memory.database import take_the_write_lock_first
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_memory_bridge import (
|
||||
HyMemoryBridge,
|
||||
HyMemoryBridgeError,
|
||||
HyMemoryIngestRequest,
|
||||
HyMemoryIngestResult,
|
||||
HyMemoryOutput,
|
||||
LocalHyEmbedService,
|
||||
_joined_thread,
|
||||
)
|
||||
from one_memory.hy_permission_coding_store import HyPermissionCodingStore
|
||||
from one_memory.hy_permission_store import (
|
||||
HyNodePermission,
|
||||
HyPermissionGraphStore,
|
||||
HyPermissionVectorStore,
|
||||
)
|
||||
from one_memory.hy_permission_transport import HyPermissionTransport, HyPermissionTransportError
|
||||
from one_memory.hy_response_guard import HyResponseGuard
|
||||
from one_memory.hy_step_timing import HyStepTimer
|
||||
from one_memory.memory_server.migrations import hy_node_permissions_ddl
|
||||
from one_memory.memory_tasks import evidence_ref
|
||||
from one_memory.model_processing_permission_guard import ModelProcessingPermissionLease
|
||||
|
||||
|
||||
class _NativeProofs:
|
||||
def __init__(self, database: Path) -> None:
|
||||
self.database = database
|
||||
self._loaded_states: dict[str, tuple[int, str]] | None = None
|
||||
with main_database(database).write() as db, db:
|
||||
# Schema owned by `memory_server.migrations` (migration 26).
|
||||
db.execute(hy_node_permissions_ddl("sqlite"))
|
||||
|
||||
def load(self, lease: ModelProcessingPermissionLease) -> dict[str, HyNodePermission]:
|
||||
allowed = {record.evidence_id: evidence_ref(record) for record in lease.records}
|
||||
with main_database(self.database).read() as db:
|
||||
rows = db.execute(
|
||||
"SELECT p.node_id,p.fingerprint,p.evidence_json,g.revision,g.status "
|
||||
"FROM hy_node_permissions p LEFT JOIN governed_memories g ON g.memory_id=p.node_id "
|
||||
# Superseded nodes stay attested: every later head links back to them.
|
||||
"WHERE g.status IS NULL OR g.status IN ('confirmed','superseded')"
|
||||
).fetchall()
|
||||
proofs = {}
|
||||
states = {}
|
||||
for identifier, fingerprint, evidence_json, revision, status in rows:
|
||||
refs = json.loads(evidence_json)
|
||||
if (
|
||||
not isinstance(refs, list)
|
||||
or not refs
|
||||
or not all(isinstance(ref, dict) for ref in refs)
|
||||
):
|
||||
continue
|
||||
if all(allowed.get(ref.get("evidence_id")) == ref for ref in refs):
|
||||
proofs[identifier] = HyNodePermission(
|
||||
fingerprint, tuple(ref["evidence_id"] for ref in refs)
|
||||
)
|
||||
if revision is not None:
|
||||
states[identifier] = (revision, status)
|
||||
self._loaded_states = states
|
||||
return proofs
|
||||
|
||||
def guard(self, values: dict[str, HyNodePermission]) -> Callable[[], None]:
|
||||
identifiers = tuple(values)
|
||||
|
||||
def state() -> dict[str, tuple[int, str]]:
|
||||
found: dict[str, tuple[int, str]] = {}
|
||||
with main_database(self.database).read() as db:
|
||||
for start in range(0, len(identifiers), 900):
|
||||
batch = identifiers[start : start + 900]
|
||||
placeholders = ",".join("?" for _ in batch)
|
||||
for identifier, revision, status in db.execute(
|
||||
"SELECT memory_id,revision,status FROM governed_memories "
|
||||
f"WHERE memory_id IN ({placeholders})",
|
||||
batch,
|
||||
):
|
||||
found[identifier] = (revision, status)
|
||||
return found
|
||||
|
||||
if self._loaded_states is None:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
before = dict(self._loaded_states)
|
||||
|
||||
def check() -> None:
|
||||
if state() != before:
|
||||
raise HyMemoryBridgeError("model_content_context_changed")
|
||||
|
||||
check()
|
||||
return check
|
||||
|
||||
def save(
|
||||
self, lease: ModelProcessingPermissionLease, values: dict[str, HyNodePermission]
|
||||
) -> None:
|
||||
refs = {record.evidence_id: evidence_ref(record) for record in lease.records}
|
||||
with main_database(self.database).write(autocommit=True) as db:
|
||||
take_the_write_lock_first(db)
|
||||
with db:
|
||||
lease.recheck()
|
||||
db.executemany(
|
||||
"INSERT INTO hy_node_permissions VALUES (?,?,?) ON CONFLICT(node_id) "
|
||||
"DO UPDATE SET fingerprint=excluded.fingerprint,"
|
||||
"evidence_json=excluded.evidence_json",
|
||||
[
|
||||
(
|
||||
identifier,
|
||||
proof.fingerprint,
|
||||
json.dumps([refs[key] for key in proof.evidence_ids]),
|
||||
)
|
||||
for identifier, proof in values.items()
|
||||
],
|
||||
)
|
||||
|
||||
def refresh(
|
||||
self, lease: ModelProcessingPermissionLease, node_id: str
|
||||
) -> HyNodePermission | None:
|
||||
del lease
|
||||
with main_database(self.database).read() as db:
|
||||
row = db.execute(
|
||||
"SELECT fingerprint,evidence_json FROM hy_node_permissions WHERE node_id=?",
|
||||
(node_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
fingerprint = str(row["fingerprint"])
|
||||
refs = json.loads(row["evidence_json"])
|
||||
if (
|
||||
len(fingerprint) != 64
|
||||
or any(value not in "0123456789abcdef" for value in fingerprint)
|
||||
or not isinstance(refs, list)
|
||||
or not refs
|
||||
or not all(isinstance(ref, dict) for ref in refs)
|
||||
):
|
||||
return None
|
||||
evidence_ids = tuple(ref.get("evidence_id") for ref in refs)
|
||||
if any(not isinstance(value, str) or not value for value in evidence_ids):
|
||||
return None
|
||||
return HyNodePermission(fingerprint, evidence_ids)
|
||||
|
||||
|
||||
class _BoundLease:
|
||||
def __init__(
|
||||
self,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proof_check: Callable[[], None],
|
||||
step_timer: HyStepTimer | None = None,
|
||||
) -> None:
|
||||
self.lease = lease
|
||||
self.step_timer = step_timer
|
||||
self.user_id = lease.user_id
|
||||
self.profile = lease.profile
|
||||
self.task_id = lease.task_id
|
||||
self.records = lease.records
|
||||
self.source_context = lease.source_context
|
||||
self.proof_check = proof_check
|
||||
self.vector: HyPermissionVectorStore | None = None
|
||||
self.coding: HyPermissionCodingStore | None = None
|
||||
|
||||
def recheck(self) -> None:
|
||||
timer = self.step_timer
|
||||
with timer.step("permission_check") if timer is not None else nullcontext():
|
||||
self.lease.recheck()
|
||||
self.proof_check()
|
||||
if self.vector is not None:
|
||||
self.vector.recheck()
|
||||
if self.coding is not None:
|
||||
self.coding.recheck()
|
||||
|
||||
|
||||
class _Wiring:
|
||||
def __init__(
|
||||
self,
|
||||
client: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proofs_store: _NativeProofs,
|
||||
proofs: dict[str, HyNodePermission],
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
transport: httpx.AsyncBaseTransport | None,
|
||||
proof_check: Callable[[], None],
|
||||
step_timer: HyStepTimer,
|
||||
) -> None:
|
||||
self.client, self.lease, self.proofs_store, self.proofs = (
|
||||
client,
|
||||
lease,
|
||||
proofs_store,
|
||||
proofs,
|
||||
)
|
||||
self.step_timer = step_timer
|
||||
self.agent_id, self.session_id, self.http_transport = agent_id, session_id, transport
|
||||
self.cross_evidence_scope = False
|
||||
self.proof_check = proof_check
|
||||
self.saved: list[tuple[Any, str, Any]] = []
|
||||
self.transport: HyPermissionTransport
|
||||
self.vector: HyPermissionVectorStore
|
||||
self.graph: HyPermissionGraphStore | None = None
|
||||
self.coding: HyPermissionCodingStore
|
||||
self.active = False
|
||||
|
||||
def replace(self, target: Any, name: str, value: Any) -> None:
|
||||
self.saved.append((target, name, getattr(target, name)))
|
||||
setattr(target, name, value)
|
||||
|
||||
async def enter(self) -> None:
|
||||
client, registry = self.client, self.client._registry
|
||||
config = client._config
|
||||
if config.llm.enable_summary or config.llm.enable_query_rewrite:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
bound = _BoundLease(self.lease, self.proof_check, self.step_timer)
|
||||
self.transport = await HyPermissionTransport(
|
||||
bound, http_transport=self.http_transport, step_timer=self.step_timer
|
||||
).__aenter__()
|
||||
self.active = True
|
||||
try:
|
||||
for guard in getattr(client, "_one_memory_write_guards", ()):
|
||||
if hasattr(type(guard), "step_timer"):
|
||||
self.replace(guard, "step_timer", self.step_timer)
|
||||
system2 = await registry.get_system2_writer()
|
||||
writers = (await registry.get_writer(), system2._writer)
|
||||
native_vector = client._vector_store
|
||||
store_lease = _BoundLease(self.lease, self.proof_check, self.step_timer)
|
||||
self.vector = HyPermissionVectorStore(
|
||||
native_vector,
|
||||
cast(ModelProcessingPermissionLease, store_lease),
|
||||
agent_id=self.agent_id,
|
||||
session_id=self.session_id,
|
||||
cross_evidence_scope=self.cross_evidence_scope,
|
||||
node_permissions=self.proofs,
|
||||
refresh_permission=lambda node_id: asyncio.to_thread(
|
||||
self.proofs_store.refresh, self.lease, node_id
|
||||
),
|
||||
step_timer=self.step_timer,
|
||||
)
|
||||
bound.vector = self.vector
|
||||
embed = client._embed_service
|
||||
if isinstance(embed, LocalHyEmbedService):
|
||||
self.transport.bind_local_embed_service(embed)
|
||||
else:
|
||||
self.transport.bind_embed_service(embed)
|
||||
for writer in writers:
|
||||
provider = writer.mem_agent.llm_provider
|
||||
self.transport.bind_provider(
|
||||
provider.provider if isinstance(provider, HyResponseGuard) else provider
|
||||
)
|
||||
reconciler = writer.reconciler
|
||||
provider = reconciler._get_llm()
|
||||
self.transport.bind_provider(
|
||||
provider.provider if isinstance(provider, HyResponseGuard) else provider
|
||||
)
|
||||
self.replace(writer, "_external_vector_store", self.vector)
|
||||
self.replace(writer, "_vector_store", self.vector)
|
||||
self.replace(writer, "_vector_store_initialized", True)
|
||||
self.replace(writer, "_cache", None)
|
||||
self.replace(client, "_vector_store", self.vector)
|
||||
native_graph = client._graph_store
|
||||
if self.cross_evidence_scope and native_graph is not None:
|
||||
self.graph = HyPermissionGraphStore(native_graph, self.proof_check)
|
||||
self.replace(client, "_graph_store", self.graph)
|
||||
else:
|
||||
self.replace(client, "_graph_store", None)
|
||||
self.replace(registry, "_shared_vector_store", self.vector)
|
||||
self.replace(
|
||||
registry,
|
||||
"_shared_graph_store",
|
||||
self.graph if self.graph is not None else None,
|
||||
)
|
||||
self.replace(registry, "_shared_cache", None)
|
||||
self.replace(system2, "_vector_store", self.vector)
|
||||
self.replace(
|
||||
system2,
|
||||
"_graph_store",
|
||||
self.graph if self.graph is not None else None,
|
||||
)
|
||||
self.replace(system2, "_cache", None)
|
||||
# Result fallback may call search; rebuild only its non-generative reader.
|
||||
self.replace(registry, "_reader", None)
|
||||
self.replace(registry, "_readers", {})
|
||||
await self._enter_coding(bound, embed)
|
||||
self.transport.assert_allowed()
|
||||
except BaseException:
|
||||
await self.exit()
|
||||
raise
|
||||
|
||||
async def _enter_coding(self, bound: _BoundLease, embed: Any) -> None:
|
||||
"""HY judges and writes the coding route itself; every seam it uses is bound here."""
|
||||
client = self.client
|
||||
# Created now, so the judge and every coding stage share one bound provider.
|
||||
writer = await client._get_coding_writer()
|
||||
stages = (writer, writer.extractor, writer.reconciler, writer.fact_reconciler)
|
||||
native_store = client._coding_store
|
||||
if any(
|
||||
stage.store is not native_store
|
||||
for stage in (writer, writer.reconciler, writer.fact_reconciler)
|
||||
) or any(
|
||||
item.embed_service is not embed
|
||||
for item in (native_store, writer, writer.reconciler, writer.fact_reconciler)
|
||||
):
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
providers = {
|
||||
id(p): p for p in (client._coding_llm_provider, *(s.llm_provider for s in stages))
|
||||
}
|
||||
for provider in providers.values():
|
||||
self.transport.bind_provider(provider)
|
||||
self.coding = HyPermissionCodingStore(
|
||||
native_store,
|
||||
cast(
|
||||
ModelProcessingPermissionLease,
|
||||
_BoundLease(self.lease, self.proof_check, self.step_timer),
|
||||
),
|
||||
agent_id=self.agent_id,
|
||||
session_id=self.session_id,
|
||||
node_permissions=self.proofs,
|
||||
)
|
||||
bound.coding = self.coding
|
||||
for stage in (writer, writer.reconciler, writer.fact_reconciler):
|
||||
self.replace(stage, "store", self.coding)
|
||||
self.replace(client, "_coding_store", self.coding)
|
||||
for stage in (writer, writer.extractor, writer.reconciler):
|
||||
self.replace(stage, "cache", None)
|
||||
# The judge logs its prompt through the client's own cache.
|
||||
self.replace(client, "_cache", None)
|
||||
|
||||
async def finish(self) -> dict[str, HyNodePermission]:
|
||||
self.transport.assert_allowed()
|
||||
self.vector.recheck()
|
||||
self.coding.recheck()
|
||||
return {**self.vector.written_nodes, **self.coding.written_nodes}
|
||||
|
||||
async def retired(self) -> tuple[tuple[str, ...], dict[str, tuple[str, ...]]]:
|
||||
self.transport.assert_allowed()
|
||||
return (
|
||||
tuple(sorted(self.vector.retired_node_ids | self.coding.retired_ids)),
|
||||
{**self.vector.superseded_nodes, **self.coding.superseded_ids},
|
||||
)
|
||||
|
||||
async def exit(self) -> None:
|
||||
if not self.active:
|
||||
return
|
||||
self.active = False
|
||||
try:
|
||||
await self.transport.__aexit__(None, None, None)
|
||||
finally:
|
||||
for target, name, value in reversed(self.saved):
|
||||
setattr(target, name, value)
|
||||
self.saved.clear()
|
||||
if hasattr(self, "vector"):
|
||||
await self.vector.close()
|
||||
if self.graph is not None:
|
||||
await self.graph.close()
|
||||
if hasattr(self, "coding"):
|
||||
await self.coding.close()
|
||||
|
||||
|
||||
class _LeasedBridge:
|
||||
def __init__(
|
||||
self,
|
||||
bridge: HyMemoryBridge,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proofs: _NativeProofs,
|
||||
transport: httpx.AsyncBaseTransport | None,
|
||||
) -> None:
|
||||
self.bridge, self.lease, self.proofs, self.transport = bridge, lease, proofs, transport
|
||||
self.wiring: _Wiring | None = None
|
||||
self.finished = False
|
||||
self.step_timer = HyStepTimer()
|
||||
self.request_id: str | None = None
|
||||
|
||||
def _step(self, name: str) -> Any:
|
||||
return self.step_timer.step(name)
|
||||
|
||||
async def _recheck(self) -> None:
|
||||
def recheck() -> None:
|
||||
with self._step("permission_check"):
|
||||
self.lease.recheck()
|
||||
|
||||
await _joined_thread(recheck)
|
||||
|
||||
async def _start(self, request: HyMemoryIngestRequest) -> None:
|
||||
if self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_reused")
|
||||
if request.user_id != self.lease.user_id or set(request.evidence_ids) - {
|
||||
item.evidence_id for item in self.lease.records
|
||||
}:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._start_operation(
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
request_id=request.request_id,
|
||||
session_id=request.session_id,
|
||||
)
|
||||
|
||||
async def _start_operation(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
request_id: str,
|
||||
session_id: str,
|
||||
cross_evidence_scope: bool = False,
|
||||
) -> None:
|
||||
if self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_reused")
|
||||
if user_id != self.lease.user_id:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
self.request_id = request_id
|
||||
await self._recheck()
|
||||
client = cast(Any, await self.bridge._get_client())
|
||||
with self._step("permission_store_read"):
|
||||
known = await asyncio.to_thread(self.proofs.load, self.lease)
|
||||
proof_check = await asyncio.to_thread(self.proofs.guard, known)
|
||||
self.wiring = _Wiring(
|
||||
client,
|
||||
self.lease,
|
||||
self.proofs,
|
||||
known,
|
||||
agent_id,
|
||||
session_id,
|
||||
self.transport,
|
||||
proof_check,
|
||||
self.step_timer,
|
||||
)
|
||||
self.wiring.cross_evidence_scope = cross_evidence_scope
|
||||
await _joined_thread(client._loop_thread.run, self.wiring.enter())
|
||||
|
||||
async def list_agent_ids(self, *, user_id: str, limit: int = 1_000) -> tuple[str, ...]:
|
||||
if user_id != self.lease.user_id or self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._recheck()
|
||||
result = await self.bridge.list_agent_ids(user_id=user_id, limit=limit)
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def digest(self, *, user_id: str, agent_id: str) -> dict[str, object]:
|
||||
await self._start_operation(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
request_id=f"digest:{self.lease.task_id}:{agent_id}",
|
||||
session_id=f"digest:{self.lease.task_id}",
|
||||
cross_evidence_scope=True,
|
||||
)
|
||||
try:
|
||||
result = dict(await self.bridge.digest(user_id=user_id, agent_id=agent_id))
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
proofs = await self._check()
|
||||
with self._step("permission_store_write"):
|
||||
await asyncio.to_thread(self.proofs.save, self.lease, proofs)
|
||||
await self.close()
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def graph_snapshot(self, *, user_id: str, agent_id: str, limit: int = 300) -> Any:
|
||||
await self._start_operation(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
request_id=f"digest-graph:{self.lease.task_id}:{agent_id}",
|
||||
session_id=f"digest:{self.lease.task_id}",
|
||||
cross_evidence_scope=True,
|
||||
)
|
||||
try:
|
||||
result = await self.bridge.graph_snapshot(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
limit=limit,
|
||||
)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
await self._check()
|
||||
await self.close()
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
|
||||
await self._start(request)
|
||||
try:
|
||||
result = await self.bridge.ingest(request)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
return await self._with_retired(result)
|
||||
|
||||
async def ingest_extracted(
|
||||
self, request: HyMemoryIngestRequest, contents: tuple[str, ...]
|
||||
) -> HyMemoryIngestResult:
|
||||
await self._start(request)
|
||||
try:
|
||||
result = await self.bridge.ingest_extracted(request, contents)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
return await self._with_retired(result)
|
||||
|
||||
async def _with_retired(self, result: HyMemoryIngestResult) -> HyMemoryIngestResult:
|
||||
await self._check()
|
||||
if self.wiring is None:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
try:
|
||||
retired, superseded = await _joined_thread(
|
||||
self.wiring.client._loop_thread.run, self.wiring.retired()
|
||||
)
|
||||
except HyPermissionTransportError as error:
|
||||
await self._recheck()
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
return result.model_copy(
|
||||
update={"retired_memory_ids": retired, "superseded_memory_ids": superseded}
|
||||
)
|
||||
|
||||
async def _check(self) -> dict[str, HyNodePermission]:
|
||||
if self.wiring is None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._recheck()
|
||||
try:
|
||||
return await _joined_thread(self.wiring.client._loop_thread.run, self.wiring.finish())
|
||||
except HyPermissionTransportError as error:
|
||||
await self._recheck()
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
|
||||
async def profile_memories(self, *, user_id: str, agent_id: str) -> tuple[HyMemoryOutput, ...]:
|
||||
if self.wiring is None or user_id != self.lease.user_id or agent_id != self.wiring.agent_id:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
result = await self.bridge.profile_memories(user_id=user_id, agent_id=agent_id)
|
||||
proofs = await self._check()
|
||||
with self._step("permission_store_write"):
|
||||
await asyncio.to_thread(self.proofs.save, self.lease, proofs)
|
||||
# SDK context closes before governance can update a previously known memory.
|
||||
await self.close()
|
||||
return result
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.wiring is not None and not self.finished:
|
||||
self.finished = True
|
||||
try:
|
||||
await _joined_thread(self.wiring.client._loop_thread.run, self.wiring.exit())
|
||||
finally:
|
||||
self.step_timer.emit(request_id=self.request_id, task_id=self.lease.task_id)
|
||||
|
||||
|
||||
def permission_bridge_factory(
|
||||
resolve_bridge: Callable[..., Any],
|
||||
*,
|
||||
http_transport: httpx.AsyncBaseTransport | None = None,
|
||||
) -> Callable[..., Any]:
|
||||
@asynccontextmanager
|
||||
async def factory(
|
||||
lease: ModelProcessingPermissionLease, secret: str | None, data_dir: Path
|
||||
) -> AsyncIterator[Any]:
|
||||
bridge = resolve_bridge(lease.profile, secret, data_dir)
|
||||
if not isinstance(bridge, HyMemoryBridge):
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
proofs = _NativeProofs(data_dir / "one-memory.sqlite3")
|
||||
async with bridge.exclusive_access():
|
||||
await bridge.prepare()
|
||||
bound = _LeasedBridge(bridge, lease, proofs, http_transport)
|
||||
try:
|
||||
yield bound
|
||||
except BaseException as error:
|
||||
with suppress(Exception):
|
||||
await bound.close()
|
||||
if isinstance(error, HyPermissionTransportError):
|
||||
await _joined_thread(lease.recheck)
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
raise
|
||||
else:
|
||||
try:
|
||||
await bound.close()
|
||||
except HyPermissionTransportError as error:
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
|
||||
return factory
|
||||
@@ -1,880 +0,0 @@
|
||||
"""Operation-bound read/write capability for the pinned native HY chat vector store.
|
||||
|
||||
No semantic storage or extraction lives here. Pre-existing nodes require a full
|
||||
native-node attestation with complete approved Evidence provenance; known content
|
||||
alone cannot authorize tags, custom fields, vectors or links. Unknown nodes stay
|
||||
in the same HY backend but are not exposed to this operation.
|
||||
|
||||
The caller must exclusively own the native backend's writes during execution:
|
||||
its API has no compare-and-swap, and a swallowed get_by_id error can look missing.
|
||||
This component cannot make that check/upsert interval atomic. Factory integration
|
||||
must replace client/registry/both writers/cached-reader references, bind physical
|
||||
LLM/embedding sends to the lease AND this denial latch, and guard publication.
|
||||
Coding rows go through hy_permission_coding_store instead; graph, entities,
|
||||
digest, sweeper and summary remain unsupported.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import math
|
||||
import struct
|
||||
import weakref
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable, Mapping
|
||||
from contextlib import asynccontextmanager, nullcontext
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from hashlib import sha256
|
||||
from types import MappingProxyType
|
||||
from typing import Any, NoReturn
|
||||
|
||||
from hy_memory.models.memory import ( # type: ignore[import-untyped]
|
||||
MemoryLayer,
|
||||
MemoryNode,
|
||||
MemoryStatus,
|
||||
)
|
||||
|
||||
from one_memory.hy_step_timing import HyStepTimer
|
||||
from one_memory.model_processing_permission_guard import (
|
||||
ModelProcessingPermissionLease,
|
||||
permission_error,
|
||||
)
|
||||
|
||||
_WRITE_LOCKS: weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
|
||||
weakref.WeakKeyDictionary()
|
||||
)
|
||||
|
||||
|
||||
def _native_write_lock() -> asyncio.Lock:
|
||||
loop = asyncio.get_running_loop()
|
||||
lock = _WRITE_LOCKS.get(loop)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
_WRITE_LOCKS[loop] = lock
|
||||
return lock
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def native_store_read_slot() -> AsyncIterator[bool]:
|
||||
"""Admit one native HY read only while no write is active.
|
||||
|
||||
Chroma's native reader may retain the GIL after an asyncio timeout. Running
|
||||
it concurrently with HY's multi-step store writes has deadlocked the Core
|
||||
under sustained imports. Reads therefore share the physical store lock,
|
||||
but never wait behind a writer: host recall can immediately use the local
|
||||
governance ledger instead.
|
||||
"""
|
||||
lock = _native_write_lock()
|
||||
if lock.locked():
|
||||
yield False
|
||||
return
|
||||
await lock.acquire()
|
||||
try:
|
||||
yield True
|
||||
finally:
|
||||
lock.release()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HyNodePermission:
|
||||
"""Trusted caller's full-node attestation, not permission inferred from an ID."""
|
||||
|
||||
fingerprint: str
|
||||
evidence_ids: tuple[str, ...]
|
||||
|
||||
|
||||
def hy_node_fingerprint(node: MemoryNode) -> str:
|
||||
"""Hash the complete native serialized node, including vectors and metadata.
|
||||
|
||||
Timestamp precision follows HY's to_dict contract (integer Unix seconds).
|
||||
Attest the node read from the backend, not an unpersisted input object.
|
||||
"""
|
||||
if type(node) is not MemoryNode:
|
||||
raise permission_error("hy_store_node_shape")
|
||||
return sha256(
|
||||
json.dumps(node.to_dict(), sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
_CHAT_LAYERS = frozenset(
|
||||
{
|
||||
MemoryLayer.L0_BASIC_INFO,
|
||||
MemoryLayer.L1_RAW,
|
||||
MemoryLayer.L2_FACT,
|
||||
MemoryLayer.L4_IDENTITY,
|
||||
MemoryLayer.L7_INTENTION,
|
||||
}
|
||||
)
|
||||
_UPDATE_FIELDS = frozenset(
|
||||
{
|
||||
"content",
|
||||
"embedding",
|
||||
"tags",
|
||||
"tag_ners",
|
||||
"layer",
|
||||
"memory_at",
|
||||
"gmt_modified",
|
||||
"is_latest",
|
||||
"status",
|
||||
"supersedes",
|
||||
"superseded_by",
|
||||
"source_raw_memory_id",
|
||||
# SDK 2.0.2 sends both of these on an in-place reconcile UPDATE.
|
||||
# They are metadata only and must never carry identity, permission or
|
||||
# tenant state: `owner` is a provenance label whose only values are
|
||||
# "user" / "agent" (who said this line), and `custom` is the SDK's own
|
||||
# free-form notes field. Neither may hold a user_id, scope, grant or
|
||||
# tenancy marker, so allowing the field names does not widen the
|
||||
# cross-user boundary that the store still enforces per node.
|
||||
"owner",
|
||||
"custom",
|
||||
}
|
||||
)
|
||||
_DEDUP_AUDIT_FIELDS = frozenset({"shadow_reason", "dedup_kept_id"})
|
||||
_DIGEST_COUNTER_FIELD = "s2_evidence_count"
|
||||
|
||||
|
||||
def _write_view(node: MemoryNode) -> dict[str, Any]:
|
||||
"""Only native Chroma's empty-string and float32 storage normalization."""
|
||||
payload: dict[str, Any] = node.to_dict()
|
||||
payload["source_session_id"] = payload["source_session_id"] or ""
|
||||
# The pinned public SDK serializes tag_ners as JSON, but its Chroma read path omits this
|
||||
# field from the JSON decode list. Decode only the exact list-of-strings
|
||||
# shape that the write path produced; malformed values still fail the
|
||||
# complete-node comparison below.
|
||||
tag_ners = payload.get("tag_ners")
|
||||
if (
|
||||
isinstance(tag_ners, list)
|
||||
and tag_ners
|
||||
and all(isinstance(value, str) and len(value) == 1 for value in tag_ners)
|
||||
and tag_ners[0] == "["
|
||||
and tag_ners[-1] == "]"
|
||||
):
|
||||
tag_ners = "".join(tag_ners)
|
||||
if isinstance(tag_ners, str):
|
||||
try:
|
||||
decoded = json.loads(tag_ners)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
decoded = None
|
||||
if isinstance(decoded, list) and all(isinstance(value, str) for value in decoded):
|
||||
payload["tag_ners"] = decoded
|
||||
if payload.get("tag_ners") in ([], ["[", "]"]):
|
||||
payload["tag_ners"] = []
|
||||
# HY Chroma encodes None as ""; these JSON fields keep "" when json.loads
|
||||
# fails (vector_store_chroma.py:741-788). Only an absent link is equivalent.
|
||||
for field in ("supersedes", "superseded_by"):
|
||||
if payload[field] == "":
|
||||
payload[field] = None
|
||||
if payload["embedding"] is not None:
|
||||
payload["embedding"] = [
|
||||
struct.unpack("f", struct.pack("f", n))[0] for n in payload["embedding"]
|
||||
]
|
||||
if not all(math.isfinite(n) for n in payload["embedding"]):
|
||||
raise ValueError("nonfinite_embedding")
|
||||
return payload
|
||||
|
||||
|
||||
def _same_written_vector(actual: list[float] | None, expected: list[float] | None) -> bool:
|
||||
if actual == expected:
|
||||
return True
|
||||
if actual is None or expected is None or len(actual) != len(expected):
|
||||
return False
|
||||
# Chroma's cosine index can renormalize float32 vectors on readback. Bound
|
||||
# only this write roundtrip; the saved full-node proof uses exact readback.
|
||||
return all(
|
||||
math.isclose(left, right, rel_tol=4 * 2**-23, abs_tol=0.0)
|
||||
for left, right in zip(actual, expected, strict=True)
|
||||
)
|
||||
|
||||
|
||||
_NATIVE_WRITES = frozenset(
|
||||
{"upsert", "upsert_batch", "update_payload", "update_embedding", "delete"}
|
||||
)
|
||||
|
||||
|
||||
class HyPermissionGraphStore:
|
||||
"""Lease-gated graph store for HY System 2's digest transaction.
|
||||
|
||||
The native graph is deliberately absent from ordinary ingest leases. Digest
|
||||
needs it to write L6 schemas, but every operation must remain inside the
|
||||
same frozen permission lease rather than exposing the raw store.
|
||||
"""
|
||||
|
||||
def __init__(self, native: Any, recheck: Any) -> None:
|
||||
self._native = native
|
||||
self._recheck = recheck
|
||||
self._closed = False
|
||||
|
||||
def recheck(self) -> None:
|
||||
if self._closed:
|
||||
raise permission_error("hy_graph_store_closed")
|
||||
self._recheck()
|
||||
|
||||
async def close(self) -> None:
|
||||
self._closed = True
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
if name.startswith("_") and name != "_run":
|
||||
raise AttributeError(name)
|
||||
attribute = getattr(self._native, name)
|
||||
if not callable(attribute):
|
||||
return attribute
|
||||
|
||||
async def guarded(*args: Any, **kwargs: Any) -> Any:
|
||||
self.recheck()
|
||||
result = await attribute(*args, **kwargs)
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
return guarded
|
||||
|
||||
|
||||
class HyPermissionVectorStore:
|
||||
"""Closed, task-exclusive adapter; initialize/close never own the native store.
|
||||
|
||||
node_permissions may attest additional native raw/profile nodes not represented
|
||||
in known_memories. Every attestation must cover all its contributors, and all
|
||||
Evidence IDs must already be in this lease. Known entries also constrain exact
|
||||
content/owner. Without a full attestation even a known entry is denied.
|
||||
|
||||
written_nodes records successful writes with the entire lease footprint, a
|
||||
conservative provenance bound, not a claim of semantic citation accuracy.
|
||||
These transient fingerprints must be persisted by the integrating factory if
|
||||
future operations are to reuse raw/profile nodes. No content is stored here.
|
||||
"""
|
||||
|
||||
__slots__ = (
|
||||
"__native",
|
||||
"__lease",
|
||||
"__agent_id",
|
||||
"__session_id",
|
||||
"__cross_evidence_scope",
|
||||
"__user_id",
|
||||
"__known",
|
||||
"__permissions",
|
||||
"__written",
|
||||
"__evidence_ids",
|
||||
"__deleted",
|
||||
"__shadowed",
|
||||
"__superseded",
|
||||
"__heads",
|
||||
"__lock",
|
||||
"__closed",
|
||||
"__failure",
|
||||
"__step_timer",
|
||||
"__refresh_permission",
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
native: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
*,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
cross_evidence_scope: bool = False,
|
||||
node_permissions: Mapping[str, HyNodePermission] | None = None,
|
||||
refresh_permission: Callable[[str], Awaitable[HyNodePermission | None]] | None = None,
|
||||
step_timer: HyStepTimer | None = None,
|
||||
) -> None:
|
||||
self.__step_timer = step_timer
|
||||
self.__native = native
|
||||
self.__lease = lease
|
||||
self.__agent_id = agent_id
|
||||
self.__session_id = session_id
|
||||
self.__cross_evidence_scope = cross_evidence_scope
|
||||
self.__user_id = lease.user_id
|
||||
self.__failure: str | None = None
|
||||
self.__closed = False
|
||||
self.__lock = asyncio.Lock()
|
||||
self.__deleted: set[str] = set()
|
||||
self.__written: dict[str, HyNodePermission] = {}
|
||||
self.__shadowed: set[str] = set()
|
||||
self.__superseded: dict[str, tuple[str, ...]] = {}
|
||||
self.__heads: dict[str, tuple[str, ...]] = {}
|
||||
self.__refresh_permission = refresh_permission
|
||||
self.recheck()
|
||||
self.__evidence_ids = tuple(record.evidence_id for record in lease.records)
|
||||
current = lease.source_context.current if lease.source_context else lease.records
|
||||
conversation_mismatch = any(r.conversation_id != session_id for r in current)
|
||||
if (
|
||||
not lease.task_id
|
||||
or not agent_id
|
||||
or agent_id == "*"
|
||||
or not session_id
|
||||
or not current
|
||||
or (conversation_mismatch and not cross_evidence_scope)
|
||||
or not self.__evidence_ids
|
||||
or len(set(self.__evidence_ids)) != len(lease.records)
|
||||
or any(r.user_id != self.__user_id for r in lease.records)
|
||||
):
|
||||
self.__deny("scope")
|
||||
known = lease.source_context.known_memories if lease.source_context else ()
|
||||
self.__known = {item.hy_memory_id: item for item in known}
|
||||
if len(self.__known) != len(known):
|
||||
self.__deny("provenance")
|
||||
self.__permissions = dict(node_permissions or {})
|
||||
for key, proof in self.__permissions.items():
|
||||
if (
|
||||
not key
|
||||
or not isinstance(proof, HyNodePermission)
|
||||
or len(proof.fingerprint) != 64
|
||||
or any(c not in "0123456789abcdef" for c in proof.fingerprint)
|
||||
or not isinstance(proof.evidence_ids, tuple)
|
||||
or not proof.evidence_ids
|
||||
or len(set(proof.evidence_ids)) != len(proof.evidence_ids)
|
||||
or not set(proof.evidence_ids) <= set(self.__evidence_ids)
|
||||
):
|
||||
self.__deny("provenance")
|
||||
for key, item in self.__known.items():
|
||||
if not item.evidence_ids or not set(item.evidence_ids) <= set(self.__evidence_ids):
|
||||
self.__deny("provenance")
|
||||
known_proof = self.__permissions.get(key)
|
||||
if known_proof and not set(item.evidence_ids) <= set(known_proof.evidence_ids):
|
||||
self.__deny("provenance")
|
||||
|
||||
@property
|
||||
def written_nodes(self) -> Mapping[str, HyNodePermission]:
|
||||
"""Latest verified write proofs; each access returns a new immutable snapshot.
|
||||
|
||||
Export after SDK writes have drained and before close/governance projection.
|
||||
Merge these rows into the lineage ledger, then remove deleted_node_ids;
|
||||
this write delta does not include untouched pre-existing permissions.
|
||||
"""
|
||||
self.recheck()
|
||||
return MappingProxyType(dict(self.__written))
|
||||
|
||||
@property
|
||||
def deleted_node_ids(self) -> frozenset[str]:
|
||||
"""Verified native deletions in this operation, for lineage-ledger cleanup."""
|
||||
self.recheck()
|
||||
return frozenset(self.__deleted)
|
||||
|
||||
@property
|
||||
def retired_node_ids(self) -> frozenset[str]:
|
||||
"""Nodes HY deleted or shadowed (its logical delete) in this operation."""
|
||||
self.recheck()
|
||||
return frozenset(self.__deleted | self.__shadowed)
|
||||
|
||||
@property
|
||||
def superseded_nodes(self) -> Mapping[str, tuple[str, ...]]:
|
||||
"""Nodes HY marked superseded in this operation, with what replaced them.
|
||||
|
||||
HY writes the two directions separately and may fail either one, so a
|
||||
replacement counts whether it was recorded on the old node or only on
|
||||
the new head.
|
||||
"""
|
||||
self.recheck()
|
||||
result = {}
|
||||
for key, by in self.__superseded.items():
|
||||
heads = set(by) | {head for head, olds in self.__heads.items() if key in olds}
|
||||
heads -= self.__deleted | {key}
|
||||
if key not in self.__deleted and heads:
|
||||
result[key] = tuple(sorted(heads))
|
||||
return MappingProxyType(result)
|
||||
|
||||
@property
|
||||
def supports_fulltext(self) -> bool:
|
||||
return False
|
||||
|
||||
def recheck(self) -> None:
|
||||
"""Must also gate transport/publication after HY swallows a store failure."""
|
||||
self.__assert_open()
|
||||
try:
|
||||
self.__lease.recheck()
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
self.__failure = (
|
||||
code
|
||||
if code.startswith("model_content_") and len(code) <= 100
|
||||
else "model_content_hy_store_lease_failed"
|
||||
)
|
||||
raise permission_error(self.__failure) from None
|
||||
|
||||
def __assert_open(self) -> None:
|
||||
if self.__closed:
|
||||
self.__deny("closed")
|
||||
if self.__failure:
|
||||
raise permission_error(self.__failure)
|
||||
|
||||
async def initialize(self) -> None:
|
||||
self.recheck()
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Revoke this wrapper locally without rechecking context or closing HY.
|
||||
|
||||
Cleanup must remain possible after governance projection changes the
|
||||
lease's known-memory revisions, or after permission has been revoked.
|
||||
"""
|
||||
async with self.__lock:
|
||||
self.__closed = True
|
||||
self.__permissions.clear()
|
||||
self.__written.clear()
|
||||
|
||||
async def has_tag_embedding(self, user_id: str, tag: str) -> bool:
|
||||
self.recheck()
|
||||
# Native chat calls this best-effort helper. NotImplemented prevents it
|
||||
# from embedding tags or falling through to an unguarded entity store.
|
||||
raise NotImplementedError("hy_permission_store_tag_index_unsupported")
|
||||
|
||||
def __deny(self, suffix: str) -> NoReturn:
|
||||
self.__failure = self.__failure or f"model_content_hy_store_{suffix}"
|
||||
raise permission_error(self.__failure)
|
||||
|
||||
async def __invoke(self, method: str, *args: Any, **kwargs: Any) -> Any:
|
||||
writing = method in _NATIVE_WRITES
|
||||
self.recheck() if writing else self.__assert_open()
|
||||
step = "vector_store_write" if writing else "vector_store_read"
|
||||
first = args[0] if args else None
|
||||
try:
|
||||
with (
|
||||
self.__step_timer.step(step, items=len(first) if isinstance(first, list) else 1)
|
||||
if self.__step_timer is not None
|
||||
else nullcontext()
|
||||
):
|
||||
result = await getattr(self.__native, method)(*args, **kwargs)
|
||||
except Exception:
|
||||
self.__deny("backend_failed")
|
||||
self.recheck() if writing else self.__assert_open()
|
||||
return result
|
||||
|
||||
async def __read_native(self, key: str) -> MemoryNode | None:
|
||||
value = await self.__invoke("get_by_id", key)
|
||||
if value is not None and (type(value) is not MemoryNode or value.node_id != key):
|
||||
self.__deny("node_shape")
|
||||
return deepcopy(value)
|
||||
|
||||
def __fingerprint(self, node: MemoryNode) -> str:
|
||||
try:
|
||||
return hy_node_fingerprint(node)
|
||||
except (TypeError, ValueError, AttributeError, OverflowError):
|
||||
self.__deny("node_shape")
|
||||
|
||||
def __write_view(self, node: MemoryNode) -> dict[str, Any]:
|
||||
self.__fingerprint(node)
|
||||
try:
|
||||
return _write_view(node)
|
||||
except (TypeError, ValueError, OverflowError, struct.error):
|
||||
self.__deny("node_shape")
|
||||
|
||||
def __in_scope(self, node: MemoryNode) -> bool:
|
||||
if node.user_id != self.__user_id or node.layer not in _CHAT_LAYERS:
|
||||
return False
|
||||
if node.layer is MemoryLayer.L0_BASIC_INFO:
|
||||
# Public SDK 2.0.2 migrates verified legacy agent-level profiles into one
|
||||
# user-level chain. Existing nodes still require a full proof in
|
||||
# __load; accepting their historical agent id does not authorize
|
||||
# unproven cross-agent memories.
|
||||
return True
|
||||
if self.__cross_evidence_scope:
|
||||
return bool(node.agent_id == self.__agent_id)
|
||||
return bool(node.agent_id == self.__agent_id and node.session_id == self.__session_id)
|
||||
|
||||
def __new_node_in_scope(self, node: MemoryNode) -> bool:
|
||||
if not self.__in_scope(node):
|
||||
return False
|
||||
if node.layer is MemoryLayer.L0_BASIC_INFO:
|
||||
return node.agent_id in ("", self.__agent_id) and node.session_id in (
|
||||
"",
|
||||
self.__session_id,
|
||||
)
|
||||
return True
|
||||
|
||||
def __references(self, node: MemoryNode) -> list[str]:
|
||||
refs = []
|
||||
for field in ("supersedes", "superseded_by", "evidence_chain"):
|
||||
values = getattr(node, field)
|
||||
if field != "evidence_chain" and values == "":
|
||||
continue
|
||||
if values is not None:
|
||||
if not isinstance(values, list) or any(
|
||||
not isinstance(v, str) or not v for v in values
|
||||
):
|
||||
self.__deny("node_shape")
|
||||
refs.extend(values)
|
||||
if node.source_raw_memory_id is not None:
|
||||
if not isinstance(node.source_raw_memory_id, str) or not node.source_raw_memory_id:
|
||||
self.__deny("node_shape")
|
||||
refs.append(node.source_raw_memory_id)
|
||||
return refs
|
||||
|
||||
async def __load(
|
||||
self,
|
||||
key: str,
|
||||
first: MemoryNode | None = None,
|
||||
*,
|
||||
allow_deleted_links: bool = False,
|
||||
) -> MemoryNode | None:
|
||||
pending = [key]
|
||||
checked: dict[str, MemoryNode] = {}
|
||||
while pending:
|
||||
current = pending.pop()
|
||||
if current in checked:
|
||||
continue
|
||||
# HY dedup deletes head then ancestors. Only this operation's
|
||||
# verified tombstones may terminate a closure during deletion.
|
||||
if allow_deleted_links and current != key and current in self.__deleted:
|
||||
continue
|
||||
proof = self.__permissions.get(current)
|
||||
if proof is None or current in self.__deleted:
|
||||
return None
|
||||
value = (
|
||||
first if current == key and first is not None else await self.__read_native(current)
|
||||
)
|
||||
if value is None or not self.__in_scope(value):
|
||||
return None
|
||||
if self.__fingerprint(value) != proof.fingerprint:
|
||||
refreshed = (
|
||||
await self.__refresh_permission(current)
|
||||
if self.__refresh_permission is not None
|
||||
else None
|
||||
)
|
||||
if refreshed is None or self.__fingerprint(value) != refreshed.fingerprint:
|
||||
self.__deny("node_changed")
|
||||
self.__permissions[current] = refreshed
|
||||
proof = refreshed
|
||||
known = self.__known.get(current) if current not in self.__written else None
|
||||
if known and (value.content != known.content or value.owner != known.owner):
|
||||
self.__deny("node_changed")
|
||||
checked[current] = value
|
||||
pending.extend(self.__references(value))
|
||||
return deepcopy(checked[key])
|
||||
|
||||
async def get_by_id(self, node_id: str) -> MemoryNode | None:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
result = await self.__load(node_id)
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
async def get_by_ids(self, node_ids: list[str]) -> list[MemoryNode]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
result = [
|
||||
n for key in dict.fromkeys(node_ids) if (n := await self.__load(key)) is not None
|
||||
]
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
async def get_embeddings(self, node_ids: list[str]) -> dict[str, list[float]]:
|
||||
return {
|
||||
n.node_id: list(n.embedding)
|
||||
for n in await self.get_by_ids(node_ids)
|
||||
if n.embedding is not None
|
||||
}
|
||||
|
||||
def __scope(self, user_id: str | None, agent_id: str | None) -> None:
|
||||
if user_id not in (None, self.__user_id) or agent_id not in (None, self.__agent_id):
|
||||
self.__deny("scope")
|
||||
|
||||
async def list_by_user(
|
||||
self,
|
||||
user_id: str | None = None,
|
||||
agent_id: str | None = None,
|
||||
limit: int = 10000,
|
||||
status_filter: list[Any] | None = None,
|
||||
layers: list[Any] | None = None,
|
||||
user_ids: list[str] | None = None,
|
||||
agent_ids: list[str] | None = None,
|
||||
only_latest: bool = False,
|
||||
with_vectors: bool = True,
|
||||
allow_raw_layer: bool = False,
|
||||
) -> list[MemoryNode]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
effective_users = user_ids or ([user_id] if user_id else [])
|
||||
effective_agents = agent_ids or ([agent_id] if agent_id else [])
|
||||
if effective_users != [self.__user_id] or any(
|
||||
value != self.__agent_id for value in effective_agents
|
||||
):
|
||||
self.__deny("scope")
|
||||
rows = await self.__invoke(
|
||||
"list_by_user",
|
||||
user_id=self.__user_id,
|
||||
agent_id=agent_id,
|
||||
limit=limit,
|
||||
status_filter=status_filter,
|
||||
layers=layers,
|
||||
user_ids=[self.__user_id],
|
||||
agent_ids=effective_agents or None,
|
||||
only_latest=only_latest,
|
||||
with_vectors=with_vectors,
|
||||
allow_raw_layer=allow_raw_layer,
|
||||
)
|
||||
result = []
|
||||
for row in rows:
|
||||
if type(row) is not MemoryNode:
|
||||
self.__deny("node_shape")
|
||||
value = await self.__load(row.node_id)
|
||||
if (
|
||||
value is not None
|
||||
and (not layers or value.layer in layers or value.layer.value in layers)
|
||||
and (
|
||||
not status_filter
|
||||
or value.status in status_filter
|
||||
or value.status.value in status_filter
|
||||
)
|
||||
):
|
||||
result.append(value)
|
||||
self.recheck()
|
||||
return result[:limit]
|
||||
|
||||
async def search(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
isolation_key: str = "",
|
||||
isolation_keys: list[str] | None = None,
|
||||
user_id: str | None = None,
|
||||
user_ids: list[str] | None = None,
|
||||
agent_ids: list[str] | None = None,
|
||||
layers: list[Any] | None = None,
|
||||
limit: int = 10,
|
||||
score_threshold: float = 0.0,
|
||||
status_filter: list[Any] | None = None,
|
||||
only_latest: bool = True,
|
||||
tags_match_any: list[str] | None = None,
|
||||
created_after: float | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
self.__scope(user_id, None)
|
||||
key = MemoryNode.build_isolation_key(self.__user_id, self.__agent_id, self.__session_id)
|
||||
if (
|
||||
any(uid != self.__user_id for uid in user_ids or [])
|
||||
or any(aid != self.__agent_id for aid in agent_ids or [])
|
||||
or isolation_key not in ("", key)
|
||||
or any(ik != key for ik in isolation_keys or [])
|
||||
):
|
||||
self.__deny("scope")
|
||||
hits = await self.__invoke(
|
||||
"search",
|
||||
query_embedding=deepcopy(query_embedding),
|
||||
isolation_key=isolation_key,
|
||||
isolation_keys=isolation_keys,
|
||||
user_id=self.__user_id,
|
||||
user_ids=[self.__user_id],
|
||||
agent_ids=[self.__agent_id],
|
||||
layers=layers,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold,
|
||||
status_filter=status_filter,
|
||||
only_latest=only_latest,
|
||||
tags_match_any=tags_match_any,
|
||||
created_after=created_after,
|
||||
)
|
||||
result = []
|
||||
for hit in hits:
|
||||
value = await self.__load(hit["node_id"])
|
||||
if value is None:
|
||||
continue
|
||||
score = float(hit["score"])
|
||||
if not math.isfinite(score):
|
||||
self.__deny("node_shape")
|
||||
# Do not forward arbitrary backend hit metadata. Rehydrate the
|
||||
# full fingerprinted node; native search often omits embedding.
|
||||
result.append({"node_id": value.node_id, "node": value, "score": score})
|
||||
self.recheck()
|
||||
return result[:limit]
|
||||
|
||||
async def __target(
|
||||
self,
|
||||
key: str,
|
||||
first: MemoryNode | None = None,
|
||||
*,
|
||||
allow_deleted_links: bool = False,
|
||||
) -> MemoryNode:
|
||||
value = await self.__load(key, first, allow_deleted_links=allow_deleted_links)
|
||||
if value is None:
|
||||
self.__deny("target_denied")
|
||||
return value
|
||||
|
||||
async def __check_links(self, value: MemoryNode) -> None:
|
||||
for ref in self.__references(value):
|
||||
if await self.__load(ref) is None:
|
||||
self.__deny("link_denied")
|
||||
|
||||
async def __plan_upsert(self, value: MemoryNode) -> MemoryNode:
|
||||
value = deepcopy(value)
|
||||
if type(value) is not MemoryNode or not value.node_id or not self.__in_scope(value):
|
||||
self.__deny("scope")
|
||||
existing = await self.__read_native(value.node_id)
|
||||
if existing is not None:
|
||||
await self.__target(value.node_id, existing)
|
||||
elif (
|
||||
not self.__new_node_in_scope(value)
|
||||
or value.node_id in self.__permissions
|
||||
or value.node_id in self.__known
|
||||
or value.node_id in self.__deleted
|
||||
):
|
||||
self.__deny("scope")
|
||||
await self.__check_links(value)
|
||||
self.__write_view(value)
|
||||
return value
|
||||
|
||||
async def __capture(
|
||||
self,
|
||||
expected: MemoryNode,
|
||||
*,
|
||||
allow_native_modified_time: bool = False,
|
||||
) -> None:
|
||||
actual = await self.__read_native(expected.node_id)
|
||||
if actual is None:
|
||||
self.__deny("write_mismatch")
|
||||
actual_view, expected_view = self.__write_view(actual), self.__write_view(expected)
|
||||
if allow_native_modified_time:
|
||||
if actual.gmt_modified < expected.gmt_modified:
|
||||
self.__deny("write_mismatch")
|
||||
actual_view["gmt_modified"] = expected_view["gmt_modified"]
|
||||
actual_vector = actual_view.pop("embedding")
|
||||
expected_vector = expected_view.pop("embedding")
|
||||
if actual_view != expected_view or not _same_written_vector(actual_vector, expected_vector):
|
||||
self.__deny("write_mismatch")
|
||||
proof = HyNodePermission(self.__fingerprint(actual), self.__evidence_ids)
|
||||
self.__permissions[actual.node_id] = proof
|
||||
self.__written[actual.node_id] = proof
|
||||
status = getattr(actual.status, "value", actual.status)
|
||||
if status == MemoryStatus.SHADOW.value:
|
||||
self.__shadowed.add(actual.node_id)
|
||||
else:
|
||||
self.__shadowed.discard(actual.node_id)
|
||||
if status == MemoryStatus.SUPERSEDED.value:
|
||||
by = actual.superseded_by
|
||||
self.__superseded[actual.node_id] = tuple(by) if isinstance(by, list) else ()
|
||||
else:
|
||||
self.__superseded.pop(actual.node_id, None)
|
||||
olds = actual.supersedes
|
||||
if isinstance(olds, list) and olds:
|
||||
self.__heads[actual.node_id] = tuple(olds)
|
||||
else:
|
||||
self.__heads.pop(actual.node_id, None)
|
||||
|
||||
async def upsert(self, node: MemoryNode) -> str:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
expected = await self.__plan_upsert(node)
|
||||
result = await self.__invoke("upsert", deepcopy(expected))
|
||||
if result != expected.node_id:
|
||||
self.__deny("write_mismatch")
|
||||
await self.__capture(expected)
|
||||
return str(result)
|
||||
|
||||
async def upsert_batch(self, nodes: list[MemoryNode]) -> list[str]:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
if len({n.node_id for n in nodes}) != len(nodes):
|
||||
self.__deny("node_shape")
|
||||
# No forward links to uncommitted batch members. Preflight every
|
||||
# existing target before the backend can make a partial write.
|
||||
planned = [await self.__plan_upsert(n) for n in nodes]
|
||||
ids = [n.node_id for n in planned]
|
||||
result = await self.__invoke("upsert_batch", deepcopy(planned))
|
||||
if result != ids:
|
||||
self.__deny("write_mismatch")
|
||||
for expected in planned:
|
||||
await self.__capture(expected)
|
||||
return ids
|
||||
|
||||
async def update_payload(self, node_id: str, updates: dict[str, Any]) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
current = await self.__target(node_id)
|
||||
updates = deepcopy(updates)
|
||||
dedup_audit = False
|
||||
if set(updates) & _DEDUP_AUDIT_FIELDS:
|
||||
if (
|
||||
not set(updates) <= (_UPDATE_FIELDS | _DEDUP_AUDIT_FIELDS)
|
||||
or updates.get("status") != MemoryStatus.SHADOW.value
|
||||
or updates.get("is_latest") is not False
|
||||
or not isinstance(updates.get("shadow_reason"), str)
|
||||
or not updates["shadow_reason"].startswith("dedup:")
|
||||
or not isinstance(updates.get("dedup_kept_id"), str)
|
||||
or not updates["dedup_kept_id"]
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
await self.__target(updates["dedup_kept_id"])
|
||||
custom = deepcopy(current.custom)
|
||||
custom["hy_dedup"] = {
|
||||
"reason": updates.pop("shadow_reason"),
|
||||
"kept_id": updates.pop("dedup_kept_id"),
|
||||
}
|
||||
updates["custom"] = custom
|
||||
dedup_audit = True
|
||||
allowed = _UPDATE_FIELDS | ({"custom"} if dedup_audit else set())
|
||||
if self.__cross_evidence_scope and set(updates) == {"custom"}:
|
||||
custom = updates.get("custom")
|
||||
if not isinstance(custom, dict):
|
||||
self.__deny("update_fields")
|
||||
current_custom = current.custom if isinstance(current.custom, dict) else {}
|
||||
previous = current_custom.get(_DIGEST_COUNTER_FIELD, 0)
|
||||
proposed = custom.get(_DIGEST_COUNTER_FIELD)
|
||||
if (
|
||||
not isinstance(previous, int)
|
||||
or isinstance(previous, bool)
|
||||
or not isinstance(proposed, int)
|
||||
or isinstance(proposed, bool)
|
||||
or proposed != previous + 1
|
||||
or set(custom) != set(current_custom) | {_DIGEST_COUNTER_FIELD}
|
||||
or any(custom[key] != value for key, value in current_custom.items())
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
allowed |= {"custom"}
|
||||
# `custom` is an allowed field name (SDK 2.0.2 sends it on a
|
||||
# reconcile UPDATE), but the System 2 evidence counter it can hold
|
||||
# stays digest-only: only the digest scope may advance it, and only
|
||||
# by one per write. Letting the name through must not let an
|
||||
# ordinary write move that counter.
|
||||
if not self.__cross_evidence_scope and _DIGEST_COUNTER_FIELD in (
|
||||
updates.get("custom") or {}
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
if not set(updates) <= allowed:
|
||||
self.__deny("update_fields")
|
||||
try:
|
||||
if "layer" in updates:
|
||||
MemoryLayer(updates["layer"])
|
||||
if "status" in updates:
|
||||
MemoryStatus(updates["status"])
|
||||
if "tag_ners" in updates and (
|
||||
not isinstance(updates["tag_ners"], list)
|
||||
or any(not isinstance(value, str) for value in updates["tag_ners"])
|
||||
):
|
||||
self.__deny("node_shape")
|
||||
expected = MemoryNode.from_dict({**current.to_dict(), **updates})
|
||||
self.__fingerprint(expected)
|
||||
except (TypeError, ValueError):
|
||||
self.__deny("node_shape")
|
||||
if not self.__in_scope(expected):
|
||||
self.__deny("scope")
|
||||
await self.__check_links(expected)
|
||||
self.__write_view(expected)
|
||||
if await self.__invoke("update_payload", node_id, updates) is not True:
|
||||
self.__deny("write_failed")
|
||||
await self.__capture(expected, allow_native_modified_time=True)
|
||||
return True
|
||||
|
||||
async def update_embedding(self, node_id: str, embedding: list[float]) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
expected = await self.__target(node_id)
|
||||
expected.embedding = deepcopy(embedding)
|
||||
self.__write_view(expected)
|
||||
if await self.__invoke("update_embedding", node_id, deepcopy(embedding)) is not True:
|
||||
self.__deny("write_failed")
|
||||
await self.__capture(expected, allow_native_modified_time=True)
|
||||
return True
|
||||
|
||||
async def delete(self, node_id: str) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
await self.__target(node_id, allow_deleted_links=True)
|
||||
if await self.__invoke("delete", node_id) is not True:
|
||||
self.__deny("write_failed")
|
||||
if await self.__read_native(node_id) is not None:
|
||||
self.__deny("write_mismatch")
|
||||
self.__deleted.add(node_id)
|
||||
self.__permissions.pop(node_id, None)
|
||||
self.__written.pop(node_id, None)
|
||||
self.__superseded.pop(node_id, None)
|
||||
self.__heads.pop(node_id, None)
|
||||
return True
|
||||
File diff suppressed because it is too large.
Load diff
@@ -144,6 +144,10 @@ class HyReconcileGuard:
|
||||
self.failures: list[str] = []
|
||||
self.bindings: HyWriteBindings | None = None
|
||||
self.imported_contents: frozenset[str] | None = None
|
||||
self.mutation_target_ids: set[str] = set()
|
||||
|
||||
def begin_write(self) -> None:
|
||||
self.mutation_target_ids.clear()
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(self.reconciler, name)
|
||||
@@ -210,6 +214,15 @@ class HyReconcileGuard:
|
||||
for operation in result.ops:
|
||||
raw_name = getattr(operation, "op", "")
|
||||
name = str(getattr(raw_name, "value", raw_name)).upper()
|
||||
if name in {"UPDATE", "SUPERSEDE", "DELETE", "SHADOW"}:
|
||||
target = getattr(operation, "memory_id", None)
|
||||
if isinstance(target, str) and target:
|
||||
self.mutation_target_ids.add(target)
|
||||
self.mutation_target_ids.update(
|
||||
key
|
||||
for key in getattr(operation, "memory_ids", ())
|
||||
if isinstance(key, str) and key
|
||||
)
|
||||
if name == "ADD":
|
||||
operation_counts["ADD"] += 1
|
||||
elif name in {"UPDATE", "MERGE", "NOOP"}:
|
||||
@@ -232,20 +245,6 @@ class HyReconcileGuard:
|
||||
"result_count": len(result.ops),
|
||||
},
|
||||
)
|
||||
timer = self.step_timer
|
||||
try:
|
||||
with (
|
||||
timer.step("write_verification", items=len(result.ops))
|
||||
if timer is not None
|
||||
else nullcontext()
|
||||
):
|
||||
await self._check_targets(result, kwargs)
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
if not code.startswith("hy_memory_write_verification_") or len(code) > 100:
|
||||
code = "hy_memory_write_verification_failed"
|
||||
self.failures.append(code)
|
||||
raise ValueError(code) from error
|
||||
return result
|
||||
|
||||
async def _check_targets(self, result: Any, kwargs: dict[str, Any]) -> None:
|
||||
|
||||
@@ -14,7 +14,7 @@ from pathlib import Path
|
||||
from typing import Literal, cast
|
||||
|
||||
HY_MEMORY_DISTRIBUTION = "hy-memory"
|
||||
HY_MEMORY_VERSION = "2.0.2"
|
||||
HY_MEMORY_VERSION = "2.0.3"
|
||||
HY_MEMORY_NAMESPACE = "governed-v2"
|
||||
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
|
||||
|
||||
@@ -35,7 +35,7 @@ _COLLECTION_LOGGERS = (
|
||||
# One Memory deliberately stays on ultra because its existing, user-visible
|
||||
# profile/graph flow calls digest(), which the pinned public SDK exposes only
|
||||
# for ultra clients. Reader and non-reranker behavior remain pinned for
|
||||
# compatibility with public SDK 2.0.2.
|
||||
# compatibility with public SDK 2.0.3.
|
||||
HY_MEMORY_MODE = "ultra"
|
||||
# The SDK only recognises `legacy` and `hybrid` (see
|
||||
# hy_memory/pipelines/_retrieval/config.py: `ALL_READERS = (READER_LEGACY,
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
"""Process-local coordination for HY's native stores."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import weakref
|
||||
from collections.abc import AsyncIterator
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
_WRITE_LOCKS: weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
|
||||
weakref.WeakKeyDictionary()
|
||||
)
|
||||
|
||||
|
||||
def native_store_write_lock() -> asyncio.Lock:
|
||||
"""Return the write lock for the current event loop."""
|
||||
loop = asyncio.get_running_loop()
|
||||
lock = _WRITE_LOCKS.get(loop)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
_WRITE_LOCKS[loop] = lock
|
||||
return lock
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def native_store_read_slot() -> AsyncIterator[bool]:
|
||||
"""Admit a native HY read only while no write is active.
|
||||
|
||||
Chroma's native reader may retain the GIL after an asyncio timeout. Reads
|
||||
therefore share the physical-store lock, but never wait behind a writer:
|
||||
host recall can immediately fall back to the governance ledger.
|
||||
"""
|
||||
lock = native_store_write_lock()
|
||||
if lock.locked():
|
||||
yield False
|
||||
return
|
||||
await lock.acquire()
|
||||
try:
|
||||
yield True
|
||||
finally:
|
||||
lock.release()
|
||||
@@ -316,6 +316,10 @@ class ImportSupervisor:
|
||||
"source_id": current.source_id,
|
||||
"mode": current.operation,
|
||||
"count": current.stored,
|
||||
"stored_count": current.stored,
|
||||
"revised_count": current.revised,
|
||||
"skipped_count": current.skipped,
|
||||
"error_count": current.errors,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -10,7 +10,6 @@ from fastapi.params import Depends
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.imports.models import (
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
@@ -186,6 +185,10 @@ class MemoryNoteStore:
|
||||
]
|
||||
if task is None or task.status == MemoryTaskStatus.WAITING_FOR_MODEL:
|
||||
state = "waiting_for_model"
|
||||
elif task.status == MemoryTaskStatus.WAITING_FOR_PERMISSION:
|
||||
# Upgrade compatibility: old rows are claimable and will leave
|
||||
# this state as soon as the direct SDK runner claims them.
|
||||
state = "waiting_for_permission"
|
||||
elif task.status == MemoryTaskStatus.COMPLETED:
|
||||
state = "ready" if memory_ids else "no_memories"
|
||||
elif task.status == MemoryTaskStatus.RUNNING:
|
||||
@@ -195,22 +198,6 @@ class MemoryNoteStore:
|
||||
else:
|
||||
state = "failed"
|
||||
error = task.last_error_code if task else None
|
||||
if self.processor.model_content_config is not None and state in {
|
||||
"waiting_for_model",
|
||||
"failed",
|
||||
}:
|
||||
try:
|
||||
active = self.processor.profiles.active(self.processor.secrets)
|
||||
if active is not None:
|
||||
self.processor._guard_records((record,), active)
|
||||
except HyMemoryBridgeError as failure:
|
||||
if str(failure).startswith("model_content_"):
|
||||
state = "waiting_for_permission"
|
||||
error = str(failure)
|
||||
else:
|
||||
raise
|
||||
except SecretStoreUnavailableError:
|
||||
error = "secure_storage_unavailable"
|
||||
return MemoryNoteReceipt(
|
||||
evidence_id=record.evidence_id,
|
||||
text=sanitize_model_input(record.verbatim),
|
||||
@@ -274,11 +261,8 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
# unchanged; replaying the same request returns the original
|
||||
# `entry:<evidence_id>` job instead of a second HY call.
|
||||
if readmit:
|
||||
# `/{evidence_id}/process` is the explicit "the condition
|
||||
# that parked it may have changed" signal: the person just
|
||||
# granted consent or fixed the model. `submit_evidence`
|
||||
# alone would return an existing parked row unchanged, so
|
||||
# without this readmit the note stays parked forever.
|
||||
# `/{evidence_id}/process` is the explicit retry signal.
|
||||
# It also revives pre-upgrade permission waiters.
|
||||
await asyncio.to_thread(notes.intake.queue.readmit_parked)
|
||||
submitted = await asyncio.to_thread(
|
||||
notes.intake.submit_evidence,
|
||||
@@ -296,10 +280,6 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
last_round=1,
|
||||
priority="live",
|
||||
)
|
||||
except HyMemoryBridgeError as error:
|
||||
if not str(error).startswith("model_content_"):
|
||||
raise
|
||||
return await asyncio.to_thread(notes.receipt, record.evidence_id)
|
||||
except SecretStoreUnavailableError:
|
||||
saved = await asyncio.to_thread(notes.receipt, record.evidence_id)
|
||||
return saved.model_copy(
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -29,7 +29,6 @@ class ConfigDocument(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
|
||||
|
||||
schema_version: int = Field(default=1, ge=1, le=1)
|
||||
model_content_allowed: StrictBool = False
|
||||
sources: dict[str, SourceConfig] = Field(default_factory=dict)
|
||||
|
||||
@field_validator("sources")
|
||||
@@ -64,10 +63,6 @@ class ConfigStore:
|
||||
source = self._document.sources.get(source_id)
|
||||
return bool(source and source.sync_enabled)
|
||||
|
||||
def model_content_allowed(self) -> bool:
|
||||
with self._lock:
|
||||
return self._document.model_content_allowed
|
||||
|
||||
def set_sync(self, source_id: str, enabled: bool) -> ConfigDocument:
|
||||
return self.set_sync_many({source_id: enabled})
|
||||
|
||||
@@ -86,20 +81,17 @@ class ConfigStore:
|
||||
self._document = updated
|
||||
return updated
|
||||
|
||||
def set_model_content_allowed(self, allowed: bool) -> ConfigDocument:
|
||||
if type(allowed) is not bool:
|
||||
raise ValueError("invalid_model_content_config")
|
||||
with self._lock:
|
||||
updated = self._document.model_copy(update={"model_content_allowed": allowed})
|
||||
self._write(updated)
|
||||
self._document = updated
|
||||
return updated
|
||||
|
||||
def _read(self) -> ConfigDocument:
|
||||
if not self.path.exists():
|
||||
return ConfigDocument()
|
||||
try:
|
||||
return ConfigDocument.model_validate_json(self.path.read_text(encoding="utf-8"))
|
||||
value = json.loads(self.path.read_text(encoding="utf-8"))
|
||||
if isinstance(value, dict):
|
||||
# Builds through 1.0.2 persisted this global consent bit.
|
||||
# The embedded SDK is now trusted to use all local memory, so
|
||||
# accept existing files while discarding the retired setting.
|
||||
value.pop("model_content_allowed", None)
|
||||
return ConfigDocument.model_validate(value)
|
||||
except (OSError, ValueError) as error:
|
||||
raise ConfigFileError("config_invalid") from error
|
||||
|
||||
|
||||
@@ -306,7 +306,7 @@ LEGACY_MEMORY_WRITE_JOB_COLUMNS = (
|
||||
("last_error", "TEXT"),
|
||||
)
|
||||
|
||||
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL, UNKNOWN)
|
||||
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -1492,6 +1492,59 @@ class DurableWriteQueue:
|
||||
)
|
||||
return len(job_ids)
|
||||
|
||||
def retry_first_unknown_after_reconcile(
|
||||
self,
|
||||
*,
|
||||
older_than: datetime,
|
||||
limit: int = 32,
|
||||
) -> int:
|
||||
"""Retry one unresolved unknown outcome once, after reconciliation.
|
||||
|
||||
The caller first asks HY and checks linked governed output. Rows still
|
||||
unknown may have lost the response after the SDK call, so one bounded
|
||||
retry is useful. `attempts=1` is the guard: the retry claim increments
|
||||
it to two, and a second unknown is left for explicit user recovery.
|
||||
"""
|
||||
if type(limit) is not int or limit < 1:
|
||||
raise ValueError("limit must be positive")
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with (
|
||||
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
|
||||
connection,
|
||||
):
|
||||
rows = connection.execute(
|
||||
"SELECT job_id FROM memory_write_jobs "
|
||||
"WHERE state=? AND attempts=1 AND attempts < max_attempts "
|
||||
"AND updated_at <= ? ORDER BY updated_at, job_id LIMIT ?",
|
||||
(UNKNOWN, older_than.isoformat(), limit),
|
||||
).fetchall()
|
||||
job_ids = tuple(str(row["job_id"]) for row in rows)
|
||||
if not job_ids:
|
||||
return 0
|
||||
placeholders = ",".join("?" for _ in job_ids)
|
||||
connection.execute(
|
||||
f"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
|
||||
f"lease_expires_at=NULL, hy_invoked=0, "
|
||||
f"last_error='model_content_unknown_auto_retry', updated_at=? "
|
||||
f"WHERE state=? AND attempts=1 AND job_id IN ({placeholders})",
|
||||
(PENDING, now, UNKNOWN, *job_ids),
|
||||
)
|
||||
if connection.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_tasks'"
|
||||
).fetchone():
|
||||
connection.execute(
|
||||
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
|
||||
f"last_error_code='model_content_unknown_auto_retry', updated_at=? "
|
||||
f"WHERE status=? AND task_id IN ("
|
||||
f"SELECT inputs.task_id FROM memory_task_inputs AS inputs "
|
||||
f"JOIN memory_write_job_evidence AS linked ON EXISTS ("
|
||||
f"SELECT 1 FROM json_each(inputs.refs_json) AS ref "
|
||||
f"WHERE json_extract(ref.value, '$.evidence_id')=linked.evidence_id) "
|
||||
f"WHERE linked.job_id IN ({placeholders}))",
|
||||
(MemoryTaskStatus.PENDING, now, MemoryTaskStatus.UNKNOWN, *job_ids),
|
||||
)
|
||||
return len(job_ids)
|
||||
|
||||
def resolve_unknown_from_linked_output(self, job_id: str) -> bool:
|
||||
"""Close an unknown durable row when local governance output proves success."""
|
||||
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
|
||||
@@ -1896,6 +1949,13 @@ class DurableWriteDrain:
|
||||
#: Reaping is a write; it runs at most once per this interval on the
|
||||
#: idle path so an empty queue does not issue one write per poll.
|
||||
self._reap_interval = reap_interval
|
||||
# Unknown recovery is user-visible and lightweight enough to check
|
||||
# sooner than general database housekeeping. Tests that set the common
|
||||
# interval to zero keep immediate maintenance.
|
||||
self._unknown_reconcile_interval = min(
|
||||
reap_interval,
|
||||
timedelta(seconds=10),
|
||||
)
|
||||
#: The maintenance writes below are recoverability/retention work, not
|
||||
#: idle housekeeping, so each has its own interval and runs at the start
|
||||
#: of every drain tick. Sharing one timestamp would let a busy tick
|
||||
@@ -2308,7 +2368,8 @@ class DurableWriteDrain:
|
||||
)
|
||||
if self.reconcile_unknown is not None and (
|
||||
self._last_unknown_reconcile is None
|
||||
or tick_started - self._last_unknown_reconcile >= self._reap_interval
|
||||
or tick_started - self._last_unknown_reconcile
|
||||
>= self._unknown_reconcile_interval
|
||||
):
|
||||
self._last_unknown_reconcile = tick_started
|
||||
try:
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
The result decides whether the `unknown` job state can ever be retired, so it
|
||||
is probed against the installed SDK rather than written down from memory.
|
||||
|
||||
What was verified against `hy-memory==2.0.2`:
|
||||
What was verified against `hy-memory==2.0.3`:
|
||||
|
||||
- `HyMemoryClient.add(..., request_id=...)` accepts a caller-supplied id and
|
||||
echoes it back. It is described as a request/receipt handle, and the SDK
|
||||
|
||||
@@ -1404,7 +1404,14 @@ async def _024_matter_model_cache(cursor: Any) -> None:
|
||||
|
||||
|
||||
async def _026_hy_node_permissions(cursor: Any) -> None:
|
||||
await cursor.execute(hy_node_permissions_ddl(_dialect(cursor)))
|
||||
# Historical migration retained as a ledger slot. Fresh databases no
|
||||
# longer create this retired proof table; migration 51 removes it from
|
||||
# databases that ran the original v26 body.
|
||||
return None
|
||||
|
||||
|
||||
async def _051_drop_hy_node_permissions(cursor: Any) -> None:
|
||||
await cursor.execute("DROP TABLE IF EXISTS hy_node_permissions")
|
||||
|
||||
|
||||
def model_profiles_ddl(dialect: str = "mysql") -> str:
|
||||
@@ -2756,7 +2763,7 @@ MIGRATIONS: tuple[Migration, ...] = (
|
||||
),
|
||||
Migration(23, "source_candidates", _023_source_candidates, ("source_candidates",)),
|
||||
Migration(24, "matter_model_cache", _024_matter_model_cache, ("matter_model_cache",)),
|
||||
Migration(26, "hy_node_permissions", _026_hy_node_permissions, ("hy_node_permissions",)),
|
||||
Migration(26, "hy_node_permissions", _026_hy_node_permissions),
|
||||
Migration(
|
||||
27,
|
||||
"model_profiles",
|
||||
@@ -2841,4 +2848,5 @@ MIGRATIONS: tuple[Migration, ...] = (
|
||||
"model_probe_operations_reserved_at",
|
||||
_050_model_probe_operations_reserved_at,
|
||||
),
|
||||
Migration(51, "drop_hy_node_permissions", _051_drop_hy_node_permissions),
|
||||
)
|
||||
@@ -267,15 +267,13 @@ class MemoryTaskOutput(BaseModel):
|
||||
MEMORY_TASK_SCHEMA = memory_tasks_ddl("sqlite")
|
||||
|
||||
|
||||
#: Transport refusals raised before the physical send, so provably nothing
|
||||
#: reached HY: nothing can have been written and retrying cannot duplicate.
|
||||
#: Legacy transport refusals raised before the physical send, so provably
|
||||
#: nothing reached HY: nothing can have been written and retrying cannot
|
||||
#: duplicate. Kept to recover rows written by builds through 1.0.2.
|
||||
#:
|
||||
#: Every name here is denied by `HyPermissionTransport._check_request` (the
|
||||
#: httpx request hook) or by `_PhysicalTransport.handle_async_request` before
|
||||
#: it counts the request — both run inside httpx, before the socket. Explicit
|
||||
#: on purpose (D88): the transport also denies `budget_exceeded`, `redirect`,
|
||||
#: `lease_*` and `request_failed`, all of which can arrive after HY persisted
|
||||
#: a node, so a prefix rule would requeue work that may already exist.
|
||||
#: The retired request hook denied every name below before opening a socket.
|
||||
#: This stays explicit: other old transport codes may have happened after HY
|
||||
#: persisted a node and must not be blindly requeued.
|
||||
_PRE_SEND_REFUSALS = (
|
||||
"model_content_transport_headers_mismatch",
|
||||
"model_content_transport_body_mismatch",
|
||||
@@ -2163,6 +2161,43 @@ class MemoryTaskStore:
|
||||
)
|
||||
return changed.rowcount
|
||||
|
||||
def retry_first_unknown_after_reconcile(
|
||||
self,
|
||||
*,
|
||||
older_than: datetime,
|
||||
limit: int = 128,
|
||||
) -> int:
|
||||
"""Automatically retry a task-only unknown exactly once.
|
||||
|
||||
Reconciliation runs before this transition. Attempts are preserved, so
|
||||
the retry claim moves `attempts=1` to two and another unknown is left
|
||||
for explicit user recovery.
|
||||
"""
|
||||
if type(limit) is not int or limit < 1:
|
||||
raise ValueError("limit must be positive")
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with self._connect(writing=True) as connection:
|
||||
changed = connection.execute(
|
||||
"""
|
||||
UPDATE memory_tasks
|
||||
SET status=?, runner_token=NULL,
|
||||
last_error_code='model_content_unknown_auto_retry', updated_at=?
|
||||
WHERE task_id IN (
|
||||
SELECT task_id FROM memory_tasks
|
||||
WHERE status=? AND attempts=1 AND updated_at <= ?
|
||||
ORDER BY updated_at, task_id LIMIT ?
|
||||
)
|
||||
""",
|
||||
(
|
||||
MemoryTaskStatus.PENDING,
|
||||
now,
|
||||
MemoryTaskStatus.UNKNOWN,
|
||||
older_than.isoformat(),
|
||||
limit,
|
||||
),
|
||||
)
|
||||
return changed.rowcount
|
||||
|
||||
def recover_running(self) -> int:
|
||||
"""Return tasks orphaned by a previous process to the pending queue."""
|
||||
return self.recover_orphan_running()
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
"""One global final-state permission for remote model content egress."""
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Sequence
|
||||
from typing import Annotated, cast
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.params import Depends
|
||||
from pydantic import BaseModel, ConfigDict, StrictBool
|
||||
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
|
||||
|
||||
class ModelContentPermissionState(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
allowed: bool
|
||||
|
||||
|
||||
class SetModelContentPermission(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
allowed: StrictBool
|
||||
|
||||
|
||||
def model_content_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
router = APIRouter(prefix="/api/v1/model-content-permission", dependencies=dependencies)
|
||||
|
||||
def config(request: Request) -> ConfigStore:
|
||||
return cast(ConfigStore, request.app.state.config_store)
|
||||
|
||||
Config = Annotated[ConfigStore, Depends(config)]
|
||||
|
||||
@router.get("", response_model=ModelContentPermissionState)
|
||||
async def current(store: Config) -> ModelContentPermissionState:
|
||||
return ModelContentPermissionState(allowed=store.model_content_allowed())
|
||||
|
||||
@router.post("", response_model=ModelContentPermissionState)
|
||||
async def set_permission(
|
||||
request: Request,
|
||||
value: SetModelContentPermission,
|
||||
store: Config,
|
||||
) -> ModelContentPermissionState:
|
||||
await asyncio.to_thread(store.set_model_content_allowed, value.allowed)
|
||||
if value.allowed:
|
||||
request.app.state.start_memory_backfill()
|
||||
return ModelContentPermissionState(allowed=value.allowed)
|
||||
|
||||
return router
|
||||
@@ -1,180 +0,0 @@
|
||||
"""Live global permission checks for remote model-content egress.
|
||||
|
||||
``config.json`` is the only durable permission state. Tasks do not carry a
|
||||
permission receipt, revision, source scope, or model pin. Every remote send
|
||||
and every publication boundary comes back through the lease and reads the
|
||||
current final-state boolean.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from one_memory.hy_source_context import HySourceContext
|
||||
from one_memory.imports.models import EvidenceRecord
|
||||
from one_memory.memory_governance import MemoryGovernanceStore, MemoryStatus
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
from one_memory.memory_tasks import MemoryTask, MemoryTaskStatus
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ModelProfileStore
|
||||
|
||||
|
||||
def permission_error(code: str) -> RuntimeError:
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
|
||||
return HyMemoryBridgeError(
|
||||
code if code.startswith("model_content_") else f"model_content_{code}"
|
||||
)
|
||||
|
||||
|
||||
def error_is_policy_stop(code: str) -> bool:
|
||||
"""A disabled final-state permission is an intentional stop."""
|
||||
|
||||
return code in {
|
||||
"model_content_source_disabled",
|
||||
"hy_memory_evidence_changed",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelProcessingPermissionLease:
|
||||
task_id: str
|
||||
user_id: str
|
||||
profile: ModelProfileSnapshot
|
||||
records: tuple[EvidenceRecord, ...]
|
||||
source_context: HySourceContext | None
|
||||
_guard: ModelProcessingPermissionGuard = field(repr=False, compare=False)
|
||||
|
||||
def recheck(self) -> None:
|
||||
"""Read current config/profile immediately before an egress boundary."""
|
||||
|
||||
self.recheck_permission()
|
||||
self._guard.check_context(self.records, self.source_context)
|
||||
|
||||
def recheck_permission(self) -> None:
|
||||
self._guard.check(self.profile, self.records)
|
||||
|
||||
|
||||
class ModelProcessingPermissionGuard:
|
||||
def __init__(
|
||||
self,
|
||||
config: ConfigStore,
|
||||
governance: MemoryGovernanceStore,
|
||||
profiles: ModelProfileStore,
|
||||
) -> None:
|
||||
self.config = config
|
||||
self.governance = governance
|
||||
self.profiles = profiles
|
||||
|
||||
def require_allowed(self) -> None:
|
||||
if not self.config.model_content_allowed():
|
||||
raise permission_error("permission_required")
|
||||
|
||||
def check(
|
||||
self,
|
||||
profile: ModelProfileSnapshot,
|
||||
records: tuple[EvidenceRecord, ...] = (),
|
||||
) -> None:
|
||||
"""Validate only live global consent and the currently active model."""
|
||||
|
||||
self.require_allowed()
|
||||
if self.profiles.active_identity() != (profile.profile_id, profile.revision):
|
||||
raise permission_error("profile_changed")
|
||||
if records and any(record.user_id != records[0].user_id for record in records):
|
||||
raise permission_error("context_scope_invalid")
|
||||
|
||||
def lease(
|
||||
self,
|
||||
task: MemoryTask,
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
context: HySourceContext | None,
|
||||
) -> ModelProcessingPermissionLease:
|
||||
combined = self.context_records(records, context)
|
||||
lease = ModelProcessingPermissionLease(
|
||||
task_id=task.task_id,
|
||||
user_id=task.user_id,
|
||||
profile=task.profile_snapshot,
|
||||
records=combined,
|
||||
source_context=context,
|
||||
_guard=self,
|
||||
)
|
||||
lease.recheck()
|
||||
return lease
|
||||
|
||||
@staticmethod
|
||||
def context_records(
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
context: HySourceContext | None,
|
||||
) -> tuple[EvidenceRecord, ...]:
|
||||
by_id = {record.evidence_id: record for record in records}
|
||||
if len(by_id) != len(records):
|
||||
raise permission_error("context_scope_invalid")
|
||||
if context is not None:
|
||||
for record in (*context.current, *context.history, *context.supporting):
|
||||
previous = by_id.get(record.evidence_id)
|
||||
if previous is not None and previous != record:
|
||||
raise permission_error("evidence_changed")
|
||||
by_id[record.evidence_id] = record
|
||||
return tuple(by_id.values())
|
||||
|
||||
def check_context(
|
||||
self,
|
||||
records: tuple[EvidenceRecord, ...],
|
||||
context: HySourceContext | None,
|
||||
) -> None:
|
||||
if context is None:
|
||||
return
|
||||
ids = {record.evidence_id for record in records}
|
||||
if not context.current:
|
||||
raise permission_error("context_scope_invalid")
|
||||
context_records = (*context.current, *context.history, *context.supporting)
|
||||
if len({record.evidence_id for record in context_records}) != len(context_records):
|
||||
raise permission_error("context_scope_invalid")
|
||||
for known in context.known_memories:
|
||||
if not known.evidence_ids or not set(known.evidence_ids) <= ids:
|
||||
raise permission_error("context_support_missing")
|
||||
memory = self.governance.get(
|
||||
known.governed_memory_id,
|
||||
user_id=context.current[0].user_id,
|
||||
)
|
||||
if (
|
||||
memory is None
|
||||
or memory.revision != known.revision
|
||||
or memory.content != known.content
|
||||
or set(memory.evidence_ids) != set(known.evidence_ids)
|
||||
or memory.status is not MemoryStatus.CONFIRMED
|
||||
):
|
||||
raise permission_error("context_changed")
|
||||
|
||||
def reject_unstarted(self, task_id: str, code: str) -> None:
|
||||
stopped = error_is_policy_stop(code)
|
||||
with self._task_write() as tasks:
|
||||
tasks.execute(
|
||||
"UPDATE memory_tasks SET status=?, last_error_code=?, updated_at=? "
|
||||
"WHERE task_id=? AND status IN (?,?,?,?)",
|
||||
(
|
||||
MemoryTaskStatus.CANCELLED if stopped else MemoryTaskStatus.FAILED,
|
||||
str(permission_error(code)),
|
||||
datetime.now(UTC).isoformat(),
|
||||
task_id,
|
||||
MemoryTaskStatus.PENDING,
|
||||
MemoryTaskStatus.WAITING_FOR_PERMISSION,
|
||||
MemoryTaskStatus.WAITING_FOR_MODEL,
|
||||
MemoryTaskStatus.DEFERRED_OUTSIDE_WINDOW,
|
||||
),
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def _task_write(self) -> Iterator[object]:
|
||||
from one_memory.database import take_the_write_lock_first
|
||||
from one_memory.database_access import main_database
|
||||
|
||||
with main_database(self.governance.database_path).write(
|
||||
autocommit=True,
|
||||
foreign_keys=True,
|
||||
) as connection:
|
||||
take_the_write_lock_first(connection)
|
||||
with connection:
|
||||
yield connection
|
||||
@@ -68,6 +68,16 @@ COMPLETE_INSERT_MYSQL = (
|
||||
"singleton, status, selected_source_ids_json, started_at, completed_at, updated_at"
|
||||
") VALUES (1, 'completed', '[]', NULL, %s, %s)"
|
||||
)
|
||||
RESET_UPDATE_SQLITE = (
|
||||
"UPDATE onboarding_state SET status = 'not_started', "
|
||||
"selected_source_ids_json = '[]', started_at = NULL, completed_at = NULL, "
|
||||
"updated_at = ? WHERE singleton = 1"
|
||||
)
|
||||
RESET_UPDATE_MYSQL = (
|
||||
"UPDATE onboarding_state SET status = 'not_started', "
|
||||
"selected_source_ids_json = '[]', started_at = NULL, completed_at = NULL, "
|
||||
"updated_at = %s WHERE singleton = 1"
|
||||
)
|
||||
|
||||
|
||||
class OnboardingState(BaseModel):
|
||||
@@ -140,6 +150,30 @@ class OnboardingStore:
|
||||
)
|
||||
return self.get()
|
||||
|
||||
def reset(self) -> OnboardingState:
|
||||
"""Return onboarding to its first-run state, touching nothing else.
|
||||
|
||||
The completion flag is stored beside the memories on purpose: a fresh
|
||||
install of the app must not greet someone who already has memories as a
|
||||
first-time user. That makes the way back to the cold-start screens an
|
||||
explicit action rather than a side effect of reinstalling, and it is
|
||||
the whole reason this exists: QA and long-time users need to see those
|
||||
screens again without giving up a single memory.
|
||||
|
||||
Sources, memories and settings are separate tables and are not read or
|
||||
written here.
|
||||
"""
|
||||
now = datetime.now(UTC).isoformat()
|
||||
mysql = is_mysql(self._store)
|
||||
with self._connect(writing=True) as connection:
|
||||
execute(
|
||||
connection,
|
||||
mysql,
|
||||
RESET_UPDATE_MYSQL if mysql else RESET_UPDATE_SQLITE,
|
||||
(now,),
|
||||
)
|
||||
return self.get()
|
||||
|
||||
@contextmanager
|
||||
def _connect(
|
||||
self,
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
"""Continuous recovery monitor for model processing.
|
||||
|
||||
The monitor is intentionally retained. It observes the global config final
|
||||
state plus ``memory_tasks`` and asks the supervisor to recover work; it owns no
|
||||
second request/receipt state machine.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from collections.abc import Hashable
|
||||
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
|
||||
|
||||
class ProcessingCoordinator:
|
||||
RESCAN_SECONDS = 60.0
|
||||
POLL_SECONDS = 2.0
|
||||
|
||||
def __init__(self, config: ConfigStore, supervisor: MemoryProcessingSupervisor):
|
||||
self.config = config
|
||||
self.supervisor = supervisor
|
||||
self._stopping = False
|
||||
self._monitor: asyncio.Task[None] | None = None
|
||||
self._error: str | None = None
|
||||
self._observed: tuple[Hashable, ...] | None = None
|
||||
self._last_scan = 0.0
|
||||
self._clock = time.monotonic
|
||||
|
||||
async def recover(self) -> None:
|
||||
"""Start the one process-owned monitor, whether permission is on or off."""
|
||||
|
||||
if self._stopping:
|
||||
return
|
||||
if self._monitor is None or self._monitor.done():
|
||||
self._monitor = asyncio.create_task(
|
||||
self._monitor_loop(),
|
||||
name="one-memory-processing-monitor",
|
||||
)
|
||||
|
||||
def _identity(self) -> tuple[Hashable, ...]:
|
||||
counts = self.supervisor.tasks.counts()
|
||||
return (
|
||||
self.config.model_content_allowed(),
|
||||
tuple(sorted((str(status), count) for status, count in counts.items())),
|
||||
)
|
||||
|
||||
async def _tick(self) -> None:
|
||||
identity = await asyncio.to_thread(self._identity)
|
||||
now = self._clock()
|
||||
if identity[0] and (
|
||||
identity != self._observed or now - self._last_scan >= self.RESCAN_SECONDS
|
||||
):
|
||||
await self.supervisor.resume_allowed()
|
||||
self._last_scan = now
|
||||
identity = await asyncio.to_thread(self._identity)
|
||||
self._observed = identity
|
||||
self._error = None
|
||||
|
||||
async def _monitor_loop(self) -> None:
|
||||
while not self._stopping:
|
||||
try:
|
||||
await self._tick()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
self._error = "model_content_recovery_blocked"
|
||||
await asyncio.sleep(self.POLL_SECONDS)
|
||||
|
||||
def status(self) -> dict[str, object]:
|
||||
return {
|
||||
"monitoring": self._monitor is not None and not self._monitor.done(),
|
||||
"allowed": self._observed[0] if self._observed else None,
|
||||
"error": self._error,
|
||||
}
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
self._stopping = True
|
||||
if self._monitor is not None:
|
||||
self._monitor.cancel()
|
||||
await asyncio.gather(self._monitor, return_exceptions=True)
|
||||
@@ -13,10 +13,7 @@
|
||||
| `test_archive_extract.py` | `contract` | Unpacking an archive a stranger sent, onto this user's disk. |
|
||||
| `test_ask_workspace_path.py` | `contract` | Ask must honour the same workspace_path expansion recall uses. |
|
||||
| `test_async_bridge.py` | `contract` | 覆盖 reader rejects parallel workers that could close during next、cancelling a read stops the worker before the batch finishes、starting a read does not stop the app answering 等 10 个场景。 |
|
||||
| `test_async_dispatch_checks.py` | `contract` | 覆盖 cancelling quiet timer does not start any permission reader、dispatch check does not queue work after shutdown、dispatch permission check denies before queue after classification 等 6 个场景。 |
|
||||
| `test_auth.py` | `integration` | 覆盖 local api requires bearer token、browser origin is checked in addition to token 场景。 |
|
||||
| `test_backfill_acceptance_lifecycle.py` | `contract` | A processing acknowledgement belongs to one consent revision and live worker. |
|
||||
| `test_backfill_conversation_isolation.py` | `contract` | One stale conversation cannot invalidate an entire backfill scan. |
|
||||
| `test_background_work.py` | `contract` | 覆盖 a request does not wait behind a catch up、bulk work is kept to a few threads at a time、the pool outlives one hand off 等 4 个场景。 |
|
||||
| `test_backup_destroy_restore_reconciliation.py` | `contract` | Backup -> destroy -> restore, reconciled row for row. |
|
||||
| `test_backup_endpoint.py` | `integration` | The 'POST /api/v1/backup' endpoint the CLI's 'backup create' calls. |
|
||||
@@ -64,7 +61,6 @@
|
||||
| `test_cli_lifecycle.py` | `integration` | 覆盖 cli starts and stops without residual runtime 场景。 |
|
||||
| `test_cli_restore.py` | `contract` | 覆盖 restore requires a source、restore refuses while the runtime is running、restore previews without confirmation 等 4 个场景。 |
|
||||
| `test_codex_header_provenance.py` | `contract` | 覆盖 header workspace versions and native ids reach file origin、new adapter rereads header after a real checkpoint、machine and unknown session sources do not establish human or agent sender 等 14 个场景。 |
|
||||
| `test_compact_model_content.py` | `contract` | Server-held previews preserve exact grants without round-tripping all refs. |
|
||||
| `test_connection_lifetime.py` | `integration` | Every SQLite connection a request opens must be closed when the request ends. |
|
||||
| `test_context_access.py` | `contract` | 覆盖 memory only never reads raw authority、raw permission does not grant memory and removes revoked sources、raw requires all apps and source 等 13 个场景。 |
|
||||
| `test_context_api.py` | `integration` | 覆盖 no model local query returns real raw and governed items、existing single lane authorization never runs denied reader、authentication and origin remain required 等 14 个场景。 |
|
||||
@@ -76,12 +72,6 @@
|
||||
| `test_context_search.py` | `contract` | 覆盖 authorization and requested paths never invoke other lane、empty partial zero unavailable and safe failures、parallel common deadline and finalizers wait for both reads 等 24 个场景。 |
|
||||
| `test_context_search_both_paths.py` | `integration` | The objective names 'recall' and requires BOTH paths: raw and memory. |
|
||||
| `test_context_v2_contract_parity.py` | `unit` | Mechanical parity checks for the shared v2 coverage receipt contract. |
|
||||
| `test_continuous_completed_input_review.py` | `contract` | Independent review of completed-input reuse; real stores and controlled HY only. |
|
||||
| `test_continuous_coordinator.py` | `contract` | 覆盖 activation hook and startup recovery start single monitor then stop、failed preparation does not advance observed watermark、new journal event and source regrant wake replay without legacy request 等 6 个场景。 |
|
||||
| `test_continuous_outcome_unknown.py` | `contract` | 覆盖 a timed out continuous task becomes unknown not pending、a timed out non continuous task also becomes unknown、an unsettled continuous send becomes unknown not failed 等 8 个场景。 |
|
||||
| `test_continuous_policy.py` | `contract` | 覆盖 backlog snapshot over one hundred thousand is compact、caller rollback includes policy and backlog、enable idempotence rejects changed payload and superseded operation 等 19 个场景。 |
|
||||
| `test_continuous_processing_bridge.py` | `contract` | 覆盖 bound source regrant refuses old task、continuous inactive never falls back to legacy、continuous prepare lease ingest has no legacy receipt or old context 等 17 个场景。 |
|
||||
| `test_continuous_processing_integration.py` | `contract` | Real temporary Core/stores with controlled probe and HY boundaries, never real data. |
|
||||
| `test_contract_parity.py` | `contract` | Python 的枚举和 TypeScript 里那份手写的联合类型必须一致。 |
|
||||
| `test_conversation_search.py` | `integration` | Searching imported conversations before any generation model is configured. |
|
||||
| `test_conversation_seed_ids.py` | `contract` | 覆盖 seed ids read metadata only without hydrating、seed eligibility keeps person and memory content only、latest activity includes nonseed turns and import time fallback 等 13 个场景。 |
|
||||
@@ -97,7 +87,6 @@
|
||||
| `test_database_contention.py` | `contract` | Writers meet in this one file, and the loser used to lose data. |
|
||||
| `test_database_file_permissions.py` | `contract` | 覆盖 a new main database is readable by its owner alone、the wal sidecars are not left world readable、a database from an older build is tightened on open 场景。 |
|
||||
| `test_db_session_utc.py` | `contract` | The MySQL session must be pinned to UTC. |
|
||||
| `test_delete_no_conversation_freeze.py` | `contract` | 删掉一条记忆,不得让同一对话之后的新 turn 停止沉淀(RC10)。 |
|
||||
| `test_delivery_board.py` | `contract` | 覆盖 parses only the marked structured record、verified requires evidence with a positive control、rejects unknown dependency 等 5 个场景。 |
|
||||
| `test_derived_echo.py` | `contract` | A host that saves our injection back must not mint a new user decision. |
|
||||
| `test_detailed_logging.py` | `contract` | 覆盖 detailed pipeline logging defaults off and survives restart、live log level applies to full hy and provider call chain、info collects hy info without enabling debug pipeline content 等 7 个场景。 |
|
||||
@@ -111,7 +100,6 @@
|
||||
| `test_dual_engine_helper.py` | `contract` | Tests for the dual-engine harness itself. |
|
||||
| `test_event_loop_blocking.py` | `contract` | Proof, as a test, that blocking work on the loop freezes it — and the fix works. |
|
||||
| `test_evidence_reconcile_primitive.py` | `integration` | A bounded, read-only discovery primitive for the H23 Evidence reconcile. |
|
||||
| `test_evidence_revision_read_review.py` | `contract` | Read invalidation review: real guard/provenance, controlled HY, no model/network. |
|
||||
| `test_evidence_store.py` | `contract` | 覆盖 same source message merges origins without duplication、identical text from different sources is not deduplicated、rescanning an unchanged file does not write the message again 等 26 个场景。 |
|
||||
| `test_export_probe.py` | `contract` | What a probe may say about a stranger's chat history, and what it may not. |
|
||||
| `test_external_endpoint_scopes.py` | `integration` | Core boundary tests using the broker's envelope, never a host's claimed identity. |
|
||||
@@ -125,10 +113,9 @@
|
||||
| `test_file_import_operations.py` | `integration` | Bounded synthetic uploads; operation identity is not a content dedup key. |
|
||||
| `test_file_import_operations_review.py` | `contract` | Upload refusal/legacy compatibility review with bounded synthetic files only. |
|
||||
| `test_file_publish_job_boundary.py` | `contract` | Staged uploads cannot widen an earlier directory import's uncaptured input. |
|
||||
| `test_four_app_live_ingest.py` | `integration` | Actual source readers + OS watch + Core routes; no model or native host calls. |
|
||||
| `test_four_app_live_ingest.py` | `contract` | Actual source readers + OS watch + Core routes; no model or native host calls. |
|
||||
| `test_frozen_core_layout.py` | `contract` | 覆盖 single file layout remains available without private preset、folder layout keeps same core and all native dependencies、unknown layout does not silently build another package 等 5 个场景。 |
|
||||
| `test_golden_corpus.py` | `contract` | A7-01: the golden corpus comparison. |
|
||||
| `test_governance_authority_review.py` | `contract` | D04 authority and workspace scope through the real continuous permission guard. |
|
||||
| `test_governance_permission_commit.py` | `contract` | 覆盖 permission guard is rechecked inside write transaction and rolls back、guard context does not leak into subsequent manual actions、guard check and commit hold database write authority 场景。 |
|
||||
| `test_governance_read_path_does_not_take_the_writer.py` | `contract` | The recall read path must not acquire the single global writer. |
|
||||
| `test_governance_store_root.py` | `contract` | 生产代码里治理 store 只在 Core 启动的组合根建一次。 |
|
||||
@@ -144,32 +131,19 @@
|
||||
| `test_hourly_c1_peak_replay_contract.py` | `contract` | Guard the package peak-replay probe's error accounting. |
|
||||
| `test_hourly_c1_recall_stage_probe_contract.py` | `contract` | Guard the packaged recall stage probe's command and error semantics. |
|
||||
| `test_hy_baseline_source.py` | `contract` | 覆盖 first import replay copies raw prefix without selecting message ids、frozen transcript keeps middle user message and excludes later records 场景。 |
|
||||
| `test_hy_coding_lease.py` | `contract` | HY's own coding/working route under the permission lease, on the pinned SDK. |
|
||||
| `test_hy_coding_reply.py` | `contract` | R-07: HY's coding-shaped replies read as "no memories at all". |
|
||||
| `test_hy_lease_responsiveness.py` | `contract` | Blocked lease checks, local failing doubles only; no network or credentials. |
|
||||
| `test_hy_lifecycle_sync.py` | `contract` | HY decides deletions; the ledger mirrors them. Pinned SDK, scripted HTTP. |
|
||||
| `test_hy_log_privacy.py` | `contract` | HY SDK records are not rewritten by One Memory's logging setup. |
|
||||
| `test_hy_memory_bridge.py` | `contract` | 覆盖 hy client initialization is serial across bridges、native hy clients share one runtime per data directory、ollama profile ignores ambient remote storage and models 等 41 个场景。 |
|
||||
| `test_hy_metrics_lifecycle.py` | `contract` | What the pinned public SDK's MetricsCollector does, and what that costs us. |
|
||||
| `test_hy_native_write_chain.py` | `contract` | HY writes with its own prompts and judgment; the product adds no audit of its own. |
|
||||
| `test_hy_network_egress.py` | `contract` | Where a real HY client is allowed to send bytes, and where it is not. |
|
||||
| `test_hy_permission_context.py` | `contract` | Whole native HY client with synthetic content and HTTP MockTransport, never cloud. |
|
||||
| `test_hy_permission_context_races.py` | `contract` | Bounded factory races with a temporary native HY client and no model calls. |
|
||||
| `test_hy_permission_raw_context.py` | `contract` | Real pinned HY SDK/Chroma and guards; scripted HTTP is not model acceptance. |
|
||||
| `test_hy_permission_store.py` | `contract` | Native HY node/algorithm tests with synthetic data and an in-memory backend only. |
|
||||
| `test_hy_permission_transport.py` | `contract` | Native HY entry points, synthetic inputs and MockTransport only. |
|
||||
| `test_hy_response_guard.py` | `contract` | 覆盖 invalid response retries same hy provider without changing prompt、repeated empty response fails instead of becoming no op、failure receipt survives a later successful model call 等 23 个场景。 |
|
||||
| `test_hy_response_shapes.py` | `contract` | HY response shape selection only; no provider, write routing or model calls. |
|
||||
| `test_hy_runtime_config.py` | `contract` | MEMORY.md is the audited record of what we send HY; keep it answerable. |
|
||||
| `test_hy_sdk_idempotency_probe.py` | `contract` | A0-03: probe whether the HY SDK can make a write idempotent. |
|
||||
| `test_hy_source_context.py` | `contract` | 覆盖 bound context rejects a same named conversation in another workspace、bound context rejects a duplicated evidence id、bound context rejects a same conversation from another source 等 9 个场景。 |
|
||||
| `test_hy_stage_trace.py` | `contract` | 覆盖 observer returns native results without editing or requesting models、observer redacts content without recording prompts or exception details 场景。 |
|
||||
| `test_hy_state_source.py` | `contract` | RC4:记忆生命周期(有效 / 被取代 / 去重隐藏 / 过期)直接读 HY,产品账本不再逐类镜像。 |
|
||||
| `test_hy_step_timing.py` | `contract` | Where an HY write spends the time that is not a model call. |
|
||||
| `test_hy_step_trace_switch.py` | `contract` | Rule 15: the per-step HY trace is off unless the debug tier asks for it. |
|
||||
| `test_hy_supersede_sync.py` | `contract` | HY decides supersessions; the ledger mirrors them. Pinned SDK, scripted HTTP. |
|
||||
| `test_hy_trial_budget.py` | `contract` | Offline proofs for the real-model acceptance budget (no network, no Keychain). |
|
||||
| `test_hy_turn_input.py` | `contract` | HY gets one turn at a time, exactly as it happened, and picks its own route. |
|
||||
| `test_hy_writeback.py` | `integration` | HY 的结果只有一扇门进治理表,而且每次进门都带着调用开始时的证据快照。 |
|
||||
| `test_import_api.py` | `integration` | 覆盖 resume during cleanup returns conflict without internal error、authorize detect import and revoke api flow、protected api rejects missing token and foreign origin 等 5 个场景。 |
|
||||
| `test_import_control_races_review.py` | `contract` | Read-only review repro: real runner/store, synthetic delayed adapter cleanup. |
|
||||
@@ -181,7 +155,6 @@
|
||||
| `test_import_three_scenarios.py` | `contract` | The three import scenarios a user actually meets. |
|
||||
| `test_import_three_scenarios_real.py` | `contract` | The three import scenarios, on a real Core at a real volume. |
|
||||
| `test_imported_index_background.py` | `contract` | The ordering index is built off the startup path. |
|
||||
| `test_incremental_memory_backfill.py` | `contract` | 覆盖 accept backfill returns after first batch while later history is still read、backfill budget yields between rounds and resumes same cursor、backfill holds an unanswered raw turn until its reply arrives 等 18 个场景。 |
|
||||
| `test_ingest_journal.py` | `contract` | 覆盖 baseline and empty initialize are explicit transactional and idempotent、old date new import and revise noop and rollbacks、import metadata changes invalidate without authorizing content 等 12 个场景。 |
|
||||
| `test_input_sanitization.py` | `unit` | 覆盖 redacts cli and assignment secrets without removing context、redacts bearer known tokens urls and private keys、redacts quoted json assignments and basic authorization 等 8 个场景。 |
|
||||
| `test_installer.py` | `contract` | 覆盖 installer create verify and rollback is transactional、installer updates owned file and restores backup、installer refuses unowned or changed target 场景。 |
|
||||
@@ -209,7 +182,6 @@
|
||||
| `test_main_database_fairness.py` | `contract` | Waiting for the one database has to be a queue, not a scramble. |
|
||||
| `test_main_database_isolation.py` | `contract` | What one checkout of the main database can do, and what it leaves for the next. |
|
||||
| `test_managed_restore.py` | `contract` | 覆盖 restore rejects a source outside the managed backup directory、restore rejects a symlinked source、restore rejects a non one memory database 等 9 个场景。 |
|
||||
| `test_manual_memory_via_hy.py` | `contract` | T04: what the person explicitly asks to remember is written through HY's import path. |
|
||||
| `test_manual_transcript_import.py` | `integration` | 覆盖 manual json keeps authors unknown time and exact text、marked text keeps code fences and never invents timestamps、ambiguous partial or invalid manual data never leaves partial import 等 4 个场景。 |
|
||||
| `test_matter_collection.py` | `integration` | 覆盖 hy windows cover every turn including leading and boundary assistants、oversize turn is split without silently dropping content、partial timestamps preserve source sequence 等 8 个场景。 |
|
||||
| `test_matter_discovery.py` | `contract` | 覆盖 synthesis materializes only known verbatim citations、duplicate claims keep all governed refs and distinct states、synthesis cannot echo the original sixty claim list 等 27 个场景。 |
|
||||
@@ -267,30 +239,18 @@
|
||||
| `test_memory_write_registry.py` | `contract` | A3-01: acceptance and outcome recording for the unified write path. |
|
||||
| `test_memory_write_routes.py` | `integration` | A3-01: the HTTP write entry, tested against a real FastAPI app. |
|
||||
| `test_migrate_off_sqlite.py` | `contract` | A1-06: the migration is only done when reconciliation agrees. |
|
||||
| `test_model_content_bulk.py` | `contract` | Bulk consent reads preserve exact identities and SQLite snapshot boundaries. |
|
||||
| `test_model_content_parse_cost.py` | `contract` | Memoize parsing only; every permission decision still reads current SQLite state. |
|
||||
| `test_model_content_permissions.py` | `contract` | Content permission tests use only a temporary database and synthetic sources. |
|
||||
| `test_model_content_preview_races.py` | `contract` | Confirmation must tolerate append-only import without broadening consent. |
|
||||
| `test_model_content_routes.py` | `integration` | 覆盖 permission route reads and writes only config final state、core composition starts with config permission and durable drain 场景。 |
|
||||
| `test_model_content_scope_seals.py` | `contract` | Indexed scope invalidation retains complete-snapshot semantics without rereading it. |
|
||||
| `test_model_content_snapshot_cost.py` | `contract` | Reuse derived fingerprints only after reading every raw row in this transaction. |
|
||||
| `test_model_creation_api.py` | `integration` | Normal protected model creation routes with temporary data and in-memory secrets. |
|
||||
| `test_model_creation_operations.py` | `contract` | 覆盖 lookup without table is read only and legacy create does not initialize、operation commits with profile and version without autoactivation、any replay conflicts before secret put 等 21 个场景。 |
|
||||
| `test_model_creation_validation_boundary.py` | `integration` | Real ASGI validation must precede creation reservation and secret writes. |
|
||||
| `test_model_policy_activation.py` | `contract` | 覆盖 activation helper is transactional and restores row factory、backlog scan does not hold the single writer、cloud credential is checked without keychain write 等 21 个场景。 |
|
||||
| `test_model_policy_routes.py` | `contract` | 覆盖 all routes require auth、backlog is required not defaulted、enable and transition safe shape and callback once after commit 等 10 个场景。 |
|
||||
| `test_model_probe.py` | `contract` | 覆盖 probe still measures embedding when structured output fails、probe records the real embedding width on the profile、openai compatible probe checks model and structured output 等 23 个场景。 |
|
||||
| `test_model_probe_input_revision.py` | `contract` | 覆盖 input revision precedes measured width update、concurrent update still rejects width or capability cas、early returns record the actual starting revision 等 6 个场景。 |
|
||||
| `test_model_probe_operations.py` | `integration` | Persisted probe identity, real SQLite/API and synthetic transport only. |
|
||||
| `test_model_processing_permissions.py` | `contract` | 覆盖 lease reads current allowed before each boundary、lease refuses a task profile that is no longer active 场景。 |
|
||||
| `test_model_profile_api.py` | `integration` | 覆盖 model profile api never returns key and enforces revision、model profile probe route returns only machine results、model profile validation never echoes api key 等 4 个场景。 |
|
||||
| `test_model_profile_context_reset.py` | `integration` | 覆盖 create context default and inclusive bounds、explicit null clears override and preserves history and secret、omitted or same value is noop with unchanged revision and metadata 等 11 个场景。 |
|
||||
| `test_model_profile_identity.py` | `contract` | Stable model editing and configured-vs-available credentials; no Keychain/network. |
|
||||
| `test_model_profile_queue_handover.py` | `contract` | Queued work when the model changes or its configuration is deleted (D75). |
|
||||
| `test_model_profiles.py` | `contract` | 覆盖 profile uses one model and keeps key out of database、api key updates reuse one stable secret reference、startup cleanup removes only secrets not referenced by current profiles 等 15 个场景。 |
|
||||
| `test_model_request_budget.py` | `contract` | 覆盖 persists failed request reservations without content、parallel leases cannot exceed request limit、unknown or excessive output and body are denied 等 6 个场景。 |
|
||||
| `test_model_request_budget_integration.py` | `contract` | Real HY leaf retries plus durable budget accounting; no network or provider. |
|
||||
| `test_model_scope_lifecycle.py` | `contract` | Bound derived/temporary data without changing consent or original history. |
|
||||
| `test_mysql_lifecycle.py` | `contract` | A1-07: the decisions Runtime makes around the bundled MySQL. |
|
||||
| `test_mysql_platform.py` | `contract` | A1-07: the machine-facing half of managing the bundled MySQL. |
|
||||
| `test_obs03_health_probe.py` | `contract` | OBS-03: '/health' answered with the web app instead of a health payload. |
|
||||
@@ -299,14 +259,11 @@
|
||||
| `test_oversized_file_guard.py` | `contract` | One pass over a source file is bounded by a byte budget, not skipped. |
|
||||
| `test_parent_watchdog.py` | `integration` | A Core that outlives its launcher makes the next launch impossible. |
|
||||
| `test_policy_api_wiring.py` | `integration` | 覆盖 global model permission off does not backfill、backfill puts history on write jobs without scheduling memory tasks、source pause skips history until the source is reenabled 场景。 |
|
||||
| `test_policy_operation_lookup.py` | `contract` | 覆盖 corrupt current row cannot make history or missing operation look safe、current operation uses safe receipt without callback or writes、lookup connection rejects writes and preserves database 等 12 个场景。 |
|
||||
| `test_policy_production_entry.py` | `contract` | 覆盖 production serve passes only supported app options、maintenance serve does not own the runtime 场景。 |
|
||||
| `test_precipitation_concurrency_setting.py` | `contract` | Rule 2/9: precipitation concurrency is configurable, not hardcoded to 1. |
|
||||
| `test_primary_source_adapters.py` | `contract` | 覆盖 pipeline json is host generated even when truncated、workbuddy record parsing never blocks the event loop、cursor store user info envelope is not attributed to the person 等 34 个场景。 |
|
||||
| `test_probe_code_contract_parity.py` | `contract` | 探测码在三处必须同时存在:python 枚举、TS 契约、界面文案表。 |
|
||||
| `test_probe_payment_required.py` | `contract` | 覆盖 http 402 without a useful body is still understood as unpaid、a wrong address is still a missing model not an unpaid one 场景。 |
|
||||
| `test_processing_coordinator.py` | `contract` | 覆盖 monitor is retained and observes config plus tasks 场景。 |
|
||||
| `test_processing_merged_context.py` | `contract` | 覆盖 new chunk omits known memory with changed support but keeps original history 场景。 |
|
||||
| `test_processing_requests.py` | `contract` | 覆盖 concurrent claim only one snapshot wins、concurrent create same grant has one id and profile、concurrent different profile never reuses another scope 等 14 个场景。 |
|
||||
| `test_quality_acceptance.py` | `contract` | 覆盖 acceptance pack is deterministic and scores personal boundary、acceptance score fails missing personal memory 场景。 |
|
||||
| `test_raw_search_without_model.py` | `integration` | Search imported conversations with no generative model configured at all. |
|
||||
@@ -375,8 +332,6 @@
|
||||
| `test_session_inbox.py` | `contract` | A session inbox is not a shared mailbox and a pull is not adoption. |
|
||||
| `test_session_lease.py` | `contract` | A session file on disk is not proof that the session is online. |
|
||||
| `test_session_registry.py` | `contract` | Trusted session identity is not a workspace path. |
|
||||
| `test_settings_policy_lifecycle.py` | `contract` | Settings data-plane lifecycle: real temporary Core, fake probe/HY, no UI or package. |
|
||||
| `test_simplified_model_permission_migrations.py` | `contract` | 覆盖 fresh schema omits removed permission state machines 场景。 |
|
||||
| `test_snapshot_adapter.py` | `contract` | Selection-only binding reopens checksum-verified frozen copies. |
|
||||
| `test_snapshot_pool.py` | `contract` | A whole-directory snapshot must not hold a thread everything else is waiting for. |
|
||||
| `test_soak_probe_effectiveness.py` | `integration` | The soak probe must observe the load it reports on. |
|
||||
@@ -392,7 +347,7 @@
|
||||
| `test_source_parsing.py` | `contract` | 覆盖 reminders are removed only at message edges、original supported outer context and query envelopes、attributed reminder is opaque before finding the outer query 等 11 个场景。 |
|
||||
| `test_source_permission_races.py` | `integration` | Permission execution regressions using temporary stores and Core ASGI instances. |
|
||||
| `test_source_query_preview.py` | `contract` | 覆盖 preserves exact user text and span、supported outer wrapper is stripped without paraphrasing、cursor host prefixes are not shown as the user query 等 8 个场景。 |
|
||||
| `test_source_read_revocation.py` | `integration` | 覆盖 source revocation stops every outward read but keeps other sources、revocation during model wait prevents late answer 场景。 |
|
||||
| `test_source_read_revocation.py` | `contract` | 覆盖 revocation during model wait prevents late answer、source revocation stops every outward read but keeps other sources 场景。 |
|
||||
| `test_source_readers.py` | `contract` | 覆盖 jsonl is resumable and does not advance over an incomplete tail、sqlite reader uses monotonic row checkpoint、zstd jsonl reader resumes by completed record 等 7 个场景。 |
|
||||
| `test_source_registry_and_grants.py` | `contract` | 覆盖 catalog resolves defaults and marks environment overrides、recall hosts that can append join the live sync poller、declarations with no reader are named rather than mistaken for behaviour 等 15 个场景。 |
|
||||
| `test_source_revocation_parking.py` | `integration` | Revoking a source parks its unfinished extraction instead of failing it (D73). |
|
||||
@@ -402,7 +357,7 @@
|
||||
| `test_source_sync_idempotent.py` | `integration` | Final-state Harness sync is idempotent and resumes missed source work. |
|
||||
| `test_source_sync_permissions.py` | `integration` | S1.3 source permission tests use temporary files only, never real host history. |
|
||||
| `test_source_totals.py` | `contract` | 覆盖 the first ask is answered from the rows、same named sessions in two workspaces count as two conversations、insert during count is not marked as already counted 等 7 个场景。 |
|
||||
| `test_source_upload_intents.py` | `integration` | Upload identity binds retries without replacing file receipts or directory consent. |
|
||||
| `test_source_upload_intents.py` | `contract` | Upload identity binds retries without replacing file receipts or directory consent. |
|
||||
| `test_source_write_switch.py` | `contract` | A frozen Harness produces no extraction tasks. |
|
||||
| `test_sql_compat.py` | `contract` | The shared compatibility layer, especially 'rowcount'. |
|
||||
| `test_sqlite_semantics_gate.py` | `contract` | D56: prevent SQLite implementation details spreading outside the data layer. |
|
||||
@@ -411,13 +366,10 @@
|
||||
| `test_store_binding.py` | `contract` | A2: which store is in force, and honest reporting of it. |
|
||||
| `test_store_binding_ratchet.py` | `contract` | D56: ratchet the retired dual-engine 'store_for' seam out of the Core. |
|
||||
| `test_store_port.py` | `contract` | A1-04/A2: the store port, and that real implementations satisfy it. |
|
||||
| `test_subagent_sessions.py` | `contract` | D98: a sub agent's transcript is its own conversation, not part of the main one. |
|
||||
| `test_subagent_turns.py` | `contract` | D98: a sub agent's turn is ordinary user-assistant data. |
|
||||
| `test_subscription_visibility.py` | `unit` | A shared directory is not a read grant. |
|
||||
| `test_sync_facade.py` | `contract` | A2: the sync facade must keep the sync shape call sites already use. |
|
||||
| `test_tail_metadata_lane.py` | `contract` | Post-response metadata retries must be bounded and never block the route. |
|
||||
| `test_terminal_payload_reaping.py` | `contract` | Rule 14: dead-end payloads are actually reaped, not just reapable. |
|
||||
| `test_tool_evidence_hy_turns.py` | `contract` | Every adapter's tool records reach HY inside the turn they happened in. |
|
||||
| `test_two_level_discovery.py` | `contract` | 3925a kernel item 1: two-level discovery. |
|
||||
| `test_ui_session.py` | `contract` | 覆盖 exchange is single use and creates a session、expired exchange cannot create a session、exchange is bound to the core instance 等 5 个场景。 |
|
||||
| `test_ui_session_api.py` | `integration` | 覆盖 ui session exchange sets httponly strict cookie、ui session requires bearer for exchange、ui session reports missing web assets 场景。 |
|
||||
@@ -429,8 +381,8 @@
|
||||
| `test_windows_process_control.py` | `contract` | R-29:Windows 上 'os.kill' 不是发信号,是无条件杀进程。 |
|
||||
| `test_workbuddy_boundary.py` | `contract` | WorkBuddy only injects on SessionStart and UserPromptSubmit. |
|
||||
| `test_workbuddy_parsing.py` | `contract` | 覆盖 automation preserves query without claiming fresh human authorship、automation context attributes do not depend on order or quote style、quoted full automation envelope is not scheduled provenance 等 17 个场景。 |
|
||||
| `test_workbuddy_prompt_backfill.py` | `integration` | When the prompt hook's capture fails, the transcript file is what brings the words in. |
|
||||
| `test_workbuddy_prompt_backfill.py` | `contract` | When the prompt hook's capture fails, the transcript file is what brings the words in. |
|
||||
| `test_workspace_recall.py` | `contract` | Workspace memories were unreachable from every host; these pin the way in. |
|
||||
| `test_write_queue_lease_renewal.py` | `contract` | Rule 8: a lease must outlive the work it covers, and be renewed while it runs. |
|
||||
|
||||
共 428 个测试文件。
|
||||
共 380 个测试文件。
|
||||
@@ -14,8 +14,8 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
PRIMARY_MARKERS = ("unit", "contract", "integration", "performance", "live")
|
||||
EXPECTED_TEST_FILE_COUNT = 428
|
||||
EXPECTED_TEST_FILES_SHA256 = "bd3bf4fc9cda2a8e4427d106283c40d22dad026fd452db954046b6f28d21b896"
|
||||
EXPECTED_TEST_FILE_COUNT = 380
|
||||
EXPECTED_TEST_FILES_SHA256 = "9f075d18403328882c25325d29ebe77d0e3365d675ba19d56134a6a0fb9b64b4"
|
||||
|
||||
# Pure transformations, validation models, parity checks and state machines.
|
||||
# This list is intentionally small: uncertain files default to contract.
|
||||
@@ -68,7 +68,6 @@ INTEGRATION_FILES = frozenset(
|
||||
"test_external_installation_permissions.py",
|
||||
"test_fault_matrix.py",
|
||||
"test_file_import_operations.py",
|
||||
"test_four_app_live_ingest.py",
|
||||
"test_graph_width_endpoint.py",
|
||||
"test_home_summary.py",
|
||||
"test_hy_writeback.py",
|
||||
@@ -100,7 +99,6 @@ INTEGRATION_FILES = frozenset(
|
||||
"test_memory_supersede.py",
|
||||
"test_memory_visibility.py",
|
||||
"test_memory_write_routes.py",
|
||||
"test_model_content_routes.py",
|
||||
"test_model_creation_api.py",
|
||||
"test_model_creation_validation_boundary.py",
|
||||
"test_model_probe_operations.py",
|
||||
@@ -124,17 +122,14 @@ INTEGRATION_FILES = frozenset(
|
||||
"test_source_connection_intents.py",
|
||||
"test_source_import_receipts.py",
|
||||
"test_source_permission_races.py",
|
||||
"test_source_read_revocation.py",
|
||||
"test_source_revocation_parking.py",
|
||||
"test_source_session_count_api.py",
|
||||
"test_source_status_latency.py",
|
||||
"test_source_sync_idempotent.py",
|
||||
"test_source_sync_permissions.py",
|
||||
"test_source_upload_intents.py",
|
||||
"test_ui_session_api.py",
|
||||
"test_unified_write_entry_wiring.py",
|
||||
"test_visibility_background.py",
|
||||
"test_workbuddy_prompt_backfill.py",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@@ -26,67 +26,6 @@ async def thaw_source(client, source_id, receipt):
|
||||
return response.json()
|
||||
|
||||
|
||||
async def confirm_model_content(
|
||||
client,
|
||||
*,
|
||||
source_ids,
|
||||
evidence_ids=(),
|
||||
all_imported=None,
|
||||
future_additions=True,
|
||||
process=True,
|
||||
):
|
||||
"""Grant model processing the same way the product does: preview, then confirm.
|
||||
|
||||
Live suites need this because importing Evidence is not consent to send it
|
||||
to a model; without it the processing guard refuses, which is the correct
|
||||
product behaviour and not something a test may bypass by writing the store.
|
||||
"""
|
||||
profiles = await client.get("/api/v1/model-profiles")
|
||||
profiles.raise_for_status()
|
||||
active = next(item for item in profiles.json() if item["is_active"])
|
||||
preview = await client.post(
|
||||
"/api/v1/model-content-permission/preview",
|
||||
json={
|
||||
"profile_id": active["profile_id"],
|
||||
"profile_revision": active["revision"],
|
||||
"source_ids": list(source_ids),
|
||||
"evidence_ids": list(evidence_ids),
|
||||
# The manual source only accepts named originals, so "everything
|
||||
# from these sources" is not a default the caller can ignore.
|
||||
"all_imported_from_selected_sources": (
|
||||
not evidence_ids if all_imported is None else all_imported
|
||||
),
|
||||
"include_future_additions": future_additions,
|
||||
},
|
||||
)
|
||||
# Report the refusal code, not just the status: every content refusal is a
|
||||
# named product decision and a live run that hides it cannot be diagnosed.
|
||||
assert preview.status_code == 200, preview.text
|
||||
current = await client.get("/api/v1/model-content-permission")
|
||||
current.raise_for_status()
|
||||
existing = current.json()
|
||||
confirmed = await client.post(
|
||||
"/api/v1/model-content-permission",
|
||||
json={
|
||||
"snapshot": preview.json(),
|
||||
"expected_revision": existing["revision"] if existing else 0,
|
||||
"user_confirmed": True,
|
||||
},
|
||||
)
|
||||
confirmed.raise_for_status()
|
||||
receipt = confirmed.json()
|
||||
if process:
|
||||
# What the product does after the user confirms: take the already
|
||||
# imported Evidence through the pipeline instead of waiting for the
|
||||
# next import to trigger a debounce.
|
||||
accepted = await client.post(
|
||||
"/api/v1/model-content-permission/process",
|
||||
json={"expected_revision": receipt["revision"], "user_confirmed": True},
|
||||
)
|
||||
accepted.raise_for_status()
|
||||
return receipt
|
||||
|
||||
|
||||
async def connection_request(client, source_ids, **options):
|
||||
roots = {}
|
||||
for source_id in source_ids:
|
||||
|
||||
@@ -65,12 +65,7 @@ HY_BRIDGE = f"{PACKAGE_LABEL}/hy_memory_bridge.py"
|
||||
SDK_CALLER = HY_BRIDGE
|
||||
|
||||
#: Type-only importers of HY datatypes; they hold no client and write nothing.
|
||||
HY_TYPE_ONLY = frozenset(
|
||||
{
|
||||
f"{PACKAGE_LABEL}/hy_permission_store.py",
|
||||
f"{PACKAGE_LABEL}/hy_permission_coding_store.py",
|
||||
}
|
||||
)
|
||||
HY_TYPE_ONLY: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
def _relative(path: Path) -> str:
|
||||
|
||||
@@ -1,248 +0,0 @@
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.model_profiles import ModelProfilePatch
|
||||
from tests.test_model_processing_permissions import Harness
|
||||
|
||||
|
||||
def test_quiet_arrival_guard_is_off_loop_and_5000_events_coalesce(tmp_path, monkeypatch):
|
||||
h = Harness(tmp_path)
|
||||
supervisor = h.supervisor()
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
checked = []
|
||||
original = supervisor._guard_records
|
||||
|
||||
async def scenario():
|
||||
loop_thread = threading.get_ident()
|
||||
|
||||
def guard(records, *args, **kwargs):
|
||||
checked.append(threading.get_ident())
|
||||
assert threading.get_ident() != loop_thread, "arrival parsed permission on the loop"
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
return original(records, *args, **kwargs)
|
||||
|
||||
def credentials(*_args, **_kwargs):
|
||||
pytest.fail("denied preparation reached credential/profile resolution")
|
||||
|
||||
monkeypatch.setattr(supervisor, "_guard_records", guard)
|
||||
monkeypatch.setattr(h.profiles, "active", credentials)
|
||||
monkeypatch.setattr(h.profiles, "resolve_snapshot_secret", credentials)
|
||||
try:
|
||||
for _ in range(5000):
|
||||
supervisor.enqueue_evidence(h.record)
|
||||
assert checked == []
|
||||
assert len(supervisor._debounces) == 1
|
||||
assert await asyncio.to_thread(entered.wait, 3)
|
||||
# This callback must run while the actual permission reader is blocked.
|
||||
responsive = asyncio.Event()
|
||||
asyncio.get_running_loop().call_soon(responsive.set)
|
||||
await asyncio.wait_for(responsive.wait(), 0.5)
|
||||
assert checked == [checked[0]]
|
||||
release.set()
|
||||
await asyncio.gather(*tuple(supervisor._debounces.values()))
|
||||
assert supervisor._preparations == {}
|
||||
assert h.tasks.list() == []
|
||||
assert len(supervisor._queue) == 0 and supervisor._workers == {}
|
||||
assert h.factory_calls == [] and h.bridge.calls == []
|
||||
finally:
|
||||
release.set()
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stop_mode", ["cancel", "shutdown"])
|
||||
def test_live_preparation_cancel_joins_thread_without_late_dispatch(
|
||||
tmp_path, monkeypatch, stop_mode
|
||||
):
|
||||
h = Harness(tmp_path)
|
||||
h.grant()
|
||||
supervisor = h.supervisor()
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
original = supervisor._prepare_conversation_work
|
||||
completed = []
|
||||
|
||||
def blocked(seed, **kwargs):
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
result = original(seed, **kwargs)
|
||||
completed.append(True)
|
||||
return result
|
||||
|
||||
monkeypatch.setattr(supervisor, "_prepare_conversation_work", blocked)
|
||||
|
||||
async def scenario():
|
||||
stopping = None
|
||||
preparation = None
|
||||
try:
|
||||
supervisor.enqueue_evidence(h.record)
|
||||
assert await asyncio.to_thread(entered.wait, 3)
|
||||
preparation = next(iter(supervisor._preparations.values()))
|
||||
if stop_mode == "cancel":
|
||||
preparation.cancel()
|
||||
await asyncio.sleep(0.02)
|
||||
preparation.cancel()
|
||||
else:
|
||||
stopping = asyncio.create_task(supervisor.shutdown())
|
||||
await asyncio.sleep(0.02)
|
||||
assert not preparation.done(), "cancel detached the actual preparation thread"
|
||||
if stopping is not None:
|
||||
assert not stopping.done()
|
||||
finally:
|
||||
release.set()
|
||||
if preparation is not None:
|
||||
await asyncio.gather(preparation, return_exceptions=True)
|
||||
if stopping is not None:
|
||||
await stopping
|
||||
await supervisor.shutdown()
|
||||
assert completed == [True]
|
||||
assert supervisor._preparations == {} and supervisor._debounces == {}
|
||||
assert len(supervisor._queue) == 0 and supervisor._workers == {}
|
||||
assert h.factory_calls == [] and h.bridge.calls == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"change", ["model-revoked", "source-revoked", "profile-changed", "regranted-other-input"]
|
||||
)
|
||||
def test_quiet_preparation_rechecks_permission_before_credentials_or_durable_work(
|
||||
tmp_path, monkeypatch, change
|
||||
):
|
||||
h = Harness(tmp_path)
|
||||
receipt = h.grant()
|
||||
supervisor = h.supervisor()
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
original = supervisor._prepare_conversation_work
|
||||
guarded = []
|
||||
original_guard = supervisor._guard_records
|
||||
|
||||
def blocked(seed, **kwargs):
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
return original(seed, **kwargs)
|
||||
|
||||
def guard(*args, **kwargs):
|
||||
guarded.append(True)
|
||||
return original_guard(*args, **kwargs)
|
||||
|
||||
def credentials(*_args, **_kwargs):
|
||||
pytest.fail("denied preparation reached credential/profile resolution")
|
||||
|
||||
monkeypatch.setattr(supervisor, "_prepare_conversation_work", blocked)
|
||||
monkeypatch.setattr(supervisor, "_guard_records", guard)
|
||||
|
||||
async def scenario():
|
||||
try:
|
||||
supervisor.enqueue_evidence(h.record)
|
||||
assert await asyncio.to_thread(entered.wait, 3)
|
||||
if change in {"model-revoked", "regranted-other-input"}:
|
||||
h.permissions.revoke(
|
||||
user_id="local", expected_revision=receipt.revision, user_confirmed=True
|
||||
)
|
||||
if change == "regranted-other-input":
|
||||
h.grant((h.put("other", conversation="other"),))
|
||||
elif change == "source-revoked":
|
||||
h.sources.revoke(user_id="local", source_id="cursor")
|
||||
else:
|
||||
h.profiles.update(
|
||||
h.profile.profile_id,
|
||||
ModelProfilePatch(expected_revision=h.profile.revision, model="changed"),
|
||||
h.secrets,
|
||||
)
|
||||
monkeypatch.setattr(h.profiles, "active", credentials)
|
||||
monkeypatch.setattr(h.profiles, "resolve_snapshot_secret", credentials)
|
||||
release.set()
|
||||
await asyncio.gather(*tuple(supervisor._debounces.values()))
|
||||
assert guarded == [True]
|
||||
assert h.tasks.list() == []
|
||||
assert h.factory_calls == [] and h.bridge.calls == []
|
||||
assert len(supervisor._queue) == 0 and supervisor._workers == {}
|
||||
finally:
|
||||
release.set()
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_cancelling_quiet_timer_does_not_start_any_permission_reader(tmp_path, monkeypatch):
|
||||
h = Harness(tmp_path)
|
||||
h.grant()
|
||||
supervisor = h.supervisor()
|
||||
checks = []
|
||||
original = supervisor._guard_records
|
||||
|
||||
def guard(*args, **kwargs):
|
||||
checks.append(True)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(supervisor, "_guard_records", guard)
|
||||
|
||||
async def scenario():
|
||||
try:
|
||||
supervisor.enqueue_evidence(h.record)
|
||||
timer = next(iter(supervisor._debounces.values()))
|
||||
await asyncio.sleep(0)
|
||||
timer.cancel()
|
||||
await asyncio.gather(timer, return_exceptions=True)
|
||||
assert checks == []
|
||||
assert supervisor._debounces == {} and supervisor._preparations == {}
|
||||
assert h.tasks.list() == [] and h.factory_calls == [] and h.bridge.calls == []
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_dispatch_permission_check_denies_before_queue_after_classification(tmp_path, monkeypatch):
|
||||
h = Harness(tmp_path)
|
||||
h.grant()
|
||||
supervisor = h.supervisor()
|
||||
work = supervisor._classify(supervisor.prepare_conversation(h.record))
|
||||
checked = []
|
||||
|
||||
def reject(task):
|
||||
checked.append(task.task_id)
|
||||
return False
|
||||
|
||||
async def scenario():
|
||||
monkeypatch.setattr(supervisor, "_guard_task", reject)
|
||||
try:
|
||||
supervisor._schedule_tasks(work)
|
||||
assert len(checked) == 1
|
||||
assert len(supervisor._queue) == 0 and supervisor._workers == {}
|
||||
assert h.factory_calls == []
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_dispatch_check_does_not_queue_work_after_shutdown(tmp_path, monkeypatch):
|
||||
h = Harness(tmp_path)
|
||||
h.grant()
|
||||
supervisor = h.supervisor()
|
||||
work = supervisor._classify(supervisor.prepare_conversation(h.record))
|
||||
checked = []
|
||||
|
||||
def allowed(task):
|
||||
checked.append(task.task_id)
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(supervisor, "_guard_task", allowed)
|
||||
|
||||
async def scenario():
|
||||
stopping = asyncio.create_task(supervisor.shutdown())
|
||||
try:
|
||||
await stopping
|
||||
supervisor._schedule_tasks(work)
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
assert len(supervisor._queue) == 0 and supervisor._workers == {}
|
||||
assert checked == []
|
||||
assert h.factory_calls == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,244 +0,0 @@
|
||||
"""A processing acknowledgement belongs to one consent revision and live worker."""
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from tests.test_model_processing_permissions import Harness
|
||||
|
||||
|
||||
def test_changed_consent_does_not_acknowledge_an_older_running_request(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
first = h.grant()
|
||||
supervisor = h.supervisor()
|
||||
started = asyncio.Event()
|
||||
release = asyncio.Event()
|
||||
calls = []
|
||||
|
||||
async def backfill(*, expected_permission_revision=None, on_first_scheduled=None):
|
||||
calls.append(expected_permission_revision)
|
||||
if on_first_scheduled is not None:
|
||||
on_first_scheduled()
|
||||
started.set()
|
||||
await release.wait()
|
||||
|
||||
monkeypatch.setattr(supervisor, "backfill_off_thread", backfill)
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
supervisor.accept_backfill_off_thread(expected_permission_revision=first.revision),
|
||||
timeout=5,
|
||||
)
|
||||
await asyncio.wait_for(started.wait(), timeout=5)
|
||||
second = h.grant()
|
||||
await asyncio.wait_for(
|
||||
supervisor.accept_backfill_off_thread(expected_permission_revision=second.revision),
|
||||
timeout=5,
|
||||
)
|
||||
started.clear()
|
||||
await asyncio.wait_for(started.wait(), timeout=5)
|
||||
assert calls == [first.revision, second.revision]
|
||||
finally:
|
||||
release.set()
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_late_failure_is_observed_in_processing_status(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
receipt = h.grant()
|
||||
supervisor = h.supervisor()
|
||||
release = asyncio.Event()
|
||||
|
||||
async def backfill(*, expected_permission_revision=None, on_first_scheduled=None):
|
||||
if on_first_scheduled is not None:
|
||||
on_first_scheduled()
|
||||
await release.wait()
|
||||
raise HyMemoryBridgeError("model_content_source_permission_changed")
|
||||
|
||||
monkeypatch.setattr(supervisor, "backfill_off_thread", backfill)
|
||||
try:
|
||||
await supervisor.accept_backfill_off_thread(
|
||||
expected_permission_revision=receipt.revision
|
||||
)
|
||||
assert supervisor.accepted_backfill_status() == ("running", None)
|
||||
release.set()
|
||||
await supervisor._live_backfill
|
||||
assert supervisor.accepted_backfill_status() == (
|
||||
"failed",
|
||||
"model_content_source_permission_changed",
|
||||
)
|
||||
finally:
|
||||
release.set()
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_same_consent_concurrent_requests_share_one_worker(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
receipt = h.grant()
|
||||
supervisor = h.supervisor()
|
||||
entered, acknowledge, finish = asyncio.Event(), asyncio.Event(), asyncio.Event()
|
||||
calls = []
|
||||
|
||||
async def backfill(*, expected_permission_revision=None, on_first_scheduled=None):
|
||||
calls.append(expected_permission_revision)
|
||||
entered.set()
|
||||
await acknowledge.wait()
|
||||
if on_first_scheduled is not None:
|
||||
on_first_scheduled()
|
||||
await finish.wait()
|
||||
|
||||
monkeypatch.setattr(supervisor, "backfill_off_thread", backfill)
|
||||
first = asyncio.create_task(
|
||||
supervisor.accept_backfill_off_thread(expected_permission_revision=receipt.revision)
|
||||
)
|
||||
try:
|
||||
await entered.wait()
|
||||
second = asyncio.create_task(
|
||||
supervisor.accept_backfill_off_thread(expected_permission_revision=receipt.revision)
|
||||
)
|
||||
await asyncio.sleep(0)
|
||||
acknowledge.set()
|
||||
await asyncio.wait_for(asyncio.gather(first, second), 2)
|
||||
assert calls == [receipt.revision]
|
||||
finally:
|
||||
acknowledge.set()
|
||||
finish.set()
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_background_failure_is_reported_after_acceptance_and_can_retry(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
receipt = h.grant()
|
||||
supervisor = h.supervisor()
|
||||
|
||||
async def broken(**_kwargs):
|
||||
raise RuntimeError("secret or provider body must not be returned")
|
||||
|
||||
monkeypatch.setattr(supervisor, "backfill_off_thread", broken)
|
||||
try:
|
||||
await supervisor.accept_backfill_off_thread(
|
||||
expected_permission_revision=receipt.revision
|
||||
)
|
||||
await supervisor._live_backfill
|
||||
assert supervisor.accepted_backfill_status() == (
|
||||
"failed",
|
||||
"model_content_processing_unavailable",
|
||||
)
|
||||
|
||||
async def complete(**kwargs):
|
||||
callback = kwargs.get("on_first_scheduled")
|
||||
if callback is not None:
|
||||
callback()
|
||||
|
||||
monkeypatch.setattr(supervisor, "backfill_off_thread", complete)
|
||||
await supervisor.accept_backfill_off_thread(
|
||||
expected_permission_revision=receipt.revision
|
||||
)
|
||||
await supervisor._live_backfill
|
||||
assert supervisor.accepted_backfill_status() == ("done", None)
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_shutdown_rejects_new_processing_acceptance(tmp_path):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
receipt = h.grant()
|
||||
supervisor = h.supervisor()
|
||||
await supervisor.shutdown()
|
||||
with pytest.raises(HyMemoryBridgeError, match="model_content_processing_unavailable"):
|
||||
await supervisor.accept_backfill_off_thread(
|
||||
expected_permission_revision=receipt.revision
|
||||
)
|
||||
assert h.tasks.list() == []
|
||||
assert h.factory_calls == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_health_reports_late_explicit_processing_failure(tmp_path, monkeypatch):
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
|
||||
monkeypatch.setattr("one_memory.model_presets._preset_data", lambda: None)
|
||||
|
||||
async def scenario():
|
||||
app = create_app(Settings(_env_file=None, data_dir=tmp_path, imports_enabled=False))
|
||||
release = asyncio.Event()
|
||||
async with app.router.lifespan_context(app):
|
||||
await app.state.memory_backfill
|
||||
supervisor = app.state.memory_processing_supervisor
|
||||
supervisor._permission_guard = None
|
||||
|
||||
async def failed_later(**kwargs):
|
||||
callback = kwargs.get("on_first_scheduled")
|
||||
if callback is not None:
|
||||
callback()
|
||||
await release.wait()
|
||||
raise HyMemoryBridgeError("model_content_source_permission_changed")
|
||||
|
||||
monkeypatch.setattr(supervisor, "backfill_off_thread", failed_later)
|
||||
try:
|
||||
await supervisor.accept_backfill_off_thread()
|
||||
async with AsyncClient(
|
||||
transport=ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
running = (await client.get("/api/v1/health")).json()
|
||||
assert running["memory_backfill"] == "running"
|
||||
release.set()
|
||||
await supervisor._live_backfill
|
||||
failed = (await client.get("/api/v1/health")).json()
|
||||
assert failed["memory_backfill"] == "failed"
|
||||
assert (
|
||||
failed["memory_backfill_error"] == "model_content_source_permission_changed"
|
||||
)
|
||||
finally:
|
||||
release.set()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_completed_caller_does_not_cancel_accepted_background_work(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
receipt = h.grant()
|
||||
supervisor = h.supervisor()
|
||||
entered, release = asyncio.Event(), asyncio.Event()
|
||||
|
||||
async def backfill(**kwargs):
|
||||
entered.set()
|
||||
await release.wait()
|
||||
callback = kwargs.get("on_first_scheduled")
|
||||
if callback is not None:
|
||||
callback()
|
||||
|
||||
monkeypatch.setattr(supervisor, "backfill_off_thread", backfill)
|
||||
caller = asyncio.create_task(
|
||||
supervisor.accept_backfill_off_thread(expected_permission_revision=receipt.revision)
|
||||
)
|
||||
try:
|
||||
await asyncio.wait_for(caller, timeout=2)
|
||||
assert not supervisor._live_backfill.done()
|
||||
await asyncio.wait_for(entered.wait(), timeout=5)
|
||||
release.set()
|
||||
await supervisor._live_backfill
|
||||
assert supervisor.accepted_backfill_status() == ("done", None)
|
||||
finally:
|
||||
release.set()
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,54 +0,0 @@
|
||||
"""One stale conversation cannot invalidate an entire backfill scan."""
|
||||
|
||||
import asyncio
|
||||
from contextlib import suppress
|
||||
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.imports.models import EvidenceInput
|
||||
from tests.test_continuous_processing_integration import IntegrationCore
|
||||
|
||||
|
||||
def test_one_changed_conversation_does_not_block_other_conversations(tmp_path, monkeypatch):
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
async def scenario():
|
||||
async with core.running(automatic_memory_processing=False):
|
||||
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
|
||||
core.grant_source()
|
||||
core.put_complete("conversation-good", conversation="good")
|
||||
changed = core.put_complete("conversation-changed", conversation="changed")
|
||||
await core.save_and_probe()
|
||||
core.kernel_enable("include")
|
||||
|
||||
evidence_store = core.app.state.evidence_store
|
||||
changed_input = EvidenceInput.model_validate(
|
||||
{
|
||||
**changed.model_dump(include=set(EvidenceInput.model_fields)),
|
||||
"verbatim": changed.verbatim + " edited",
|
||||
}
|
||||
)
|
||||
evidence_store.put(changed_input)
|
||||
assert evidence_store.get(changed.evidence_id).revision > changed.revision
|
||||
|
||||
# Production reaches completed-input reuse with the live chunk when
|
||||
# an earlier task already covered it; that reuse rechecks the
|
||||
# stored input against the row now in Evidence. Model the observed
|
||||
# stale-row refusal at the exact boundary that raises in production.
|
||||
guard = core.app.state.memory_processing_supervisor._permission_guard
|
||||
original_completed_lookup = guard.completed_input_task
|
||||
|
||||
def refuse_changed_conversation(receipt, records, **kwargs):
|
||||
if any(record.conversation_id == "changed" for record in records):
|
||||
raise HyMemoryBridgeError("model_content_evidence_changed")
|
||||
return original_completed_lookup(receipt, records, **kwargs)
|
||||
|
||||
monkeypatch.setattr(guard, "completed_input_task", refuse_changed_conversation)
|
||||
|
||||
supervisor = core.app.state.memory_processing_supervisor
|
||||
with suppress(Exception):
|
||||
await supervisor.backfill_off_thread()
|
||||
|
||||
await core.wait_completed(timeout=10, minimum_tasks=1)
|
||||
assert len(core.bridge.requests) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,9 +1,8 @@
|
||||
"""Diagnostic: why does an all-imported preview see zero Evidence?
|
||||
|
||||
The C0 integration red asserts a preview snapshot has two Evidence rows after a
|
||||
real WorkBuddy import. Reproduces the exact query path from
|
||||
`ModelContentPermissionStore.snapshot_from_ids` and reports what the import
|
||||
actually wrote, so the owner can be named from evidence instead of inference.
|
||||
The C0 integration red asserted a preview snapshot had two Evidence rows after
|
||||
a real WorkBuddy import. This keeps the useful import-path diagnostic after
|
||||
the retired model-content preview layer was removed.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
@@ -1,105 +0,0 @@
|
||||
"""Server-held previews preserve exact grants without round-tripping all refs."""
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.model_content_permissions import (
|
||||
ModelContentPermissionError,
|
||||
ModelContentPermissionStore,
|
||||
)
|
||||
from tests.test_model_content_bulk import many_records
|
||||
from tests.test_model_content_permissions import USER, evidence_value
|
||||
from tests.test_model_content_permissions import env as env
|
||||
|
||||
|
||||
def test_compact_preview_grants_exact_scope_and_survives_restart(env):
|
||||
records = many_records(env, 1000)
|
||||
snapshot = env.snapshot(records=tuple(records), future=True)
|
||||
result = env.permissions.retain_preview(snapshot)
|
||||
assert result["evidence"] == [] and result["evidence_count"] == 1000
|
||||
assert len(json.dumps(result)) < 5000
|
||||
assert env.permissions.receipt(user_id=USER) is None
|
||||
restarted = ModelContentPermissionStore(env.database)
|
||||
receipt = restarted.grant_preview(
|
||||
result["preview_id"], user_id=USER, expected_revision=0, user_confirmed=True
|
||||
)
|
||||
assert receipt.snapshot == snapshot
|
||||
assert env.check(receipt) == receipt
|
||||
with pytest.raises(ModelContentPermissionError, match="permission_changed"):
|
||||
restarted.grant_preview(
|
||||
result["preview_id"], user_id=USER, expected_revision=0, user_confirmed=True
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"change", ["edit", "revoke-source", "grant-race", "expiry", "wrong-owner", "confirmation"]
|
||||
)
|
||||
def test_compact_preview_does_not_bypass_live_checks(env, change):
|
||||
result = env.permissions.retain_preview(env.snapshot(future=True))
|
||||
if change == "edit":
|
||||
with sqlite3.connect(env.database) as db:
|
||||
db.execute("UPDATE evidence SET verbatim='changed raw original'")
|
||||
elif change == "revoke-source":
|
||||
env.sources.revoke(user_id=USER, source_id=env.record.source_id)
|
||||
elif change == "grant-race":
|
||||
env.grant()
|
||||
elif change == "expiry":
|
||||
with sqlite3.connect(env.database) as db:
|
||||
db.execute("UPDATE model_content_previews SET expires_at='2000-01-01T00:00:00+00:00'")
|
||||
with pytest.raises(ModelContentPermissionError):
|
||||
env.permissions.grant_preview(
|
||||
result["preview_id"],
|
||||
user_id="other" if change == "wrong-owner" else USER,
|
||||
expected_revision=0,
|
||||
user_confirmed=change != "confirmation",
|
||||
)
|
||||
|
||||
|
||||
def test_new_originals_keep_explicit_future_scope_but_not_exact_scope(env):
|
||||
result = env.permissions.retain_preview(env.snapshot(future=True))
|
||||
later = env.evidence.put(evidence_value("new-arrival")).record
|
||||
receipt = env.permissions.grant_preview(
|
||||
result["preview_id"], user_id=USER, expected_revision=0, user_confirmed=True
|
||||
)
|
||||
assert len(receipt.snapshot.evidence) == 1
|
||||
assert env.check(receipt, records=(later,)) == receipt
|
||||
|
||||
|
||||
def test_compact_receipt_is_metadata_only_and_version_bound(env):
|
||||
result = env.permissions.retain_preview(env.snapshot())
|
||||
receipt = env.permissions.grant_preview(
|
||||
result["preview_id"], user_id=USER, expected_revision=0, user_confirmed=True
|
||||
)
|
||||
compact = env.permissions.compact_receipt(receipt)
|
||||
assert compact["revision"] == receipt.revision
|
||||
assert compact["snapshot"]["snapshot_hash"] == result["snapshot_hash"]
|
||||
assert compact["snapshot"]["source_counts"] == {env.record.source_id: 1}
|
||||
assert env.record.verbatim not in json.dumps(compact)
|
||||
assert compact["snapshot"]["evidence"] == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("future", [False, True])
|
||||
def test_preview_cannot_rebase_over_a_grant_or_revoke_between_reads(env, future):
|
||||
first = env.grant(future=future)
|
||||
preview = env.snapshot(future=future)
|
||||
env.permissions.revoke(user_id=USER, expected_revision=first.revision, user_confirmed=True)
|
||||
with pytest.raises(ModelContentPermissionError, match="permission_changed"):
|
||||
env.permissions.retain_preview(preview)
|
||||
assert env.permissions.receipt(user_id=USER).revoked_at is not None
|
||||
|
||||
|
||||
def test_shared_migration_declares_the_retained_preview_table() -> None:
|
||||
"""Retained previews are product state; migration 28 must recreate them.
|
||||
|
||||
This is a red-state guard for the A1-04 schema extraction. The table exists
|
||||
in the historical f6d624a schema and is still read/written by this module,
|
||||
so migration 28's generator must declare it for both SQLite and MySQL.
|
||||
"""
|
||||
from one_memory.memory_server import migrations
|
||||
|
||||
sqlite_schema = migrations.model_content_permissions_ddl("sqlite")
|
||||
mysql_schema = migrations.model_content_permissions_ddl("mysql")
|
||||
assert "model_content_previews" in sqlite_schema
|
||||
assert "model_content_previews" in mysql_schema
|
||||
@@ -12,15 +12,12 @@ import asyncio
|
||||
import gc
|
||||
import sqlite3
|
||||
from contextlib import contextmanager
|
||||
from types import SimpleNamespace
|
||||
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import SecretStr
|
||||
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
from one_memory.database_access import close_main_databases, main_database_stats
|
||||
from one_memory.hy_permission_context import _NativeProofs
|
||||
from one_memory.imports.sources.registry import build_source_catalog
|
||||
|
||||
|
||||
@@ -81,24 +78,3 @@ def test_polled_endpoints_close_every_connection_they_open(tmp_path, monkeypatch
|
||||
assert left_open == 0, f"{left_open} connections left for the collector"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_hy_permission_proofs_close_their_connections(tmp_path):
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
with sqlite3.connect(database) as db:
|
||||
db.execute("CREATE TABLE governed_memories (memory_id TEXT, revision INT, status TEXT)")
|
||||
db.close()
|
||||
proofs = _NativeProofs(database)
|
||||
lease = SimpleNamespace(records=())
|
||||
# The first read opens the pooled reader; it is meant to stay open.
|
||||
proofs.guard(proofs.load(lease))()
|
||||
|
||||
with _without_cyclic_collection():
|
||||
before = _open_connections()
|
||||
for _ in range(5):
|
||||
check = proofs.guard(proofs.load(lease))
|
||||
check()
|
||||
left_open = _open_connections() - before
|
||||
assert left_open == 0, f"{left_open} connections left for the collector"
|
||||
close_main_databases()
|
||||
assert main_database_stats()["open_connections"] == 0
|
||||
@@ -1,341 +0,0 @@
|
||||
"""Independent review of completed-input reuse; real stores and controlled HY only."""
|
||||
|
||||
import asyncio
|
||||
import sqlite3
|
||||
from contextlib import closing
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory import continuous_policy
|
||||
from one_memory.continuous_permission_bridge import ContinuousPermissionReceipt
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.imports.models import EvidenceInput
|
||||
from one_memory.imports.sources.registry import RootCandidateKind
|
||||
from one_memory.memory_processing import MEMORY_INGEST_POLICY_VERSION
|
||||
from one_memory.memory_tasks import MemoryTaskStatus
|
||||
from tests.test_continuous_processing_integration import IntegrationCore
|
||||
|
||||
|
||||
def transition(core, action):
|
||||
with closing(sqlite3.connect(core.database)) as db, db:
|
||||
db.execute("BEGIN IMMEDIATE")
|
||||
policy = continuous_policy.current(db)
|
||||
return continuous_policy.transition(
|
||||
db,
|
||||
action=action,
|
||||
expected_revision=policy.revision,
|
||||
operation_id=str(uuid4()),
|
||||
user_confirmed=True,
|
||||
)
|
||||
|
||||
|
||||
def bindings(core):
|
||||
with closing(sqlite3.connect(core.database)) as db:
|
||||
return db.execute(
|
||||
"SELECT task_id,policy_revision,policy_epoch,receipt_json "
|
||||
"FROM memory_task_continuous_permissions ORDER BY task_id"
|
||||
).fetchall()
|
||||
|
||||
|
||||
async def completed(core):
|
||||
core.grant_source()
|
||||
await core.save_and_probe()
|
||||
core.kernel_enable("none")
|
||||
record = core.put_complete("review-completed", conversation="review")
|
||||
tasks = await core.run_prepared(record)
|
||||
assert len(tasks) == 1 and tasks[0].status == MemoryTaskStatus.COMPLETED
|
||||
assert len(core.bridge.requests) == 1
|
||||
return record, tasks[0]
|
||||
|
||||
|
||||
def frozen(core, records):
|
||||
guard = core.app.state.memory_processing_supervisor._permission_guard
|
||||
receipt = guard.freeze(guard.current("local"), records)
|
||||
assert isinstance(receipt, ContinuousPermissionReceipt)
|
||||
return guard, receipt
|
||||
|
||||
|
||||
def lookup(guard, receipt, records, seed):
|
||||
return guard.completed_input_task(
|
||||
receipt,
|
||||
records,
|
||||
input_ref=seed.evidence_id,
|
||||
request_prefix=f"conversation:{MEMORY_INGEST_POLICY_VERSION}:{seed.source_id}:",
|
||||
)
|
||||
|
||||
|
||||
def test_resume_reuses_completed_without_rebinding_historical_receipt(tmp_path, monkeypatch):
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
record, task = await completed(core)
|
||||
before = bindings(core)
|
||||
transition(core, "pause")
|
||||
resumed = transition(core, "resume")
|
||||
supervisor = core.app.state.memory_processing_supervisor
|
||||
profile = core.app.state.model_profile_store.active(core.secrets)
|
||||
|
||||
def cannot_rebind(*args, **kwargs):
|
||||
pytest.fail("completed history was rebound to a new execution receipt")
|
||||
|
||||
with monkeypatch.context() as patch:
|
||||
patch.setattr(supervisor._permission_guard, "bind", cannot_rebind)
|
||||
reused = supervisor._enqueue_chunk(core.turn_records(record), profile)
|
||||
assert reused.task_id == task.task_id
|
||||
assert reused.status == MemoryTaskStatus.COMPLETED
|
||||
assert resumed.revision == 3 and before[0][1] == 1
|
||||
assert bindings(core) == before
|
||||
assert supervisor._classify([reused]) == []
|
||||
assert len(core.bridge.requests) == 1
|
||||
assert len(core.app.state.memory_task_store.list()) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change", ["chunk", "event-revision", "profile"])
|
||||
def test_changed_input_identity_is_not_reused_as_completed(tmp_path, monkeypatch, change):
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
record, task = await completed(core)
|
||||
before = bindings(core)
|
||||
records = core.turn_records(record)
|
||||
if change == "chunk":
|
||||
other = core.put("review-added", conversation="review")
|
||||
# Same input_ref deliberately: input_ref alone cannot prove chunk identity.
|
||||
records = (other, *records)
|
||||
elif change == "event-revision":
|
||||
changed = EvidenceInput.model_validate(
|
||||
{
|
||||
**record.model_dump(include=set(EvidenceInput.model_fields)),
|
||||
"verbatim": record.verbatim + " changed",
|
||||
}
|
||||
)
|
||||
record = core.app.state.evidence_store.put(changed).record
|
||||
assert record.revision > 1
|
||||
records = (record, *records[1:])
|
||||
elif change == "profile":
|
||||
transition(core, "revoke")
|
||||
# Persist a new profile through the real API; probe remains controlled.
|
||||
saved = await core.client.post(
|
||||
"/api/v1/model-profiles",
|
||||
json={
|
||||
"name": "review-second-profile",
|
||||
"provider_kind": "openai_compatible",
|
||||
"base_url": "https://integration.invalid/v1",
|
||||
"model": "integration-model",
|
||||
"api_key": "synthetic-key-not-a-credential",
|
||||
"embedding_model": "integration-vector",
|
||||
"embedding_provider_kind": "openai_compatible",
|
||||
"embedding_base_url": "https://integration.invalid/v1",
|
||||
},
|
||||
)
|
||||
assert saved.status_code == 200
|
||||
result = await core.client.post(
|
||||
f"/api/v1/model-profiles/{saved.json()['profile_id']}/probe"
|
||||
)
|
||||
assert result.status_code == 200
|
||||
core.profile = next(
|
||||
p
|
||||
for p in (await core.client.get("/api/v1/model-profiles")).json()
|
||||
if p["profile_id"] == saved.json()["profile_id"]
|
||||
)
|
||||
current = core.policy_current()
|
||||
core.app.state.model_policy_activation.enable(
|
||||
operation_id=str(uuid4()),
|
||||
profile_id=core.profile["profile_id"],
|
||||
profile_revision=core.profile["revision"],
|
||||
expected_policy_revision=current.revision,
|
||||
expected_legacy_revision=current.legacy_revision,
|
||||
backlog_mode="include",
|
||||
user_confirmed=True,
|
||||
)
|
||||
guard, receipt = frozen(core, records)
|
||||
assert lookup(guard, receipt, records, record) is None
|
||||
assert bindings(core) == before
|
||||
assert (
|
||||
core.app.state.memory_task_store.require(task.task_id).status
|
||||
== MemoryTaskStatus.COMPLETED
|
||||
)
|
||||
assert len(core.bridge.requests) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_changed_source_root_is_not_reused_as_completed(tmp_path, monkeypatch):
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
record, task = await completed(core)
|
||||
before = bindings(core)
|
||||
grants = core.app.state.source_grant_store
|
||||
grants.revoke(user_id="local", source_id="cursor")
|
||||
replacement = core.home / "cursor-replacement"
|
||||
replacement.mkdir()
|
||||
grants.grant(
|
||||
user_id="local",
|
||||
source_id="cursor",
|
||||
root=replacement,
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
records = core.turn_records(record)
|
||||
guard, receipt = frozen(core, records)
|
||||
|
||||
assert lookup(guard, receipt, records, record) is None
|
||||
assert bindings(core) == before
|
||||
assert (
|
||||
core.app.state.memory_task_store.require(task.task_id).status
|
||||
== MemoryTaskStatus.COMPLETED
|
||||
)
|
||||
assert len(core.bridge.requests) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_reenabled_policy_reuses_exact_completed_input_without_duplicate_hy_call(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
record, task = await completed(core)
|
||||
before = bindings(core)
|
||||
transition(core, "revoke")
|
||||
prior = core.policy_current()
|
||||
core.app.state.model_policy_activation.enable(
|
||||
operation_id=str(uuid4()),
|
||||
profile_id=core.profile["profile_id"],
|
||||
profile_revision=core.profile["revision"],
|
||||
expected_policy_revision=prior.revision,
|
||||
expected_legacy_revision=prior.legacy_revision,
|
||||
backlog_mode="include",
|
||||
user_confirmed=True,
|
||||
)
|
||||
assert core.policy_current().epoch != before[0][2]
|
||||
|
||||
supervisor = core.app.state.memory_processing_supervisor
|
||||
profile = core.app.state.model_profile_store.active(core.secrets)
|
||||
reused = supervisor._enqueue_chunk(core.turn_records(record), profile)
|
||||
|
||||
assert reused.task_id == task.task_id
|
||||
assert reused.status == MemoryTaskStatus.COMPLETED
|
||||
assert bindings(core) == before
|
||||
assert len(core.bridge.requests) == 1
|
||||
assert len(core.app.state.memory_task_store.list()) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("when", ["before", "after-first-check"])
|
||||
@pytest.mark.parametrize("authority", ["source", "policy"])
|
||||
def test_current_source_permission_is_rechecked_around_completed_lookup(
|
||||
tmp_path, monkeypatch, when, authority
|
||||
):
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
record, _ = await completed(core)
|
||||
before = bindings(core)
|
||||
records = core.turn_records(record)
|
||||
guard, receipt = frozen(core, records)
|
||||
|
||||
def revoke():
|
||||
if authority == "source":
|
||||
core.app.state.source_grant_store.revoke(user_id="local", source_id="cursor")
|
||||
else:
|
||||
transition(core, "pause")
|
||||
|
||||
calls = []
|
||||
original = guard.check
|
||||
|
||||
def checked(*args, **kwargs):
|
||||
value = original(*args, **kwargs)
|
||||
calls.append(True)
|
||||
if len(calls) == 1:
|
||||
revoke()
|
||||
return value
|
||||
|
||||
if when == "before":
|
||||
revoke()
|
||||
else:
|
||||
monkeypatch.setattr(guard, "check", checked)
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
lookup(guard, receipt, records, record)
|
||||
assert bindings(core) == before
|
||||
assert len(core.bridge.requests) == 1
|
||||
if when == "after-first-check":
|
||||
assert calls == [True]
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_corrupt_historical_binding_is_not_accepted_as_completion(tmp_path, monkeypatch):
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
record, task = await completed(core)
|
||||
with sqlite3.connect(core.database) as db:
|
||||
db.execute(
|
||||
"UPDATE memory_task_continuous_permissions "
|
||||
"SET policy_revision=99 WHERE task_id=?",
|
||||
(task.task_id,),
|
||||
)
|
||||
records = core.turn_records(record)
|
||||
guard, receipt = frozen(core, records)
|
||||
with pytest.raises(HyMemoryBridgeError, match="continuous_binding_invalid"):
|
||||
lookup(guard, receipt, records, record)
|
||||
assert len(core.bridge.requests) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_sdk_admitted_unknown_is_not_reused_or_resent_on_resume(tmp_path, monkeypatch):
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
core.grant_source()
|
||||
await core.save_and_probe()
|
||||
core.kernel_enable()
|
||||
record = core.put_complete("review-unknown")
|
||||
supervisor = core.app.state.memory_processing_supervisor
|
||||
tasks = supervisor.prepare_conversation(record)
|
||||
entered = asyncio.Event()
|
||||
|
||||
async def interrupted(request):
|
||||
core.bridge.requests.append(request)
|
||||
entered.set()
|
||||
await asyncio.Event().wait()
|
||||
|
||||
monkeypatch.setattr(core.bridge, "ingest", interrupted)
|
||||
running = asyncio.create_task(supervisor._create_runner().run(tasks[0].task_id))
|
||||
try:
|
||||
await asyncio.wait_for(entered.wait(), 3)
|
||||
finally:
|
||||
running.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await running
|
||||
# Actual restart recovery marks a previously admitted RUNNING task unknown.
|
||||
core.app.state.memory_task_store.recover_running()
|
||||
before = bindings(core)
|
||||
transition(core, "pause")
|
||||
transition(core, "resume")
|
||||
with pytest.raises(HyMemoryBridgeError, match="outcome_unknown"):
|
||||
supervisor._enqueue_chunk(
|
||||
core.turn_records(record),
|
||||
core.app.state.model_profile_store.active(core.secrets),
|
||||
)
|
||||
assert bindings(core) == before
|
||||
assert len(core.bridge.requests) == 1
|
||||
task = core.app.state.memory_task_store.require(tasks[0].task_id)
|
||||
assert task.last_error_code == "model_content_outcome_unknown" and task.attempts == 1
|
||||
assert len(core.app.state.memory_task_store.list()) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,190 +0,0 @@
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.processing_coordinator import ProcessingCoordinator
|
||||
from one_memory.processing_requests import ProcessingRequestStore
|
||||
from tests.test_continuous_processing_bridge import enable, put_after_policy, transition
|
||||
from tests.test_model_processing_permissions import h as h # noqa: F401
|
||||
|
||||
|
||||
def coordinator(h, monkeypatch):
|
||||
supervisor = h.supervisor()
|
||||
scheduled = []
|
||||
|
||||
async def schedule(work):
|
||||
scheduled.extend(work)
|
||||
|
||||
monkeypatch.setattr(supervisor, "_schedule_tasks_off_thread", schedule)
|
||||
return ProcessingCoordinator(ProcessingRequestStore(h.database), supervisor), scheduled
|
||||
|
||||
|
||||
def test_restart_replays_lost_callback_into_same_durable_task_queue(h, monkeypatch):
|
||||
policy = enable(h)
|
||||
fresh = put_after_policy(h, "committed-before-crash")
|
||||
owner, scheduled = coordinator(h, monkeypatch)
|
||||
|
||||
async def run():
|
||||
try:
|
||||
await owner._continuous_tick()
|
||||
assert scheduled
|
||||
task_ids = {task.task_id for task in h.tasks.list()}
|
||||
assert len(task_ids) == 1
|
||||
assert {
|
||||
ref["evidence_id"]
|
||||
for task in h.tasks.list()
|
||||
for ref in h.tasks.inputs(task.task_id)
|
||||
} == {fresh.evidence_id}
|
||||
assert owner.continuous_status()["observed_seq"] > policy.after_seq
|
||||
await owner._continuous_tick()
|
||||
assert {task.task_id for task in h.tasks.list()} == task_ids
|
||||
restarted, _ = coordinator(h, monkeypatch)
|
||||
try:
|
||||
await restarted._continuous_tick()
|
||||
assert {task.task_id for task in h.tasks.list()} == task_ids
|
||||
finally:
|
||||
await restarted.shutdown()
|
||||
await restarted.supervisor.shutdown()
|
||||
finally:
|
||||
await owner.shutdown()
|
||||
await owner.supervisor.shutdown()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_new_journal_event_and_source_regrant_wake_replay_without_legacy_request(h, monkeypatch):
|
||||
from one_memory.imports.sources.registry import RootCandidateKind
|
||||
|
||||
enable(h)
|
||||
owner, scheduled = coordinator(h, monkeypatch)
|
||||
|
||||
async def run():
|
||||
try:
|
||||
await owner._continuous_tick()
|
||||
assert not scheduled
|
||||
fresh = put_after_policy(h, "new")
|
||||
h.sources.revoke(user_id="local", source_id="cursor")
|
||||
await owner._continuous_tick()
|
||||
assert not scheduled
|
||||
h.sources.grant(
|
||||
user_id="local",
|
||||
source_id="cursor",
|
||||
root=h.root / "synthetic-source",
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
await owner._continuous_tick()
|
||||
assert scheduled
|
||||
assert h.tasks.inputs(scheduled[-1].task_id)[0]["evidence_id"] == fresh.evidence_id
|
||||
with h.evidence._connect() as db:
|
||||
assert db.execute("SELECT count(*) FROM processing_requests").fetchone()[0] == 0
|
||||
finally:
|
||||
await owner.shutdown()
|
||||
await owner.supervisor.shutdown()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_activation_hook_and_startup_recovery_start_single_monitor_then_stop(h, monkeypatch):
|
||||
policy = enable(h)
|
||||
put_after_policy(h, "new")
|
||||
owner, scheduled = coordinator(h, monkeypatch)
|
||||
|
||||
async def run():
|
||||
try:
|
||||
await owner.recover()
|
||||
worker = owner._continuous_worker
|
||||
assert worker is not None
|
||||
await owner.resume_continuous(expected_policy_revision=policy.revision)
|
||||
assert owner._continuous_worker is worker
|
||||
for _ in range(100):
|
||||
if scheduled:
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
assert scheduled
|
||||
finally:
|
||||
await owner.shutdown()
|
||||
await owner.supervisor.shutdown()
|
||||
assert worker.done()
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
await owner.resume_continuous(expected_policy_revision=policy.revision)
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_pause_refuses_recovery_without_fallback_and_resume_replays(h, monkeypatch):
|
||||
h.grant()
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
owner, scheduled = coordinator(h, monkeypatch)
|
||||
transition(h, "pause")
|
||||
|
||||
async def run():
|
||||
try:
|
||||
await owner.recover()
|
||||
assert not scheduled and owner._continuous_worker is None
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
await owner.resume_continuous()
|
||||
transition(h, "resume")
|
||||
await owner._continuous_tick()
|
||||
assert scheduled
|
||||
assert h.tasks.inputs(scheduled[-1].task_id)[0]["evidence_id"] == fresh.evidence_id
|
||||
finally:
|
||||
await owner.shutdown()
|
||||
await owner.supervisor.shutdown()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_failed_preparation_does_not_advance_observed_watermark(h, monkeypatch):
|
||||
enable(h)
|
||||
put_after_policy(h, "new")
|
||||
owner, _ = coordinator(h, monkeypatch)
|
||||
original = owner.supervisor.resume_continuous
|
||||
|
||||
async def fail(**kwargs):
|
||||
raise RuntimeError("synthetic failed preparation")
|
||||
|
||||
monkeypatch.setattr(owner.supervisor, "resume_continuous", fail)
|
||||
|
||||
async def run():
|
||||
try:
|
||||
with pytest.raises(RuntimeError):
|
||||
await owner._continuous_tick()
|
||||
assert owner.continuous_status()["observed_seq"] is None
|
||||
monkeypatch.setattr(owner.supervisor, "resume_continuous", original)
|
||||
await owner._continuous_tick()
|
||||
assert owner.continuous_status()["observed_seq"] is not None
|
||||
finally:
|
||||
await owner.shutdown()
|
||||
await owner.supervisor.shutdown()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_unchanged_continuous_identity_is_rescanned_every_sixty_seconds(h, monkeypatch):
|
||||
enable(h)
|
||||
owner, _scheduled = coordinator(h, monkeypatch)
|
||||
calls = []
|
||||
now = 100.0
|
||||
|
||||
async def resume(*, expected_policy_revision=None):
|
||||
calls.append(expected_policy_revision)
|
||||
|
||||
monkeypatch.setattr(owner.supervisor, "resume_continuous", resume)
|
||||
owner._clock = lambda: now
|
||||
|
||||
async def run():
|
||||
nonlocal now
|
||||
try:
|
||||
await owner._continuous_tick()
|
||||
now = 159.9
|
||||
await owner._continuous_tick()
|
||||
now = 160.0
|
||||
await owner._continuous_tick()
|
||||
assert calls == [1, 1]
|
||||
finally:
|
||||
await owner.shutdown()
|
||||
await owner.supervisor.shutdown()
|
||||
|
||||
asyncio.run(run())
|
||||
@@ -1,176 +0,0 @@
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.memory_tasks import MemoryTaskOwnershipError, MemoryTaskStatus
|
||||
from tests.test_continuous_processing_bridge import enable, put_after_policy
|
||||
from tests.test_model_processing_permissions import h as h # noqa: F401
|
||||
|
||||
|
||||
def test_crashed_continuous_send_is_not_replayed_but_new_event_progresses(h):
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "sent-before-crash")
|
||||
supervisor = h.supervisor()
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
claimed, _ = h.tasks.claim(task.task_id)
|
||||
runner = supervisor._create_runner()
|
||||
lease, request, _ = runner._prepare_permission_lease(claimed)
|
||||
h.bridge.fail = False
|
||||
asyncio.run(h.bridge.ingest(request)) # Controlled send returned, process lost its receipt.
|
||||
assert len(h.bridge.calls) == 1
|
||||
assert h.tasks.recover_running() == 1
|
||||
recovered = h.tasks.require(task.task_id)
|
||||
assert recovered.status == MemoryTaskStatus.UNKNOWN
|
||||
assert recovered.last_error_code == "model_content_outcome_unknown"
|
||||
with pytest.raises(MemoryTaskOwnershipError):
|
||||
h.tasks.claim(task.task_id)
|
||||
assert supervisor.prepare_resume() == []
|
||||
assert supervisor.prepare_backfill() == []
|
||||
# The continuous-policy filter now excludes an already possibly-sent
|
||||
# Evidence before task preparation. Either explicit unknown or outside
|
||||
# policy proves it cannot be blindly replayed.
|
||||
with pytest.raises(HyMemoryBridgeError, match="outcome_unknown|evidence_outside_policy"):
|
||||
asyncio.run(supervisor.process_records((fresh,)))
|
||||
# A new message in the same conversation must not resend the unknown original as context.
|
||||
newest = put_after_policy(h, "new-after-restart")
|
||||
tasks = supervisor.prepare_conversation(newest)
|
||||
assert len(tasks) == 1 and tasks[0].task_id != task.task_id
|
||||
|
||||
async def run():
|
||||
try:
|
||||
result = await supervisor._create_runner().run(tasks[0].task_id)
|
||||
assert result.status == MemoryTaskStatus.COMPLETED
|
||||
sent = [
|
||||
request
|
||||
for kind, request in h.bridge.calls
|
||||
if kind in {"ingest", "ingest_extracted"}
|
||||
]
|
||||
assert len(sent) == 2
|
||||
assert sent[-1].evidence_ids == (newest.evidence_id,)
|
||||
assert all(r.evidence_id != fresh.evidence_id for r in h.leases[-1].records)
|
||||
assert supervisor.prepare_backfill() == []
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(run())
|
||||
assert h.tasks.require(task.task_id).last_error_code == "model_content_outcome_unknown"
|
||||
|
||||
|
||||
def test_legacy_running_recovery_remains_pending_and_no_continuous_table(h):
|
||||
legacy = h.legacy_task()
|
||||
h.tasks.claim(legacy.task_id)
|
||||
assert h.tasks.recover_running() == 1
|
||||
restored = h.tasks.require(legacy.task_id)
|
||||
assert restored.status == MemoryTaskStatus.PENDING
|
||||
assert restored.last_error_code == "worker_restarted"
|
||||
|
||||
|
||||
def test_a_timed_out_non_continuous_task_also_becomes_unknown(h):
|
||||
"""The timeout reason alone is enough to make a task possibly-sent.
|
||||
|
||||
The continuous-permission record is not the only way HY can have been
|
||||
reached: `MemoryTaskRunner.run` wraps the HY call in `asyncio.wait_for`, so
|
||||
a wall-clock timeout can fire after the request was in flight. A
|
||||
non-continuous task must therefore also become sticky `unknown`, or the
|
||||
scheduler re-selects it and sends a second copy (HY has no idempotency key).
|
||||
"""
|
||||
# A legacy (non-continuous) task: no continuous permission record exists.
|
||||
task = h.legacy_task()
|
||||
_claimed, runner_token = h.tasks.claim(task.task_id)
|
||||
h.tasks.record_hy_invocation(task.task_id, runner_token=runner_token)
|
||||
assert h.tasks.recover_one_running(task.task_id, reason="hy_memory_timeout") is True
|
||||
recovered = h.tasks.require(task.task_id)
|
||||
assert recovered.status == MemoryTaskStatus.UNKNOWN
|
||||
assert recovered.last_error_code == "model_content_outcome_unknown"
|
||||
with pytest.raises(MemoryTaskOwnershipError):
|
||||
h.tasks.claim(task.task_id)
|
||||
|
||||
|
||||
def test_claim_without_proven_send_is_still_conservatively_unknown(h):
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "claim-only")
|
||||
task = h.supervisor().prepare_conversation(fresh)[0]
|
||||
h.tasks.claim(task.task_id)
|
||||
h.tasks.recover_running()
|
||||
recovered = h.tasks.require(task.task_id)
|
||||
assert recovered.status == MemoryTaskStatus.UNKNOWN
|
||||
assert recovered.last_error_code == "model_content_outcome_unknown"
|
||||
assert h.bridge.calls == []
|
||||
|
||||
|
||||
def test_a_timed_out_continuous_task_becomes_unknown_not_pending(h):
|
||||
"""The one-shot recovery must be continuous-aware, like process-start recovery.
|
||||
|
||||
`recover_one_running` is the path a timeout takes. A continuous task may
|
||||
already have reached HY, so returning it to `pending` would let it be
|
||||
re-sent and duplicate the memory; it must become sticky `unknown` exactly
|
||||
as `recover_running` marks it at process start.
|
||||
"""
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "timed-out-continuous-send")
|
||||
task = h.supervisor().prepare_conversation(fresh)[0]
|
||||
_claimed, runner_token = h.tasks.claim(task.task_id)
|
||||
h.tasks.record_hy_invocation(task.task_id, runner_token=runner_token)
|
||||
|
||||
assert h.tasks.recover_one_running(task.task_id, reason="hy_memory_timeout") is True
|
||||
|
||||
recovered = h.tasks.require(task.task_id)
|
||||
assert recovered.status == MemoryTaskStatus.UNKNOWN
|
||||
assert recovered.last_error_code == "model_content_outcome_unknown"
|
||||
with pytest.raises(MemoryTaskOwnershipError):
|
||||
h.tasks.claim(task.task_id)
|
||||
|
||||
|
||||
def test_restart_preserves_the_original_unknown_ordering_time(h):
|
||||
"""Recovery must not collapse old in-flight work into the restart instant.
|
||||
|
||||
`unknown_jobs()` pages in `(updated_at, job_id)` order. If a process
|
||||
restart rewrote every recovered task's `updated_at`, a crash with many
|
||||
in-flight tasks would erase their original request order and make the
|
||||
inquiry drain start from an arbitrary same-timestamp batch. The recovery
|
||||
transition changes status/error only; the timestamp remains the durable
|
||||
moment the row last changed before the process died.
|
||||
"""
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "restart-order")
|
||||
task = h.supervisor().prepare_conversation(fresh)[0]
|
||||
h.tasks.claim(task.task_id)
|
||||
original = h.tasks.require(task.task_id).updated_at
|
||||
|
||||
assert h.tasks.recover_running() == 1
|
||||
|
||||
recovered = h.tasks.require(task.task_id)
|
||||
assert recovered.status is MemoryTaskStatus.UNKNOWN
|
||||
assert recovered.updated_at == original
|
||||
|
||||
|
||||
def test_an_unsettled_continuous_send_becomes_unknown_not_failed(h):
|
||||
"""`failed` is retryable, so an abandoned continuous send must be `unknown`.
|
||||
|
||||
`fail_unsettled` runs when a runner raised without settling. That exception
|
||||
may have arrived after HY was already called; `failed` would be re-claimed
|
||||
and re-sent, duplicating the memory. Only a task that never reached
|
||||
`running` (still pending) is provably unsent and may be `failed`.
|
||||
"""
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "unsettled-continuous-send")
|
||||
task = h.supervisor().prepare_conversation(fresh)[0]
|
||||
h.tasks.claim(task.task_id)
|
||||
|
||||
assert h.tasks.fail_unsettled(task.task_id, error_code="memory_task_dispatch_failed") is True
|
||||
|
||||
settled = h.tasks.require(task.task_id)
|
||||
assert settled.status == MemoryTaskStatus.UNKNOWN
|
||||
with pytest.raises(MemoryTaskOwnershipError):
|
||||
h.tasks.claim(task.task_id)
|
||||
|
||||
|
||||
def test_an_unsettled_non_continuous_task_is_still_failed(h):
|
||||
"""The continuous rule must not change the ordinary path."""
|
||||
legacy = h.legacy_task()
|
||||
h.tasks.claim(legacy.task_id)
|
||||
|
||||
assert h.tasks.fail_unsettled(legacy.task_id, error_code="memory_task_dispatch_failed") is True
|
||||
|
||||
assert h.tasks.require(legacy.task_id).status == MemoryTaskStatus.FAILED
|
||||
@@ -1,336 +0,0 @@
|
||||
import sqlite3
|
||||
from contextlib import contextmanager
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory import continuous_policy as policy
|
||||
from one_memory import ingest_journal as journal
|
||||
from one_memory.imports.sources.registry import RootCandidateKind
|
||||
from one_memory.model_profiles import ModelProfilePatch
|
||||
from tests.test_model_content_permissions import SOURCE, evidence_value
|
||||
from tests.test_model_content_permissions import env as env # noqa: F401
|
||||
|
||||
|
||||
@contextmanager
|
||||
def transaction(environment):
|
||||
db = sqlite3.connect(environment.database)
|
||||
db.row_factory = sqlite3.Row
|
||||
db.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
journal.initialize(db)
|
||||
policy.initialize(db)
|
||||
yield db
|
||||
db.commit()
|
||||
except BaseException:
|
||||
db.rollback()
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def enable(db, environment, **overrides):
|
||||
values = dict(
|
||||
profile=environment.profile,
|
||||
expected_revision=0,
|
||||
expected_legacy_revision=0,
|
||||
operation_id=str(uuid4()),
|
||||
backlog_mode="none",
|
||||
user_confirmed=True,
|
||||
)
|
||||
values.update(overrides)
|
||||
return policy.enable(db, **values)
|
||||
|
||||
|
||||
def test_explicit_policy_never_upgrades_legacy_or_sends_or_creates_tasks(env):
|
||||
with transaction(env) as db:
|
||||
assert policy.current(db) is None
|
||||
receipt = enable(db, env)
|
||||
assert policy.current(db) == receipt
|
||||
assert db.execute("SELECT COUNT(*) FROM model_content_permissions").fetchone()[0] == 0
|
||||
assert receipt.backlog_events == ()
|
||||
with pytest.raises(policy.ContinuousPolicyError, match="evidence_outside_policy"):
|
||||
policy.check(db, receipt, evidence_ids=(env.record.evidence_id,))
|
||||
|
||||
|
||||
def test_backlog_snapshot_over_one_hundred_thousand_is_compact(monkeypatch, env) -> None:
|
||||
count = 100_001
|
||||
|
||||
class Database:
|
||||
in_transaction = True
|
||||
|
||||
def execute(self, _sql, _values):
|
||||
return iter(((SOURCE, count, 1),))
|
||||
|
||||
monkeypatch.setattr(policy.ingest_journal, "watermark", lambda _db: count)
|
||||
monkeypatch.setattr(
|
||||
policy.ModelContentPermissionStore,
|
||||
"_source",
|
||||
lambda _db, _user, _source: object(),
|
||||
)
|
||||
|
||||
snapshot = policy.snapshot_backlog(Database()) # type: ignore[arg-type]
|
||||
|
||||
assert snapshot.after_seq == count
|
||||
assert snapshot.source_ids == (SOURCE,)
|
||||
assert snapshot.evidence_count == count
|
||||
assert snapshot.conversation_count == 1
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env, backlog_mode="include", backlog_snapshot=snapshot)
|
||||
assert receipt.backlog_through_seq == count
|
||||
assert receipt.backlog_source_ids == (SOURCE,)
|
||||
assert receipt.backlog_count == count
|
||||
assert receipt.backlog_conversation_count == 1
|
||||
assert receipt.backlog_events == ()
|
||||
assert len(receipt.model_dump_json()) < 5_000
|
||||
|
||||
|
||||
def test_legacy_backlog_receipt_remains_readable(env) -> None:
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env)
|
||||
legacy = receipt.model_dump(
|
||||
exclude={"backlog_through_seq", "backlog_source_ids", "backlog_count"}
|
||||
)
|
||||
legacy["backlog_mode"] = "include"
|
||||
legacy["backlog_events"] = (1,)
|
||||
restored = policy.ContinuousPolicy.model_validate(legacy)
|
||||
db.execute(
|
||||
"UPDATE continuous_policy_history SET receipt_json=? WHERE user_id=?",
|
||||
(restored.model_dump_json(), restored.user_id),
|
||||
)
|
||||
assert policy.check(db, restored, evidence_ids=(env.record.evidence_id,)) == restored
|
||||
|
||||
|
||||
@pytest.mark.parametrize("confirmation", [False, 1, "true", None])
|
||||
def test_explicit_confirmation_is_required(env, confirmation):
|
||||
with transaction(env) as db:
|
||||
with pytest.raises(policy.ContinuousPolicyError):
|
||||
enable(db, env, user_confirmed=confirmation)
|
||||
assert policy.current(db) is None
|
||||
|
||||
|
||||
def test_future_old_dated_import_and_new_allowed_source_are_eligible(env, tmp_path):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env)
|
||||
other_root = tmp_path / "other-source"
|
||||
other_root.mkdir()
|
||||
env.sources.grant(
|
||||
user_id="local",
|
||||
source_id="cursor",
|
||||
root=other_root,
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
with transaction(env) as db:
|
||||
value = evidence_value("after", source_id="cursor")
|
||||
result = env.evidence.put(value, connection=db)
|
||||
event = journal.record(db, result.record.evidence_id)
|
||||
assert event.seq > receipt.after_seq
|
||||
assert policy.check(db, receipt, evidence_ids=(result.record.evidence_id,)) == receipt
|
||||
|
||||
|
||||
def test_include_uses_current_source_authority_not_the_enable_time_snapshot(env, tmp_path):
|
||||
old_ungranted = env.evidence.put(evidence_value("old-cursor", source_id="cursor")).record
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env, backlog_mode="include")
|
||||
assert receipt.backlog_source_ids == (SOURCE,)
|
||||
|
||||
other_root = tmp_path / "later-authorized-source"
|
||||
other_root.mkdir()
|
||||
env.sources.grant(
|
||||
user_id="local",
|
||||
source_id="cursor",
|
||||
root=other_root,
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
with transaction(env) as db:
|
||||
assert policy.check(db, receipt, evidence_ids=(old_ungranted.evidence_id,)) == receipt
|
||||
|
||||
|
||||
def test_include_keeps_legacy_backlog_fields_readable_without_using_them_as_a_boundary(env):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env, backlog_mode="include")
|
||||
assert receipt.backlog_events == ()
|
||||
assert receipt.backlog_through_seq == 1
|
||||
assert receipt.backlog_source_ids == (SOURCE,)
|
||||
policy.check(db, receipt, evidence_ids=(env.record.evidence_id,))
|
||||
db.execute(
|
||||
"UPDATE evidence SET workspace='new identity' WHERE evidence_id=?",
|
||||
(env.record.evidence_id,),
|
||||
)
|
||||
journal.record(db, env.record.evidence_id, cause="maintenance")
|
||||
assert policy.check(db, receipt, evidence_ids=(env.record.evidence_id,)) == receipt
|
||||
|
||||
|
||||
def test_only_new_mode_keeps_the_enable_time_boundary(env):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env, backlog_mode="none")
|
||||
with pytest.raises(policy.ContinuousPolicyError, match="evidence_outside_policy"):
|
||||
policy.check(db, receipt, evidence_ids=(env.record.evidence_id,))
|
||||
|
||||
|
||||
def test_pause_resume_and_revoke_are_cas_and_old_receipts_never_reactivate(env):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env)
|
||||
paused = policy.transition(
|
||||
db, action="pause", operation_id=str(uuid4()), user_confirmed=True, expected_revision=1
|
||||
)
|
||||
with pytest.raises(policy.ContinuousPolicyError):
|
||||
policy.check(db, receipt)
|
||||
with pytest.raises(policy.ContinuousPolicyError):
|
||||
policy.check(db, paused)
|
||||
resumed = policy.transition(
|
||||
db, action="resume", operation_id=str(uuid4()), user_confirmed=True, expected_revision=2
|
||||
)
|
||||
assert resumed.epoch == receipt.epoch and resumed.after_seq == receipt.after_seq
|
||||
revoked = policy.transition(
|
||||
db, action="revoke", operation_id=str(uuid4()), user_confirmed=True, expected_revision=3
|
||||
)
|
||||
with pytest.raises(policy.ContinuousPolicyError):
|
||||
policy.check(db, resumed)
|
||||
with pytest.raises(policy.ContinuousPolicyError, match="requires_new_enable"):
|
||||
policy.transition(
|
||||
db,
|
||||
action="resume",
|
||||
operation_id=str(uuid4()),
|
||||
user_confirmed=True,
|
||||
expected_revision=4,
|
||||
)
|
||||
new = enable(db, env, expected_revision=revoked.revision)
|
||||
assert new.epoch != receipt.epoch
|
||||
|
||||
|
||||
def test_enable_idempotence_rejects_changed_payload_and_superseded_operation(env):
|
||||
with transaction(env) as db:
|
||||
request_id = str(uuid4())
|
||||
receipt = enable(db, env, operation_id=request_id)
|
||||
assert enable(db, env, operation_id=request_id) == receipt
|
||||
assert db.execute("SELECT COUNT(*) FROM continuous_policy_history").fetchone()[0] == 1
|
||||
with pytest.raises(policy.ContinuousPolicyError, match="payload_changed"):
|
||||
enable(db, env, operation_id=request_id, backlog_mode="include")
|
||||
policy.transition(
|
||||
db, action="revoke", operation_id=str(uuid4()), user_confirmed=True, expected_revision=1
|
||||
)
|
||||
with pytest.raises(policy.ContinuousPolicyError, match="superseded"):
|
||||
enable(db, env, operation_id=request_id)
|
||||
|
||||
|
||||
def test_source_revoke_profile_change_and_legacy_changes_block_existing_policy(env):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env, backlog_mode="include")
|
||||
env.sources.revoke(user_id="local", source_id=SOURCE)
|
||||
with transaction(env) as db, pytest.raises(RuntimeError):
|
||||
policy.check(db, receipt, evidence_ids=(env.record.evidence_id,))
|
||||
env.profiles.update(
|
||||
env.profile.profile_id,
|
||||
ModelProfilePatch(expected_revision=1, model="different"),
|
||||
env.secrets,
|
||||
)
|
||||
with transaction(env) as db, pytest.raises(RuntimeError):
|
||||
policy.check(db, receipt)
|
||||
|
||||
|
||||
def test_caller_rollback_includes_policy_and_backlog(env):
|
||||
with pytest.raises(RuntimeError, match="rollback"), transaction(env) as db:
|
||||
enable(db, env)
|
||||
raise RuntimeError("rollback")
|
||||
with transaction(env) as db:
|
||||
assert policy.current(db) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [True, -1, "0", 0.5])
|
||||
def test_revision_values_are_not_coerced(env, value):
|
||||
with (
|
||||
transaction(env) as db,
|
||||
pytest.raises(policy.ContinuousPolicyError, match="invalid_revision"),
|
||||
):
|
||||
enable(db, env, expected_revision=value)
|
||||
|
||||
|
||||
def test_legacy_permission_changes_do_not_invalidate_continuous_policy(env):
|
||||
legacy = env.grant()
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env, expected_legacy_revision=legacy.revision)
|
||||
env.permissions.revoke(user_id="local", expected_revision=legacy.revision, user_confirmed=True)
|
||||
with transaction(env) as db:
|
||||
assert policy.check(db, receipt) == receipt
|
||||
paused = policy.transition(
|
||||
db,
|
||||
expected_revision=receipt.revision,
|
||||
action="pause",
|
||||
operation_id=str(uuid4()),
|
||||
user_confirmed=True,
|
||||
)
|
||||
resumed = policy.transition(
|
||||
db,
|
||||
expected_revision=paused.revision,
|
||||
action="resume",
|
||||
operation_id=str(uuid4()),
|
||||
user_confirmed=True,
|
||||
)
|
||||
assert resumed.state == "enabled"
|
||||
|
||||
|
||||
def test_paused_policy_cannot_resume_after_model_identity_changes(env):
|
||||
with transaction(env) as db:
|
||||
enable(db, env)
|
||||
policy.transition(
|
||||
db, expected_revision=1, action="pause", operation_id=str(uuid4()), user_confirmed=True
|
||||
)
|
||||
env.profiles.update(
|
||||
env.profile.profile_id,
|
||||
ModelProfilePatch(expected_revision=1, model="changed-model"),
|
||||
env.secrets,
|
||||
)
|
||||
with transaction(env) as db:
|
||||
with pytest.raises(RuntimeError):
|
||||
policy.transition(
|
||||
db,
|
||||
expected_revision=2,
|
||||
action="resume",
|
||||
operation_id=str(uuid4()),
|
||||
user_confirmed=True,
|
||||
)
|
||||
assert policy.current(db).state == "paused"
|
||||
|
||||
|
||||
def test_read_check_restores_callers_row_factory(env):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env)
|
||||
db.row_factory = None
|
||||
assert policy.check(db, receipt) == receipt
|
||||
assert db.row_factory is None
|
||||
|
||||
|
||||
def test_prepared_input_cannot_be_replaced_by_a_newer_allowed_revision(env):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env)
|
||||
first = env.evidence.put(evidence_value("new-input"), connection=db).record
|
||||
lease = policy.bind_inputs(db, receipt, (first.evidence_id,))
|
||||
policy.recheck_inputs(db, lease)
|
||||
env.evidence.put(evidence_value("new-input", text="changed material"), connection=db)
|
||||
policy.check(db, receipt, evidence_ids=(first.evidence_id,))
|
||||
with pytest.raises(policy.ContinuousPolicyError, match="bound_input_changed"):
|
||||
policy.recheck_inputs(db, lease)
|
||||
|
||||
|
||||
def test_prepared_source_cannot_survive_regrant_without_revalidation(env):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env, backlog_mode="include")
|
||||
lease = policy.bind_inputs(db, receipt, (env.record.evidence_id,))
|
||||
env.sources.revoke(user_id="local", source_id=SOURCE)
|
||||
env.sources.grant(
|
||||
user_id="local", source_id=SOURCE, root=env.root, candidate_kind=RootCandidateKind.DEFAULT
|
||||
)
|
||||
with (
|
||||
transaction(env) as db,
|
||||
pytest.raises(policy.ContinuousPolicyError, match="bound_source_authority_changed"),
|
||||
):
|
||||
policy.recheck_inputs(db, lease)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("inputs", [(), ("duplicate", "duplicate")])
|
||||
def test_lease_requires_explicit_unique_inputs(env, inputs):
|
||||
with transaction(env) as db:
|
||||
receipt = enable(db, env)
|
||||
with pytest.raises(policy.ContinuousPolicyError, match="unique_inputs"):
|
||||
policy.bind_inputs(db, receipt, inputs)
|
||||
@@ -1,472 +0,0 @@
|
||||
import asyncio
|
||||
import json
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory import continuous_policy as cp
|
||||
from one_memory import ingest_journal
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.imports.sources.registry import RootCandidateKind
|
||||
from one_memory.memory_tasks import MemoryTaskStatus
|
||||
from one_memory.model_processing_permission_guard import ModelProcessingPermissionGuard
|
||||
from tests.test_model_processing_permissions import h as h # noqa: F401
|
||||
|
||||
|
||||
def enable(h, backlog="none"):
|
||||
with h.evidence._connect(writing=True) as db:
|
||||
ingest_journal.initialize(db)
|
||||
cp.initialize(db)
|
||||
legacy = h.permissions.receipt(user_id="local")
|
||||
return cp.enable(
|
||||
db,
|
||||
profile=h.profiles.snapshot(h.profile.profile_id, h.profile.revision),
|
||||
expected_revision=0,
|
||||
expected_legacy_revision=legacy.revision if legacy else 0,
|
||||
operation_id=str(uuid4()),
|
||||
backlog_mode=backlog,
|
||||
user_confirmed=True,
|
||||
)
|
||||
|
||||
|
||||
def transition(h, action):
|
||||
with h.evidence._connect(writing=True) as db:
|
||||
policy = cp.current(db)
|
||||
return cp.transition(
|
||||
db,
|
||||
expected_revision=policy.revision,
|
||||
action=action,
|
||||
operation_id=str(uuid4()),
|
||||
user_confirmed=True,
|
||||
)
|
||||
|
||||
|
||||
def put_after_policy(h, key, *, source="cursor"):
|
||||
"""Create one complete continuous input and record its ingest identity.
|
||||
|
||||
These policy tests used a raw user-only record, which the complete-turn
|
||||
contract correctly holds back. MEMORY_CONTENT is a complete input by
|
||||
itself; recording it after policy enable proves the `after_seq` boundary
|
||||
without inventing an assistant reply that the assertions do not inspect.
|
||||
"""
|
||||
record = h.put(key, source=source, memory_content=True)
|
||||
with h.evidence._connect(writing=True) as db:
|
||||
ingest_journal.record(db, record.evidence_id, cause="import")
|
||||
return h.evidence.get(record.evidence_id)
|
||||
|
||||
|
||||
def test_continuous_prepare_lease_ingest_has_no_legacy_receipt_or_old_context(h):
|
||||
policy = enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
|
||||
async def scenario():
|
||||
try:
|
||||
tasks = supervisor.prepare_conversation(fresh)
|
||||
assert len(tasks) == 1
|
||||
result = await supervisor._create_runner().run(tasks[0].task_id)
|
||||
assert result.status == MemoryTaskStatus.WAITING_FOR_MODEL # Controlled HY failure.
|
||||
assert result.last_error_code == "hy_memory_unavailable"
|
||||
lease = h.leases[0]
|
||||
assert lease.receipt.schema_version == 2
|
||||
assert lease.receipt.authority_kind == "continuous"
|
||||
assert lease.receipt.policy == policy
|
||||
assert {r.evidence_id for r in lease.records} == {fresh.evidence_id}
|
||||
assert h.record.evidence_id not in {r.evidence_id for r in lease.records}
|
||||
assert h.permissions.receipt(user_id="local") is None
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("action", ["pause", "revoke"])
|
||||
def test_continuous_inactive_never_falls_back_to_legacy(h, action):
|
||||
h.grant()
|
||||
enable(h, "include")
|
||||
transition(h, action)
|
||||
supervisor = h.supervisor()
|
||||
with pytest.raises(HyMemoryBridgeError, match="model_content_"):
|
||||
supervisor.prepare_conversation(h.record)
|
||||
assert not h.bridge.calls
|
||||
|
||||
|
||||
@pytest.mark.parametrize("phase", ["before_run", "ingest", "profile"])
|
||||
def test_revocation_rechecked_at_runner_and_publication_boundaries(h, phase):
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
h.bridge.fail = False
|
||||
if phase == "before_run":
|
||||
transition(h, "revoke")
|
||||
elif phase == "ingest":
|
||||
h.bridge.on_ingest = lambda: transition(h, "revoke")
|
||||
else:
|
||||
h.bridge.on_profile = lambda: transition(h, "revoke")
|
||||
|
||||
async def scenario():
|
||||
try:
|
||||
result = await supervisor._create_runner().run(task.task_id)
|
||||
assert result.status == MemoryTaskStatus.CANCELLED
|
||||
assert result.last_error_code.startswith("model_content_")
|
||||
assert h.governance.list(user_id="local") == []
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_durable_binding_restarts_and_cannot_adopt_new_policy_or_new_revision(h):
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
guard = ModelProcessingPermissionGuard(h.permissions, h.governance)
|
||||
saved = guard.bound(task)
|
||||
assert saved.authority_kind == "continuous" and saved.inputs
|
||||
guard.lease(task, (fresh,), None).recheck()
|
||||
with h.evidence._connect(writing=True) as db:
|
||||
db.execute(
|
||||
"UPDATE evidence SET revision=revision+1,verbatim='changed' WHERE evidence_id=?",
|
||||
(fresh.evidence_id,),
|
||||
)
|
||||
ingest_journal.record(db, fresh.evidence_id, cause="import")
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
guard.lease(task, (fresh,), None).recheck()
|
||||
|
||||
|
||||
def test_legacy_task_not_promoted_by_continuous_policy(h):
|
||||
legacy = h.grant()
|
||||
task = h.legacy_task()
|
||||
guard = ModelProcessingPermissionGuard(h.permissions, h.governance)
|
||||
guard.bind(task, legacy)
|
||||
enable(h, "include")
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
guard.bound(task)
|
||||
|
||||
|
||||
def test_future_new_source_old_date_uses_continuous_ingest_boundary(h, tmp_path):
|
||||
enable(h)
|
||||
folder = tmp_path / "second"
|
||||
folder.mkdir()
|
||||
h.sources.grant(
|
||||
user_id="local",
|
||||
source_id="new-source",
|
||||
root=folder,
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
fresh = put_after_policy(h, "new", source="new-source")
|
||||
with h.evidence._connect(writing=True) as db:
|
||||
db.execute(
|
||||
"UPDATE evidence SET created_at='1990-01-01T00:00:00+00:00' WHERE evidence_id=?",
|
||||
(fresh.evidence_id,),
|
||||
)
|
||||
# Actual import material revision, not authorizing an attribution repair.
|
||||
db.execute(
|
||||
"UPDATE evidence SET verbatim='Newly imported old-dated content' WHERE evidence_id=?",
|
||||
(fresh.evidence_id,),
|
||||
)
|
||||
ingest_journal.record(db, fresh.evidence_id, cause="import")
|
||||
fresh = h.evidence.get(fresh.evidence_id)
|
||||
supervisor = h.supervisor()
|
||||
tasks = supervisor.prepare_conversation(fresh)
|
||||
assert len(tasks) == 1
|
||||
assert [work.task_id for work in supervisor._classify(tasks)] == [tasks[0].task_id]
|
||||
assert supervisor.tasks.require(tasks[0].task_id).status is MemoryTaskStatus.PENDING
|
||||
|
||||
|
||||
def test_schema2_roundtrip_rejects_unknown_and_preserves_bound_events(h):
|
||||
from one_memory.continuous_permission_bridge import decode_authority
|
||||
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
receipt = supervisor._permission_guard.bound(task)
|
||||
assert decode_authority(receipt.model_dump_json()) == receipt
|
||||
payload = receipt.model_dump(mode="json")
|
||||
payload["schema_version"] = 99
|
||||
with pytest.raises(ValueError):
|
||||
decode_authority(json.dumps(payload))
|
||||
|
||||
|
||||
def test_transport_rechecks_continuous_lease_without_forged_snapshot(h):
|
||||
from one_memory.hy_permission_transport import HyPermissionTransport, HyPermissionTransportError
|
||||
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
lease = supervisor._create_runner()._prepare_permission_lease(task)[0]
|
||||
transport = HyPermissionTransport(lease)
|
||||
transport._active = True
|
||||
transport.assert_allowed()
|
||||
transition(h, "pause")
|
||||
with pytest.raises(HyPermissionTransportError, match="lease_denied"):
|
||||
transport.assert_allowed()
|
||||
|
||||
|
||||
def test_transport_denial_names_why_the_lease_failed(h):
|
||||
from one_memory.hy_permission_transport import HyPermissionTransport, HyPermissionTransportError
|
||||
|
||||
enable(h)
|
||||
task = h.supervisor().prepare_conversation(put_after_policy(h, "new"))[0]
|
||||
lease = h.supervisor()._create_runner()._prepare_permission_lease(task)[0]
|
||||
transport = HyPermissionTransport(lease)
|
||||
transport._active = True
|
||||
transition(h, "pause")
|
||||
with pytest.raises(Exception) as direct:
|
||||
lease.recheck()
|
||||
reason = str(direct.value).removeprefix("model_content_")
|
||||
assert reason and reason != "lease_denied"
|
||||
|
||||
with pytest.raises(HyPermissionTransportError) as denied:
|
||||
transport.assert_allowed()
|
||||
|
||||
assert denied.value.code == f"model_content_transport_lease_denied_{reason}"
|
||||
|
||||
|
||||
def test_successful_continuous_runner_completes_without_old_backlog(h):
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
h.bridge.fail = False
|
||||
|
||||
async def run():
|
||||
try:
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
result = await supervisor._create_runner().run(task.task_id)
|
||||
assert result.status == MemoryTaskStatus.COMPLETED
|
||||
assert [kind for kind, _ in h.bridge.calls] == [
|
||||
"ingest_extracted",
|
||||
"profile",
|
||||
]
|
||||
request = h.bridge.calls[0][1]
|
||||
assert request.evidence_ids == (fresh.evidence_id,)
|
||||
assert request.source_context is None
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_prepare_backfill_filters_old_inputs_and_paused_policy_refuses(h):
|
||||
policy = enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
tasks = supervisor.prepare_backfill(expected_permission_revision=policy.revision)
|
||||
assert tasks
|
||||
for work in tasks:
|
||||
task = h.tasks.require(work.task_id)
|
||||
assert {ref["evidence_id"] for ref in h.tasks.inputs(task.task_id)} == {fresh.evidence_id}
|
||||
transition(h, "pause")
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
supervisor.prepare_backfill()
|
||||
|
||||
|
||||
def test_bound_source_regrant_refuses_old_task(h):
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
h.sources.revoke(user_id="local", source_id="cursor")
|
||||
h.sources.grant(
|
||||
user_id="local",
|
||||
source_id="cursor",
|
||||
root=h.root / "synthetic-source",
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
with pytest.raises(HyMemoryBridgeError):
|
||||
supervisor._create_runner()._prepare_permission_lease(task)
|
||||
|
||||
|
||||
def test_schema2_native_sdk_child_send_denies_after_policy_revoke(h):
|
||||
import httpx
|
||||
|
||||
from one_memory.hy_permission_transport import HyPermissionTransport, HyPermissionTransportError
|
||||
from tests.test_hy_permission_transport import native, response
|
||||
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
lease = supervisor._create_runner()._prepare_permission_lease(task)[0]
|
||||
provider, _ = native(lease)
|
||||
requests = []
|
||||
|
||||
def transport(request):
|
||||
requests.append(request.url.path)
|
||||
return response(request)
|
||||
|
||||
async def run():
|
||||
with pytest.raises(HyPermissionTransportError):
|
||||
async with HyPermissionTransport(
|
||||
lease, http_transport=httpx.MockTransport(transport)
|
||||
) as boundary:
|
||||
boundary.bind_provider(provider)
|
||||
await provider.complete("Synthetic permitted input")
|
||||
assert len(requests) == 1
|
||||
transition(h, "revoke")
|
||||
await provider.complete("This request must not leave")
|
||||
assert len(requests) == 1
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_recreated_event_identity_produces_new_task_even_same_revision(h):
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
first = supervisor.prepare_conversation(fresh)[0]
|
||||
with h.evidence._connect(writing=True) as db:
|
||||
db.execute(
|
||||
"UPDATE evidence SET verbatim='changed without revision' WHERE evidence_id=?",
|
||||
(fresh.evidence_id,),
|
||||
)
|
||||
ingest_journal.record(db, fresh.evidence_id, cause="import")
|
||||
new = h.evidence.get(fresh.evidence_id)
|
||||
second = supervisor.prepare_conversation(new)[0]
|
||||
assert second.task_id != first.task_id
|
||||
|
||||
|
||||
def test_default_guard_does_not_create_continuous_schema(h):
|
||||
ModelProcessingPermissionGuard(h.permissions, h.governance)
|
||||
with h.evidence._connect() as db:
|
||||
assert not db.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE name='memory_task_continuous_permissions'"
|
||||
).fetchone()
|
||||
assert not db.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE name='continuous_policy_history'"
|
||||
).fetchone()
|
||||
|
||||
|
||||
def test_publish_guard_rechecks_continuous_after_last_bridge_read(h, monkeypatch):
|
||||
enable(h)
|
||||
fresh = put_after_policy(h, "new")
|
||||
supervisor = h.supervisor()
|
||||
h.bridge.fail = False
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
runner = supervisor._create_runner()
|
||||
original = runner._record_outputs
|
||||
|
||||
def revoke_at_commit(**kwargs):
|
||||
transition(h, "revoke")
|
||||
return original(**kwargs)
|
||||
|
||||
monkeypatch.setattr(runner, "_record_outputs", revoke_at_commit)
|
||||
result = asyncio.run(runner.run(task.task_id))
|
||||
assert result.status == MemoryTaskStatus.CANCELLED
|
||||
assert result.last_error_code.startswith("model_content_")
|
||||
assert h.tasks.extraction(task.task_id) is None
|
||||
assert h.governance.list(user_id="local") == []
|
||||
|
||||
|
||||
def test_native_hy_continuous_source_to_governance_with_mock_network(tmp_path):
|
||||
import httpx
|
||||
|
||||
from one_memory.hy_permission_context import permission_bridge_factory
|
||||
from one_memory.imports.models import (
|
||||
EvidenceContentKind,
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
)
|
||||
from tests.test_hy_permission_context import whole_client
|
||||
|
||||
async def run():
|
||||
with whole_client(tmp_path) as (supervisor, old, permissions, legacy):
|
||||
with supervisor.evidence._connect(writing=True) as db:
|
||||
ingest_journal.initialize(db)
|
||||
cp.initialize(db)
|
||||
cp.enable(
|
||||
db,
|
||||
profile=legacy.snapshot.profile,
|
||||
expected_revision=0,
|
||||
expected_legacy_revision=legacy.revision,
|
||||
operation_id=str(uuid4()),
|
||||
backlog_mode="none",
|
||||
user_confirmed=True,
|
||||
)
|
||||
fresh = supervisor.evidence.put(
|
||||
EvidenceInput(
|
||||
source_id=old.source_id,
|
||||
conversation_id=old.conversation_id,
|
||||
message_id="new",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content_kind=EvidenceContentKind.MEMORY_CONTENT,
|
||||
verbatim="Synthetic fresh continuous memory, not old backlog.",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
sends = []
|
||||
|
||||
def handler(request):
|
||||
body = json.loads(request.content)
|
||||
sends.append(body)
|
||||
assert old.verbatim not in request.content.decode()
|
||||
if request.url.path.endswith("embeddings"):
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"object": "list",
|
||||
"model": "test-embedding",
|
||||
"data": [
|
||||
{
|
||||
"index": i,
|
||||
"object": "embedding",
|
||||
"embedding": [1.0] + [0.0] * 383,
|
||||
}
|
||||
for i, _ in enumerate(body["input"])
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "total_tokens": 1},
|
||||
},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "fixture",
|
||||
"object": "chat.completion",
|
||||
"created": 1,
|
||||
"model": "test-generation",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"finish_reason": "stop",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": json.dumps(
|
||||
[
|
||||
{
|
||||
"op": "ADD",
|
||||
"content": fresh.verbatim,
|
||||
"layer": "L2_FACT",
|
||||
"owner": "user",
|
||||
}
|
||||
]
|
||||
),
|
||||
},
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
||||
},
|
||||
)
|
||||
|
||||
supervisor.permission_bridge_factory = permission_bridge_factory(
|
||||
supervisor._bridge, http_transport=httpx.MockTransport(handler)
|
||||
)
|
||||
try:
|
||||
task = supervisor.prepare_conversation(fresh)[0]
|
||||
result = await supervisor._create_runner().run(task.task_id)
|
||||
assert result.status == "completed", result.last_error_code
|
||||
memories = supervisor.governance.list(user_id="local")
|
||||
assert sends and len(memories) == 1
|
||||
assert memories[0].content == fresh.verbatim
|
||||
assert memories[0].evidence_ids == (fresh.evidence_id,)
|
||||
assert permissions.receipt(user_id="local") == legacy
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(run())
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,223 +0,0 @@
|
||||
"""删掉一条记忆,不得让同一对话之后的新 turn 停止沉淀(RC10)。
|
||||
|
||||
沙箱真 HY 实验里实测过:人删掉一条记忆后,同一对话之后的新消息也不再成为记忆。
|
||||
处理一段对话时,每个 chunk 都要过一遍上下文预检;已处理过的旧 turn 的证据
|
||||
正托着被删的那条记忆,于是预检在旧 chunk 上抛 evidence_changed,整段对话中止,
|
||||
新 turn 连同它一起被丢下。live 路径还把这个错误吞成一行日志,实时同步就此静默停住。
|
||||
|
||||
删除删的是那条记忆实体,不是"永远不许再说这件事":
|
||||
新 turn 是新的陈述,应当照常沉淀;同一段旧原话(按证据 id)不得把已删记忆抽回。
|
||||
|
||||
三条入口各守一条:即时 prepare、live 到达、backfill 回放。
|
||||
全部走真实的临时 Core、真实的连续处理授权与守卫,只把 HY 换成可控的替身。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.imports.models import WrittenBy
|
||||
from one_memory.imports.runner import live_sync_arrival
|
||||
from one_memory.memory_governance import MemoryStatus
|
||||
from one_memory.memory_tasks import MemoryTaskStatus
|
||||
from tests.test_continuous_processing_integration import IntegrationCore
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def core(tmp_path, monkeypatch):
|
||||
return IntegrationCore(tmp_path, monkeypatch)
|
||||
|
||||
|
||||
def _memory_for(core, record):
|
||||
return core.app.state.memory_governance_store.get(
|
||||
"integration-hy-" + record.evidence_id, user_id="local"
|
||||
)
|
||||
|
||||
|
||||
async def _first_turn_then_delete(core):
|
||||
"""第一个 turn 沉淀成记忆,然后人把它删掉。返回那一轮的原话与连续授权。"""
|
||||
core.grant_source()
|
||||
await core.save_and_probe()
|
||||
receipt = core.kernel_enable()
|
||||
first = core.put_complete("first-turn")
|
||||
results = await core.run_prepared(first)
|
||||
assert [t.status for t in results] == [MemoryTaskStatus.COMPLETED]
|
||||
memory = _memory_for(core, first)
|
||||
# 正向前提:第一轮确实沉淀出了一条可删的记忆。
|
||||
assert memory is not None and memory.status is MemoryStatus.CONFIRMED
|
||||
forgotten = []
|
||||
|
||||
async def forget_in_hy(memory_id):
|
||||
# HY 那头的遗忘另有守护;这里只要治理侧的删除照常落地。
|
||||
forgotten.append(memory_id)
|
||||
return {"deleted": True}
|
||||
|
||||
core.app.state.memory_processing_supervisor.delete_hy_memory = forget_in_hy
|
||||
deleted = await core.client.post(
|
||||
f"/api/v1/memories/{memory.memory_id}/actions/delete",
|
||||
json={"request_id": "delete-first", "expected_revision": memory.revision},
|
||||
)
|
||||
assert deleted.status_code == 200, deleted.text
|
||||
assert _memory_for(core, first).status is MemoryStatus.DELETED
|
||||
assert forgotten == [memory.memory_id]
|
||||
return first, receipt
|
||||
|
||||
|
||||
def test_prepare_after_a_deletion_still_settles_the_new_turn(core):
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
first, _receipt = await _first_turn_then_delete(core)
|
||||
second = core.put_complete("second-turn")
|
||||
supervisor = core.app.state.memory_processing_supervisor
|
||||
|
||||
prepared = await asyncio.to_thread(supervisor.prepare_conversation, second)
|
||||
# 已完成的旧 turn 在合并前剔除、不再预检重跑;只有新 turn 是待做的。
|
||||
by_input = {t.input_ref: t for t in prepared}
|
||||
assert set(by_input) == {second.evidence_id}
|
||||
pending = [t for t in prepared if t.status is not MemoryTaskStatus.COMPLETED]
|
||||
assert [t.input_ref for t in pending] == [second.evidence_id]
|
||||
for task in pending:
|
||||
await supervisor._create_runner().run(task.task_id)
|
||||
|
||||
memory = _memory_for(core, second)
|
||||
assert memory is not None and memory.status is MemoryStatus.CONFIRMED
|
||||
assert _memory_for(core, first).status is MemoryStatus.DELETED
|
||||
# 新 turn 照常处理,但被删的记忆和它托着的原话都不进这一轮的上下文。
|
||||
sent = next(r for r in core.bridge.requests if second.evidence_id in r.evidence_ids)
|
||||
context = sent.source_context
|
||||
assert context is not None
|
||||
assert first.evidence_id not in {
|
||||
r.evidence_id for r in (*context.history, *context.supporting)
|
||||
}
|
||||
deleted_id = "integration-hy-" + first.evidence_id
|
||||
assert deleted_id not in {m.governed_memory_id for m in context.known_memories}
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
async def _live_settled(supervisor, tasks):
|
||||
"""等防抖、准备与派发都走完,而不是等固定秒数:慢机器上也不会早判。"""
|
||||
loop = asyncio.get_running_loop()
|
||||
end = loop.time() + 30
|
||||
while loop.time() < end:
|
||||
idle = not supervisor._debounces and not supervisor._preparations
|
||||
busy = any(
|
||||
t.status in {MemoryTaskStatus.PENDING, MemoryTaskStatus.RUNNING} for t in tasks.list()
|
||||
)
|
||||
if idle and not busy:
|
||||
return
|
||||
await asyncio.sleep(0.05)
|
||||
pytest.fail("live 到达的准备与派发没有在界限内走完")
|
||||
|
||||
|
||||
def test_live_arrival_after_a_deletion_still_settles_the_new_turn(core):
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
first, _receipt = await _first_turn_then_delete(core)
|
||||
second = core.put("second-turn")
|
||||
reply = core.put(
|
||||
"second-turn-reply",
|
||||
conversation="same-conversation",
|
||||
role="assistant",
|
||||
written_by="agent",
|
||||
)
|
||||
supervisor = core.app.state.memory_processing_supervisor
|
||||
assert _memory_for(core, second) is None
|
||||
|
||||
token = live_sync_arrival.set(True)
|
||||
try:
|
||||
supervisor.enqueue_evidence(second)
|
||||
supervisor.enqueue_evidence(reply)
|
||||
finally:
|
||||
live_sync_arrival.reset(token)
|
||||
await _live_settled(supervisor, core.app.state.memory_task_store)
|
||||
|
||||
memory = _memory_for(core, second)
|
||||
assert memory is not None and memory.status is MemoryStatus.CONFIRMED
|
||||
assert _memory_for(core, first).status is MemoryStatus.DELETED
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("deleted", [True, False], ids=["deleted", "control-kept"])
|
||||
def test_reprocessing_the_same_old_evidence_does_not_bring_a_deleted_memory_back(core, deleted):
|
||||
"""按证据 id 挡住:同一段旧原话再交给 HY 一次,也不得重新长出被删的那条记忆。
|
||||
|
||||
对照组不删:同一条重跑路径确实会把同一段原话再抽成一条新记忆——
|
||||
证明挡住它的是删除,而不是这条路径本来就到不了写入。
|
||||
"""
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
if deleted:
|
||||
first, _receipt = await _first_turn_then_delete(core)
|
||||
else:
|
||||
core.grant_source()
|
||||
await core.save_and_probe()
|
||||
core.kernel_enable()
|
||||
first = core.put_complete("first-turn")
|
||||
await core.run_prepared(first)
|
||||
original = core.bridge.ingest
|
||||
|
||||
async def reextract(request):
|
||||
# 真 HY 重抽同一段原话时给的是新 id,内容相同。
|
||||
result = await original(request)
|
||||
return result.model_copy(
|
||||
update={
|
||||
"memories": tuple(
|
||||
m.model_copy(update={"memory_id": "reextracted-" + m.memory_id})
|
||||
for m in result.memories
|
||||
)
|
||||
}
|
||||
)
|
||||
|
||||
core.bridge.ingest = reextract
|
||||
tasks = core.app.state.memory_task_store
|
||||
done = next(t for t in tasks.list() if t.input_ref == first.evidence_id)
|
||||
tasks.requeue_completed(done.task_id, reason="reprocess_same_evidence")
|
||||
|
||||
finished = await core.app.state.memory_processing_supervisor._create_runner().run(
|
||||
done.task_id
|
||||
)
|
||||
|
||||
back = core.app.state.memory_governance_store.get(
|
||||
"reextracted-integration-hy-" + first.evidence_id, user_id="local"
|
||||
)
|
||||
if deleted:
|
||||
assert back is None or back.status is not MemoryStatus.CONFIRMED
|
||||
assert finished.status is not MemoryTaskStatus.COMPLETED
|
||||
else:
|
||||
assert finished.status is MemoryTaskStatus.COMPLETED, finished.last_error_code
|
||||
assert back is not None and back.status is MemoryStatus.CONFIRMED
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_backfill_after_a_deletion_still_settles_the_new_turn(core):
|
||||
"""backfill 修复前就不冻结:已完成的旧 chunk 在入队时跳过,出错的 chunk 也只丢它自己。
|
||||
|
||||
只拿掉其中一道它仍是绿的;两道都拿掉时才红(实测)。
|
||||
它守的是 backfill 这一路的结果,不单独守"已完成跳过"——那一道由 prepare 的守卫看住。
|
||||
"""
|
||||
|
||||
async def scenario():
|
||||
async with core.running():
|
||||
first, receipt = await _first_turn_then_delete(core)
|
||||
second = core.put_complete("second-turn")
|
||||
core.put(
|
||||
"second-turn-answer",
|
||||
conversation=second.conversation_id,
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
)
|
||||
supervisor = core.app.state.memory_processing_supervisor
|
||||
assert _memory_for(core, second) is None
|
||||
|
||||
await supervisor.backfill_off_thread(expected_permission_revision=receipt.revision)
|
||||
await core.wait_completed(minimum_tasks=2)
|
||||
|
||||
memory = _memory_for(core, second)
|
||||
assert memory is not None and memory.status is MemoryStatus.CONFIRMED
|
||||
assert _memory_for(core, first).status is MemoryStatus.DELETED
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,10 +1,8 @@
|
||||
import json
|
||||
import logging
|
||||
from importlib import import_module
|
||||
from inspect import getsource
|
||||
from pathlib import Path
|
||||
|
||||
from one_memory.hy_permission_transport import HyPermissionTransport
|
||||
from one_memory.hy_runtime_config import (
|
||||
configure_hy_runtime_environment,
|
||||
detailed_logging_enabled,
|
||||
@@ -66,13 +64,6 @@ def test_info_collects_hy_info_without_enabling_debug_pipeline_content(
|
||||
assert __import__("os").environ["MEMORY_LOG_LLM_CONTENT"] == "false"
|
||||
|
||||
|
||||
def test_transport_does_not_reject_debug_wire_logging() -> None:
|
||||
source = getsource(HyPermissionTransport)
|
||||
|
||||
assert "_check_wire_logging" not in source
|
||||
assert "unsafe_wire_logging" not in source
|
||||
|
||||
|
||||
def test_live_debug_switch_makes_sdk_pipeline_writer_keep_original_content(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
|
||||
@@ -165,6 +165,13 @@ def test_import_request_trace_reaches_the_background_job_without_original_text(t
|
||||
assert "import.queued" in names
|
||||
assert "import.started" in names
|
||||
assert "import.finished" in names
|
||||
finished = next(event for event in events if event["event"] == "import.finished")
|
||||
assert {
|
||||
"stored_count",
|
||||
"revised_count",
|
||||
"skipped_count",
|
||||
"error_count",
|
||||
} <= finished["details"].keys()
|
||||
assert secret not in written
|
||||
|
||||
|
||||
|
||||
@@ -1,142 +0,0 @@
|
||||
"""Read invalidation review: real guard/provenance, controlled HY, no model/network."""
|
||||
|
||||
import asyncio
|
||||
import sqlite3
|
||||
from contextlib import closing
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory import context_routes
|
||||
from one_memory.hy_memory_bridge import HyMemoryOutput
|
||||
from one_memory.imports.models import WrittenBy
|
||||
from one_memory.memory_governance import MemoryStatus
|
||||
from one_memory.memory_tasks import evidence_ref
|
||||
from tests.test_continuous_processing_integration import IntegrationCore
|
||||
from tests.test_governance_authority_review import WORKSPACE, prepare, process, put, search
|
||||
|
||||
OLD = "revisionneedle the project deploys only on Windows"
|
||||
NEW = "revisionneedle the project deploys only on macOS"
|
||||
|
||||
|
||||
async def derived(core):
|
||||
await prepare(core)
|
||||
original = put(core, "edited-turn", OLD, WrittenBy.PERSON)
|
||||
put(core, "edited-turn-reply", "Acknowledged.", WrittenBy.AGENT)
|
||||
outputs = await process(
|
||||
core,
|
||||
original,
|
||||
[
|
||||
HyMemoryOutput(
|
||||
memory_id="review-old-claim",
|
||||
content=OLD,
|
||||
layer="L2_FACT",
|
||||
owner="user",
|
||||
evidence_ids=(original.evidence_id,),
|
||||
)
|
||||
],
|
||||
)
|
||||
output = outputs[0]
|
||||
assert output.support_refs == (evidence_ref(original),)
|
||||
assert output.memory_revision == 1
|
||||
before = await search(core, "revisionneedle", [WORKSPACE])
|
||||
assert before.status_code == 200 and len(before.json()["items"]) == 1
|
||||
assert before.json()["items"][0]["content"] == OLD
|
||||
return original, output
|
||||
|
||||
|
||||
@pytest.mark.parametrize("during_read", [False, True], ids=["before-query", "before-publish"])
|
||||
def test_revised_evidence_does_not_supply_old_derived_claim(tmp_path, monkeypatch, during_read):
|
||||
async def scenario():
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
async with core.running():
|
||||
original, output = await derived(core)
|
||||
revised = None
|
||||
|
||||
def revise():
|
||||
nonlocal revised
|
||||
revised = put(core, "edited-turn", NEW, WrittenBy.PERSON)
|
||||
assert revised.evidence_id == original.evidence_id
|
||||
assert revised.revision == original.revision + 1
|
||||
assert revised.content_hash != original.content_hash
|
||||
|
||||
if during_read:
|
||||
read = context_routes.read_memory_context
|
||||
|
||||
def revision_after_selection(*args, **kwargs):
|
||||
result = read(*args, **kwargs)
|
||||
assert len(result.items) == 1
|
||||
revise()
|
||||
return result
|
||||
|
||||
monkeypatch.setattr(context_routes, "read_memory_context", revision_after_selection)
|
||||
else:
|
||||
revise()
|
||||
response = await search(core, "revisionneedle", [WORKSPACE])
|
||||
assert response.status_code == 200
|
||||
assert revised is not None
|
||||
stored = core.app.state.memory_task_store.outputs(output.task_id)[0]
|
||||
assert stored.support_refs == output.support_refs
|
||||
memory = core.app.state.memory_governance_store.get(output.memory_id, user_id="local")
|
||||
assert memory.status is MemoryStatus.CONFIRMED and memory.content == OLD
|
||||
assert len(core.bridge.requests) == 1
|
||||
with closing(sqlite3.connect(core.database)) as db:
|
||||
versions = db.execute(
|
||||
"SELECT revision FROM evidence_versions WHERE evidence_id=? ORDER BY revision",
|
||||
(original.evidence_id,),
|
||||
).fetchall()
|
||||
events = db.execute(
|
||||
"SELECT evidence_revision FROM evidence_ingest_events "
|
||||
"WHERE evidence_id=? ORDER BY seq",
|
||||
(original.evidence_id,),
|
||||
).fetchall()
|
||||
assert versions == [(1,), (2,)] and events == [(1,), (2,)]
|
||||
print("revision review:", during_read, output.support_refs, response.json())
|
||||
assert response.json()["items"] == []
|
||||
library = await core.client.get("/api/v1/memory-library")
|
||||
# D22: the total skips the per-row freshness scan (product accepted a
|
||||
# small drift), but the stale claim itself must never be listed.
|
||||
assert library.status_code == 200 and library.json()["items"] == []
|
||||
home = await core.client.get("/api/v1/home-summary")
|
||||
# Same D22 drift for the homepage count; the latest card must not be it.
|
||||
assert home.status_code == 200 and home.json()["latest"] is None
|
||||
assert (await core.client.get("/api/v1/memories")).json() == []
|
||||
detail = await core.client.get("/api/v1/memories/" + output.memory_id)
|
||||
assert detail.status_code == 403
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_explicit_correction_is_distinguishable_from_unchanged_sdk_projection(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
async def scenario():
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
async with core.running():
|
||||
original, output = await derived(core)
|
||||
corrected = core.app.state.person_writes.correct(
|
||||
output.memory_id,
|
||||
user_id="local",
|
||||
request_id="explicit-user-correction",
|
||||
expected_revision=output.memory_revision,
|
||||
content=NEW,
|
||||
)
|
||||
assert corrected.revision > output.memory_revision
|
||||
assert corrected.origin.value == "extracted"
|
||||
assert corrected.evidence_ids == (original.evidence_id,)
|
||||
with closing(sqlite3.connect(core.database)) as db:
|
||||
operation = db.execute(
|
||||
"SELECT operation FROM governed_memory_mutations WHERE request_id=?",
|
||||
("explicit-user-correction",),
|
||||
).fetchone()
|
||||
version = db.execute(
|
||||
"SELECT action,content FROM governed_memory_versions "
|
||||
"WHERE memory_id=? AND revision=?",
|
||||
(corrected.memory_id, corrected.revision),
|
||||
).fetchone()
|
||||
assert operation == ("correct",) and version == ("correct", NEW)
|
||||
stored = core.app.state.memory_task_store.outputs(output.task_id)[0]
|
||||
assert stored.memory_revision == 1 and stored.support_refs == output.support_refs
|
||||
# Inspect metadata only: no new policy that old refs authorize a manual correction.
|
||||
print("correction distinction:", corrected.revision, stored.memory_revision, operation)
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -79,9 +79,3 @@ def test_default_distribution_avoids_per_launch_self_extraction(monkeypatch):
|
||||
calls, _, _ = evaluate_spec(monkeypatch, None)
|
||||
assert calls["exe"][1]["exclude_binaries"] is True
|
||||
assert calls["collect"][1]["name"] == "one-memory-runtime"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("layout", ["folder", "single-file"])
|
||||
def test_runtime_permission_factory_is_explicitly_bundled(monkeypatch, layout):
|
||||
calls, _, _ = evaluate_spec(monkeypatch, layout)
|
||||
assert "one_memory.hy_permission_context" in calls["analysis"][1]["hiddenimports"]
|
||||
@@ -1,201 +0,0 @@
|
||||
"""D04 authority and workspace scope through the real continuous permission guard.
|
||||
|
||||
Only HY/model transport is controlled. These assert the documented boundary,
|
||||
not that a confirmed machine projection constitutes human factual acceptance.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import HyMemoryIngestResult, HyMemoryOutput
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.memory_governance import MemoryDraft, MemoryKind, MemoryOrigin, MemoryStatus
|
||||
from one_memory.memory_tasks import MemoryTaskStatus
|
||||
from tests.test_continuous_processing_integration import IntegrationCore
|
||||
|
||||
WORKSPACE = "workspace:v1:" + "a" * 64
|
||||
OTHER_WORKSPACE = "workspace:v1:" + "b" * 64
|
||||
AGENT_TEXT = "authorityneedle assistant proposes project delivery without user acceptance"
|
||||
PROJECT_TEXT = "projectneedle this project requires a staging approval before deployment"
|
||||
|
||||
|
||||
def envelope(scopes, tools=("om_recall",)):
|
||||
# Simulated authenticated Broker envelope, not a real host installation/grant.
|
||||
return {
|
||||
"x-one-memory-read-context": "external",
|
||||
"x-one-memory-installation-id": "00000000-0000-0000-0000-000000000001",
|
||||
"x-one-memory-installation-revision": "1",
|
||||
"x-one-memory-allowed-scopes": json.dumps(scopes),
|
||||
"x-one-memory-allowed-tools": json.dumps(tools),
|
||||
}
|
||||
|
||||
|
||||
async def search(core, query, scopes, tools=("om_recall",)):
|
||||
return await core.client.post(
|
||||
"/api/v1/context/search",
|
||||
json={"query": query, "kinds": ["memory"]},
|
||||
headers=envelope(scopes, tools),
|
||||
)
|
||||
|
||||
|
||||
def put(core, key, text, author):
|
||||
return core.app.state.evidence_store.put(
|
||||
EvidenceInput(
|
||||
user_id="local",
|
||||
source_id="cursor",
|
||||
conversation_id="authority-review",
|
||||
message_id=key,
|
||||
workspace=WORKSPACE,
|
||||
role="assistant" if author is WrittenBy.AGENT else "user",
|
||||
written_by=author,
|
||||
created_at=datetime.now(UTC),
|
||||
verbatim=text,
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
|
||||
|
||||
async def prepare(core):
|
||||
core.grant_source()
|
||||
core.app.state.config_store.set_sync("cursor", True)
|
||||
await core.save_and_probe()
|
||||
await core.enable_api()
|
||||
runner = core.app.state.memory_processing_supervisor._create_runner()
|
||||
assert runner._permission_guard is not None
|
||||
|
||||
|
||||
async def process(core, record, outputs):
|
||||
async def ingest(request):
|
||||
core.bridge.requests.append(request)
|
||||
assert record.evidence_id in request.evidence_ids
|
||||
return HyMemoryIngestResult(success=True, memories=tuple(outputs))
|
||||
|
||||
core.bridge.ingest = ingest
|
||||
results = await core.run_prepared(record)
|
||||
assert results and all(task.status == MemoryTaskStatus.COMPLETED for task in results)
|
||||
assert core.bridge.requests and core.bridge.leases
|
||||
provenance = [
|
||||
output
|
||||
for task in results
|
||||
for output in core.app.state.memory_task_store.outputs(task.task_id)
|
||||
]
|
||||
assert len(provenance) == len(outputs)
|
||||
return provenance
|
||||
|
||||
|
||||
@pytest.mark.parametrize("existing", [False, True])
|
||||
def test_agent_output_lands_usable_without_demoting_an_existing_memory(
|
||||
tmp_path, monkeypatch, existing
|
||||
):
|
||||
async def scenario():
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
async with core.running():
|
||||
await prepare(core)
|
||||
person = put(core, "person", "Please review the project proposal", WrittenBy.PERSON)
|
||||
agent = put(core, "agent", AGENT_TEXT, WrittenBy.AGENT)
|
||||
governance = core.app.state.memory_governance_store
|
||||
if existing:
|
||||
previous = core.app.state.person_writes.create(
|
||||
MemoryDraft(
|
||||
user_id="local",
|
||||
kind=MemoryKind.CHAT,
|
||||
content=AGENT_TEXT,
|
||||
scopes=(WORKSPACE,),
|
||||
evidence_ids=(agent.evidence_id,),
|
||||
origin=MemoryOrigin.AGENT,
|
||||
),
|
||||
request_id="fixture-prior",
|
||||
memory_id="prior-agent-memory",
|
||||
)
|
||||
await process(
|
||||
core,
|
||||
person,
|
||||
[
|
||||
HyMemoryOutput(
|
||||
memory_id="hy-agent-proposal",
|
||||
content=AGENT_TEXT,
|
||||
layer="L2_FACT",
|
||||
owner="agent",
|
||||
evidence_ids=(agent.evidence_id,),
|
||||
)
|
||||
],
|
||||
)
|
||||
# HY's id is mirrored as its own row; the ledger does not fold it
|
||||
# into an older row that happens to share the words.
|
||||
memory = governance.get("hy-agent-proposal", user_id="local")
|
||||
assert memory is not None and memory.origin is MemoryOrigin.AGENT
|
||||
assert memory.evidence_ids == (agent.evidence_id,)
|
||||
assert memory.status is MemoryStatus.CONFIRMED
|
||||
prior_visible = 0
|
||||
if existing:
|
||||
prior = governance.get("prior-agent-memory", user_id="local")
|
||||
assert prior.status is MemoryStatus.CONFIRMED
|
||||
assert prior.revision == previous.revision
|
||||
prior_visible = 1
|
||||
matching = await search(core, "authorityneedle", [WORKSPACE])
|
||||
assert matching.status_code == 200, matching.text
|
||||
assert len(matching.json()["items"]) == 1 + prior_visible
|
||||
assert memory.scopes == (WORKSPACE,) or WORKSPACE in memory.scopes
|
||||
denied = await search(core, "authorityneedle", [WORKSPACE], ("om_conversations",))
|
||||
assert denied.status_code == 403
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_project_fact_is_shared_across_apps_and_leaves_with_its_grant(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
core = IntegrationCore(tmp_path, monkeypatch)
|
||||
async with core.running():
|
||||
await prepare(core)
|
||||
record = put(core, "project", PROJECT_TEXT, WrittenBy.PERSON)
|
||||
put(core, "project-reply", "Acknowledged.", WrittenBy.AGENT)
|
||||
await process(
|
||||
core,
|
||||
record,
|
||||
[
|
||||
HyMemoryOutput(
|
||||
memory_id="hy-project-rule",
|
||||
content=PROJECT_TEXT,
|
||||
layer="L2_FACT",
|
||||
owner="user",
|
||||
evidence_ids=(record.evidence_id,),
|
||||
),
|
||||
*[
|
||||
HyMemoryOutput(
|
||||
memory_id="hy-profile-" + layer,
|
||||
content="profileneedle " + layer + " user prefers concise explanations",
|
||||
layer=layer,
|
||||
owner="user",
|
||||
evidence_ids=(record.evidence_id,),
|
||||
)
|
||||
for layer in ("L0_BASIC_INFO", "L6_SCHEMA")
|
||||
],
|
||||
],
|
||||
)
|
||||
memory = core.app.state.memory_governance_store.get("hy-project-rule", user_id="local")
|
||||
assert memory is not None and memory.status is MemoryStatus.CONFIRMED
|
||||
assert memory.origin is MemoryOrigin.EXTRACTED
|
||||
workspace = await search(core, "projectneedle", [WORKSPACE])
|
||||
assert workspace.status_code == 200 and len(workspace.json()["items"]) == 1
|
||||
unrelated = await search(core, "projectneedle", [OTHER_WORKSPACE])
|
||||
assert unrelated.status_code == 200 and unrelated.json()["items"] == []
|
||||
global_read = await search(core, "projectneedle", ["all-apps", OTHER_WORKSPACE])
|
||||
assert global_read.status_code == 200 and len(global_read.json()["items"]) == 1
|
||||
assert memory.scopes == ("all-apps", WORKSPACE)
|
||||
profiles = await search(core, "profileneedle", ["all-apps"])
|
||||
assert profiles.status_code == 200 and len(profiles.json()["items"]) == 2
|
||||
denied = await search(core, "projectneedle", [WORKSPACE], ("om_conversations",))
|
||||
assert denied.status_code == 403
|
||||
core.app.state.source_grant_store.revoke(user_id="local", source_id="cursor")
|
||||
revoked = await search(core, "projectneedle", [WORKSPACE])
|
||||
assert revoked.status_code == 200 and revoked.json()["items"] == []
|
||||
revoked_global = await search(core, "projectneedle", ["all-apps"])
|
||||
assert revoked_global.status_code == 200 and revoked_global.json()["items"] == []
|
||||
revoked_profile = await search(core, "profileneedle", ["all-apps"])
|
||||
assert revoked_profile.status_code == 200 and revoked_profile.json()["items"] == []
|
||||
assert len(core.bridge.requests) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -3,13 +3,10 @@ import logging
|
||||
import sqlite3
|
||||
import threading
|
||||
import traceback
|
||||
from collections.abc import Callable
|
||||
from contextlib import suppress
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient, Response
|
||||
from one_memory.model_content_permissions import ModelContentPermissionStore
|
||||
from pydantic import SecretStr
|
||||
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
@@ -18,10 +15,10 @@ from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetad
|
||||
from one_memory.imports.sources.registry import RootCandidateKind
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import MemoryGovernanceStore
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_server.durable_queue import DurableWriteQueue
|
||||
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
|
||||
from one_memory.model_profiles import ModelCapabilities, ModelProfileInput, ModelProfileStore
|
||||
from one_memory.secrets import InMemorySecretStore, SecretStoreUnavailableError
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
|
||||
|
||||
async def request(
|
||||
@@ -86,45 +83,6 @@ def test_core_refuses_to_start_a_database_written_by_a_newer_build(tmp_path, mon
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_core_stays_ready_when_legacy_policy_upgrade_cannot_read_keychain(
|
||||
tmp_path, monkeypatch
|
||||
) -> None:
|
||||
monkeypatch.setattr("one_memory.model_presets._preset_data", lambda: None)
|
||||
monkeypatch.setattr(
|
||||
"one_memory.model_policy_activation.ModelPolicyActivation.upgrade_legacy_none_policy",
|
||||
lambda _self: (_ for _ in ()).throw(
|
||||
SecretStoreUnavailableError("Operating system secure storage read backoff")
|
||||
),
|
||||
)
|
||||
settings = Settings(
|
||||
_env_file=None,
|
||||
data_dir=tmp_path / "data",
|
||||
source_home=tmp_path / "source",
|
||||
api_token=SecretStr("test"),
|
||||
imports_enabled=False,
|
||||
automatic_memory_processing=False,
|
||||
live_sync_enabled=False,
|
||||
)
|
||||
app = create_app(settings, secret_store=InMemorySecretStore())
|
||||
|
||||
async def scenario() -> None:
|
||||
async with (
|
||||
app.router.lifespan_context(app),
|
||||
AsyncClient(
|
||||
transport=ASGITransport(app=app),
|
||||
base_url="http://test",
|
||||
headers={"Authorization": "Bearer test"},
|
||||
) as client,
|
||||
):
|
||||
health = await client.get("/api/v1/health")
|
||||
recall = await client.post("/api/v1/recall", json={"query": "still ready"})
|
||||
assert health.status_code == 200
|
||||
assert health.json()["status"] == "ok"
|
||||
assert recall.status_code == 200
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_lifespan_starts_the_content_candidate_backfill_and_exposes_running(
|
||||
tmp_path, monkeypatch
|
||||
) -> None:
|
||||
@@ -211,7 +169,7 @@ def test_lifespan_drains_committed_pending_dispatch_through_enqueue(tmp_path, mo
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
evidence.put(
|
||||
_ = evidence.put(
|
||||
EvidenceInput(
|
||||
source_id="fixture",
|
||||
conversation_id="conversation",
|
||||
@@ -360,7 +318,7 @@ def test_diagnostic_log_level_allows_desktop_put_preflight() -> None:
|
||||
assert "PUT" in response.headers["access-control-allow-methods"]
|
||||
|
||||
|
||||
def core(tmp_path, *, permission=True):
|
||||
def core(tmp_path):
|
||||
database = tmp_path / "data/one-memory.sqlite3"
|
||||
root = tmp_path / "source"
|
||||
root.mkdir()
|
||||
@@ -369,7 +327,7 @@ def core(tmp_path, *, permission=True):
|
||||
user_id="local", source_id="fixture", root=root, candidate_kind=RootCandidateKind.DEFAULT
|
||||
)
|
||||
evidence = EvidenceStore(database)
|
||||
record = evidence.put(
|
||||
_record = evidence.put(
|
||||
EvidenceInput(
|
||||
source_id="fixture",
|
||||
conversation_id="health",
|
||||
@@ -399,15 +357,6 @@ def core(tmp_path, *, permission=True):
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
store = ModelContentPermissionStore(database)
|
||||
if permission:
|
||||
snapshot = store.snapshot(
|
||||
user_id="local",
|
||||
source_ids=("fixture",),
|
||||
profile=profiles.snapshot(profile.profile_id, profile.revision),
|
||||
evidence=(record,),
|
||||
)
|
||||
store.grant(snapshot, user_confirmed=True, expected_revision=0)
|
||||
return create_app(
|
||||
Settings(
|
||||
data_dir=tmp_path / "data",
|
||||
@@ -420,25 +369,6 @@ def core(tmp_path, *, permission=True):
|
||||
)
|
||||
|
||||
|
||||
def test_configured_model_without_content_consent_does_not_start_backfill(tmp_path, monkeypatch):
|
||||
started = []
|
||||
|
||||
def prepare(*args, **kwargs):
|
||||
started.append(True)
|
||||
yield []
|
||||
|
||||
monkeypatch.setattr(MemoryProcessingSupervisor, "_prepare_backfill_batches", prepare)
|
||||
|
||||
async def scenario():
|
||||
app = core(tmp_path, permission=False)
|
||||
async with app.router.lifespan_context(app):
|
||||
await app.state.memory_backfill
|
||||
assert started == []
|
||||
assert app.state.memory_task_store.list() == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_startup_finishes_before_the_memory_backfill_does(tmp_path, monkeypatch) -> None:
|
||||
"""Startup must not wait on work that grows with the corpus.
|
||||
|
||||
@@ -451,16 +381,16 @@ def test_startup_finishes_before_the_memory_backfill_does(tmp_path, monkeypatch)
|
||||
release = threading.Event()
|
||||
|
||||
def blocking(
|
||||
_: MemoryProcessingSupervisor,
|
||||
should_stop: Callable[[], bool] = lambda: False,
|
||||
_: DurableWriteQueue,
|
||||
*,
|
||||
expected_permission_revision: int | None = None,
|
||||
limit: int = 512,
|
||||
after_cursor: str | None = None,
|
||||
):
|
||||
walking.set()
|
||||
assert release.wait(10)
|
||||
yield []
|
||||
return ()
|
||||
|
||||
monkeypatch.setattr(MemoryProcessingSupervisor, "_prepare_backfill_batches", blocking)
|
||||
monkeypatch.setattr(DurableWriteQueue, "unsubmitted_evidence_ids", blocking)
|
||||
|
||||
async def scenario() -> None:
|
||||
app = core(tmp_path)
|
||||
@@ -489,15 +419,14 @@ def test_startup_finishes_before_the_memory_backfill_does(tmp_path, monkeypatch)
|
||||
|
||||
def test_a_backfill_that_fails_says_so_rather_than_disappearing(tmp_path, monkeypatch) -> None:
|
||||
def broken(
|
||||
_: MemoryProcessingSupervisor,
|
||||
should_stop: Callable[[], bool] = lambda: False,
|
||||
_: DurableWriteQueue,
|
||||
*,
|
||||
expected_permission_revision: int | None = None,
|
||||
limit: int = 512,
|
||||
after_cursor: str | None = None,
|
||||
):
|
||||
raise RuntimeError("the catch-up broke")
|
||||
yield []
|
||||
|
||||
monkeypatch.setattr(MemoryProcessingSupervisor, "_prepare_backfill_batches", broken)
|
||||
monkeypatch.setattr(DurableWriteQueue, "unsubmitted_evidence_ids", broken)
|
||||
|
||||
async def scenario() -> None:
|
||||
app = core(tmp_path)
|
||||
@@ -521,16 +450,16 @@ def test_shutting_down_mid_backfill_leaves_nothing_running(tmp_path, monkeypatch
|
||||
started = threading.Event()
|
||||
|
||||
def slow(
|
||||
_: MemoryProcessingSupervisor,
|
||||
should_stop: Callable[[], bool] = lambda: False,
|
||||
_: DurableWriteQueue,
|
||||
*,
|
||||
expected_permission_revision: int | None = None,
|
||||
limit: int = 512,
|
||||
after_cursor: str | None = None,
|
||||
):
|
||||
started.set()
|
||||
threading.Event().wait(0.2)
|
||||
yield []
|
||||
return ()
|
||||
|
||||
monkeypatch.setattr(MemoryProcessingSupervisor, "_prepare_backfill_batches", slow)
|
||||
monkeypatch.setattr(DurableWriteQueue, "unsubmitted_evidence_ids", slow)
|
||||
|
||||
async def scenario() -> None:
|
||||
app = core(tmp_path)
|
||||
@@ -553,15 +482,14 @@ def test_a_failed_catch_up_leaves_a_traceback_behind(tmp_path, monkeypatch, capl
|
||||
"""
|
||||
|
||||
def failing(
|
||||
_: MemoryProcessingSupervisor,
|
||||
should_stop: Callable[[], bool] = lambda: False,
|
||||
_: DurableWriteQueue,
|
||||
*,
|
||||
expected_permission_revision: int | None = None,
|
||||
limit: int = 512,
|
||||
after_cursor: str | None = None,
|
||||
):
|
||||
raise sqlite3.OperationalError("database is locked")
|
||||
yield []
|
||||
|
||||
monkeypatch.setattr(MemoryProcessingSupervisor, "_prepare_backfill_batches", failing)
|
||||
monkeypatch.setattr(DurableWriteQueue, "unsubmitted_evidence_ids", failing)
|
||||
|
||||
async def scenario() -> dict:
|
||||
app = core(tmp_path)
|
||||
@@ -591,7 +519,7 @@ def test_a_failed_catch_up_leaves_a_traceback_behind(tmp_path, monkeypatch, capl
|
||||
if record.exc_info is not None
|
||||
for frame in traceback.extract_tb(record.exc_info[2])
|
||||
]
|
||||
assert any(frame.name == "backfill_off_thread" for frame in frames)
|
||||
assert any(frame.name == "_run_backfill" for frame in frames)
|
||||
assert any(frame.name == "failing" and frame.filename == __file__ for frame in frames), (
|
||||
"栈里没有出事的那个调用"
|
||||
)
|
||||
@@ -1,384 +0,0 @@
|
||||
"""HY's own coding/working route under the permission lease, on the pinned SDK.
|
||||
|
||||
HY decides the route; scripted HTTP stands in for the model, not for HY.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sqlite3
|
||||
import traceback
|
||||
from collections.abc import Mapping
|
||||
|
||||
import httpx
|
||||
from hy_memory.coding.types import CodingMemory
|
||||
|
||||
from one_memory.memory_governance import MemoryKind, MemoryStatus
|
||||
from tests.test_hy_permission_raw_context import (
|
||||
bind_oracle,
|
||||
grant_records,
|
||||
native_client,
|
||||
on_native_loop,
|
||||
put_turn,
|
||||
raw_client,
|
||||
)
|
||||
from tests.test_hy_permission_store import Lease
|
||||
|
||||
TOOL_TEXT = "[Bash] pytest -q\n12 passed"
|
||||
TASK = "Run the synthetic test suite"
|
||||
SEARCH_KEY = "pytest quiet run"
|
||||
SOLUTION = "Run `pytest -q` from the repository root."
|
||||
STATEMENT = "The agent ran pytest -q in the synthetic repository and 12 tests passed."
|
||||
DENIED_EXPERIENCE = "denied-coding-experience"
|
||||
DENIED_TASK = "DENIED_CODING_TASK_5d2e run the suite"
|
||||
DENIED_SOLUTION = "DENIED_CODING_SOLUTION_7f10: skip the tests"
|
||||
|
||||
JUDGE = "You are a single-label scene classifier."
|
||||
EXTRACT = "You extract durable WORKING memories"
|
||||
RECONCILE = "You decide how to merge new coding memory drafts"
|
||||
FACT_RECONCILE = "You maintain session-level WORKING FACTS"
|
||||
|
||||
|
||||
def coding_vector():
|
||||
# One shared direction: every stored experience and fact is a candidate.
|
||||
return [float(index == 5) for index in range(384)]
|
||||
|
||||
|
||||
class FixedEmbeddings:
|
||||
async def embed_batch(self, texts):
|
||||
return [coding_vector() for _ in texts]
|
||||
|
||||
async def embed(self, text):
|
||||
return coding_vector()
|
||||
|
||||
|
||||
class CodingOracle:
|
||||
def __init__(
|
||||
self,
|
||||
records,
|
||||
*,
|
||||
solution=SOLUTION,
|
||||
statement=STATEMENT,
|
||||
update_target=None,
|
||||
forbidden=(),
|
||||
after_response=None,
|
||||
):
|
||||
self.records = records
|
||||
self.solution, self.statement = solution, statement
|
||||
self.update_target = update_target
|
||||
self.forbidden = forbidden
|
||||
self.after_response = after_response
|
||||
self.requests = []
|
||||
self.stages = []
|
||||
self.reconcile_prompts = []
|
||||
self.errors = []
|
||||
self.revoked_at = None
|
||||
|
||||
def __call__(self, request):
|
||||
try:
|
||||
return self.respond(request)
|
||||
except Exception:
|
||||
self.errors.append(traceback.format_exc())
|
||||
raise
|
||||
|
||||
def respond(self, request):
|
||||
body = json.loads(request.content)
|
||||
self.requests.append(body)
|
||||
assert self.revoked_at is None, "native child request escaped after revocation"
|
||||
serialized = json.dumps(body, ensure_ascii=False)
|
||||
for forbidden in self.forbidden:
|
||||
assert forbidden not in serialized, f"unapproved coding row reached HTTP: {forbidden}"
|
||||
if request.url.path == "/v1/embeddings":
|
||||
texts = body["input"] if isinstance(body["input"], list) else [body["input"]]
|
||||
self.stages.append("embedding")
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"object": "list",
|
||||
"model": body["model"],
|
||||
"data": [
|
||||
{"index": i, "object": "embedding", "embedding": coding_vector()}
|
||||
for i in range(len(texts))
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "total_tokens": 1},
|
||||
},
|
||||
)
|
||||
prompt = body["messages"][-1]["content"]
|
||||
if prompt.startswith(JUDGE):
|
||||
result = {"is_coding": True, "reason": "synthetic tool run"}
|
||||
stage = "judge"
|
||||
elif prompt.startswith(EXTRACT):
|
||||
for record in self.records:
|
||||
assert record.verbatim in prompt, record.role
|
||||
result = {
|
||||
"experiences": [
|
||||
{
|
||||
"task": TASK,
|
||||
"search_keys": [SEARCH_KEY],
|
||||
"solution": self.solution,
|
||||
"boundary_envs": "",
|
||||
"confidence": 0.9,
|
||||
}
|
||||
],
|
||||
"fact": {
|
||||
"subject": "synthetic repository",
|
||||
"statement": self.statement,
|
||||
"files": [],
|
||||
"details": "",
|
||||
},
|
||||
}
|
||||
stage = "extract"
|
||||
elif prompt.startswith(RECONCILE):
|
||||
self.reconcile_prompts.append(prompt)
|
||||
op = (
|
||||
{"action": "UPDATE", "draft_idx": 0, "target_experience_id": self.update_target}
|
||||
if self.update_target
|
||||
else {"action": "ADD", "draft_idx": 0}
|
||||
)
|
||||
result = {"ops": [op]}
|
||||
stage = "reconcile"
|
||||
elif prompt.startswith(FACT_RECONCILE):
|
||||
result = {"ops": []}
|
||||
stage = "fact_reconcile"
|
||||
else:
|
||||
raise AssertionError(f"unexpected native prompt: {prompt[:80]!r}")
|
||||
self.stages.append(stage)
|
||||
response = httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "synthetic-coding-response",
|
||||
"object": "chat.completion",
|
||||
"created": 1,
|
||||
"model": body["model"],
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"finish_reason": "stop",
|
||||
"message": {"role": "assistant", "content": json.dumps(result)},
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
||||
},
|
||||
)
|
||||
if self.after_response is not None and self.after_response(stage):
|
||||
self.revoked_at = len(self.requests)
|
||||
return response
|
||||
|
||||
|
||||
def proof_ids(supervisor):
|
||||
with sqlite3.connect(supervisor.data_dir / "one-memory.sqlite3") as db:
|
||||
return {row[0] for row in db.execute("SELECT node_id FROM hy_node_permissions")}
|
||||
|
||||
|
||||
async def run_coding_turn(supervisor, seed, oracle):
|
||||
# Renewed consent also re-queues earlier turns; run only the seed's turn.
|
||||
(task,) = (
|
||||
task
|
||||
for task in supervisor.prepare_conversation(seed)
|
||||
if seed.evidence_id in {ref["evidence_id"] for ref in supervisor.tasks.inputs(task.task_id)}
|
||||
)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(task.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
return task, result
|
||||
|
||||
|
||||
def by_kind(supervisor):
|
||||
memories = supervisor.governance.list(user_id="local")
|
||||
grouped = {}
|
||||
for memory in memories:
|
||||
grouped.setdefault(memory.kind, []).append(memory)
|
||||
return grouped
|
||||
|
||||
|
||||
def test_hy_routes_a_tool_turn_to_coding_and_both_layers_are_kept(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, tool=TOOL_TEXT) as (supervisor, records, _):
|
||||
oracle = CodingOracle(records)
|
||||
task, result = await run_coding_turn(supervisor, records[0], oracle)
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
assert oracle.stages[:2] == ["judge", "extract"]
|
||||
assert "reconcile" in oracle.stages
|
||||
kinds = by_kind(supervisor)
|
||||
assert set(kinds) == {MemoryKind.CODING, MemoryKind.WORKING}
|
||||
(coding,), (working,) = kinds[MemoryKind.CODING], kinds[MemoryKind.WORKING]
|
||||
assert coding.content == f"{TASK}\n{SOLUTION}"
|
||||
assert working.content == STATEMENT
|
||||
every_record = {record.evidence_id for record in records}
|
||||
for memory in (coding, working):
|
||||
assert set(memory.evidence_ids) == every_record
|
||||
assert memory.status is MemoryStatus.CONFIRMED
|
||||
outputs = supervisor.tasks.outputs(task.task_id)
|
||||
assert {output.layer for output in outputs} == {"coding_experience", "working_fact"}
|
||||
assert {output.hy_memory_id for output in outputs} <= proof_ids(supervisor)
|
||||
# HY's pipeline log copies every coding prompt it is handed a cache for.
|
||||
(log_dir,) = tmp_path.rglob("hy-memory/*/logs")
|
||||
for path in log_dir.rglob("*"):
|
||||
if path.is_file():
|
||||
for text in (records[0].verbatim, TASK):
|
||||
assert text.encode() not in path.read_bytes(), (path.name, text)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_an_unapproved_experience_never_reaches_the_model_or_changes(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, tool=TOOL_TEXT) as (supervisor, records, _):
|
||||
task = supervisor.prepare_conversation(records[0])[0]
|
||||
_, request, _ = supervisor._create_runner()._prepare_permission_lease(task)
|
||||
client = await native_client(supervisor, task)
|
||||
|
||||
async def seed():
|
||||
await client._get_coding_writer()
|
||||
store = client._coding_store
|
||||
native_embeddings = store.embed_service
|
||||
store.embed_service = FixedEmbeddings()
|
||||
try:
|
||||
await store.insert(
|
||||
CodingMemory(
|
||||
experience_id=DENIED_EXPERIENCE,
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
session_id=request.session_id,
|
||||
task=DENIED_TASK,
|
||||
solution=DENIED_SOLUTION,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
store.embed_service = native_embeddings
|
||||
found = await store.search_by_query_embedding(
|
||||
coding_vector(), user_id=request.user_id
|
||||
)
|
||||
assert [hit["experience_id"] for hit in found] == [DENIED_EXPERIENCE]
|
||||
return (await store.get_by_id(DENIED_EXPERIENCE)).to_dict()
|
||||
|
||||
before = await on_native_loop(client, seed())
|
||||
oracle = CodingOracle(
|
||||
records, forbidden=(DENIED_EXPERIENCE, DENIED_TASK, DENIED_SOLUTION)
|
||||
)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(task.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
assert oracle.reconcile_prompts
|
||||
after = await on_native_loop(client, client._coding_store.get_by_id(DENIED_EXPERIENCE))
|
||||
assert after.to_dict() == before
|
||||
assert DENIED_EXPERIENCE not in proof_ids(supervisor)
|
||||
contents = {memory.content for memory in supervisor.governance.list(user_id="local")}
|
||||
assert contents == {f"{TASK}\n{SOLUTION}", STATEMENT}
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_later_turn_updating_an_experience_updates_the_same_memory(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, tool=TOOL_TEXT) as (supervisor, records, permission):
|
||||
_, first = await run_coding_turn(supervisor, records[0], CodingOracle(records))
|
||||
assert first.status == "completed", first.last_error_code
|
||||
(original,) = by_kind(supervisor)[MemoryKind.CODING]
|
||||
question = put_turn(
|
||||
supervisor.evidence, "user-again", "user", "Run it again verbosely.", 4
|
||||
)
|
||||
tool = put_turn(
|
||||
supervisor.evidence, "tool-again", "tool", "[Bash] pytest -v\n12 passed", 5
|
||||
)
|
||||
everything = (*records, question, tool)
|
||||
grant_records(supervisor, everything, permission.revision)
|
||||
verbose = "Run `pytest -v` from the repository root for per-test output."
|
||||
oracle = CodingOracle(
|
||||
(question, tool),
|
||||
solution=verbose,
|
||||
statement="The agent re-ran the suite verbosely; 12 passed.",
|
||||
update_target=original.memory_id,
|
||||
)
|
||||
_, second = await run_coding_turn(supervisor, question, oracle)
|
||||
assert second.status == "completed", (second.last_error_code, oracle.stages)
|
||||
assert original.memory_id in oracle.reconcile_prompts[0]
|
||||
kinds = by_kind(supervisor)
|
||||
(updated,) = kinds[MemoryKind.CODING]
|
||||
assert updated.memory_id == original.memory_id
|
||||
assert updated.content == f"{TASK}\n{verbose}"
|
||||
assert updated.revision > original.revision
|
||||
assert {question.evidence_id, tool.evidence_id} <= set(updated.evidence_ids)
|
||||
assert len(kinds[MemoryKind.WORKING]) == 2
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_revoking_during_coding_extraction_stops_the_route(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, tool=TOOL_TEXT) as (supervisor, records, permission):
|
||||
|
||||
def revoke(stage):
|
||||
if stage != "extract":
|
||||
return False
|
||||
supervisor.model_content_permissions.revoke(
|
||||
user_id="local", expected_revision=permission.revision, user_confirmed=True
|
||||
)
|
||||
return True
|
||||
|
||||
oracle = CodingOracle(records, after_response=revoke)
|
||||
task, result = await run_coding_turn(supervisor, records[0], oracle)
|
||||
assert result.status == "cancelled"
|
||||
assert result.last_error_code == "model_content_permission_revoked", result
|
||||
assert oracle.revoked_at is not None and len(oracle.requests) == oracle.revoked_at
|
||||
assert not supervisor.governance.list(user_id="local")
|
||||
assert not supervisor.tasks.outputs(task.task_id)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_exposed_coding_config_is_a_redacted_projection_not_the_native_handle(tmp_path):
|
||||
"""Public SDK 2.0.2 reads `config` off the coding store; it must not hand out secrets.
|
||||
|
||||
The wrapper answers `config` so the SDK's routing works, but the native
|
||||
MemoryConfig is a live backend handle carrying api keys and passwords. This
|
||||
asserts callers get a plain projection: not the native object, and no
|
||||
credential value survives anywhere inside it.
|
||||
"""
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, tool=TOOL_TEXT) as (supervisor, records, _):
|
||||
task = supervisor.prepare_conversation(records[0])[0]
|
||||
client = await native_client(supervisor, task)
|
||||
await on_native_loop(client, client._get_coding_writer())
|
||||
return client._coding_store
|
||||
|
||||
native = asyncio.run(scenario())
|
||||
from one_memory.hy_permission_coding_store import HyPermissionCodingStore
|
||||
|
||||
native.config.llm.api_key = "DENIED_LLM_KEY_2f31"
|
||||
native.config.embedder.api_key = "DENIED_EMBED_KEY_9c04"
|
||||
native.config.graph_store.password = "DENIED_GRAPH_PW_77aa"
|
||||
store = HyPermissionCodingStore(
|
||||
native,
|
||||
Lease(),
|
||||
agent_id="agent",
|
||||
session_id="session",
|
||||
)
|
||||
|
||||
exposed = store.config
|
||||
assert exposed is not native.config
|
||||
assert not isinstance(exposed, type(native.config))
|
||||
# Attribute access still works: that is how the public SDK reads it.
|
||||
assert isinstance(exposed["llm"], Mapping)
|
||||
assert exposed["llm"]["model"] == native.config.llm.model
|
||||
assert exposed.llm.model == native.config.llm.model
|
||||
|
||||
def contains_secret(value, depth=0):
|
||||
if depth > 5:
|
||||
return False
|
||||
if isinstance(value, str):
|
||||
return "DENIED_" in value
|
||||
if isinstance(value, Mapping):
|
||||
return any(contains_secret(item, depth + 1) for item in value.values())
|
||||
if isinstance(value, (list, tuple)):
|
||||
return any(contains_secret(item, depth + 1) for item in value)
|
||||
return False
|
||||
|
||||
assert not contains_secret(exposed)
|
||||
assert exposed["llm"]["api_key"] is None
|
||||
assert exposed["embedder"]["api_key"] is None
|
||||
assert exposed["graph_store"]["password"] is None
|
||||
# No native handle, connection or callable escapes the projection.
|
||||
assert not hasattr(exposed, "_native")
|
||||
assert "native" not in exposed
|
||||
@@ -1,250 +0,0 @@
|
||||
"""Blocked lease checks, local failing doubles only; no network or credentials."""
|
||||
|
||||
import asyncio
|
||||
import threading
|
||||
from contextlib import asynccontextmanager
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridge, HyMemoryBridgeError
|
||||
from one_memory.hy_permission_context import _LeasedBridge, permission_bridge_factory
|
||||
from one_memory.hy_permission_transport import HyPermissionTransportError
|
||||
|
||||
|
||||
class ControlledLease:
|
||||
def __init__(self, *, blocked_call=1, error=None):
|
||||
self.receipt = SimpleNamespace(user_id="local")
|
||||
self.profile = object()
|
||||
self.task_id = "synthetic-task"
|
||||
self.records = (SimpleNamespace(evidence_id="synthetic-evidence"),)
|
||||
self.blocked_call = blocked_call
|
||||
self.error = error
|
||||
self.calls = []
|
||||
self.entered = threading.Event()
|
||||
self.release = threading.Event()
|
||||
self.exited = threading.Event()
|
||||
self.revoked = False
|
||||
|
||||
def recheck(self):
|
||||
self.calls.append(threading.get_ident())
|
||||
if len(self.calls) == self.blocked_call:
|
||||
self.entered.set()
|
||||
try:
|
||||
if not self.release.wait(2):
|
||||
raise HyMemoryBridgeError("test_check_was_not_released")
|
||||
if self.revoked:
|
||||
raise HyMemoryBridgeError("model_content_permission_revoked")
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
finally:
|
||||
self.exited.set()
|
||||
return self.receipt
|
||||
|
||||
|
||||
class ControlledLoop:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def run(self, coroutine):
|
||||
self.calls.append(threading.get_ident())
|
||||
return asyncio.run(coroutine)
|
||||
|
||||
|
||||
class ControlledWiring:
|
||||
def __init__(self, *, error=None):
|
||||
self.client = SimpleNamespace(_loop_thread=ControlledLoop())
|
||||
self.error = error
|
||||
self.finished = 0
|
||||
self.closed = 0
|
||||
|
||||
async def finish(self):
|
||||
self.finished += 1
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
return {}
|
||||
|
||||
async def exit(self):
|
||||
self.closed += 1
|
||||
|
||||
|
||||
REQUEST = SimpleNamespace(
|
||||
user_id="local",
|
||||
evidence_ids=("synthetic-evidence",),
|
||||
agent_id="agent",
|
||||
session_id="session",
|
||||
request_id="synthetic-request",
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def blocked_operation(lease, operation):
|
||||
main_thread = threading.get_ident()
|
||||
pending = asyncio.create_task(operation())
|
||||
try:
|
||||
# A timer on this loop must run while the synchronous check is held.
|
||||
async with asyncio.timeout(0.5):
|
||||
while not lease.entered.is_set():
|
||||
await asyncio.sleep(0.005)
|
||||
ticked = asyncio.Event()
|
||||
asyncio.get_running_loop().call_later(0.02, ticked.set)
|
||||
await ticked.wait()
|
||||
assert not lease.exited.is_set(), "lease check blocked the main event loop"
|
||||
assert main_thread not in lease.calls, "lease check executed on the main thread"
|
||||
yield pending
|
||||
finally:
|
||||
lease.release.set()
|
||||
if not pending.done():
|
||||
pending.cancel()
|
||||
await asyncio.gather(pending, return_exceptions=True)
|
||||
assert lease.exited.is_set()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("operation", ["ingest", "ingest_extracted"])
|
||||
def test_cancelled_start_joins_check_without_later_sdk_entry(operation):
|
||||
async def scenario():
|
||||
lease = ControlledLease()
|
||||
bridge = SimpleNamespace(
|
||||
_get_client=AsyncMock(), ingest=AsyncMock(), ingest_extracted=AsyncMock()
|
||||
)
|
||||
bound = _LeasedBridge(bridge, lease, object(), None)
|
||||
|
||||
async def start():
|
||||
try:
|
||||
if operation == "ingest":
|
||||
await bound.ingest(REQUEST)
|
||||
else:
|
||||
await bound.ingest_extracted(REQUEST, ("Synthetic original",))
|
||||
finally:
|
||||
await bound.close()
|
||||
|
||||
async with blocked_operation(lease, start) as pending:
|
||||
pending.cancel()
|
||||
await asyncio.sleep(0.02)
|
||||
pending.cancel()
|
||||
await asyncio.sleep(0.02)
|
||||
assert not pending.done(), "cancel abandoned the active lease check"
|
||||
assert not bridge._get_client.called
|
||||
lease.release.set()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await pending
|
||||
assert not bridge._get_client.called
|
||||
assert not bridge.ingest.called and not bridge.ingest_extracted.called
|
||||
assert bound.wiring is None
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_revoked_start_preserves_error_and_never_opens_sdk():
|
||||
async def scenario():
|
||||
lease = ControlledLease()
|
||||
bridge = SimpleNamespace(_get_client=AsyncMock())
|
||||
bound = _LeasedBridge(bridge, lease, object(), None)
|
||||
async with blocked_operation(lease, lambda: bound._start(REQUEST)) as pending:
|
||||
lease.revoked = True
|
||||
lease.release.set()
|
||||
with pytest.raises(HyMemoryBridgeError, match="^model_content_permission_revoked$"):
|
||||
await pending
|
||||
await bound.close()
|
||||
assert not bridge._get_client.called and bound.wiring is None
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_released_start_reaches_client_once_and_preserves_client_error():
|
||||
async def scenario():
|
||||
lease = ControlledLease()
|
||||
expected = HyMemoryBridgeError("hy_memory_init_failed")
|
||||
bridge = SimpleNamespace(_get_client=AsyncMock(side_effect=expected))
|
||||
bound = _LeasedBridge(bridge, lease, object(), None)
|
||||
async with blocked_operation(lease, lambda: bound._start(REQUEST)) as pending:
|
||||
assert not bridge._get_client.called
|
||||
lease.release.set()
|
||||
with pytest.raises(HyMemoryBridgeError) as raised:
|
||||
await pending
|
||||
assert raised.value is expected
|
||||
bridge._get_client.assert_awaited_once_with()
|
||||
await bound.close()
|
||||
assert len(lease.calls) == 1 and bound.wiring is None
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("transport_failure", [False, True])
|
||||
@pytest.mark.parametrize("cancel", [False, True])
|
||||
def test_postcheck_and_transport_error_recheck_are_joined(transport_failure, cancel):
|
||||
async def scenario():
|
||||
expected = HyMemoryBridgeError("model_content_evidence_changed")
|
||||
lease = ControlledLease(blocked_call=2 if transport_failure else 1, error=expected)
|
||||
bound = _LeasedBridge(object(), lease, object(), None)
|
||||
wiring = ControlledWiring(
|
||||
error=HyPermissionTransportError("controlled_transport_failure")
|
||||
if transport_failure
|
||||
else None
|
||||
)
|
||||
bound.wiring = wiring
|
||||
|
||||
async def checked():
|
||||
try:
|
||||
await bound._check()
|
||||
finally:
|
||||
await bound.close()
|
||||
|
||||
async with blocked_operation(lease, checked) as pending:
|
||||
if cancel:
|
||||
pending.cancel()
|
||||
await asyncio.sleep(0.02)
|
||||
assert not pending.done() and wiring.closed == 0
|
||||
lease.release.set()
|
||||
if cancel:
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await pending
|
||||
else:
|
||||
with pytest.raises(HyMemoryBridgeError) as raised:
|
||||
await pending
|
||||
assert raised.value is expected
|
||||
assert wiring.finished == int(transport_failure)
|
||||
assert wiring.closed == 1 and bound.finished
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("revoked", [False, True])
|
||||
@pytest.mark.parametrize("cancel", [False, True])
|
||||
def test_factory_error_check_holds_exclusive_access_until_joined(tmp_path, revoked, cancel):
|
||||
async def scenario():
|
||||
lease = ControlledLease()
|
||||
bridge = HyMemoryBridge(
|
||||
profile=lease.profile,
|
||||
api_key=None,
|
||||
data_dir=tmp_path,
|
||||
client=SimpleNamespace(close=lambda: None),
|
||||
)
|
||||
factory = permission_bridge_factory(lambda *_: bridge)
|
||||
expected = HyPermissionTransportError("controlled_transport_failure")
|
||||
|
||||
async def failed_context():
|
||||
async with factory(lease, None, tmp_path):
|
||||
raise expected
|
||||
|
||||
async with blocked_operation(lease, failed_context) as pending:
|
||||
assert bridge._access_lock.locked()
|
||||
lease.revoked = revoked
|
||||
if cancel:
|
||||
pending.cancel()
|
||||
await asyncio.sleep(0.02)
|
||||
assert not pending.done() and bridge._access_lock.locked()
|
||||
lease.release.set()
|
||||
if cancel:
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await pending
|
||||
else:
|
||||
code = "model_content_permission_revoked" if revoked else expected.code
|
||||
with pytest.raises(HyMemoryBridgeError, match=f"^{code}$"):
|
||||
await pending
|
||||
assert bridge._access_owner is None and not bridge._access_lock.locked()
|
||||
await bridge.close()
|
||||
assert bridge._client is None
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,145 +0,0 @@
|
||||
"""HY decides deletions; the ledger mirrors them. Pinned SDK, scripted HTTP."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
import httpx
|
||||
from hy_memory.models.memory import MemoryStatus as HyStatus
|
||||
|
||||
from one_memory.memory_governance import DeletionStatus, MemoryKind, MemoryStatus
|
||||
from tests.test_hy_coding_lease import (
|
||||
TOOL_TEXT,
|
||||
CodingOracle,
|
||||
by_kind,
|
||||
run_coding_turn,
|
||||
)
|
||||
from tests.test_hy_permission_raw_context import (
|
||||
AGENT_TEXT,
|
||||
USER_TEXT,
|
||||
ScriptedOracle,
|
||||
assert_outputs,
|
||||
bind_oracle,
|
||||
grant_records,
|
||||
native_client,
|
||||
on_native_loop,
|
||||
put_turn,
|
||||
raw_client,
|
||||
)
|
||||
|
||||
|
||||
class DeletingOracle(ScriptedOracle):
|
||||
"""HY's reconciler withdraws the user's claim and keeps the agent's."""
|
||||
|
||||
def respond(self, request):
|
||||
response = super().respond(request)
|
||||
if self.stages[-1] != "reconcile":
|
||||
return response
|
||||
body = json.loads(response.content)
|
||||
ops = json.loads(body["choices"][0]["message"]["content"])
|
||||
for op in ops:
|
||||
if op["content"] == USER_TEXT and "memory_id" in op:
|
||||
op.clear()
|
||||
op.update({"op": "DELETE", "memory_id": self.withdrawn})
|
||||
body["choices"][0]["message"]["content"] = json.dumps(ops)
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
|
||||
def test_a_memory_hy_deletes_is_deleted_in_the_ledger_and_the_rest_stay(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, permission):
|
||||
first = supervisor.prepare_conversation(records[0])[0]
|
||||
client = await native_client(supervisor, first)
|
||||
bind_oracle(supervisor, ScriptedOracle(records))
|
||||
assert (await supervisor._create_runner().run(first.task_id)).status == "completed"
|
||||
await assert_outputs(supervisor, first, records, client)
|
||||
by_content = {m.content: m for m in supervisor.governance.list(user_id="local")}
|
||||
withdrawn, kept = by_content[USER_TEXT], by_content[AGENT_TEXT]
|
||||
|
||||
added = put_turn(supervisor.evidence, "user-withdraw", "user", USER_TEXT, 3)
|
||||
records = (*records, added)
|
||||
grant_records(supervisor, records, permission.revision)
|
||||
second = supervisor.prepare_conversation(added)[0]
|
||||
oracle = DeletingOracle(records)
|
||||
oracle.withdrawn = withdrawn.memory_id
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(second.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
|
||||
native = await on_native_loop(
|
||||
client, client._vector_store.get_by_id(withdrawn.memory_id)
|
||||
)
|
||||
assert native is not None and native.status == HyStatus.SHADOW
|
||||
mirrored = supervisor.governance.get(withdrawn.memory_id, user_id="local")
|
||||
assert mirrored.status is MemoryStatus.DELETED
|
||||
deletion = supervisor.governance.deletion(withdrawn.memory_id, user_id="local")
|
||||
assert deletion is not None and deletion.status is DeletionStatus.COMPLETED
|
||||
untouched = supervisor.governance.get(kept.memory_id, user_id="local")
|
||||
assert untouched.status is MemoryStatus.CONFIRMED
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
class RetiringCodingOracle(CodingOracle):
|
||||
def __init__(self, records, *, retire, **kwargs):
|
||||
super().__init__(records, **kwargs)
|
||||
self.retire = retire
|
||||
|
||||
def respond(self, request):
|
||||
response = super().respond(request)
|
||||
if self.stages[-1] != "reconcile":
|
||||
return response
|
||||
body = json.loads(response.content)
|
||||
body["choices"][0]["message"]["content"] = json.dumps(
|
||||
{
|
||||
"ops": [
|
||||
{"action": "ADD", "draft_idx": 0},
|
||||
{
|
||||
"action": "DELETE",
|
||||
"draft_idx": 0,
|
||||
"target_experience_id": self.retire,
|
||||
"reason": "the new run explicitly replaces the old procedure",
|
||||
},
|
||||
]
|
||||
}
|
||||
)
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
|
||||
def test_an_experience_hy_deletes_is_deleted_in_the_ledger(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, tool=TOOL_TEXT) as (supervisor, records, permission):
|
||||
_, first = await run_coding_turn(supervisor, records[0], CodingOracle(records))
|
||||
assert first.status == "completed", first.last_error_code
|
||||
(original,) = by_kind(supervisor)[MemoryKind.CODING]
|
||||
question = put_turn(supervisor.evidence, "user-again", "user", "Use tox now.", 4)
|
||||
tool = put_turn(supervisor.evidence, "tool-again", "tool", "[Bash] tox\nok", 5)
|
||||
grant_records(supervisor, (*records, question, tool), permission.revision)
|
||||
oracle = RetiringCodingOracle(
|
||||
(question, tool),
|
||||
retire=original.memory_id,
|
||||
solution="Run `tox` from the repository root.",
|
||||
statement="The agent switched the synthetic suite to tox.",
|
||||
)
|
||||
_, second = await run_coding_turn(supervisor, question, oracle)
|
||||
assert second.status == "completed", (second.last_error_code, oracle.stages)
|
||||
client = await native_client(supervisor, second)
|
||||
assert (
|
||||
await on_native_loop(client, client._coding_store.get_by_id(original.memory_id))
|
||||
is None
|
||||
)
|
||||
|
||||
assert (
|
||||
supervisor.governance.get(original.memory_id, user_id="local").status
|
||||
is MemoryStatus.DELETED
|
||||
)
|
||||
live = [
|
||||
m
|
||||
for m in by_kind(supervisor)[MemoryKind.CODING]
|
||||
if m.status is MemoryStatus.CONFIRMED
|
||||
]
|
||||
assert [m.content for m in live] == [
|
||||
"Run the synthetic test suite\nRun `tox` from the repository root."
|
||||
]
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -18,10 +18,12 @@ from one_memory.hy_memory_bridge import (
|
||||
HyMemoryBridge,
|
||||
HyMemoryBridgeError,
|
||||
HyMemoryIngestRequest,
|
||||
HyMemoryIngestResult,
|
||||
HyMemoryTurn,
|
||||
UnsupportedHyMemoryProviderError,
|
||||
_basic_info_memories,
|
||||
_collection_name,
|
||||
_with_native_mutation_delta,
|
||||
build_hy_memory_config,
|
||||
configure_hy_memory_environment,
|
||||
reattach_legacy_vector_collection,
|
||||
@@ -1324,7 +1326,7 @@ def test_hy_runtime_matches_public_sdk_without_reranker_or_service_backends(
|
||||
config = build_hy_memory_config(profile, api_key=None, data_dir=tmp_path)
|
||||
|
||||
assert HY_MEMORY_DISTRIBUTION == "hy-memory"
|
||||
assert HY_MEMORY_VERSION == "2.0.2"
|
||||
assert HY_MEMORY_VERSION == "2.0.3"
|
||||
assert HY_MEMORY_MODE == "ultra"
|
||||
assert HY_MEMORY_READER == "hybrid"
|
||||
assert os.environ["MEMORY_MODE"] == "ultra"
|
||||
@@ -1507,3 +1509,58 @@ def test_the_client_mode_follows_the_graph_store_switch(tmp_path, monkeypatch) -
|
||||
hy_memory_bridge.create_hy_memory_client(profile, api_key=None, data_dir=tmp_path).close()
|
||||
|
||||
assert seen == ["pro", HY_MEMORY_MODE]
|
||||
|
||||
|
||||
def test_native_mutation_delta_reports_coding_deletes_and_supersedes() -> None:
|
||||
raw = {
|
||||
"ops": [
|
||||
{
|
||||
"action": "DELETE",
|
||||
"target_experience_id": "old-experience",
|
||||
}
|
||||
],
|
||||
"fact_ops": [
|
||||
{
|
||||
"action": "SUPERSEDE",
|
||||
"fact_id": "old-fact",
|
||||
"superseded_by": "new-fact",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
result = asyncio.run(
|
||||
_with_native_mutation_delta(
|
||||
SimpleNamespace(),
|
||||
HyMemoryIngestResult(success=True),
|
||||
raw,
|
||||
(),
|
||||
)
|
||||
)
|
||||
|
||||
assert result.retired_memory_ids == ("old-experience", "old-fact")
|
||||
assert result.superseded_memory_ids == {"old-fact": ("new-fact",)}
|
||||
|
||||
|
||||
def test_native_mutation_delta_reads_chat_status_from_the_sdk() -> None:
|
||||
class Client:
|
||||
def get(self, memory_id: str) -> Mapping[str, object]:
|
||||
assert memory_id == "old-chat"
|
||||
return {
|
||||
"memory_id": memory_id,
|
||||
"content": "old",
|
||||
"status": "superseded",
|
||||
"superseded_by": ["new-chat"],
|
||||
}
|
||||
|
||||
guard = SimpleNamespace(mutation_target_ids={"old-chat"})
|
||||
result = asyncio.run(
|
||||
_with_native_mutation_delta(
|
||||
Client(),
|
||||
HyMemoryIngestResult(success=True),
|
||||
{},
|
||||
(guard,),
|
||||
)
|
||||
)
|
||||
|
||||
assert result.retired_memory_ids == ("old-chat",)
|
||||
assert result.superseded_memory_ids == {"old-chat": ("new-chat",)}
|
||||
@@ -1,208 +0,0 @@
|
||||
"""HY writes with its own prompts and judgment; the product adds no audit of its own."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from hy_memory.agent.reconciler import ReconcileOp, ReconcileResult
|
||||
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
from one_memory.hy_source_context import HySourceContext, HySupportedMemory, HyWriteBindings
|
||||
from one_memory.hy_write_verification import HyExtractionGuard
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from tests.test_hy_permission_raw_context import (
|
||||
ScriptedOracle,
|
||||
assert_outputs,
|
||||
bind_oracle,
|
||||
native_client,
|
||||
raw_client,
|
||||
)
|
||||
from tests.test_hy_source_context import source
|
||||
|
||||
PRODUCT_RULE_MARKERS = ("# 来源与语气保真", "# 原始发言主体规则", "owner记录命题的陈述权威")
|
||||
|
||||
|
||||
def test_a_raw_turn_reaches_hy_with_its_native_prompts_and_no_product_audit(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, _):
|
||||
task = supervisor.prepare_conversation(records[0])[0]
|
||||
oracle = ScriptedOracle(records)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(task.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
assert "extract" in oracle.stages
|
||||
assert "review" not in oracle.stages
|
||||
assert "citations" not in oracle.stages
|
||||
sent = json.dumps([body for _path, body in oracle.requests], ensure_ascii=False)
|
||||
for marker in PRODUCT_RULE_MARKERS:
|
||||
assert marker not in sent, marker
|
||||
# Each memory still points at the words of its own author in this turn.
|
||||
client = await native_client(supervisor, task)
|
||||
await assert_outputs(supervisor, task, records, client)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_an_imported_memory_keeps_the_owner_hy_gives_it():
|
||||
class Native:
|
||||
received = None
|
||||
|
||||
async def reconcile(self, **kwargs):
|
||||
self.received = kwargs["new_memories_with_meta"]
|
||||
return ReconcileResult(
|
||||
ops=[ReconcileOp(op="ADD", content="用户偏好简洁表达。", owner="user")]
|
||||
)
|
||||
|
||||
native = Native()
|
||||
guard = HyReconcileGuard(native)
|
||||
guard.imported_contents = frozenset({"用户偏好简洁表达。"})
|
||||
result = asyncio.run(
|
||||
guard.reconcile(
|
||||
new_memories=["用户偏好简洁表达。"],
|
||||
new_memories_with_meta=[{"content": "用户偏好简洁表达。", "layer": "L2_FACT"}],
|
||||
)
|
||||
)
|
||||
assert "owner" not in native.received[0]
|
||||
assert result.ops[0].owner == "user"
|
||||
assert guard.failures == []
|
||||
|
||||
|
||||
def bound(tmp_path, *, known=False):
|
||||
store = EvidenceStore(tmp_path / "evidence.db")
|
||||
old = source(store, "old", "user", "Atlas用本地存储。")
|
||||
current = source(store, "now", "user", "Atlas也要能离线同步。")
|
||||
memories = (
|
||||
HySupportedMemory(
|
||||
hy_memory_id="prior",
|
||||
governed_memory_id="prior",
|
||||
revision=1,
|
||||
content="用户要求Atlas使用本地存储。",
|
||||
owner="user",
|
||||
evidence_ids=(old.evidence_id,),
|
||||
),
|
||||
)
|
||||
context = HySourceContext(
|
||||
current=(current,), history=(old,), known_memories=memories if known else ()
|
||||
)
|
||||
return HyWriteBindings(context), old, current
|
||||
|
||||
|
||||
def test_the_turn_hy_extracts_must_be_the_turn_that_was_sent(tmp_path):
|
||||
bindings, _old, current = bound(tmp_path)
|
||||
calls = []
|
||||
|
||||
class Native:
|
||||
async def extract(self, content, context, **kwargs):
|
||||
calls.append(content)
|
||||
return SimpleNamespace(success=True, info={"memory": []})
|
||||
|
||||
guard = HyExtractionGuard(Native())
|
||||
guard.bindings = bindings
|
||||
sent = json.dumps([{"role": "user", "content": current.verbatim}], ensure_ascii=False)
|
||||
assert asyncio.run(guard.extract(sent, {})).success and calls == [sent]
|
||||
with pytest.raises(ValueError, match="input_mismatch"):
|
||||
asyncio.run(guard.extract(json.dumps([{"role": "user", "content": "别的内容"}]), {}))
|
||||
assert calls == [sent]
|
||||
assert guard.failures == ["hy_memory_write_verification_input_mismatch"]
|
||||
|
||||
|
||||
class PriorStore:
|
||||
async def get_by_id(self, key):
|
||||
return SimpleNamespace(
|
||||
user_id="local", agent_id="atlas", owner="user", content="用户要求Atlas使用本地存储。"
|
||||
)
|
||||
|
||||
|
||||
def merge_into_prior(content):
|
||||
class Native:
|
||||
async def reconcile(self, **kwargs):
|
||||
return ReconcileResult(
|
||||
ops=[ReconcileOp(op="UPDATE", memory_id="prior", owner="user", content=content)]
|
||||
)
|
||||
|
||||
return Native()
|
||||
|
||||
|
||||
ARGS = {"user_id": "local", "agent_id": "atlas", "vector_store": PriorStore()}
|
||||
|
||||
|
||||
def test_hy_may_only_change_a_memory_this_write_was_allowed_to_see(tmp_path):
|
||||
allowed, _old, _current = bound(tmp_path / "allowed", known=True)
|
||||
guard = HyReconcileGuard(merge_into_prior("用户要求Atlas本地存储并支持离线同步。"))
|
||||
guard.bindings = allowed
|
||||
assert asyncio.run(guard.reconcile(**ARGS)).success
|
||||
|
||||
unseen, _old, _current = bound(tmp_path / "unseen")
|
||||
guard = HyReconcileGuard(merge_into_prior("用户要求Atlas本地存储并支持离线同步。"))
|
||||
guard.bindings = unseen
|
||||
with pytest.raises(ValueError, match="target_unbound"):
|
||||
asyncio.run(guard.reconcile(**ARGS))
|
||||
|
||||
|
||||
def test_a_memory_hy_merges_cites_this_turn_and_what_it_already_cited(tmp_path):
|
||||
bindings, old, current = bound(tmp_path, known=True)
|
||||
merged = "用户要求Atlas本地存储并支持离线同步。"
|
||||
guard = HyReconcileGuard(merge_into_prior(merged))
|
||||
guard.bindings = bindings
|
||||
assert asyncio.run(guard.reconcile(**ARGS)).success
|
||||
assert bindings.support_for(merged, "user") == (current.evidence_id, old.evidence_id)
|
||||
|
||||
|
||||
HY_USER_ONLY_LINE = "只提取 NEW user messages 中首次出现的新信息。"
|
||||
|
||||
|
||||
def _section(prompt, heading):
|
||||
_, found, rest = prompt.partition(heading)
|
||||
assert found, heading
|
||||
return rest.partition("\n---\n")[0]
|
||||
|
||||
|
||||
def _served_extract_prompts(tmp_path):
|
||||
from one_memory.hy_memory_bridge import build_hy_memory_config
|
||||
from tests.test_memory_language import _profile
|
||||
|
||||
config = build_hy_memory_config(_profile(), api_key=None, data_dir=tmp_path)
|
||||
return config.llm.extract_system_prompt_zh, config.llm.extract_system_prompt_en
|
||||
|
||||
|
||||
def test_hys_native_extraction_only_reads_what_the_user_said():
|
||||
from hy_memory.agent.prompts_zh import EXTRACT_SYSTEM_PROMPT_ZH
|
||||
|
||||
assert HY_USER_ONLY_LINE in EXTRACT_SYSTEM_PROMPT_ZH
|
||||
# HY already knows how to keep an assistant's words apart from the person's.
|
||||
assert '`"agent"` —— AI 助手' in _section(EXTRACT_SYSTEM_PROMPT_ZH, "# OWNER(归属)\n")
|
||||
|
||||
|
||||
def test_hy_extracts_both_speakers_and_keeps_its_own_authorship_rules(tmp_path):
|
||||
from hy_memory.agent.prompts_zh import EXTRACT_SYSTEM_PROMPT_ZH as native
|
||||
|
||||
for served in _served_extract_prompts(tmp_path):
|
||||
assert HY_USER_ONLY_LINE not in served
|
||||
principle = _section(served, "# 提取原则\n")
|
||||
assert "用户" in principle and "AI 助手" in principle
|
||||
assert "OWNER" in principle
|
||||
for heading in ("# OWNER(归属)\n", "# 主体规则\n"):
|
||||
assert _section(served, heading) == _section(native, heading), heading
|
||||
for marker in PRODUCT_RULE_MARKERS:
|
||||
assert marker not in served, marker
|
||||
|
||||
|
||||
def test_a_reworded_extraction_line_still_asks_for_both_speakers(tmp_path, monkeypatch, caplog):
|
||||
from hy_memory.agent.prompts_zh import EXTRACT_SYSTEM_PROMPT_ZH as native
|
||||
|
||||
from one_memory import hy_memory_bridge
|
||||
|
||||
class Reworded:
|
||||
EXTRACT_SYSTEM_PROMPT_ZH = native.replace(HY_USER_ONLY_LINE, "HY 换了提取原则的措辞。")
|
||||
|
||||
real_import = hy_memory_bridge.import_module
|
||||
monkeypatch.setattr(
|
||||
hy_memory_bridge,
|
||||
"import_module",
|
||||
lambda name: Reworded if name == "hy_memory.agent.prompts_zh" else real_import(name),
|
||||
)
|
||||
served, _ = _served_extract_prompts(tmp_path)
|
||||
assert "AI 助手提供的" in served
|
||||
assert "hy_extraction_scope_rule_not_found" in caplog.text
|
||||
@@ -1,192 +0,0 @@
|
||||
"""Whole native HY client with synthetic content and HTTP MockTransport, never cloud."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from one_memory.hy_permission_context import permission_bridge_factory
|
||||
from one_memory.imports.grants import SourceGrantStore
|
||||
from one_memory.imports.models import (
|
||||
EvidenceContentKind,
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
)
|
||||
from one_memory.imports.sources.registry import RootCandidateKind
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import MemoryGovernanceStore
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_tasks import MemoryTaskStore
|
||||
from one_memory.model_content_permissions import ModelContentPermissionStore
|
||||
from one_memory.model_profiles import (
|
||||
ModelCapabilities,
|
||||
ModelProfileInput,
|
||||
ModelProfileStore,
|
||||
ProviderKind,
|
||||
)
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
|
||||
|
||||
@contextmanager
|
||||
def whole_client(tmp_path: Path):
|
||||
root = tmp_path / "source"
|
||||
root.mkdir()
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
grants = SourceGrantStore(database, home=tmp_path)
|
||||
grants.grant(
|
||||
user_id="local", source_id="fixture", root=root, candidate_kind=RootCandidateKind.DEFAULT
|
||||
)
|
||||
evidence = EvidenceStore(database)
|
||||
text = "Synthetic supplied memory: review permissions before release."
|
||||
record = evidence.put(
|
||||
EvidenceInput(
|
||||
source_id="fixture",
|
||||
conversation_id="session",
|
||||
message_id="1",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.UNKNOWN,
|
||||
content_kind=EvidenceContentKind.MEMORY_CONTENT,
|
||||
verbatim=text,
|
||||
origin_meta={EvidenceOrigin.MANUAL_IMPORT: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
profiles, secrets = ModelProfileStore(database), InMemorySecretStore()
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Stub only",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:1/v1",
|
||||
model="test-generation",
|
||||
embedding_provider_kind=ProviderKind.OLLAMA,
|
||||
embedding_base_url="http://127.0.0.1:2/v1",
|
||||
embedding_model="test-embedding",
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True, model_available=True, embedding_available=True
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
permissions = ModelContentPermissionStore(database)
|
||||
snapshot = permissions.snapshot(
|
||||
user_id="local",
|
||||
source_ids=("fixture",),
|
||||
profile=profiles.snapshot(profile.profile_id, profile.revision),
|
||||
evidence=(record,),
|
||||
)
|
||||
permission = permissions.grant(snapshot, user_confirmed=True, expected_revision=0)
|
||||
supervisor = MemoryProcessingSupervisor(
|
||||
tasks=MemoryTaskStore(database),
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=evidence,
|
||||
governance=MemoryGovernanceStore(database),
|
||||
data_dir=tmp_path,
|
||||
model_content_permissions=permissions,
|
||||
)
|
||||
yield supervisor, record, permissions, permission
|
||||
|
||||
|
||||
def test_native_memory_content_uses_guarded_sdk_then_returns_real_native_memory(tmp_path):
|
||||
async def scenario():
|
||||
with whole_client(tmp_path) as (supervisor, record, _, _):
|
||||
sends = []
|
||||
|
||||
def handler(request: httpx.Request):
|
||||
body = json.loads(request.content)
|
||||
sends.append({"path": request.url.path, "body": body})
|
||||
if request.url.path.endswith("embeddings"):
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"object": "list",
|
||||
"model": "test-embedding",
|
||||
"data": [
|
||||
{
|
||||
"index": i,
|
||||
"object": "embedding",
|
||||
"embedding": [1.0] + [0.0] * 383,
|
||||
}
|
||||
for i, _ in enumerate(body["input"])
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "total_tokens": 1},
|
||||
},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "fixture-response",
|
||||
"object": "chat.completion",
|
||||
"created": 1,
|
||||
"model": "test-generation",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"finish_reason": "stop",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": json.dumps(
|
||||
[
|
||||
{
|
||||
"op": "ADD",
|
||||
"content": record.verbatim,
|
||||
"layer": "L2_FACT",
|
||||
"owner": "agent",
|
||||
}
|
||||
]
|
||||
),
|
||||
},
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
||||
},
|
||||
)
|
||||
|
||||
supervisor.permission_bridge_factory = permission_bridge_factory(
|
||||
supervisor._bridge, http_transport=httpx.MockTransport(handler)
|
||||
)
|
||||
tasks = supervisor.prepare_conversation(record)
|
||||
try:
|
||||
result = await supervisor._create_runner().run(tasks[0].task_id)
|
||||
assert result.status == "completed", result.last_error_code
|
||||
assert sends and all(
|
||||
item["path"].endswith(("embeddings", "chat/completions")) for item in sends
|
||||
)
|
||||
memories = supervisor.governance.list(user_id="local")
|
||||
assert len(memories) == 1 and memories[0].content == record.verbatim
|
||||
assert memories[0].status == "confirmed"
|
||||
assert supervisor.evidence.get(record.evidence_id).verbatim == record.verbatim
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_revoked_lease_never_creates_an_sdk_client_or_sends(tmp_path):
|
||||
async def scenario():
|
||||
with whole_client(tmp_path) as (supervisor, record, permissions, permission):
|
||||
tasks = supervisor.prepare_conversation(record)
|
||||
permissions.revoke(
|
||||
user_id="local", expected_revision=permission.revision, user_confirmed=True
|
||||
)
|
||||
supervisor.permission_bridge_factory = permission_bridge_factory(
|
||||
supervisor._bridge,
|
||||
http_transport=httpx.MockTransport(lambda _: pytest.fail("unauthorized request")),
|
||||
)
|
||||
result = await supervisor._create_runner().run(tasks[0].task_id)
|
||||
assert result.last_error_code == "model_content_permission_revoked"
|
||||
assert supervisor._bridges == {}
|
||||
assert not supervisor.governance.list(user_id="local")
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,380 +0,0 @@
|
||||
"""Bounded factory races with a temporary native HY client and no model calls."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import threading
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from hy_memory.models.memory import MemoryLayer, MemoryNode
|
||||
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.hy_permission_coding_store import HyPermissionCodingStore
|
||||
from one_memory.hy_permission_context import _Wiring, permission_bridge_factory
|
||||
from one_memory.hy_permission_store import HyPermissionVectorStore
|
||||
from one_memory.hy_permission_transport import HyPermissionTransportError
|
||||
from one_memory.memory_governance import MemoryDraft, MemoryKind
|
||||
from tests.person_door import as_person
|
||||
from tests.test_hy_permission_context import whole_client
|
||||
|
||||
|
||||
def test_cancelled_start_cannot_install_wiring_after_factory_releases_client(tmp_path):
|
||||
async def scenario():
|
||||
with whole_client(tmp_path) as (supervisor, record, _, _):
|
||||
task = supervisor.prepare_conversation(record)[0]
|
||||
lease, request, _ = supervisor._create_runner()._prepare_permission_lease(task)
|
||||
bridge = supervisor._bridge(lease.profile, None, tmp_path)
|
||||
client = await bridge._get_client()
|
||||
native_vector, native_graph = client._vector_store, client._graph_store
|
||||
original_coding = client._config.coding.enable
|
||||
run = client._loop_thread.run
|
||||
submitted, release, completed = (threading.Event() for _ in range(3))
|
||||
worker_errors = []
|
||||
bound_contexts = []
|
||||
sends = []
|
||||
|
||||
def mock_send(request):
|
||||
sends.append(request)
|
||||
pytest.fail("This startup-only race must not make any model request")
|
||||
|
||||
def delayed_run(coroutine):
|
||||
if coroutine.cr_code is not _Wiring.enter.__code__:
|
||||
return run(coroutine)
|
||||
# Model a worker being descheduled just before submitting to HY.
|
||||
submitted.set()
|
||||
try:
|
||||
if not release.wait(5):
|
||||
coroutine.close()
|
||||
raise TimeoutError("test did not release startup worker")
|
||||
return run(coroutine)
|
||||
except BaseException as error:
|
||||
worker_errors.append(error)
|
||||
raise
|
||||
finally:
|
||||
completed.set()
|
||||
|
||||
client._loop_thread.run = delayed_run
|
||||
factory = permission_bridge_factory(
|
||||
supervisor._bridge, http_transport=httpx.MockTransport(mock_send)
|
||||
)
|
||||
|
||||
async def ingest():
|
||||
async with factory(lease, None, tmp_path) as bound:
|
||||
bound_contexts.append(bound)
|
||||
await bound.ingest_extracted(request, (record.verbatim,))
|
||||
|
||||
pending = asyncio.create_task(ingest())
|
||||
try:
|
||||
assert await asyncio.to_thread(submitted.wait, 5)
|
||||
pending.cancel()
|
||||
# A fixed factory may keep cancellation pending until it has
|
||||
# joined startup. Let both that behavior and early exit proceed.
|
||||
await asyncio.wait({pending}, timeout=0.1)
|
||||
released_before_start = not bridge._access_lock.locked()
|
||||
release.set()
|
||||
assert await asyncio.to_thread(completed.wait, 5)
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await asyncio.wait_for(pending, 5)
|
||||
assert not worker_errors
|
||||
assert not sends
|
||||
assert bridge._access_owner is None
|
||||
assert not bridge._access_lock.locked()
|
||||
assert not isinstance(client._vector_store, HyPermissionVectorStore), (
|
||||
"Cancelled startup installed a live permission wrapper after the "
|
||||
"factory exited and released exclusive ownership; "
|
||||
f"released_before_start={released_before_start}"
|
||||
)
|
||||
assert client._vector_store is native_vector
|
||||
assert client._graph_store is native_graph
|
||||
assert client._config.coding.enable == original_coding
|
||||
finally:
|
||||
release.set()
|
||||
if not pending.done():
|
||||
pending.cancel()
|
||||
await asyncio.gather(pending, return_exceptions=True)
|
||||
assert await asyncio.to_thread(completed.wait, 5)
|
||||
client._loop_thread.run = run
|
||||
# The regression leaves finished=True with live wiring; force
|
||||
# cleanup directly so the failing test leaks no SDK resources.
|
||||
for bound in bound_contexts:
|
||||
if bound.wiring is not None:
|
||||
await asyncio.to_thread(run, bound.wiring.exit())
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_completed_start_restores_client_and_revokes_saved_vector_reference(tmp_path):
|
||||
async def scenario():
|
||||
with whole_client(tmp_path) as (supervisor, record, _, _):
|
||||
task = supervisor.prepare_conversation(record)[0]
|
||||
lease, request, _ = supervisor._create_runner()._prepare_permission_lease(task)
|
||||
bridge = supervisor._bridge(lease.profile, None, tmp_path)
|
||||
client = await bridge._get_client()
|
||||
vector, graph = client._vector_store, client._graph_store
|
||||
factory = permission_bridge_factory(
|
||||
supervisor._bridge,
|
||||
http_transport=httpx.MockTransport(lambda _: pytest.fail("unexpected send")),
|
||||
)
|
||||
try:
|
||||
async with factory(lease, None, tmp_path) as bound:
|
||||
await bound._start(request)
|
||||
wrapped = client._vector_store
|
||||
assert isinstance(wrapped, HyPermissionVectorStore)
|
||||
assert client._graph_store is None
|
||||
coding = client._coding_store
|
||||
assert isinstance(coding, HyPermissionCodingStore)
|
||||
assert client._coding_writer.reconciler.store is coding
|
||||
assert client._cache is None
|
||||
assert client._vector_store is vector
|
||||
assert client._graph_store is graph
|
||||
assert client._coding_writer.reconciler.store is client._coding_store
|
||||
assert not isinstance(client._coding_store, HyPermissionCodingStore)
|
||||
assert client._cache is not None
|
||||
with pytest.raises(HyMemoryBridgeError, match="hy_store_closed"):
|
||||
wrapped.recheck()
|
||||
with pytest.raises(HyMemoryBridgeError, match="hy_coding_store_closed"):
|
||||
coding.recheck()
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_graph_disabled_keeps_graph_absent_even_for_cross_evidence_operations(tmp_path):
|
||||
async def scenario():
|
||||
with whole_client(tmp_path) as (supervisor, record, _, _):
|
||||
task = supervisor.prepare_conversation(record)[0]
|
||||
lease, request, _ = supervisor._create_runner()._prepare_permission_lease(task)
|
||||
bridge = supervisor._bridge(lease.profile, None, tmp_path)
|
||||
client = await bridge._get_client()
|
||||
native_graph = client._graph_store
|
||||
assert native_graph is None
|
||||
factory = permission_bridge_factory(
|
||||
supervisor._bridge,
|
||||
http_transport=httpx.MockTransport(lambda _: pytest.fail("unexpected send")),
|
||||
)
|
||||
try:
|
||||
async with factory(lease, None, tmp_path) as bound:
|
||||
await bound._start(request)
|
||||
assert client._graph_store is None
|
||||
assert bound.wiring is not None and bound.wiring.graph is None
|
||||
assert client._graph_store is native_graph
|
||||
|
||||
async with factory(lease, None, tmp_path) as digest_bound:
|
||||
await digest_bound._start_operation(
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
request_id="digest-test",
|
||||
session_id="digest:test",
|
||||
cross_evidence_scope=True,
|
||||
)
|
||||
assert client._graph_store is None
|
||||
assert digest_bound.wiring is not None
|
||||
assert digest_bound.wiring.graph is None
|
||||
assert client._graph_store is native_graph
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_persisted_native_proof_does_not_authorize_deleted_governed_memory(tmp_path):
|
||||
async def scenario():
|
||||
with whole_client(tmp_path) as (supervisor, record, permissions, permission):
|
||||
runner = supervisor._create_runner()
|
||||
task = supervisor.prepare_conversation(record)[0]
|
||||
lease, request, _ = runner._prepare_permission_lease(task)
|
||||
factory = permission_bridge_factory(
|
||||
supervisor._bridge,
|
||||
http_transport=httpx.MockTransport(lambda _: pytest.fail("unexpected send")),
|
||||
)
|
||||
bridge = supervisor._bridge(lease.profile, None, tmp_path)
|
||||
client = await bridge._get_client()
|
||||
run = client._loop_thread.run
|
||||
native = MemoryNode(
|
||||
node_id="synthetic-governed-memory",
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
session_id=request.session_id,
|
||||
content=record.verbatim,
|
||||
owner="agent",
|
||||
layer=MemoryLayer.L2_FACT,
|
||||
embedding=[1.0] + [0.0] * 383,
|
||||
)
|
||||
try:
|
||||
# Seed only through the guarded native store; profile readback
|
||||
# persists the factory's own proof ledger, as in a completed run.
|
||||
async with factory(lease, None, tmp_path) as first:
|
||||
await first._start(request)
|
||||
store = first.wiring.vector
|
||||
assert await asyncio.to_thread(run, store.upsert(native)) == native.node_id
|
||||
positive = await asyncio.to_thread(run, store.get_by_id(native.node_id))
|
||||
assert positive.content == record.verbatim
|
||||
await first.profile_memories(user_id=request.user_id, agent_id=request.agent_id)
|
||||
governed = as_person(supervisor.governance).create(
|
||||
MemoryDraft(
|
||||
user_id=request.user_id,
|
||||
kind=MemoryKind.CHAT,
|
||||
content=record.verbatim,
|
||||
scopes=("synthetic-scope",),
|
||||
evidence_ids=(record.evidence_id,),
|
||||
),
|
||||
request_id="synthetic-publication",
|
||||
memory_id=native.node_id,
|
||||
)
|
||||
rejected = as_person(supervisor.governance).delete(
|
||||
governed.memory_id,
|
||||
user_id=request.user_id,
|
||||
request_id="synthetic-rejection",
|
||||
expected_revision=governed.revision,
|
||||
)
|
||||
assert rejected.status == "deleted"
|
||||
# Explicit renewed consent permits the unchanged source text,
|
||||
# but must not reinstate a deleted derived native memory.
|
||||
snapshot = permissions.snapshot(
|
||||
user_id=request.user_id,
|
||||
source_ids=(record.source_id,),
|
||||
profile=lease.profile,
|
||||
evidence=(record,),
|
||||
)
|
||||
permissions.grant(
|
||||
snapshot, user_confirmed=True, expected_revision=permission.revision
|
||||
)
|
||||
next_task = supervisor.prepare_conversation(record)[0]
|
||||
assert next_task.task_id != task.task_id
|
||||
next_lease, next_request, _ = runner._prepare_permission_lease(next_task)
|
||||
assert next_lease.source_context is None
|
||||
assert next_lease.recheck()
|
||||
async with factory(next_lease, None, tmp_path) as second:
|
||||
await second._start(next_request)
|
||||
reused = await asyncio.to_thread(
|
||||
run, second.wiring.vector.get_by_id(native.node_id)
|
||||
)
|
||||
assert reused is None, (
|
||||
"Evidence-only proof reload exposed a deleted governed memory "
|
||||
"as native context in a new permitted task"
|
||||
)
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("rejection_point", ["after_start", "after_proof_load"])
|
||||
def test_deletion_during_proof_binding_blocks_next_physical_send(tmp_path, rejection_point):
|
||||
async def scenario():
|
||||
with whole_client(tmp_path) as (supervisor, record, _, _):
|
||||
task = supervisor.prepare_conversation(record)[0]
|
||||
lease, request, _ = supervisor._create_runner()._prepare_permission_lease(task)
|
||||
bridge = supervisor._bridge(lease.profile, None, tmp_path)
|
||||
client = await bridge._get_client()
|
||||
run = client._loop_thread.run
|
||||
sends = []
|
||||
|
||||
def handler(outbound):
|
||||
body = json.loads(outbound.content)
|
||||
sends.append(body)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"object": "list",
|
||||
"model": "test-embedding",
|
||||
"data": [
|
||||
{
|
||||
"object": "embedding",
|
||||
"index": i,
|
||||
"embedding": [1.0] + [0.0] * 383,
|
||||
}
|
||||
for i, _ in enumerate(body["input"])
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "total_tokens": 1},
|
||||
},
|
||||
)
|
||||
|
||||
factory = permission_bridge_factory(
|
||||
supervisor._bridge, http_transport=httpx.MockTransport(handler)
|
||||
)
|
||||
native = MemoryNode(
|
||||
node_id="synthetic-rejection-race",
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
session_id=request.session_id,
|
||||
content=record.verbatim,
|
||||
owner="agent",
|
||||
layer=MemoryLayer.L2_FACT,
|
||||
embedding=[1.0] + [0.0] * 383,
|
||||
)
|
||||
try:
|
||||
async with factory(lease, None, tmp_path) as first:
|
||||
await first._start(request)
|
||||
await asyncio.to_thread(run, first.wiring.vector.upsert(native))
|
||||
permitted = await asyncio.to_thread(
|
||||
run, first.wiring.vector.get_by_id(native.node_id)
|
||||
)
|
||||
assert permitted is not None
|
||||
await asyncio.to_thread(
|
||||
run, client._embed_service.embed(permitted.content, use_cache=False)
|
||||
)
|
||||
assert len(sends) == 1 and sends[0]["input"] == [record.verbatim]
|
||||
await first.profile_memories(user_id=request.user_id, agent_id=request.agent_id)
|
||||
|
||||
governed = as_person(supervisor.governance).create(
|
||||
MemoryDraft(
|
||||
user_id=request.user_id,
|
||||
kind=MemoryKind.CHAT,
|
||||
content=record.verbatim,
|
||||
scopes=("synthetic-scope",),
|
||||
evidence_ids=(record.evidence_id,),
|
||||
),
|
||||
request_id="synthetic-race-publication",
|
||||
memory_id=native.node_id,
|
||||
)
|
||||
|
||||
def reject():
|
||||
result = as_person(supervisor.governance).delete(
|
||||
governed.memory_id,
|
||||
user_id=request.user_id,
|
||||
request_id="synthetic-race-rejection",
|
||||
expected_revision=governed.revision,
|
||||
)
|
||||
assert result.status == "deleted"
|
||||
|
||||
try:
|
||||
async with factory(lease, None, tmp_path) as second:
|
||||
if rejection_point == "after_proof_load":
|
||||
load = second.proofs.load
|
||||
|
||||
def reject_after_load(current_lease):
|
||||
approved = load(current_lease)
|
||||
assert native.node_id in approved
|
||||
# A governance write can commit between the two
|
||||
# separately awaited load/guard database reads.
|
||||
reject()
|
||||
return approved
|
||||
|
||||
second.proofs.load = reject_after_load
|
||||
await second._start(request)
|
||||
if rejection_point == "after_start":
|
||||
reject()
|
||||
candidate = await asyncio.to_thread(
|
||||
run, second.wiring.vector.get_by_id(native.node_id)
|
||||
)
|
||||
if candidate is not None:
|
||||
await asyncio.to_thread(
|
||||
run,
|
||||
client._embed_service.embed(candidate.content, use_cache=False),
|
||||
)
|
||||
except (HyMemoryBridgeError, HyPermissionTransportError) as error:
|
||||
assert str(error).startswith("model_content_")
|
||||
assert (
|
||||
supervisor.governance.get(native.node_id, user_id="local").status == "deleted"
|
||||
)
|
||||
assert len(sends) == 1, (
|
||||
f"Deleted proof context reached the physical transport at {rejection_point}; "
|
||||
f"expected one approved control send, observed {len(sends)} sends"
|
||||
)
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,834 +0,0 @@
|
||||
"""Real pinned HY SDK/Chroma and guards; scripted HTTP is not model acceptance."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sqlite3
|
||||
import traceback
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from hy_memory.client import HyMemoryClient
|
||||
from hy_memory.data.vector_store_chroma import ChromaVectorStore
|
||||
from hy_memory.models.memory import MemoryLayer, MemoryNode
|
||||
from hy_memory.pipelines._retrieval.reconcile_retrieval import ReconcileHybridRetriever
|
||||
|
||||
from one_memory.hy_permission_context import permission_bridge_factory
|
||||
from one_memory.hy_response_guard import HyReconcileGuard, HyResponseGuard
|
||||
from one_memory.hy_runtime_config import hy_memory_version
|
||||
from one_memory.hy_write_verification import HyExtractionGuard
|
||||
from one_memory.imports.grants import SourceGrantStore
|
||||
from one_memory.imports.models import (
|
||||
EvidenceContentKind,
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
)
|
||||
from one_memory.imports.sources.registry import RootCandidateKind
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import MemoryGovernanceStore
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_tasks import MemoryTaskStore, evidence_ref
|
||||
from one_memory.model_content_permissions import ModelContentPermissionStore
|
||||
from one_memory.model_profiles import (
|
||||
ModelCapabilities,
|
||||
ModelProfileInput,
|
||||
ModelProfileStore,
|
||||
ProviderKind,
|
||||
)
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
|
||||
USER_TEXT = "The user requires a permission review before the synthetic release."
|
||||
AGENT_TEXT = "The agent proposes a checklist for the synthetic release."
|
||||
CLAIMS = {USER_TEXT: "user", AGENT_TEXT: "agent"}
|
||||
DENIED_FACT = "DENIED_NATIVE_FACT_63a1: omit the permission review."
|
||||
DENIED_HISTORY = "DENIED_NATIVE_HISTORY_86b2: publish without asking."
|
||||
DENIED_TAG = "DENIED_NATIVE_TAG_91c3"
|
||||
DENIED_IDS = ("denied-native-fact", "denied-native-history")
|
||||
NOW = datetime(2026, 9, 13, 8, tzinfo=UTC)
|
||||
|
||||
|
||||
def vector(text):
|
||||
# Orthogonal synthetic vectors keep the two claims distinct under native dedup.
|
||||
axis = 1 if text == USER_TEXT else 2 if text == AGENT_TEXT else 0
|
||||
return [float(index == axis) for index in range(384)]
|
||||
|
||||
|
||||
def prompt_json(prompt, heading):
|
||||
payload = prompt_section(prompt, heading)
|
||||
starts = [index for marker in ("[", "{") if (index := payload.find(marker)) >= 0]
|
||||
assert starts, (heading, payload)
|
||||
payload = payload[min(starts) :]
|
||||
try:
|
||||
return json.JSONDecoder().raw_decode(payload)[0]
|
||||
except json.JSONDecodeError as error:
|
||||
raise AssertionError(f"expected JSON after {heading!r}, got {payload[:500]!r}") from error
|
||||
|
||||
|
||||
def prompt_section(prompt: str, heading: str) -> str:
|
||||
assert prompt.count(heading) == 1, heading
|
||||
section = prompt.split(heading, 1)[1]
|
||||
return section.split("\n---\n", 1)[0].strip()
|
||||
|
||||
|
||||
class ScriptedOracle:
|
||||
"""Only the two fixture claims receive support, with exact supplied Evidence IDs."""
|
||||
|
||||
def __init__(self, records, *, after_response=None):
|
||||
self.records = {record.evidence_id: record for record in records}
|
||||
self.requests = []
|
||||
self.stages = []
|
||||
self.reconciles = []
|
||||
self.errors = []
|
||||
self.after_response = after_response
|
||||
self.revoked_at = None
|
||||
self.limit = 24
|
||||
|
||||
def __call__(self, request):
|
||||
try:
|
||||
return self.respond(request)
|
||||
except Exception:
|
||||
self.errors.append(traceback.format_exc())
|
||||
raise
|
||||
|
||||
def respond(self, request):
|
||||
body = json.loads(request.content)
|
||||
self.requests.append((request.url.path, body))
|
||||
assert len(self.requests) <= self.limit, "scripted HTTP response budget exceeded"
|
||||
assert self.revoked_at is None, "native child request escaped after revocation"
|
||||
assert request.method == "POST"
|
||||
serialized = json.dumps(body)
|
||||
for forbidden in (DENIED_FACT, DENIED_HISTORY, DENIED_TAG, *DENIED_IDS):
|
||||
assert forbidden not in serialized, f"denied native context reached HTTP: {forbidden}"
|
||||
if request.url.path == "/v1/embeddings":
|
||||
assert str(request.url) == "http://127.0.0.1:2/v1/embeddings"
|
||||
assert body["model"] == "raw-fixture-embedding"
|
||||
texts = body["input"]
|
||||
assert isinstance(texts, list) and texts
|
||||
lines = [f"[{record.role}]: {record.verbatim}" for record in self.records.values()]
|
||||
# HY stores one raw copy per add, and each add is one turn.
|
||||
turns = {
|
||||
"\n".join(lines[start:end])
|
||||
for start in range(len(lines))
|
||||
for end in range(start + 1, len(lines) + 1)
|
||||
}
|
||||
for text in texts:
|
||||
if text not in CLAIMS:
|
||||
assert text in turns
|
||||
self.stages.append("embedding")
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"object": "list",
|
||||
"model": body["model"],
|
||||
"data": [
|
||||
{"index": i, "object": "embedding", "embedding": vector(text)}
|
||||
for i, text in enumerate(texts)
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "total_tokens": 1},
|
||||
},
|
||||
)
|
||||
assert str(request.url) == "http://127.0.0.1:1/v1/chat/completions"
|
||||
assert body["model"] == "raw-fixture-generation"
|
||||
prompt = body["messages"][-1]["content"]
|
||||
if prompt.startswith("You are a single-label scene classifier."):
|
||||
result = {"is_coding": False, "reason": "synthetic chat"}
|
||||
stage = "judge"
|
||||
elif "## New Messages\n" in prompt:
|
||||
self.check_turns(prompt_json(prompt, "## New Messages\n"))
|
||||
history_section = prompt_section(prompt, "## Last k Messages\n")
|
||||
history = (
|
||||
[] if history_section == "(none)" else prompt_json(prompt, "## Last k Messages\n")
|
||||
)
|
||||
for source in history:
|
||||
record = self.records[source["evidence_id"]]
|
||||
assert source["content"] == record.verbatim
|
||||
assert source["role"] == record.role
|
||||
assert source["revision"] == record.revision
|
||||
assert '"memory"' in prompt
|
||||
assert ('"intentions"' in prompt) is ("never" not in prompt)
|
||||
result = {
|
||||
"memory": [
|
||||
{"content": text, "owner": owner, "tags": []} for text, owner in CLAIMS.items()
|
||||
],
|
||||
"basic_info": {},
|
||||
}
|
||||
if '"intentions"' in prompt:
|
||||
result["intentions"] = []
|
||||
stage = "extract"
|
||||
elif "## Existing memories\n" in prompt:
|
||||
existing = prompt_json(prompt, "## Existing memories\n")
|
||||
incoming = prompt_json(prompt, "## New memories\n")
|
||||
assert existing and incoming
|
||||
for item in (*existing, *incoming):
|
||||
assert CLAIMS[item["content"]] == item["owner"]
|
||||
assert not item.get("tags")
|
||||
self.reconciles.append(existing)
|
||||
by_content = {item["content"]: item for item in existing}
|
||||
result = [
|
||||
{
|
||||
"op": "UPDATE" if item["content"] in by_content else "ADD",
|
||||
"content": item["content"],
|
||||
"owner": item["owner"],
|
||||
"layer": "L2_FACT",
|
||||
"tags": [],
|
||||
**(
|
||||
{"memory_id": by_content[item["content"]]["memory_id"]}
|
||||
if item["content"] in by_content
|
||||
else {}
|
||||
),
|
||||
}
|
||||
for item in incoming
|
||||
]
|
||||
stage = "reconcile"
|
||||
else:
|
||||
if not prompt.lstrip().startswith("{"):
|
||||
raise AssertionError(f"unknown native child prompt: {prompt[:300]!r}")
|
||||
data = json.loads(prompt)
|
||||
if set(data) == {"items", "sources", "before"}:
|
||||
result = self.review(data)
|
||||
stage = "review"
|
||||
else:
|
||||
assert set(data) == {"proofs"}, "unknown native child prompt"
|
||||
result = self.cited_review(data)
|
||||
stage = "citations"
|
||||
self.stages.append(stage)
|
||||
response = httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "synthetic-raw-response",
|
||||
"object": "chat.completion",
|
||||
"created": 1,
|
||||
"model": body["model"],
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"finish_reason": "stop",
|
||||
"message": {"role": "assistant", "content": json.dumps(result)},
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
||||
},
|
||||
)
|
||||
if self.after_response is not None and self.after_response(stage):
|
||||
self.revoked_at = len(self.requests)
|
||||
return response
|
||||
|
||||
def check_turns(self, turns):
|
||||
assert turns and isinstance(turns, list)
|
||||
allowed = {(record.role, record.verbatim) for record in self.records.values()}
|
||||
assert all(set(turn) == {"role", "content"} for turn in turns)
|
||||
assert {(turn["role"], turn["content"]) for turn in turns} <= allowed
|
||||
|
||||
def review(self, data):
|
||||
assert data["items"] and data["sources"]
|
||||
for source in data["sources"]:
|
||||
record = self.records[source["id"]]
|
||||
assert source["role"] == record.role
|
||||
assert source["revision"] == record.revision
|
||||
assert source["created_at"] == record.created_at.isoformat()
|
||||
assert "".join(span["text"] for span in source["spans"]) == record.verbatim
|
||||
reviews = []
|
||||
for item in data["items"]:
|
||||
assert CLAIMS[item["content"]] == item["owner"]
|
||||
role = "user" if item["owner"] == "user" else "assistant"
|
||||
source = next(
|
||||
source
|
||||
for source in data["sources"]
|
||||
if source["role"] == role and self.records[source["id"]].verbatim == item["content"]
|
||||
)
|
||||
reviews.append(
|
||||
{
|
||||
"id": item["id"],
|
||||
"clauses": [
|
||||
{
|
||||
"text": item["content"],
|
||||
"verdict": "supported",
|
||||
"authority": item["owner"],
|
||||
"citations": [
|
||||
{
|
||||
"source_id": source["id"],
|
||||
"span_ids": [span["id"] for span in source["spans"]],
|
||||
}
|
||||
],
|
||||
"reason": "exact synthetic fixture claim",
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
for before in data["before"]:
|
||||
assert any(
|
||||
(item["content"], item["owner"]) == (before["content"], before["owner"])
|
||||
for item in data["items"]
|
||||
)
|
||||
return {
|
||||
"reviews": reviews,
|
||||
"coverage": [
|
||||
{"id": item["id"], "verdict": "preserved", "reason": "exact fixture match"}
|
||||
for item in data["before"]
|
||||
],
|
||||
}
|
||||
|
||||
def cited_review(self, data):
|
||||
assert data["proofs"]
|
||||
for proof in data["proofs"]:
|
||||
assert CLAIMS[proof["claim"]] == proof["owner"]
|
||||
assert proof["citations"] and not proof["context_citations"]
|
||||
for citation in proof["citations"]:
|
||||
record = self.records[citation["source_id"]]
|
||||
assert citation["text"] == record.verbatim == proof["claim"]
|
||||
assert citation["role"] == record.role
|
||||
assert citation["revision"] == record.revision
|
||||
assert citation["created_at"] == record.created_at.isoformat()
|
||||
return {
|
||||
"proofs": [
|
||||
{"id": proof["id"], "verdict": "supported", "reason": "exact fixture match"}
|
||||
for proof in data["proofs"]
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def put_turn(evidence, message_id, role, text, ordinal, *, conversation_id="raw-session"):
|
||||
return evidence.put(
|
||||
EvidenceInput(
|
||||
user_id="local",
|
||||
source_id="raw-fixture",
|
||||
conversation_id=conversation_id,
|
||||
message_id=message_id,
|
||||
source_ordinal=str(ordinal),
|
||||
role=role,
|
||||
written_by={"user": WrittenBy.PERSON, "tool": WrittenBy.TOOL}.get(
|
||||
role, WrittenBy.AGENT
|
||||
),
|
||||
content_kind=EvidenceContentKind.RAW_CONVERSATION,
|
||||
created_at=NOW + timedelta(seconds=ordinal),
|
||||
verbatim=text,
|
||||
origin_meta={EvidenceOrigin.MANUAL_IMPORT: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
|
||||
|
||||
def grant_records(supervisor, records, expected_revision=0):
|
||||
profile = supervisor.profiles.active(supervisor.secrets)
|
||||
snapshot = supervisor.model_content_permissions.snapshot(
|
||||
user_id="local",
|
||||
source_ids=("raw-fixture",),
|
||||
profile=supervisor.profiles.snapshot(profile.profile_id, profile.revision),
|
||||
evidence=records,
|
||||
)
|
||||
return supervisor.model_content_permissions.grant(
|
||||
snapshot, user_confirmed=True, expected_revision=expected_revision
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def raw_client(tmp_path, *, tool=None, conversation_id="raw-session"):
|
||||
assert hy_memory_version() == "2.0.2"
|
||||
root = tmp_path / "source"
|
||||
root.mkdir()
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
grants = SourceGrantStore(database, home=tmp_path)
|
||||
grants.grant(
|
||||
user_id="local",
|
||||
source_id="raw-fixture",
|
||||
root=root,
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
evidence = EvidenceStore(database)
|
||||
records = (
|
||||
put_turn(
|
||||
evidence,
|
||||
"user-original",
|
||||
"user",
|
||||
USER_TEXT,
|
||||
1,
|
||||
conversation_id=conversation_id,
|
||||
),
|
||||
put_turn(
|
||||
evidence,
|
||||
"agent-original",
|
||||
"assistant",
|
||||
AGENT_TEXT,
|
||||
2,
|
||||
conversation_id=conversation_id,
|
||||
),
|
||||
*(
|
||||
(
|
||||
put_turn(
|
||||
evidence,
|
||||
"tool-original",
|
||||
"tool",
|
||||
tool,
|
||||
3,
|
||||
conversation_id=conversation_id,
|
||||
),
|
||||
)
|
||||
if tool
|
||||
else ()
|
||||
),
|
||||
)
|
||||
profiles, secrets = ModelProfileStore(database), InMemorySecretStore()
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Synthetic HTTP only",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:1/v1",
|
||||
model="raw-fixture-generation",
|
||||
embedding_provider_kind=ProviderKind.OLLAMA,
|
||||
embedding_base_url="http://127.0.0.1:2/v1",
|
||||
embedding_model="raw-fixture-embedding",
|
||||
embedding_dimensions=384,
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True, model_available=True, embedding_available=True
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
supervisor = MemoryProcessingSupervisor(
|
||||
tasks=MemoryTaskStore(database),
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=evidence,
|
||||
governance=MemoryGovernanceStore(database),
|
||||
data_dir=tmp_path,
|
||||
model_content_permissions=ModelContentPermissionStore(database),
|
||||
)
|
||||
try:
|
||||
permission = grant_records(supervisor, records)
|
||||
assert supervisor.governance.list(user_id="local") == []
|
||||
yield supervisor, records, permission
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
|
||||
def test_search_after_first_wb20_task_does_not_poison_the_second_task(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, conversation_id="wb-20") as (
|
||||
supervisor,
|
||||
records,
|
||||
permission,
|
||||
):
|
||||
first = supervisor.prepare_conversation(records[0])[0]
|
||||
_, request, _ = supervisor._create_runner()._prepare_permission_lease(first)
|
||||
client = await native_client(supervisor, first)
|
||||
duplicate = MemoryNode(
|
||||
node_id="older-identical-memory",
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
session_id="wb-19",
|
||||
layer=MemoryLayer.L2_FACT,
|
||||
content=USER_TEXT,
|
||||
owner="user",
|
||||
embedding=vector(USER_TEXT),
|
||||
gmt_created=NOW - timedelta(days=1),
|
||||
gmt_modified=NOW - timedelta(days=1),
|
||||
)
|
||||
await on_native_loop(client, client._vector_store.upsert(duplicate))
|
||||
|
||||
first_oracle = ScriptedOracle(records)
|
||||
bind_oracle(supervisor, first_oracle)
|
||||
first_result = await supervisor._create_runner().run(first.task_id)
|
||||
assert first_result.status == "completed", first_result.last_error_code
|
||||
|
||||
current = next(
|
||||
memory
|
||||
for memory in supervisor.governance.list(user_id="local")
|
||||
if memory.content == USER_TEXT
|
||||
)
|
||||
client._spawn_search_dedup(
|
||||
[
|
||||
{
|
||||
"memory_id": duplicate.node_id,
|
||||
"content": duplicate.content,
|
||||
"layer": duplicate.layer.value,
|
||||
"gmt_created": int(duplicate.gmt_created.timestamp()),
|
||||
},
|
||||
{
|
||||
"memory_id": current.memory_id,
|
||||
"content": current.content,
|
||||
"layer": MemoryLayer.L2_FACT.value,
|
||||
"gmt_created": int(NOW.timestamp()),
|
||||
},
|
||||
],
|
||||
request_id="wb-20-search",
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
)
|
||||
await asyncio.sleep(0.2)
|
||||
|
||||
added = put_turn(
|
||||
supervisor.evidence,
|
||||
"wb-20-second",
|
||||
"user",
|
||||
USER_TEXT,
|
||||
3,
|
||||
conversation_id="wb-20",
|
||||
)
|
||||
records = (*records, added)
|
||||
grant_records(supervisor, records, permission.revision)
|
||||
second = supervisor.prepare_conversation(added)[0]
|
||||
second_oracle = ScriptedOracle(records)
|
||||
bind_oracle(supervisor, second_oracle)
|
||||
second_result = await supervisor._create_runner().run(second.task_id)
|
||||
|
||||
assert second_result.status == "completed", second_result.last_error_code
|
||||
assert second_result.last_error_code is None
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
async def native_client(supervisor, task):
|
||||
bridge = supervisor._bridge(task.profile_snapshot, None, supervisor.data_dir)
|
||||
client = await bridge._get_client()
|
||||
assert isinstance(client, HyMemoryClient)
|
||||
assert isinstance(client._vector_store, ChromaVectorStore)
|
||||
guards = client._one_memory_write_guards
|
||||
assert isinstance(guards[0], HyResponseGuard)
|
||||
assert isinstance(guards[2], HyReconcileGuard)
|
||||
assert isinstance(guards[3], HyExtractionGuard)
|
||||
return client
|
||||
|
||||
|
||||
def test_governance_write_runs_after_shared_hy_bridge_is_released(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, _permission):
|
||||
task = supervisor.prepare_conversation(records[0])[0]
|
||||
bind_oracle(supervisor, ScriptedOracle(records))
|
||||
await native_client(supervisor, task)
|
||||
bridge = next(
|
||||
value
|
||||
for key, value in supervisor._bridges.items()
|
||||
if key[:2] == (task.profile_id, task.profile_revision)
|
||||
)
|
||||
runner = supervisor._create_runner()
|
||||
original = runner._record_outputs
|
||||
observed = []
|
||||
|
||||
def checked(*args, **kwargs):
|
||||
observed.append(bridge._access_owner)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
runner._record_outputs = checked
|
||||
result = await runner.run(task.task_id)
|
||||
|
||||
assert result.status == "completed"
|
||||
assert observed == [None], "governance writes must not hold the shared HY bridge"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
async def on_native_loop(client, operation):
|
||||
return await asyncio.to_thread(client._loop_thread.run, operation)
|
||||
|
||||
|
||||
async def seed_denied(client, request):
|
||||
store = client._vector_store
|
||||
nodes = (
|
||||
MemoryNode(
|
||||
node_id=DENIED_IDS[0],
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
session_id=request.session_id,
|
||||
layer=MemoryLayer.L2_FACT,
|
||||
content=DENIED_FACT,
|
||||
owner="user",
|
||||
embedding=vector(USER_TEXT),
|
||||
tags=[DENIED_TAG],
|
||||
),
|
||||
MemoryNode(
|
||||
node_id=DENIED_IDS[1],
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
session_id=request.session_id,
|
||||
layer=MemoryLayer.L1_RAW,
|
||||
content=f"[user]: {DENIED_HISTORY}",
|
||||
owner="user",
|
||||
embedding=vector("raw"),
|
||||
tags=[DENIED_TAG],
|
||||
),
|
||||
)
|
||||
await on_native_loop(client, store.upsert_batch(list(nodes)))
|
||||
retrieved = await on_native_loop(
|
||||
client,
|
||||
store.search(
|
||||
query_embedding=vector(USER_TEXT),
|
||||
user_id=request.user_id,
|
||||
agent_ids=[request.agent_id],
|
||||
layers=[MemoryLayer.L2_FACT],
|
||||
),
|
||||
)
|
||||
assert {hit["node_id"] for hit in retrieved} == {DENIED_IDS[0]}
|
||||
history = await on_native_loop(
|
||||
client,
|
||||
store.list_by_user(
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
layers=[MemoryLayer.L1_RAW],
|
||||
allow_raw_layer=True,
|
||||
),
|
||||
)
|
||||
assert [node.node_id for node in history] == [DENIED_IDS[1]]
|
||||
candidates = await on_native_loop(
|
||||
client,
|
||||
ReconcileHybridRetriever().search_candidates(
|
||||
USER_TEXT,
|
||||
vector(USER_TEXT),
|
||||
vector_store=store,
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
layers=[MemoryLayer.L2_FACT],
|
||||
),
|
||||
)
|
||||
assert [hit["node"].node_id for hit in candidates] == [DENIED_IDS[0]]
|
||||
pipeline = await on_native_loop(client, client._registry.get_system2_writer())
|
||||
native_history = await on_native_loop(
|
||||
client,
|
||||
pipeline._writer._get_recent_history(store, request.user_id, request.agent_id),
|
||||
)
|
||||
native_tags = await on_native_loop(
|
||||
client,
|
||||
pipeline._writer._collect_existing_tags(store, request.user_id, request.agent_id),
|
||||
)
|
||||
assert DENIED_HISTORY in native_history
|
||||
assert DENIED_TAG in native_tags
|
||||
return await node_snapshots(client, DENIED_IDS)
|
||||
|
||||
|
||||
async def node_snapshots(client, identifiers):
|
||||
result = {}
|
||||
for identifier in identifiers:
|
||||
node = await on_native_loop(client, client._vector_store.get_by_id(identifier))
|
||||
assert node is not None, identifier
|
||||
result[identifier] = node.to_dict()
|
||||
return result
|
||||
|
||||
|
||||
def bind_oracle(supervisor, oracle):
|
||||
supervisor.permission_bridge_factory = permission_bridge_factory(
|
||||
supervisor._bridge, http_transport=httpx.MockTransport(oracle)
|
||||
)
|
||||
|
||||
|
||||
async def assert_outputs(supervisor, task, records, client, *, raw_records=None):
|
||||
outputs = supervisor.tasks.outputs(task.task_id)
|
||||
assert len(outputs) == 2
|
||||
memories = supervisor.governance.list(user_id="local")
|
||||
assert {memory.content for memory in memories} == set(CLAIMS)
|
||||
by_content = {memory.content: memory for memory in memories}
|
||||
for record in records[:2]:
|
||||
memory = by_content[record.verbatim]
|
||||
output = next(output for output in outputs if output.memory_id == memory.memory_id)
|
||||
assert output.owner == CLAIMS[record.verbatim]
|
||||
assert output.engine == "hy-memory" and output.engine_version == "2.0.2"
|
||||
assert output.hy_memory_id == memory.memory_id
|
||||
assert output.memory_revision == memory.revision
|
||||
assert output.support_refs == (evidence_ref(record),)
|
||||
assert memory.evidence_ids == (record.evidence_id,)
|
||||
# Usable either way; origin is what keeps an agent report from passing as the person.
|
||||
assert memory.status == "confirmed"
|
||||
assert memory.origin == ("agent" if output.owner == "agent" else "extracted")
|
||||
stored = await on_native_loop(client, client._vector_store.get_by_id(output.hy_memory_id))
|
||||
assert stored is not None
|
||||
assert stored.content == record.verbatim and stored.owner == output.owner
|
||||
for record in records:
|
||||
assert supervisor.evidence.get(record.evidence_id) == record
|
||||
raw = await on_native_loop(
|
||||
client,
|
||||
client._vector_store.list_by_user(
|
||||
user_id="local",
|
||||
layers=[MemoryLayer.L1_RAW],
|
||||
allow_raw_layer=True,
|
||||
),
|
||||
)
|
||||
allowed_raw = [node for node in raw if node.node_id not in DENIED_IDS]
|
||||
assert allowed_raw
|
||||
# One raw copy per turn; together they hold every record.
|
||||
for record in records if raw_records is None else raw_records:
|
||||
line = f"[{record.role}]: {record.verbatim}"
|
||||
assert any(line in node.content for node in allowed_raw), line
|
||||
with sqlite3.connect(supervisor.data_dir / "one-memory.sqlite3") as db:
|
||||
proofs = db.execute("SELECT node_id, evidence_json FROM hy_node_permissions").fetchall()
|
||||
assert {output.hy_memory_id for output in outputs} <= {key for key, _ in proofs}
|
||||
assert not set(DENIED_IDS) & {key for key, _ in proofs}
|
||||
allowed_refs = [evidence_ref(record) for record in records]
|
||||
for _, refs in proofs:
|
||||
assert json.loads(refs) and all(ref in allowed_refs for ref in json.loads(refs))
|
||||
return {output.hy_memory_id for output in outputs}
|
||||
|
||||
|
||||
def record_receipt(test_node, supervisor, task, oracle, *, phase, hidden):
|
||||
test_node.user_properties.append(
|
||||
(
|
||||
phase,
|
||||
json.dumps(
|
||||
{
|
||||
"sdk": hy_memory_version(),
|
||||
"transport": "scripted-http-oracle-not-real-model",
|
||||
"request_count": len(oracle.requests),
|
||||
"response_limit": oracle.limit,
|
||||
"stages": oracle.stages,
|
||||
"oracle_errors": oracle.errors,
|
||||
"requests_after_revocation": len(oracle.requests) - oracle.revoked_at
|
||||
if oracle.revoked_at is not None
|
||||
else None,
|
||||
"hidden_native_fixture_checked": hidden,
|
||||
"status": task.status,
|
||||
"error": task.last_error_code,
|
||||
"outputs": [
|
||||
output.model_dump(mode="json")
|
||||
for output in supervisor.tasks.outputs(task.task_id)
|
||||
],
|
||||
"governance_count": len(supervisor.governance.list(user_id="local")),
|
||||
}
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("hidden", [False, True], ids=["fresh-raw", "denied-native-context"])
|
||||
def test_native_raw_input_runs_extraction_guards_and_governance(tmp_path, hidden, request):
|
||||
test_node = request.node
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, _):
|
||||
tasks = supervisor.prepare_conversation(records[0])
|
||||
assert len(tasks) == 1
|
||||
task = tasks[0]
|
||||
runner = supervisor._create_runner()
|
||||
_, request, _ = runner._prepare_permission_lease(task)
|
||||
client = await native_client(supervisor, task)
|
||||
before = await seed_denied(client, request) if hidden else {}
|
||||
oracle = ScriptedOracle(records)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(task.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
if hidden:
|
||||
assert await node_snapshots(client, DENIED_IDS) == before
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
assert oracle.stages.count("extract") == 1
|
||||
assert "review" not in oracle.stages and "citations" not in oracle.stages
|
||||
assert "embedding" in oracle.stages
|
||||
assert "reconcile" not in oracle.stages
|
||||
await assert_outputs(supervisor, task, records, client)
|
||||
record_receipt(test_node, supervisor, result, oracle, phase="raw", hidden=hidden)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_tool_turn_hy_judges_as_chat_is_extracted_without_its_tool_messages(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, tool="[Bash] pytest -q\n12 passed") as (
|
||||
supervisor,
|
||||
records,
|
||||
_,
|
||||
):
|
||||
tasks = supervisor.prepare_conversation(records[0])
|
||||
assert len(tasks) == 1
|
||||
oracle = ScriptedOracle(records)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(tasks[0].task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
assert oracle.stages[0] == "judge"
|
||||
prompt = next(
|
||||
body["messages"][-1]["content"]
|
||||
for _path, body in oracle.requests
|
||||
if "messages" in body and "## New Messages\n" in body["messages"][-1]["content"]
|
||||
)
|
||||
turns = prompt_json(prompt, "## New Messages\n")
|
||||
assert [(turn["role"], turn["content"]) for turn in turns] == [
|
||||
(record.role, record.verbatim) for record in records[:2]
|
||||
]
|
||||
client = await native_client(supervisor, tasks[0])
|
||||
await assert_outputs(supervisor, tasks[0], records, client, raw_records=records[:2])
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_revoke_during_native_extractor_response_stops_children_and_governance(tmp_path, request):
|
||||
test_node = request.node
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, permission):
|
||||
task = supervisor.prepare_conversation(records[0])[0]
|
||||
_, request, _ = supervisor._create_runner()._prepare_permission_lease(task)
|
||||
client = await native_client(supervisor, task)
|
||||
before = await seed_denied(client, request)
|
||||
|
||||
def revoke(stage):
|
||||
if stage != "extract":
|
||||
return False
|
||||
supervisor.model_content_permissions.revoke(
|
||||
user_id="local", expected_revision=permission.revision, user_confirmed=True
|
||||
)
|
||||
return True
|
||||
|
||||
oracle = ScriptedOracle(records, after_response=revoke)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(task.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
assert result.status == "cancelled"
|
||||
assert oracle.revoked_at is not None and len(oracle.requests) == oracle.revoked_at
|
||||
assert oracle.stages.count("extract") == 1
|
||||
assert "review" not in oracle.stages and "citations" not in oracle.stages
|
||||
assert not supervisor.governance.list(user_id="local")
|
||||
assert not supervisor.tasks.outputs(task.task_id)
|
||||
assert await node_snapshots(client, DENIED_IDS) == before
|
||||
for record in records:
|
||||
assert supervisor.evidence.get(record.evidence_id) == record
|
||||
record_receipt(test_node, supervisor, result, oracle, phase="revoked", hidden=True)
|
||||
assert result.last_error_code == "model_content_permission_revoked", result
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_approved_native_candidates_are_reused_without_disclosing_denied_nodes(tmp_path, request):
|
||||
test_node = request.node
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, permission):
|
||||
first = supervisor.prepare_conversation(records[0])[0]
|
||||
_, request, _ = supervisor._create_runner()._prepare_permission_lease(first)
|
||||
client = await native_client(supervisor, first)
|
||||
before = await seed_denied(client, request)
|
||||
first_oracle = ScriptedOracle(records)
|
||||
bind_oracle(supervisor, first_oracle)
|
||||
first_result = await supervisor._create_runner().run(first.task_id)
|
||||
assert not first_oracle.errors, first_oracle.errors
|
||||
assert first_result.status == "completed", first_result.last_error_code
|
||||
ids = await assert_outputs(supervisor, first, records, client)
|
||||
record_receipt(
|
||||
test_node, supervisor, first_result, first_oracle, phase="initial", hidden=True
|
||||
)
|
||||
added = put_turn(supervisor.evidence, "user-repeat", "user", USER_TEXT, 3)
|
||||
records = (*records, added)
|
||||
grant_records(supervisor, records, permission.revision)
|
||||
second = supervisor.prepare_conversation(added)[0]
|
||||
assert second.task_id != first.task_id
|
||||
oracle = ScriptedOracle(records)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(second.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
assert await node_snapshots(client, DENIED_IDS) == before
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
# SDK 2.0.2 keeps the `limit=1` pre-check removed, so
|
||||
# skip reconcile, so the second write now compares against the
|
||||
# already-approved candidates instead of blindly ADDing them. The
|
||||
# contract is the reuse itself, not the absence of the call: one
|
||||
# reconcile pass, the returning ids are the ones already governed,
|
||||
# and no third/fourth memory appears.
|
||||
assert oracle.stages.count("reconcile") == 1
|
||||
assert {item["memory_id"] for item in oracle.reconciles[0]} == ids
|
||||
assert await assert_outputs(supervisor, second, records, client) == ids
|
||||
assert len(supervisor.governance.list(user_id="local")) == 2
|
||||
record_receipt(test_node, supervisor, result, oracle, phase="reuse", hidden=True)
|
||||
|
||||
asyncio.run(scenario())
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -69,7 +69,7 @@ def test_no_idempotency_key_means_unknown_must_remain() -> None:
|
||||
|
||||
|
||||
def test_measured_sdk_verdict_is_recorded() -> None:
|
||||
"""Pin the measured answer for `hy-memory==2.0.2`.
|
||||
"""Pin the measured answer for `hy-memory==2.0.3`.
|
||||
|
||||
The finding: `add()` takes a `request_id` but nothing de-duplicates on it,
|
||||
while `MemoryInspector.get_pipeline_logs(request_id=...)` can look the
|
||||
|
||||
@@ -1,547 +0,0 @@
|
||||
"""RC4:记忆生命周期(有效 / 被取代 / 去重隐藏 / 过期)直接读 HY,产品账本不再逐类镜像。
|
||||
|
||||
产品只保留 HY 没有的三样:使用范围、来源撤权后的隐藏过滤、用户删除记录。
|
||||
这里钉住:HY 自己改了状态(导入写回从未见过的也算),读路径立刻按 HY 走;
|
||||
HY 读不到时读路径退回账本,而不是把"没读到"当成"全部有效"。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from hy_memory.models.memory import MemoryStatus as HyStatus
|
||||
|
||||
from one_memory.hy_lifecycle import HyLifecycle, HyLifecycleUnreadable
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridge
|
||||
from one_memory.imports.grants import SourceGrantStore
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.imports.sources.registry import RootCandidateKind
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import (
|
||||
HyCreate,
|
||||
MemoryDraft,
|
||||
MemoryGovernanceStore,
|
||||
MemoryKind,
|
||||
MemoryMutationConflictError,
|
||||
MemoryStatus,
|
||||
)
|
||||
from one_memory.memory_library import read_memory_library
|
||||
from tests.person_door import as_person
|
||||
from tests.test_hy_permission_raw_context import (
|
||||
AGENT_TEXT,
|
||||
USER_TEXT,
|
||||
ScriptedOracle,
|
||||
assert_outputs,
|
||||
bind_oracle,
|
||||
native_client,
|
||||
on_native_loop,
|
||||
raw_client,
|
||||
)
|
||||
|
||||
FACT = "用户的发布前检查要求一次权限评审"
|
||||
|
||||
|
||||
def _evidence(database: Path, message_id: str, *, conversation: str = "conversation") -> str:
|
||||
return (
|
||||
EvidenceStore(database)
|
||||
.put(
|
||||
EvidenceInput(
|
||||
source_id="cursor",
|
||||
conversation_id=conversation,
|
||||
message_id=message_id,
|
||||
correlation_id=message_id,
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim=f"原话 {message_id}",
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
)
|
||||
.record.evidence_id
|
||||
)
|
||||
|
||||
|
||||
def _library(tmp_path: Path) -> tuple[Path, MemoryGovernanceStore, dict[str, str]]:
|
||||
"""Four memories, newest last: a, b, c, d. Each rests on its own evidence."""
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
SourceGrantStore(database, home=tmp_path)
|
||||
store = MemoryGovernanceStore(database)
|
||||
ids: dict[str, str] = {}
|
||||
for key in ("a", "b", "c", "d"):
|
||||
evidence_id = _evidence(database, key)
|
||||
as_person(store).create(
|
||||
MemoryDraft(
|
||||
user_id="local",
|
||||
kind=MemoryKind.CHAT,
|
||||
content=f"{FACT} {key}",
|
||||
scopes=("all-apps",),
|
||||
evidence_ids=(evidence_id,),
|
||||
),
|
||||
request_id=f"create-{key}",
|
||||
memory_id=key,
|
||||
)
|
||||
ids[key] = evidence_id
|
||||
return database, store, ids
|
||||
|
||||
|
||||
def _ids(memories) -> list[str]:
|
||||
return [memory.memory_id for memory in memories]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 账本读路径
|
||||
|
||||
|
||||
def test_what_hy_hides_leaves_every_current_read_and_shows_hy_state_by_id(tmp_path) -> None:
|
||||
_, store, _ = _library(tmp_path)
|
||||
# HY deduplicated d into b during a search, and replaced a with c.
|
||||
lifecycle = HyLifecycle(hidden={"d": ("b",), "a": ("c",)})
|
||||
|
||||
assert _ids(store.list(user_id="local")) == ["d", "c", "b", "a"]
|
||||
assert _ids(store.list(user_id="local", lifecycle=lifecycle)) == ["c", "b"]
|
||||
found, more = store.find_confirmed_matching_terms(
|
||||
user_id="local", terms=(FACT,), lifecycle=lifecycle
|
||||
)
|
||||
assert _ids(found) == ["c", "b"] and more is False
|
||||
|
||||
hidden = store.get("d", user_id="local", lifecycle=lifecycle)
|
||||
assert hidden.status is MemoryStatus.SUPERSEDED
|
||||
assert hidden.superseded_by == ("b",) and hidden.deleted_at is None
|
||||
kept = store.get("b", user_id="local", lifecycle=lifecycle)
|
||||
assert kept.status is MemoryStatus.CONFIRMED and kept.supersedes == ("d",)
|
||||
assert _ids(
|
||||
store.list(user_id="local", status=MemoryStatus.SUPERSEDED, lifecycle=lifecycle)
|
||||
) == [
|
||||
"d",
|
||||
"a",
|
||||
]
|
||||
assert _ids(
|
||||
store.list(user_id="local", status=MemoryStatus.CONFIRMED, lifecycle=lifecycle)
|
||||
) == [
|
||||
"c",
|
||||
"b",
|
||||
]
|
||||
|
||||
|
||||
def test_hidden_rows_are_excluded_before_the_page_is_cut(tmp_path) -> None:
|
||||
_, store, _ = _library(tmp_path)
|
||||
lifecycle = HyLifecycle(hidden={"d": ("b",), "c": ("b",)})
|
||||
|
||||
assert _ids(store.list(user_id="local", limit=1, lifecycle=lifecycle)) == ["b"]
|
||||
assert _ids(store.list(user_id="local", limit=1, offset=1, lifecycle=lifecycle)) == ["a"]
|
||||
found, more = store.find_confirmed_matching_terms(
|
||||
user_id="local", terms=(FACT,), limit=1, lifecycle=lifecycle
|
||||
)
|
||||
assert _ids(found) == ["b"] and more is True
|
||||
|
||||
|
||||
def test_a_user_deletion_stays_a_deletion_whatever_hy_says(tmp_path) -> None:
|
||||
_, store, _ = _library(tmp_path)
|
||||
b = store.get("b", user_id="local")
|
||||
as_person(store).delete(
|
||||
"b", user_id="local", request_id="delete-b", expected_revision=b.revision
|
||||
)
|
||||
# HY still holds b as active until the deletion outbox reaches it.
|
||||
lifecycle = HyLifecycle(hidden={})
|
||||
|
||||
assert "b" not in _ids(store.list(user_id="local", lifecycle=lifecycle))
|
||||
assert store.get("b", user_id="local", lifecycle=lifecycle).status is MemoryStatus.DELETED
|
||||
assert _ids(store.list(user_id="local", status=MemoryStatus.DELETED, lifecycle=lifecycle)) == [
|
||||
"b"
|
||||
]
|
||||
|
||||
|
||||
def test_hy_restoring_a_memory_it_had_replaced_brings_it_back(tmp_path) -> None:
|
||||
"""A row an older build mirrored as superseded is current again once HY says so.
|
||||
|
||||
HY restores the predecessor when the head of a chain is deleted; no write
|
||||
reaches the ledger for that, so only reading HY can see it.
|
||||
"""
|
||||
from one_memory.memory_governance import HySupersede
|
||||
|
||||
_, store, _ = _library(tmp_path)
|
||||
a = store.get("a", user_id="local")
|
||||
store.write_from_hy(
|
||||
HySupersede(
|
||||
memory_id="a",
|
||||
request_id="hy-supersede:t:a",
|
||||
expected_revision=a.revision,
|
||||
superseded_by=("c",),
|
||||
),
|
||||
user_id="local",
|
||||
derivation=store.hy_derivation(),
|
||||
)
|
||||
assert "a" not in _ids(store.list(user_id="local"))
|
||||
|
||||
restored = HyLifecycle(hidden={})
|
||||
assert "a" in _ids(store.list(user_id="local", lifecycle=restored))
|
||||
memory = store.get("a", user_id="local", lifecycle=restored)
|
||||
assert memory.status is MemoryStatus.CONFIRMED and memory.superseded_by == ()
|
||||
assert store.get("c", user_id="local", lifecycle=restored).supersedes == ()
|
||||
|
||||
|
||||
def test_without_a_snapshot_the_ledger_reads_as_before(tmp_path) -> None:
|
||||
_, store, _ = _library(tmp_path)
|
||||
|
||||
assert _ids(store.list(user_id="local", lifecycle=None)) == ["d", "c", "b", "a"]
|
||||
assert store.get("d", user_id="local", lifecycle=None).status is MemoryStatus.CONFIRMED
|
||||
|
||||
|
||||
def test_a_superseded_memory_does_not_come_back_after_reopen(tmp_path) -> None:
|
||||
"""Reopening the library may not resurrect what HY already replaced.
|
||||
|
||||
The ledger's own mirrored supersession is the floor that holds while HY is
|
||||
not open; a second read path must not treat "no snapshot yet" as "active".
|
||||
"""
|
||||
from one_memory.memory_governance import HySupersede
|
||||
|
||||
database, store, _ = _library(tmp_path)
|
||||
a = store.get("a", user_id="local")
|
||||
store.write_from_hy(
|
||||
HySupersede(
|
||||
memory_id="a",
|
||||
request_id="hy-supersede:reopen:a",
|
||||
expected_revision=a.revision,
|
||||
superseded_by=("c",),
|
||||
),
|
||||
user_id="local",
|
||||
derivation=store.hy_derivation(),
|
||||
)
|
||||
reopened = MemoryGovernanceStore(database)
|
||||
assert "a" not in _ids(reopened.list(user_id="local"))
|
||||
page = read_memory_library(database, limit=100)
|
||||
assert "a" not in [item.memory_id for item in page.items]
|
||||
|
||||
|
||||
def test_revoked_source_stays_hidden_even_with_a_stale_lifecycle_snapshot(tmp_path) -> None:
|
||||
"""Authority outranks an old HY snapshot: no snapshot may re-expose it."""
|
||||
database, store, _ = _library(tmp_path)
|
||||
stale = HyLifecycle(hidden={})
|
||||
assert _ids(store.list(user_id="local", lifecycle=stale)) == ["d", "c", "b", "a"]
|
||||
|
||||
grants = SourceGrantStore(database, home=tmp_path)
|
||||
root = tmp_path / "cursor-root"
|
||||
root.mkdir()
|
||||
grants.grant(
|
||||
user_id="local",
|
||||
source_id="cursor",
|
||||
root=root,
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
grants.revoke(user_id="local", source_id="cursor")
|
||||
# Documented contract: the ledger itself still lists the rows it owns.
|
||||
# Revocation is enforced by the read projections and external authority,
|
||||
# so a stale lifecycle snapshot must not re-expose them there.
|
||||
assert _ids(store.list(user_id="local", lifecycle=stale)) == ["d", "c", "b", "a"]
|
||||
page = read_memory_library(database, source="cursor", limit=100, lifecycle=stale)
|
||||
assert page.total == 0
|
||||
|
||||
|
||||
def test_revocation_survives_reopen_and_keeps_the_ledger_intact(tmp_path) -> None:
|
||||
"""Revocation hides reads; it does not delete, and it must outlive a restart.
|
||||
|
||||
The governance store keeps its row by design, so a downstream consumer has
|
||||
to read through authority and must not treat the surviving row as current.
|
||||
"""
|
||||
database, _, _ = _library(tmp_path)
|
||||
grants = SourceGrantStore(database, home=tmp_path)
|
||||
root = tmp_path / "cursor-root"
|
||||
root.mkdir()
|
||||
grants.grant(
|
||||
user_id="local",
|
||||
source_id="cursor",
|
||||
root=root,
|
||||
candidate_kind=RootCandidateKind.DEFAULT,
|
||||
)
|
||||
grants.revoke(user_id="local", source_id="cursor")
|
||||
|
||||
reopened = MemoryGovernanceStore(database)
|
||||
receipt = SourceGrantStore(database, home=tmp_path).receipt(user_id="local", source_id="cursor")
|
||||
|
||||
assert receipt is not None and receipt.revoked_at is not None
|
||||
# Rows survive; they are simply not readable through this source.
|
||||
assert len(reopened.list(user_id="local")) == 4
|
||||
assert read_memory_library(database, source="cursor", limit=100).total == 0
|
||||
|
||||
|
||||
def test_the_memory_library_page_and_total_follow_hy(tmp_path) -> None:
|
||||
database, _, _ = _library(tmp_path)
|
||||
lifecycle = HyLifecycle(hidden={"d": ("b",)})
|
||||
|
||||
page = read_memory_library(database, lifecycle=lifecycle)
|
||||
assert _ids(page.items) == ["c", "b", "a"] and page.total == 3
|
||||
replaced = read_memory_library(database, status=MemoryStatus.SUPERSEDED, lifecycle=lifecycle)
|
||||
assert _ids(replaced.items) == ["d"] and replaced.total == 1
|
||||
assert replaced.items[0].superseded_by == ("b",)
|
||||
active = read_memory_library(database, active_only=True, lifecycle=lifecycle, limit=1)
|
||||
assert _ids(active.items) == ["c"] and active.total == 3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 删除语义
|
||||
|
||||
|
||||
def test_a_new_conversation_saying_a_deleted_fact_is_remembered_again(tmp_path) -> None:
|
||||
"""Deleting removes that memory, not the fact: no content tombstone.
|
||||
|
||||
Only the exact old evidence stays guarded (RC10); a different conversation
|
||||
saying the same thing later is a new statement.
|
||||
"""
|
||||
database, store, _ = _library(tmp_path)
|
||||
a = store.get("a", user_id="local")
|
||||
as_person(store).delete(
|
||||
"a", user_id="local", request_id="delete-a", expected_revision=a.revision
|
||||
)
|
||||
again = _evidence(database, "again", conversation="later-conversation")
|
||||
|
||||
created = store.write_from_hy(
|
||||
HyCreate(
|
||||
draft=MemoryDraft(
|
||||
user_id="local",
|
||||
kind=MemoryKind.CHAT,
|
||||
content=a.content,
|
||||
scopes=("all-apps",),
|
||||
evidence_ids=(again,),
|
||||
),
|
||||
request_id="extract:later:new-a",
|
||||
memory_id="new-a",
|
||||
),
|
||||
user_id="local",
|
||||
derivation=store.hy_derivation((again,)),
|
||||
)
|
||||
|
||||
assert created.status is MemoryStatus.CONFIRMED and created.content == a.content
|
||||
assert "new-a" in _ids(store.list(user_id="local"))
|
||||
assert store.blocked_source_ids(user_id="local", evidence_ids=(again,)) == set()
|
||||
|
||||
|
||||
def test_the_store_uses_its_lifecycle_source_for_reads_and_person_edits(tmp_path) -> None:
|
||||
_, store, _ = _library(tmp_path)
|
||||
hidden = HyLifecycle(hidden={"d": ("b",)})
|
||||
calls: list[tuple[str, tuple[str, ...] | None]] = []
|
||||
|
||||
def source(user_id: str, ids: tuple[str, ...] | None = None) -> HyLifecycle:
|
||||
calls.append((user_id, ids))
|
||||
return hidden
|
||||
|
||||
store.read_lifecycle_from(source)
|
||||
assert _ids(store.list(user_id="local")) == ["c", "b", "a"]
|
||||
current = store.get("d", user_id="local")
|
||||
assert current is not None and current.status is MemoryStatus.SUPERSEDED
|
||||
assert calls == [("local", None), ("local", ("d",))]
|
||||
# Editing what HY has set aside would fork the chain it already replaced.
|
||||
with pytest.raises(MemoryMutationConflictError, match="memory_superseded"):
|
||||
as_person(store).correct(
|
||||
"d",
|
||||
user_id="local",
|
||||
request_id="correct-hidden",
|
||||
expected_revision=current.revision,
|
||||
content=f"{FACT} corrected",
|
||||
)
|
||||
|
||||
|
||||
def test_a_rejected_or_missing_source_answers_degraded_not_all_active(tmp_path) -> None:
|
||||
"""Wiring contract for C0/C4: unreadable means unavailable, never 'active'.
|
||||
|
||||
A source that raises must be visible as an error to the composition root; a
|
||||
source that answers `None` keeps the ledger floor rather than inventing
|
||||
freshness. Neither may be read as 'HY says everything is current'.
|
||||
"""
|
||||
_, store, _ = _library(tmp_path)
|
||||
|
||||
def broken(_user_id: str, _ids: tuple[str, ...] | None = None) -> HyLifecycle:
|
||||
raise HyLifecycleUnreadable("hy_lifecycle_unreadable")
|
||||
|
||||
store.read_lifecycle_from(broken)
|
||||
with pytest.raises(HyLifecycleUnreadable):
|
||||
store.list(user_id="local")
|
||||
with pytest.raises(HyLifecycleUnreadable):
|
||||
store.get("d", user_id="local")
|
||||
|
||||
# Before HY is open, `None` keeps the ledger's own supersession floor:
|
||||
# a memory HY replaced stays hidden because the write was observed here.
|
||||
from one_memory.memory_governance import HySupersede
|
||||
|
||||
_, store, _ = _library(tmp_path)
|
||||
a = store.get("a", user_id="local")
|
||||
store.write_from_hy(
|
||||
HySupersede(
|
||||
memory_id="a",
|
||||
request_id="hy-supersede:floor:a",
|
||||
expected_revision=a.revision,
|
||||
superseded_by=("c",),
|
||||
),
|
||||
user_id="local",
|
||||
derivation=store.hy_derivation(),
|
||||
)
|
||||
store.read_lifecycle_from(lambda _user_id, _ids=None: None)
|
||||
assert "a" not in _ids(store.list(user_id="local"))
|
||||
|
||||
|
||||
def test_one_snapshot_per_page_not_one_per_row(tmp_path) -> None:
|
||||
"""Wiring contract for C4: a page read asks HY once, not once per row."""
|
||||
_, store, _ = _library(tmp_path)
|
||||
calls: list[tuple[str, tuple[str, ...] | None]] = []
|
||||
|
||||
def source(user_id: str, ids: tuple[str, ...] | None = None) -> HyLifecycle:
|
||||
calls.append((user_id, ids))
|
||||
return HyLifecycle(hidden={"d": ("b",)})
|
||||
|
||||
store.read_lifecycle_from(source)
|
||||
assert len(store.list(user_id="local")) == 3
|
||||
assert calls == [("local", None)]
|
||||
|
||||
|
||||
def test_an_unreadable_lifecycle_blocks_edits_but_not_a_local_delete(tmp_path) -> None:
|
||||
_, store, _ = _library(tmp_path)
|
||||
current = store.get("d", user_id="local", lifecycle=None)
|
||||
assert current is not None
|
||||
|
||||
def unreadable(_user_id: str, _ids: tuple[str, ...] | None = None) -> HyLifecycle:
|
||||
raise HyLifecycleUnreadable("hy_lifecycle_unreadable")
|
||||
|
||||
store.read_lifecycle_from(unreadable)
|
||||
with pytest.raises(HyLifecycleUnreadable):
|
||||
as_person(store).correct(
|
||||
"d",
|
||||
user_id="local",
|
||||
request_id="correct-unreadable",
|
||||
expected_revision=current.revision,
|
||||
content=f"{FACT} corrected",
|
||||
)
|
||||
|
||||
deleted = as_person(store).delete(
|
||||
"d",
|
||||
user_id="local",
|
||||
request_id="delete-unreadable",
|
||||
expected_revision=current.revision,
|
||||
)
|
||||
assert deleted.status is MemoryStatus.DELETED
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 真 HY
|
||||
|
||||
|
||||
async def _first_turn(supervisor, records):
|
||||
first = supervisor.prepare_conversation(records[0])[0]
|
||||
client = await native_client(supervisor, first)
|
||||
bind_oracle(supervisor, ScriptedOracle(records))
|
||||
assert (await supervisor._create_runner().run(first.task_id)).status == "completed"
|
||||
await assert_outputs(supervisor, first, records, client)
|
||||
by_content = {m.content: m for m in supervisor.governance.list(user_id="local")}
|
||||
bridge = supervisor._bridge(first.profile_snapshot, None, supervisor.data_dir)
|
||||
return client, bridge, by_content[USER_TEXT], by_content[AGENT_TEXT]
|
||||
|
||||
|
||||
def test_hy_search_dedup_is_read_straight_from_hy(tmp_path) -> None:
|
||||
"""HY shadows a duplicate while searching; no ingest write-back ever sees it."""
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, _):
|
||||
client, bridge, dup, kept = await _first_turn(supervisor, records)
|
||||
assert isinstance(bridge, HyMemoryBridge)
|
||||
assert bridge.lifecycle_now("local") == HyLifecycle(hidden={})
|
||||
|
||||
await on_native_loop(
|
||||
client,
|
||||
client._vector_store.update_payload(
|
||||
dup.memory_id,
|
||||
{
|
||||
"status": HyStatus.SHADOW.value,
|
||||
"is_latest": False,
|
||||
"shadow_reason": "dedup:search",
|
||||
"dedup_kept_id": kept.memory_id,
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
lifecycle = bridge.lifecycle_now("local")
|
||||
assert lifecycle == HyLifecycle(hidden={dup.memory_id: (kept.memory_id,)})
|
||||
assert _ids(supervisor.governance.list(user_id="local", lifecycle=lifecycle)) == [
|
||||
kept.memory_id
|
||||
]
|
||||
assert bridge.lifecycle_now("someone-else") == HyLifecycle(hidden={})
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_replacement_hy_records_on_the_old_node_is_read_from_hy(tmp_path) -> None:
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, _):
|
||||
client, bridge, old, head = await _first_turn(supervisor, records)
|
||||
await on_native_loop(
|
||||
client,
|
||||
client._vector_store.update_payload(
|
||||
old.memory_id,
|
||||
{
|
||||
"status": HyStatus.SUPERSEDED.value,
|
||||
"is_latest": False,
|
||||
"superseded_by": [head.memory_id],
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
lifecycle = bridge.lifecycle_now("local")
|
||||
assert lifecycle == HyLifecycle(hidden={old.memory_id: (head.memory_id,)})
|
||||
memory = supervisor.governance.get(old.memory_id, user_id="local", lifecycle=lifecycle)
|
||||
assert memory.status is MemoryStatus.SUPERSEDED
|
||||
assert memory.superseded_by == (head.memory_id,)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_an_unreadable_hy_store_raises_instead_of_reading_as_all_active(tmp_path) -> None:
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, _):
|
||||
client, bridge, dup, kept = await _first_turn(supervisor, records)
|
||||
await on_native_loop(
|
||||
client,
|
||||
client._vector_store.update_payload(
|
||||
dup.memory_id,
|
||||
{"status": HyStatus.SHADOW.value, "is_latest": False},
|
||||
),
|
||||
)
|
||||
# Positive first: this store does hold something hidden.
|
||||
assert bridge.lifecycle_now("local") == HyLifecycle(hidden={dup.memory_id: ()})
|
||||
|
||||
collection = client._vector_store._collection
|
||||
|
||||
def broken(*args, **kwargs):
|
||||
raise RuntimeError("chroma is gone")
|
||||
|
||||
original = collection.get
|
||||
collection.get = broken
|
||||
try:
|
||||
with pytest.raises(HyLifecycleUnreadable):
|
||||
bridge.lifecycle_now("local")
|
||||
finally:
|
||||
collection.get = original
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_hy_not_opened_yet_is_only_safe_when_no_store_exists(tmp_path) -> None:
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, _):
|
||||
first = supervisor.prepare_conversation(records[0])[0]
|
||||
bridge = supervisor._bridge(first.profile_snapshot, None, supervisor.data_dir)
|
||||
assert bridge.lifecycle_now("local") is None
|
||||
|
||||
(supervisor.data_dir / "hy-memory" / "governed-v2" / "graph").mkdir(parents=True)
|
||||
with pytest.raises(HyLifecycleUnreadable, match="hy_lifecycle_not_ready"):
|
||||
bridge.lifecycle_now("local")
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_malformed_hy_lifecycle_rows_fail_closed(tmp_path) -> None:
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, _):
|
||||
client, bridge, _dup, _kept = await _first_turn(supervisor, records)
|
||||
collection = client._vector_store._collection
|
||||
original = collection.get
|
||||
collection.get = lambda **_kwargs: {"ids": ["missing-metadata"], "metadatas": []}
|
||||
try:
|
||||
with pytest.raises(HyLifecycleUnreadable, match="hy_lifecycle_store_mismatch"):
|
||||
bridge.lifecycle_now("local")
|
||||
finally:
|
||||
collection.get = original
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,145 +0,0 @@
|
||||
"""Where an HY write spends the time that is not a model call."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
|
||||
import httpx
|
||||
|
||||
from one_memory.diagnostic_log import configure_diagnostics, reset_diagnostics
|
||||
from one_memory.hy_permission_context import permission_bridge_factory
|
||||
from one_memory.hy_step_timing import HyStepTimer
|
||||
from tests.test_hy_permission_context import whole_client
|
||||
|
||||
|
||||
def _handler(record):
|
||||
def handler(request: httpx.Request):
|
||||
body = json.loads(request.content)
|
||||
if request.url.path.endswith("embeddings"):
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"object": "list",
|
||||
"model": "test-embedding",
|
||||
"data": [
|
||||
{"index": i, "object": "embedding", "embedding": [1.0] + [0.0] * 383}
|
||||
for i, _ in enumerate(body["input"])
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "total_tokens": 1},
|
||||
},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "fixture-response",
|
||||
"object": "chat.completion",
|
||||
"created": 1,
|
||||
"model": "test-generation",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"finish_reason": "stop",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": json.dumps(
|
||||
[
|
||||
{
|
||||
"op": "ADD",
|
||||
"content": record.verbatim,
|
||||
"layer": "L2_FACT",
|
||||
"owner": "agent",
|
||||
}
|
||||
]
|
||||
),
|
||||
},
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
||||
},
|
||||
)
|
||||
|
||||
return handler
|
||||
|
||||
|
||||
def test_leased_ingest_logs_each_hy_step_with_duration_and_count(tmp_path, monkeypatch):
|
||||
# Per-step tracing is its own tier on top of the master switch, so a test
|
||||
# that reads the step events has to ask for both.
|
||||
monkeypatch.setenv("ONE_MEMORY_DIAGNOSTICS", "1")
|
||||
monkeypatch.setenv("ONE_MEMORY_HY_STEP_TRACE", "1")
|
||||
receipt = configure_diagnostics(tmp_path / "diagnostics")
|
||||
|
||||
async def scenario():
|
||||
with whole_client(tmp_path) as (supervisor, record, _, _):
|
||||
supervisor.permission_bridge_factory = permission_bridge_factory(
|
||||
supervisor._bridge, http_transport=httpx.MockTransport(_handler(record))
|
||||
)
|
||||
tasks = supervisor.prepare_conversation(record)
|
||||
try:
|
||||
result = await supervisor._create_runner().run(tasks[0].task_id)
|
||||
assert result.status == "completed", result.last_error_code
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
return record, tasks[0].task_id
|
||||
|
||||
try:
|
||||
record, task_id = asyncio.run(scenario())
|
||||
finally:
|
||||
reset_diagnostics()
|
||||
written = receipt.path.read_text()
|
||||
events = [json.loads(line) for line in written.splitlines()]
|
||||
steps = {
|
||||
event["details"]["phase"]: event
|
||||
for event in events
|
||||
if event["event"] == "hy.step.completed"
|
||||
}
|
||||
# Memory content with no prior candidate takes HY's no-model ADD path.
|
||||
assert "model" not in steps
|
||||
expected = {
|
||||
"embedding": "hy.embedding",
|
||||
"vector_store_read": "hy.vector_store",
|
||||
"vector_store_write": "hy.vector_store",
|
||||
"permission_check": "hy.permission_store",
|
||||
"permission_store_read": "hy.permission_store",
|
||||
"permission_store_write": "hy.permission_store",
|
||||
"write_verification": "hy.write_verification",
|
||||
"reconcile": "hy.reconcile",
|
||||
}
|
||||
assert expected.items() <= {name: event["component"] for name, event in steps.items()}.items()
|
||||
for event in steps.values():
|
||||
assert event["task_id"] == task_id
|
||||
assert event["request_id"]
|
||||
assert isinstance(event["duration_ms"], int)
|
||||
assert event["details"]["call_count"] >= 1
|
||||
assert event["details"]["summed_ms"] + 1 >= event["duration_ms"]
|
||||
summaries = [event for event in events if event["event"] == "hy.steps.summary"]
|
||||
assert len(summaries) == 1
|
||||
summary = summaries[0]
|
||||
assert summary["component"] == "hy.bridge"
|
||||
assert summary["task_id"] == task_id
|
||||
details = summary["details"]
|
||||
assert details["model_ms"] == 0
|
||||
assert details["accounted_ms"] + details["unaccounted_ms"] == summary["duration_ms"]
|
||||
assert details["accounted_ms"] + 1 >= max(event["duration_ms"] for event in steps.values())
|
||||
assert record.verbatim not in written
|
||||
|
||||
|
||||
def test_nested_same_step_counts_once_and_overlap_counts_wall_clock_once():
|
||||
timer = HyStepTimer()
|
||||
with timer.step("permission_check"), timer.step("permission_check"):
|
||||
time.sleep(0.02)
|
||||
|
||||
async def overlapping():
|
||||
async def one():
|
||||
with timer.step("embedding", items=2):
|
||||
await asyncio.sleep(0.05)
|
||||
|
||||
await asyncio.gather(one(), one())
|
||||
|
||||
asyncio.run(overlapping())
|
||||
measured = timer.measurements()
|
||||
assert measured["permission_check"].call_count == 1
|
||||
embedding = measured["embedding"]
|
||||
assert embedding.call_count == 2
|
||||
assert embedding.item_count == 4
|
||||
assert embedding.summed_ms >= 95
|
||||
assert 45 <= embedding.busy_ms < 90
|
||||
@@ -1,453 +0,0 @@
|
||||
"""HY decides supersessions; the ledger mirrors them. Pinned SDK, scripted HTTP."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from copy import deepcopy
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from hy_memory.agent.tools.basic_profile import upsert_basic_profile
|
||||
from hy_memory.coding.types import WorkingFact
|
||||
from hy_memory.models.memory import MemoryLayer
|
||||
from hy_memory.models.memory import MemoryStatus as HyStatus
|
||||
|
||||
import tests.test_hy_permission_raw_context as raw_context
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.hy_permission_coding_store import HyPermissionCodingStore, hy_coding_fingerprint
|
||||
from one_memory.hy_permission_store import HyNodePermission
|
||||
from one_memory.memory_governance import MemoryKind, MemoryStatus
|
||||
from tests.test_hy_coding_lease import TOOL_TEXT, CodingOracle, by_kind, run_coding_turn
|
||||
from tests.test_hy_permission_raw_context import (
|
||||
AGENT_TEXT,
|
||||
USER_TEXT,
|
||||
ScriptedOracle,
|
||||
assert_outputs,
|
||||
bind_oracle,
|
||||
grant_records,
|
||||
native_client,
|
||||
on_native_loop,
|
||||
put_turn,
|
||||
raw_client,
|
||||
)
|
||||
from tests.test_hy_permission_store import Backend, Lease, node, wrapper
|
||||
|
||||
NEW_TEXT = "The user now requires two permission reviews before the synthetic release."
|
||||
|
||||
|
||||
def _vector(text):
|
||||
axis = {USER_TEXT: 1, AGENT_TEXT: 2, NEW_TEXT: 3}.get(text, 0)
|
||||
return [float(index == axis) for index in range(384)]
|
||||
|
||||
|
||||
class SupersedingOracle(ScriptedOracle):
|
||||
"""HY's reconciler replaces existing claims with the new one instead of updating them."""
|
||||
|
||||
def __init__(self, records, *, chain):
|
||||
super().__init__(records)
|
||||
self.chain = chain
|
||||
|
||||
def respond(self, request):
|
||||
response = super().respond(request)
|
||||
if self.stages[-1] != "reconcile":
|
||||
return response
|
||||
body = json.loads(response.content)
|
||||
ops = json.loads(body["choices"][0]["message"]["content"])
|
||||
by_content = {op["content"]: op for op in ops}
|
||||
supersede = {
|
||||
"op": "SUPERSEDE",
|
||||
"content": NEW_TEXT,
|
||||
"owner": "user",
|
||||
"layer": "L2_FACT",
|
||||
"tags": [],
|
||||
"supersede_reason": "the user raised the requirement",
|
||||
}
|
||||
if self.chain:
|
||||
supersede["memory_ids"] = [self.chain[0], self.chain[1]]
|
||||
replaced = {USER_TEXT, AGENT_TEXT}
|
||||
else:
|
||||
supersede["memory_id"] = by_content[USER_TEXT]["memory_id"]
|
||||
replaced = {USER_TEXT}
|
||||
ops = [op for op in ops if op["content"] not in replaced | {NEW_TEXT}] + [supersede]
|
||||
body["choices"][0]["message"]["content"] = json.dumps(ops)
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
|
||||
async def _first_turn(supervisor, records):
|
||||
first = supervisor.prepare_conversation(records[0])[0]
|
||||
client = await native_client(supervisor, first)
|
||||
bind_oracle(supervisor, ScriptedOracle(records))
|
||||
assert (await supervisor._create_runner().run(first.task_id)).status == "completed"
|
||||
await assert_outputs(supervisor, first, records, client)
|
||||
by_content = {m.content: m for m in supervisor.governance.list(user_id="local")}
|
||||
return client, by_content[USER_TEXT], by_content[AGENT_TEXT]
|
||||
|
||||
|
||||
async def _superseding_turn(supervisor, records, permission, *, chain=()):
|
||||
added = put_turn(supervisor.evidence, "user-raise", "user", NEW_TEXT, 3)
|
||||
records = (*records, added)
|
||||
grant_records(supervisor, records, permission.revision)
|
||||
second = supervisor.prepare_conversation(added)[0]
|
||||
oracle = SupersedingOracle(records, chain=chain)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(second.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
return oracle
|
||||
|
||||
|
||||
def _head(supervisor):
|
||||
(head,) = [
|
||||
m for m in supervisor.governance.list(user_id="local", limit=500) if m.content == NEW_TEXT
|
||||
]
|
||||
return head
|
||||
|
||||
|
||||
def test_a_memory_hy_supersedes_is_superseded_in_the_ledger_with_its_replacement(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(raw_context, "vector", _vector)
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, permission):
|
||||
client, replaced, kept = await _first_turn(supervisor, records)
|
||||
monkeypatch.setitem(raw_context.CLAIMS, NEW_TEXT, "user")
|
||||
await _superseding_turn(supervisor, records, permission)
|
||||
|
||||
native = await on_native_loop(
|
||||
client, client._vector_store.get_by_id(replaced.memory_id)
|
||||
)
|
||||
assert native.status == HyStatus.SUPERSEDED
|
||||
head = _head(supervisor)
|
||||
assert native.superseded_by == [head.memory_id]
|
||||
|
||||
old = supervisor.governance.get(replaced.memory_id, user_id="local")
|
||||
assert old.status is MemoryStatus.SUPERSEDED
|
||||
assert old.deleted_at is None and old.content == USER_TEXT
|
||||
assert old.superseded_by == (head.memory_id,)
|
||||
assert supervisor.governance.deletion(old.memory_id, user_id="local") is None
|
||||
assert head.status is MemoryStatus.CONFIRMED
|
||||
assert head.supersedes == (old.memory_id,)
|
||||
untouched = supervisor.governance.get(kept.memory_id, user_id="local")
|
||||
assert untouched.status is MemoryStatus.CONFIRMED
|
||||
assert untouched.supersedes == untouched.superseded_by == ()
|
||||
assert {m.memory_id for m in supervisor.governance.list(user_id="local")} == {
|
||||
head.memory_id,
|
||||
kept.memory_id,
|
||||
}
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_chain_fold_supersedes_every_folded_memory(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(raw_context, "vector", _vector)
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, permission):
|
||||
client, user, agent = await _first_turn(supervisor, records)
|
||||
monkeypatch.setitem(raw_context.CLAIMS, NEW_TEXT, "user")
|
||||
await _superseding_turn(
|
||||
supervisor, records, permission, chain=(agent.memory_id, user.memory_id)
|
||||
)
|
||||
|
||||
head = _head(supervisor)
|
||||
oldest = supervisor.governance.get(agent.memory_id, user_id="local")
|
||||
newer = supervisor.governance.get(user.memory_id, user_id="local")
|
||||
assert oldest.status is newer.status is MemoryStatus.SUPERSEDED
|
||||
assert oldest.superseded_by == (user.memory_id,)
|
||||
assert newer.superseded_by == (head.memory_id,)
|
||||
assert newer.supersedes == (agent.memory_id,)
|
||||
assert head.supersedes == (user.memory_id,)
|
||||
assert head.status is MemoryStatus.CONFIRMED
|
||||
assert [m.memory_id for m in supervisor.governance.list(user_id="local")] == [
|
||||
head.memory_id
|
||||
]
|
||||
for key in (agent.memory_id, user.memory_id):
|
||||
native = await on_native_loop(client, client._vector_store.get_by_id(key))
|
||||
assert native.status == HyStatus.SUPERSEDED
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
async def _turn_after_supersede(supervisor, records, permission):
|
||||
"""The user repeats the new requirement one turn after HY superseded the old one."""
|
||||
added = next(
|
||||
r
|
||||
for r in supervisor.evidence.list_conversation(records[0].evidence_id)
|
||||
if r.verbatim == NEW_TEXT
|
||||
)
|
||||
again = put_turn(supervisor.evidence, "user-again", "user", NEW_TEXT, 4)
|
||||
records = (*records, added, again)
|
||||
grant_records(supervisor, records, permission.revision + 1)
|
||||
third = supervisor.prepare_conversation(again)[0]
|
||||
oracle = ScriptedOracle(records)
|
||||
bind_oracle(supervisor, oracle)
|
||||
result = await supervisor._create_runner().run(third.task_id)
|
||||
assert not oracle.errors, oracle.errors
|
||||
return result, oracle
|
||||
|
||||
|
||||
def test_the_turn_after_a_supersede_still_completes(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(raw_context, "vector", _vector)
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, permission):
|
||||
_, replaced, _ = await _first_turn(supervisor, records)
|
||||
monkeypatch.setitem(raw_context.CLAIMS, NEW_TEXT, "user")
|
||||
await _superseding_turn(supervisor, records, permission)
|
||||
|
||||
# The old memory's evidence is in this turn's window; being
|
||||
# superseded by HY is not the user changing what was said.
|
||||
result, oracle = await _turn_after_supersede(supervisor, records, permission)
|
||||
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
old = supervisor.governance.get(replaced.memory_id, user_id="local")
|
||||
assert old.status is MemoryStatus.SUPERSEDED
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_hy_still_sees_the_head_of_a_supersede_chain_on_the_next_turn(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(raw_context, "vector", _vector)
|
||||
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path) as (supervisor, records, permission):
|
||||
await _first_turn(supervisor, records)
|
||||
monkeypatch.setitem(raw_context.CLAIMS, NEW_TEXT, "user")
|
||||
await _superseding_turn(supervisor, records, permission)
|
||||
head = _head(supervisor)
|
||||
|
||||
result, oracle = await _turn_after_supersede(supervisor, records, permission)
|
||||
|
||||
assert result.status == "completed", (result.last_error_code, oracle.stages)
|
||||
existing = {item["memory_id"] for batch in oracle.reconciles for item in batch}
|
||||
assert head.memory_id in existing, (head.memory_id, oracle.reconciles)
|
||||
repeated = [
|
||||
m
|
||||
for m in supervisor.governance.list(user_id="local", limit=500)
|
||||
if m.content == NEW_TEXT
|
||||
]
|
||||
assert [m.memory_id for m in repeated] == [head.memory_id]
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
class SupersedingFactOracle(CodingOracle):
|
||||
"""HY's fact reconciler judges the new session to continue the old task."""
|
||||
|
||||
def __init__(self, records, *, old_fact, **kwargs):
|
||||
super().__init__(records, **kwargs)
|
||||
self.old_fact = old_fact
|
||||
|
||||
def respond(self, request):
|
||||
body = json.loads(request.content)
|
||||
if request.url.path != "/v1/embeddings":
|
||||
prompt = body["messages"][-1]["content"]
|
||||
if prompt.startswith("A user task in a working-memory store has progressed"):
|
||||
self.requests.append(body)
|
||||
self.stages.append("chain_summary")
|
||||
return _completion(body, "The suite moved from pytest to tox.")
|
||||
response = super().respond(request)
|
||||
if self.stages[-1] != "fact_reconcile":
|
||||
return response
|
||||
return _completion(
|
||||
body,
|
||||
json.dumps(
|
||||
{
|
||||
"ops": [
|
||||
{
|
||||
"draft_idx": 0,
|
||||
"supersedes": [self.old_fact],
|
||||
"reason": "continues the same test-suite task",
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _completion(body, content):
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "synthetic-coding-response",
|
||||
"object": "chat.completion",
|
||||
"created": 1,
|
||||
"model": body["model"],
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"finish_reason": "stop",
|
||||
"message": {"role": "assistant", "content": content},
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_a_working_fact_hy_supersedes_is_superseded_in_the_ledger(tmp_path):
|
||||
async def scenario():
|
||||
async with raw_client(tmp_path, tool=TOOL_TEXT) as (supervisor, records, permission):
|
||||
_, first = await run_coding_turn(supervisor, records[0], CodingOracle(records))
|
||||
assert first.status == "completed", first.last_error_code
|
||||
(old_fact,) = by_kind(supervisor)[MemoryKind.WORKING]
|
||||
question = put_turn(supervisor.evidence, "user-again", "user", "Use tox now.", 4)
|
||||
tool = put_turn(supervisor.evidence, "tool-again", "tool", "[Bash] tox\nok", 5)
|
||||
grant_records(supervisor, (*records, question, tool), permission.revision)
|
||||
oracle = SupersedingFactOracle(
|
||||
(question, tool),
|
||||
old_fact=old_fact.memory_id,
|
||||
solution="Run `tox` from the repository root.",
|
||||
statement="The agent switched the synthetic suite to tox.",
|
||||
)
|
||||
_, second = await run_coding_turn(supervisor, question, oracle)
|
||||
assert second.status == "completed", (second.last_error_code, oracle.stages)
|
||||
assert "fact_reconcile" in oracle.stages
|
||||
client = await native_client(supervisor, second)
|
||||
(native_old,) = await on_native_loop(
|
||||
client, client._coding_store.get_facts([old_fact.memory_id], user_id="local")
|
||||
)
|
||||
assert native_old.status == "superseded"
|
||||
|
||||
old = supervisor.governance.get(old_fact.memory_id, user_id="local")
|
||||
assert old.status is MemoryStatus.SUPERSEDED
|
||||
assert old.superseded_by == (native_old.fact_link,)
|
||||
new = supervisor.governance.get(native_old.fact_link, user_id="local")
|
||||
assert new is not None and new.status is MemoryStatus.CONFIRMED
|
||||
assert new.kind is MemoryKind.WORKING
|
||||
assert new.supersedes == (old.memory_id,)
|
||||
working = [
|
||||
m.memory_id
|
||||
for m in supervisor.governance.list(user_id="local")
|
||||
if m.kind is MemoryKind.WORKING
|
||||
]
|
||||
assert working == [new.memory_id]
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_the_chat_store_reports_what_hy_superseded_and_nothing_else():
|
||||
head = node("head", supersedes=["old"])
|
||||
old = node("old")
|
||||
kept = node("kept")
|
||||
backend = Backend(old, kept)
|
||||
store = wrapper(backend, approved=(old, kept))
|
||||
|
||||
async def scenario():
|
||||
# Positive control and negative in one store: an ordinary update is not reported.
|
||||
await store.update_payload("kept", {"content": "Synthetic kept, refined"})
|
||||
assert store.superseded_nodes == {}
|
||||
await store.update_payload("old", {"is_latest": False, "status": "superseded"})
|
||||
await store.upsert(head)
|
||||
await store.update_payload("old", {"superseded_by": ["head"]})
|
||||
assert store.superseded_nodes == {"old": ("head",)}
|
||||
assert "old" not in store.retired_node_ids
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_superseded_link_written_only_on_the_head_is_still_reported():
|
||||
old = node("old")
|
||||
backend = Backend(old)
|
||||
store = wrapper(backend, approved=(old,))
|
||||
|
||||
async def scenario():
|
||||
await store.update_payload("old", {"is_latest": False, "status": "superseded"})
|
||||
await store.upsert(node("head", supersedes=["old"]))
|
||||
assert store.superseded_nodes == {"old": ("head",)}
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_a_basic_profile_supersede_is_reported_for_the_ledger():
|
||||
old = node(
|
||||
"old-profile",
|
||||
agent_id="",
|
||||
session_id="",
|
||||
owner=None,
|
||||
layer=MemoryLayer.L0_BASIC_INFO,
|
||||
content="name: Before",
|
||||
custom={"basic_info_kv": {"name": "Before"}},
|
||||
)
|
||||
backend = Backend(old)
|
||||
store = wrapper(backend, approved=(old,))
|
||||
|
||||
class Embedder:
|
||||
async def embed_queued(self, text):
|
||||
return [1.0, 0.0]
|
||||
|
||||
async def scenario():
|
||||
result = await upsert_basic_profile(
|
||||
user_id="local",
|
||||
kv={"name": "After"},
|
||||
vector_store=store,
|
||||
embed_service=Embedder(),
|
||||
allowed_fields=["name"],
|
||||
)
|
||||
assert result.success and result.changed
|
||||
assert store.superseded_nodes == {"old-profile": (result.node_id,)}
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def _fact(key, **kwargs):
|
||||
return WorkingFact(
|
||||
fact_id=key, user_id="local", agent_id="agent", session_id="session", **kwargs
|
||||
)
|
||||
|
||||
|
||||
class _NativeFacts:
|
||||
def __init__(self, *facts):
|
||||
self.facts = {fact.fact_id: deepcopy(fact) for fact in facts}
|
||||
|
||||
async def get_facts(self, keys, user_id=None):
|
||||
return [deepcopy(self.facts[key]) for key in keys if key in self.facts]
|
||||
|
||||
async def supersede_fact(self, old_id, new_id):
|
||||
row = self.facts[old_id]
|
||||
row.status, row.fact_link = "superseded", new_id
|
||||
return True
|
||||
|
||||
async def insert_fact(self, fact):
|
||||
self.facts[fact.fact_id] = deepcopy(fact)
|
||||
|
||||
|
||||
def _coding_store(native, *approved):
|
||||
return HyPermissionCodingStore(
|
||||
native,
|
||||
Lease(),
|
||||
agent_id="agent",
|
||||
session_id="session",
|
||||
node_permissions={
|
||||
fact.fact_id: HyNodePermission(hy_coding_fingerprint(fact), ("current", "support"))
|
||||
for fact in approved
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_the_coding_store_lets_hy_supersede_a_fact_before_inserting_its_successor():
|
||||
old = _fact("old-fact")
|
||||
store = _coding_store(_NativeFacts(old), old)
|
||||
|
||||
async def scenario():
|
||||
assert store.superseded_ids == {}
|
||||
assert await store.supersede_fact("old-fact", "new-fact")
|
||||
await store.insert_fact(_fact("new-fact"))
|
||||
store.recheck()
|
||||
assert store.superseded_ids == {"old-fact": ("new-fact",)}
|
||||
assert "old-fact" not in store.retired_ids
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_the_coding_store_never_links_a_fact_to_an_existing_unapproved_one():
|
||||
old, foreign = _fact("old-fact"), _fact("foreign-fact", statement="DENIED")
|
||||
native = _NativeFacts(old, foreign)
|
||||
store = _coding_store(native, old)
|
||||
|
||||
async def scenario():
|
||||
with pytest.raises(HyMemoryBridgeError, match="target_denied"):
|
||||
await store.supersede_fact("old-fact", "foreign-fact")
|
||||
|
||||
asyncio.run(scenario())
|
||||
assert native.facts["old-fact"].status == "active"
|
||||
@@ -1,433 +0,0 @@
|
||||
"""HY gets one turn at a time, exactly as it happened, and picks its own route.
|
||||
|
||||
A turn is one question from the person plus everything that answered it:
|
||||
assistant replies and the tool calls made along the way. HY's `add` decides
|
||||
chat or coding itself when it sees tool messages (`scene=None`); forcing
|
||||
`scene="chat"` made it strip them before it ever looked.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sqlite3
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from one_memory.hy_memory_bridge import (
|
||||
HyMemoryBridge,
|
||||
HyMemoryIngestRequest,
|
||||
HyMemoryOutput,
|
||||
HyMemoryTurn,
|
||||
)
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
from one_memory.imports.grants import GRANT_SCHEMA
|
||||
from one_memory.imports.models import WrittenBy
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor, _conversation_chunks
|
||||
from one_memory.memory_tasks import MemoryTaskKind, MemoryTaskStatus
|
||||
from one_memory.model_profiles import ProviderKind
|
||||
from tests.test_hy_memory_bridge import FakeHyMemoryClient, snapshot
|
||||
from tests.test_memory_processing import FakeBridge, _put, _runner, _setup, _task
|
||||
from tests.test_model_processing_permissions import Harness
|
||||
|
||||
|
||||
def _turn(evidence, index: int, *, tool: str | None = None):
|
||||
records = [
|
||||
_put(
|
||||
evidence,
|
||||
message_id=f"u{index}",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content=f"第{index}个问题。",
|
||||
),
|
||||
_put(
|
||||
evidence,
|
||||
message_id=f"a{index}",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content=f"第{index}个回答。",
|
||||
),
|
||||
]
|
||||
if tool is not None:
|
||||
records.append(
|
||||
_put(
|
||||
evidence,
|
||||
message_id=f"t{index}",
|
||||
role="tool",
|
||||
written_by=WrittenBy.TOOL,
|
||||
content=tool,
|
||||
)
|
||||
)
|
||||
records.append(
|
||||
_put(
|
||||
evidence,
|
||||
message_id=f"a{index}-after-tool",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content=f"第{index}个结论。",
|
||||
)
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def _supervisor(tmp_path, stores, bridge=None):
|
||||
evidence, profiles, tasks, governance, secrets, _profile = stores
|
||||
return MemoryProcessingSupervisor(
|
||||
tasks=tasks,
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=evidence,
|
||||
governance=governance,
|
||||
data_dir=tmp_path,
|
||||
runner_factory=lambda: _runner(tmp_path, stores, bridge or FakeBridge(outputs=())),
|
||||
)
|
||||
|
||||
|
||||
def _sent_turns(tmp_path, stores, created):
|
||||
bridge = FakeBridge(outputs=())
|
||||
runner = _runner(tmp_path, stores, bridge)
|
||||
for task in created:
|
||||
asyncio.run(runner.run(task.task_id))
|
||||
return [[turn.content for turn in request.turns] for request in bridge.requests]
|
||||
|
||||
|
||||
def test_each_turn_goes_to_hy_on_its_own(tmp_path) -> None:
|
||||
evidence = _setup(tmp_path)[0]
|
||||
first = _turn(evidence, 1, tool="[Bash] ls")
|
||||
second = _turn(evidence, 2)
|
||||
|
||||
chunks = _conversation_chunks(evidence.list_conversation(first[0].evidence_id))
|
||||
|
||||
assert [[r.evidence_id for r in chunk] for chunk in chunks] == [
|
||||
[r.evidence_id for r in first],
|
||||
[r.evidence_id for r in second],
|
||||
]
|
||||
|
||||
|
||||
def test_what_came_before_the_first_question_joins_the_turn_that_answers_it(tmp_path) -> None:
|
||||
"""An opener with no reply yet is not a turn of its own.
|
||||
|
||||
C3's real-authority drift showed splitting `assistant` proposal from the
|
||||
user turn that adopts it sends HY an answer without what it answers, so an
|
||||
unanswered assistant prefix joins the following turn. The prefix is still
|
||||
sent, once, and in order — it is just no longer a window of its own.
|
||||
"""
|
||||
evidence = _setup(tmp_path)[0]
|
||||
opener = _put(
|
||||
evidence,
|
||||
message_id="a0",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content="你好,我能帮你做什么?",
|
||||
)
|
||||
first = _turn(evidence, 1)
|
||||
|
||||
chunks = _conversation_chunks(evidence.list_conversation(opener.evidence_id))
|
||||
|
||||
assert [[r.evidence_id for r in chunk] for chunk in chunks] == [
|
||||
[opener.evidence_id, *[r.evidence_id for r in first]]
|
||||
]
|
||||
|
||||
|
||||
def test_a_turn_too_long_for_one_window_is_split_but_every_window_keeps_its_question(
|
||||
tmp_path, monkeypatch
|
||||
) -> None:
|
||||
# D74 sends this turn whole; the invariant still holds on the split path.
|
||||
monkeypatch.setenv("ONE_MEMORY_SPLIT_TURNS", "1")
|
||||
evidence = _setup(tmp_path)[0]
|
||||
question = _put(
|
||||
evidence,
|
||||
message_id="u1",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content="把这个仓库的测试跑一遍。",
|
||||
)
|
||||
calls = [
|
||||
_put(
|
||||
evidence,
|
||||
message_id=f"t{index}",
|
||||
role="tool",
|
||||
written_by=WrittenBy.TOOL,
|
||||
content=f"[Bash] {index} " + "x" * 1500,
|
||||
)
|
||||
for index in range(12)
|
||||
]
|
||||
|
||||
chunks = _conversation_chunks(evidence.list_conversation(question.evidence_id))
|
||||
|
||||
assert len(chunks) > 1
|
||||
assert all(chunk[0].evidence_id == question.evidence_id for chunk in chunks)
|
||||
assert all(sum(len(r.verbatim) for r in chunk) <= 12_000 for chunk in chunks)
|
||||
assert {r.evidence_id for chunk in chunks for r in chunk} == {
|
||||
question.evidence_id,
|
||||
*(call.evidence_id for call in calls),
|
||||
}
|
||||
|
||||
|
||||
def test_a_split_turn_never_sends_a_tool_only_window(tmp_path, monkeypatch) -> None:
|
||||
"""WorkBuddy/Codex can answer, then emit many tool calls.
|
||||
|
||||
If the answer fills the first window, the remaining calls used to become a
|
||||
tool-only window; HY rejected it with ``content or messages is required``.
|
||||
The tool calls must stay with the preceding non-tool window instead.
|
||||
D74 sends this turn whole; the invariant still holds on the split path.
|
||||
"""
|
||||
monkeypatch.setenv("ONE_MEMORY_SPLIT_TURNS", "1")
|
||||
evidence = _setup(tmp_path)[0]
|
||||
question = _put(
|
||||
evidence,
|
||||
message_id="u1",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content="先回答,再调用很多工具。",
|
||||
)
|
||||
answer = _put(
|
||||
evidence,
|
||||
message_id="a1",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content="好。" + "z" * 7_000,
|
||||
)
|
||||
calls = [
|
||||
_put(
|
||||
evidence,
|
||||
message_id=f"t{index}",
|
||||
role="tool",
|
||||
written_by=WrittenBy.TOOL,
|
||||
content=f"[Bash] {index} " + "x" * 1_500,
|
||||
)
|
||||
for index in range(4)
|
||||
]
|
||||
|
||||
chunks = _conversation_chunks(evidence.list_conversation(question.evidence_id))
|
||||
|
||||
assert chunks
|
||||
assert all(any(record.role != "tool" for record in chunk) for chunk in chunks)
|
||||
assert all(sum(len(record.verbatim) for record in chunk) <= 12_000 for chunk in chunks)
|
||||
assert list(dict.fromkeys(record.evidence_id for chunk in chunks for record in chunk)) == [
|
||||
question.evidence_id,
|
||||
answer.evidence_id,
|
||||
*(call.evidence_id for call in calls),
|
||||
]
|
||||
assert any(record.role == "tool" for chunk in chunks for record in chunk)
|
||||
|
||||
|
||||
def test_tool_calls_travel_with_their_turn_redacted_and_as_tool_messages(tmp_path) -> None:
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, _governance, _secrets, profile = stores
|
||||
records = _turn(
|
||||
evidence, 1, tool="[Bash] docker login --username kristinhe --password exposed-token-value"
|
||||
)
|
||||
bridge = FakeBridge()
|
||||
|
||||
asyncio.run(_runner(tmp_path, stores, bridge).run(_task(tasks, profile, records[0]).task_id))
|
||||
|
||||
request = bridge.requests[0]
|
||||
assert [turn.role for turn in request.turns] == ["user", "assistant", "tool", "assistant"]
|
||||
assert "exposed-token-value" not in request.turns[2].content
|
||||
assert "<redacted-secret>" in request.turns[2].content
|
||||
assert records[2].evidence_id in request.evidence_ids
|
||||
|
||||
|
||||
def test_the_next_turn_reaches_hy_with_the_previous_turn_as_history(tmp_path) -> None:
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, _tasks, _governance, _secrets, profile = stores
|
||||
with sqlite3.connect(tmp_path / "one-memory.sqlite3") as db:
|
||||
db.executescript(GRANT_SCHEMA)
|
||||
db.execute(
|
||||
"INSERT INTO source_grants VALUES ('local','cursor','/authorized','default',?,NULL)",
|
||||
(datetime.now(UTC).isoformat(),),
|
||||
)
|
||||
first = _turn(evidence, 1, tool="[Bash] 12 passed")
|
||||
second = _turn(evidence, 2)
|
||||
supervisor = _supervisor(tmp_path, stores)
|
||||
bridge = FakeBridge(outputs=())
|
||||
runner = _runner(tmp_path, stores, bridge)
|
||||
|
||||
for turn in (first, second):
|
||||
task = supervisor._enqueue_chunk(tuple(turn), profile, freeze_inputs=True)
|
||||
asyncio.run(runner.run(task.task_id))
|
||||
|
||||
context = bridge.requests[1].source_context
|
||||
assert context is not None
|
||||
assert [r.evidence_id for r in context.current] == [r.evidence_id for r in second]
|
||||
assert [r.evidence_id for r in context.history] == [r.evidence_id for r in first]
|
||||
|
||||
|
||||
def test_each_later_turn_carries_the_earlier_turns_as_history(tmp_path) -> None:
|
||||
"""Later windows in one conversation must not arrive at HY alone.
|
||||
|
||||
The frozen live pair expects later batches to grow because a window is sent
|
||||
with the turns behind it as history. A run where every request holds only the
|
||||
current turn loses the context the authority/adoption gate reads, which is
|
||||
what the cross-window real run showed.
|
||||
"""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, _tasks, _governance, _secrets, profile = stores
|
||||
with sqlite3.connect(tmp_path / "one-memory.sqlite3") as db:
|
||||
db.executescript(GRANT_SCHEMA)
|
||||
db.execute(
|
||||
"INSERT INTO source_grants VALUES ('local','cursor','/authorized','default',?,NULL)",
|
||||
(datetime.now(UTC).isoformat(),),
|
||||
)
|
||||
turns = [_turn(evidence, index) for index in (1, 2, 3)]
|
||||
supervisor = _supervisor(tmp_path, stores)
|
||||
bridge = FakeBridge(outputs=())
|
||||
runner = _runner(tmp_path, stores, bridge)
|
||||
|
||||
for turn in turns:
|
||||
task = supervisor._enqueue_chunk(tuple(turn), profile, freeze_inputs=True)
|
||||
asyncio.run(runner.run(task.task_id))
|
||||
|
||||
assert len(bridge.requests) == 3
|
||||
for index, request in enumerate(bridge.requests):
|
||||
context = request.source_context
|
||||
assert context is not None
|
||||
assert [r.evidence_id for r in context.current] == [r.evidence_id for r in turns[index]]
|
||||
# Every turn after the first is sent with the ones behind it.
|
||||
expected = [r.evidence_id for turn in turns[:index] for r in turn]
|
||||
assert [r.evidence_id for r in context.history] == expected
|
||||
|
||||
|
||||
def test_the_bridge_hands_hy_the_turn_and_lets_hy_choose_the_route(tmp_path) -> None:
|
||||
client = FakeHyMemoryClient({"success": True, "memory_ids": []})
|
||||
bridge = HyMemoryBridge(
|
||||
profile=snapshot(ProviderKind.OLLAMA, base_url="http://127.0.0.1:11434", has_api_key=False),
|
||||
api_key=None,
|
||||
data_dir=tmp_path,
|
||||
client=client,
|
||||
)
|
||||
request = HyMemoryIngestRequest(
|
||||
request_id="turn-1",
|
||||
user_id="local",
|
||||
source_id="codex",
|
||||
session_id="conversation-1",
|
||||
turns=(
|
||||
HyMemoryTurn(role="user", content="跑一下测试"),
|
||||
HyMemoryTurn(role="tool", content="[Bash] 12 passed"),
|
||||
HyMemoryTurn(role="assistant", content="全部通过。"),
|
||||
),
|
||||
evidence_ids=("u", "t", "a"),
|
||||
)
|
||||
|
||||
asyncio.run(bridge.ingest(request))
|
||||
asyncio.run(bridge.close())
|
||||
|
||||
assert client.calls[0]["scene"] is None
|
||||
assert [item["role"] for item in client.calls[0]["data"]] == ["user", "tool", "assistant"]
|
||||
|
||||
|
||||
def test_write_verification_accepts_the_tool_messages_hy_extracts_from(tmp_path) -> None:
|
||||
evidence = _setup(tmp_path)[0]
|
||||
question, _reply, call, _conclusion = _turn(evidence, 1, tool="[Bash] 12 passed")
|
||||
bindings = HyWriteBindings(HySourceContext(current=(question, call)))
|
||||
|
||||
bindings.check_content(
|
||||
json.dumps(
|
||||
[
|
||||
{"role": "user", "content": question.verbatim},
|
||||
{"role": "tool", "content": call.verbatim},
|
||||
],
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
assert [source["role"] for source in bindings.sources] == ["user", "tool"]
|
||||
|
||||
|
||||
def test_every_turn_gets_its_own_task_when_nothing_was_queued_before(tmp_path) -> None:
|
||||
stores = _setup(tmp_path)
|
||||
evidence = stores[0]
|
||||
turns = [_turn(evidence, index) for index in (1, 2, 3)]
|
||||
|
||||
created = _supervisor(tmp_path, stores).prepare_conversation(turns[2][0])
|
||||
|
||||
assert _sent_turns(tmp_path, stores, created) == [[r.verbatim for r in turn] for turn in turns]
|
||||
|
||||
|
||||
def _earlier_policy_window(stores, records, *, succeeded: bool):
|
||||
_evidence, _profiles, tasks, _governance, _secrets, profile = stores
|
||||
task = tasks.create(
|
||||
request_id=f"conversation:hy-native-v22:cursor:{records[0].evidence_id}",
|
||||
user_id=records[0].user_id,
|
||||
kind=MemoryTaskKind.EXTRACT,
|
||||
input_ref=records[0].evidence_id,
|
||||
profile=profile,
|
||||
input_records=tuple(records),
|
||||
)
|
||||
_claimed, token = tasks.claim(task.task_id)
|
||||
if succeeded:
|
||||
tasks.complete(task.task_id, runner_token=token)
|
||||
else:
|
||||
tasks.fail(task.task_id, runner_token=token, error_code="hy_memory_unavailable")
|
||||
return tasks.require(task.task_id)
|
||||
|
||||
|
||||
def test_turns_an_earlier_policy_already_extracted_are_not_sent_again(tmp_path) -> None:
|
||||
stores = _setup(tmp_path)
|
||||
evidence = stores[0]
|
||||
old = [*_turn(evidence, 1), *_turn(evidence, 2)]
|
||||
assert _earlier_policy_window(stores, old, succeeded=True).status is (
|
||||
MemoryTaskStatus.COMPLETED
|
||||
)
|
||||
new = _turn(evidence, 3)
|
||||
|
||||
created = _supervisor(tmp_path, stores).prepare_conversation(new[0])
|
||||
|
||||
assert _sent_turns(tmp_path, stores, created) == [[r.verbatim for r in new]]
|
||||
|
||||
|
||||
def test_renewed_consent_still_prepares_a_turn_the_current_policy_completed(tmp_path) -> None:
|
||||
h = Harness(tmp_path)
|
||||
h.bridge.fail = False
|
||||
turn = h.put_closed_turn("selected", conversation="renewed-consent")
|
||||
h.grant(turn)
|
||||
supervisor = h.supervisor()
|
||||
|
||||
async def scenario():
|
||||
try:
|
||||
first = supervisor.prepare_conversation(turn[0])[0]
|
||||
done = await supervisor._create_runner().run(first.task_id)
|
||||
assert done.status is MemoryTaskStatus.COMPLETED
|
||||
h.grant(turn)
|
||||
return first, supervisor.prepare_conversation(turn[0])
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
first, again = asyncio.run(scenario())
|
||||
|
||||
assert len(again) == 1 and again[0].task_id != first.task_id
|
||||
|
||||
|
||||
def test_a_failed_earlier_window_does_not_count_as_already_sent(tmp_path) -> None:
|
||||
stores = _setup(tmp_path)
|
||||
old = _turn(stores[0], 1)
|
||||
assert _earlier_policy_window(stores, old, succeeded=False).status is MemoryTaskStatus.FAILED
|
||||
|
||||
created = _supervisor(tmp_path, stores).prepare_conversation(old[0])
|
||||
|
||||
assert _sent_turns(tmp_path, stores, created) == [[r.verbatim for r in old]]
|
||||
|
||||
|
||||
def test_a_tool_heavy_turn_reaches_hy_once_and_still_becomes_a_memory(tmp_path) -> None:
|
||||
"""D74's full criterion: one call to HY, and a memory comes out of it.
|
||||
|
||||
The half that was only ever checked by hand is the second one. On 17:14
|
||||
the acceptance run for this ended with `completed=false` and zero
|
||||
evidence, so nothing proved the memory half either way; the window count
|
||||
alone leaves it open.
|
||||
"""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, governance, _secrets, profile = stores
|
||||
records = _turn(evidence, 1, tool="x" * 50_000)
|
||||
task = _task(tasks, profile, records[0])
|
||||
bridge = FakeBridge(
|
||||
outputs=(HyMemoryOutput(memory_id="hy-tool-1", content="用户跑过一次全量测试。"),)
|
||||
)
|
||||
|
||||
asyncio.run(_runner(tmp_path, stores, bridge).run(task.task_id))
|
||||
|
||||
# One window, so one call.
|
||||
assert len(_conversation_chunks(records)) == 1
|
||||
assert len(bridge.requests) == 1
|
||||
# And the memory that call produced actually landed.
|
||||
assert [item.content for item in governance.list(user_id="local")] == ["用户跑过一次全量测试。"]
|
||||
@@ -1,672 +0,0 @@
|
||||
import asyncio
|
||||
import threading
|
||||
from contextlib import suppress
|
||||
|
||||
import pytest
|
||||
|
||||
from one_memory.imports.models import EvidenceContentKind, WrittenBy
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor, _merge_historical_chunks
|
||||
from one_memory.memory_tasks import MemoryTaskStatus
|
||||
from one_memory.model_profiles import ModelProfilePatch
|
||||
from tests.test_memory_processing import _put, _setup
|
||||
from tests.test_model_processing_permissions import Harness
|
||||
|
||||
|
||||
def setup(tmp_path, count=3):
|
||||
evidence, profiles, tasks, governance, secrets, _profile = _setup(tmp_path)
|
||||
records = [
|
||||
_put(
|
||||
evidence,
|
||||
message_id=f"message-{index}",
|
||||
conversation_id=f"conversation-{index}",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content=f"Synthetic conversation {index}",
|
||||
content_kind=EvidenceContentKind.MEMORY_CONTENT,
|
||||
)
|
||||
for index in range(count)
|
||||
]
|
||||
supervisor = MemoryProcessingSupervisor(
|
||||
tasks=tasks,
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=evidence,
|
||||
governance=governance,
|
||||
data_dir=tmp_path,
|
||||
)
|
||||
return supervisor, records
|
||||
|
||||
|
||||
def test_accept_backfill_returns_after_first_batch_while_later_history_is_still_read(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr("one_memory.memory_processing._BACKFILL_START_GRACE_SECONDS", 0)
|
||||
|
||||
async def scenario():
|
||||
supervisor, records = setup(tmp_path)
|
||||
later_entered, release = threading.Event(), threading.Event()
|
||||
scheduled = []
|
||||
reads = []
|
||||
original = supervisor.evidence.list_conversation
|
||||
|
||||
def blocked(evidence_id):
|
||||
reads.append(evidence_id)
|
||||
if len(reads) == 2:
|
||||
later_entered.set()
|
||||
assert release.wait(5)
|
||||
return original(evidence_id)
|
||||
|
||||
monkeypatch.setattr(supervisor.evidence, "list_conversation", blocked)
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.extend(work)
|
||||
)
|
||||
try:
|
||||
accepted = asyncio.create_task(supervisor.accept_backfill_off_thread())
|
||||
assert await asyncio.to_thread(later_entered.wait, 3)
|
||||
await asyncio.wait_for(accepted, 1)
|
||||
assert len(scheduled) == 1
|
||||
assert accepted.done()
|
||||
assert supervisor._live_backfill is not None
|
||||
assert not supervisor._live_backfill.done()
|
||||
finally:
|
||||
release.set()
|
||||
if supervisor._live_backfill is not None:
|
||||
await supervisor._live_backfill
|
||||
await supervisor.shutdown()
|
||||
assert len(scheduled) == len(records)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_first_task_is_dispatched_before_later_conversation_is_prepared(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
supervisor, records = setup(tmp_path)
|
||||
later_entered, release = threading.Event(), threading.Event()
|
||||
scheduled = []
|
||||
reads = []
|
||||
original = supervisor.evidence.list_conversation
|
||||
|
||||
def blocked(evidence_id):
|
||||
reads.append(evidence_id)
|
||||
if len(reads) == 2:
|
||||
later_entered.set()
|
||||
assert release.wait(5)
|
||||
return original(evidence_id)
|
||||
|
||||
monkeypatch.setattr(supervisor.evidence, "list_conversation", blocked)
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.extend(work)
|
||||
)
|
||||
task = asyncio.create_task(supervisor.backfill_off_thread())
|
||||
try:
|
||||
assert await asyncio.to_thread(later_entered.wait, 3)
|
||||
assert len(scheduled) == 1, "first task still waits behind later conversation reads"
|
||||
assert supervisor.tasks.require(scheduled[0].task_id).input_ref in {
|
||||
record.evidence_id for record in records
|
||||
}
|
||||
finally:
|
||||
release.set()
|
||||
await task
|
||||
await supervisor.shutdown()
|
||||
assert len(scheduled) == len(records)
|
||||
assert {supervisor.tasks.require(item.task_id).input_ref for item in scheduled} == {
|
||||
record.evidence_id for record in records
|
||||
}
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_cancellation_joins_the_preparation_thread_before_return(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
supervisor, _records = setup(tmp_path)
|
||||
entered, release, exited = threading.Event(), threading.Event(), threading.Event()
|
||||
original = supervisor.evidence.list_conversation
|
||||
scheduled = []
|
||||
|
||||
def blocked(evidence_id):
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
result = original(evidence_id)
|
||||
exited.set()
|
||||
return result
|
||||
|
||||
monkeypatch.setattr(supervisor.evidence, "list_conversation", blocked)
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.extend(work)
|
||||
)
|
||||
task = asyncio.create_task(supervisor.backfill_off_thread())
|
||||
try:
|
||||
assert await asyncio.to_thread(entered.wait, 3)
|
||||
task.cancel()
|
||||
await asyncio.sleep(0.03)
|
||||
task.cancel()
|
||||
await asyncio.sleep(0.03)
|
||||
assert not task.done(), "backfill abandoned a thread that can still write tasks"
|
||||
finally:
|
||||
release.set()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await task
|
||||
await supervisor.shutdown()
|
||||
assert exited.is_set()
|
||||
assert scheduled == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_later_failure_keeps_the_first_committed_batch_visible(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
supervisor, _records = setup(tmp_path)
|
||||
scheduled, reads = [], []
|
||||
original = supervisor.evidence.list_conversation
|
||||
|
||||
def broken(evidence_id):
|
||||
reads.append(evidence_id)
|
||||
if len(reads) == 2:
|
||||
raise RuntimeError("synthetic later read failure")
|
||||
return original(evidence_id)
|
||||
|
||||
monkeypatch.setattr(supervisor.evidence, "list_conversation", broken)
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.extend(work)
|
||||
)
|
||||
try:
|
||||
with pytest.raises(RuntimeError, match="synthetic later"):
|
||||
await supervisor.backfill_off_thread()
|
||||
assert len(scheduled) == 1
|
||||
assert len(supervisor.tasks.list()) == 1
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change", ["revoke", "regrant"])
|
||||
def test_permission_change_between_batches_stops_without_new_tasks(tmp_path, monkeypatch, change):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
records = [
|
||||
h.put("first", conversation="first", memory_content=True),
|
||||
h.put("second", conversation="second", memory_content=True),
|
||||
]
|
||||
receipt = h.grant(records)
|
||||
supervisor = h.supervisor()
|
||||
scheduled = []
|
||||
|
||||
def dispatch(work):
|
||||
scheduled.extend(work)
|
||||
if len(scheduled) == 1:
|
||||
if change == "revoke":
|
||||
h.permissions.revoke(
|
||||
user_id="local", user_confirmed=True, expected_revision=receipt.revision
|
||||
)
|
||||
else:
|
||||
h.grant(records)
|
||||
|
||||
monkeypatch.setattr(supervisor, "_queue_validated_work", dispatch)
|
||||
try:
|
||||
with pytest.raises(RuntimeError, match="model_content_permission_"):
|
||||
await supervisor.backfill_off_thread(expected_permission_revision=receipt.revision)
|
||||
assert len(h.tasks.list()) == len(scheduled) == 1
|
||||
assert h.factory_calls == [] and h.bridge.calls == []
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_profile_change_after_first_batch_preserves_old_task_identity(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
h.grant(
|
||||
[
|
||||
h.put("first", conversation="first", memory_content=True),
|
||||
h.put("second", conversation="second", memory_content=True),
|
||||
]
|
||||
)
|
||||
supervisor = h.supervisor()
|
||||
scheduled = []
|
||||
|
||||
def dispatch(work):
|
||||
scheduled.extend(work)
|
||||
h.profiles.update(
|
||||
h.profile.profile_id,
|
||||
ModelProfilePatch(model="changed-model", expected_revision=h.profile.revision),
|
||||
h.secrets,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(supervisor, "_queue_validated_work", dispatch)
|
||||
try:
|
||||
with pytest.raises(RuntimeError, match="model_content_profile_changed"):
|
||||
await supervisor.backfill_off_thread()
|
||||
assert len(h.tasks.list()) == len(scheduled) == 1
|
||||
assert h.tasks.list()[0].profile_revision == h.profile.revision
|
||||
assert h.factory_calls == []
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_batched_and_sync_preparation_keep_all_task_inputs_and_idempotency(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
supervisor, records = setup(tmp_path, count=21)
|
||||
scheduled = []
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda batch: scheduled.append(batch)
|
||||
)
|
||||
try:
|
||||
expected = supervisor.prepare_backfill()
|
||||
expected_ids = {work.task_id for work in expected}
|
||||
assert len(expected_ids) == 21
|
||||
before = {
|
||||
record.evidence_id: supervisor.evidence.get(record.evidence_id)
|
||||
for record in records
|
||||
}
|
||||
await supervisor.backfill_off_thread()
|
||||
assert len(scheduled[0]) == 1
|
||||
assert all(len(batch) <= 8 for batch in scheduled)
|
||||
assert {work.task_id for batch in scheduled for work in batch} == expected_ids
|
||||
assert len(supervisor.tasks.list()) == 21
|
||||
assert {
|
||||
record.evidence_id: supervisor.evidence.get(record.evidence_id)
|
||||
for record in records
|
||||
} == before
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_history_backfill_merges_adjacent_unsent_turns_in_one_conversation(tmp_path):
|
||||
supervisor, records = setup(tmp_path, count=0)
|
||||
profile = supervisor.profiles.active(supervisor.secrets)
|
||||
assert profile is not None
|
||||
conversation = []
|
||||
for index in range(2):
|
||||
conversation.extend(
|
||||
(
|
||||
_put(
|
||||
supervisor.evidence,
|
||||
message_id=f"user-{index}",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content=f"问题 {index}",
|
||||
conversation_id="shared-history",
|
||||
),
|
||||
_put(
|
||||
supervisor.evidence,
|
||||
message_id=f"assistant-{index}",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content=f"回答 {index}",
|
||||
conversation_id="shared-history",
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
chunks = supervisor._prepare_conversation_chunks(conversation, profile, None)
|
||||
merged = _merge_historical_chunks(chunks)
|
||||
|
||||
assert len(chunks) == 2
|
||||
assert len(merged) == 1
|
||||
assert {record.evidence_id for record in merged[0]} == {
|
||||
record.evidence_id for record in conversation
|
||||
}
|
||||
|
||||
|
||||
def test_backfill_holds_an_unanswered_raw_turn_until_its_reply_arrives(tmp_path):
|
||||
supervisor, _records = setup(tmp_path, count=0)
|
||||
question = _put(
|
||||
supervisor.evidence,
|
||||
message_id="open-question",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content="发布包上传了吗?",
|
||||
conversation_id="open-turn",
|
||||
)
|
||||
|
||||
assert supervisor.prepare_conversation(question) == []
|
||||
assert supervisor.prepare_backfill() == []
|
||||
|
||||
answer = _put(
|
||||
supervisor.evidence,
|
||||
message_id="closing-answer",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content="已经上传完成。",
|
||||
conversation_id="open-turn",
|
||||
)
|
||||
|
||||
prepared = supervisor.prepare_conversation(answer)
|
||||
|
||||
assert len(prepared) == 1
|
||||
assert prepared[0].input_ref == question.evidence_id
|
||||
|
||||
|
||||
def test_history_merge_supersedes_an_already_queued_shorter_session_task(tmp_path):
|
||||
supervisor, _records = setup(tmp_path, count=0)
|
||||
first = _put(
|
||||
supervisor.evidence,
|
||||
message_id="merge-user-1",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content="第一问",
|
||||
conversation_id="growing-history",
|
||||
)
|
||||
first_reply = _put(
|
||||
supervisor.evidence,
|
||||
message_id="merge-assistant-1",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content="第一答",
|
||||
conversation_id="growing-history",
|
||||
)
|
||||
profile = supervisor.profiles.active(supervisor.secrets)
|
||||
assert profile is not None
|
||||
old = supervisor._enqueue_chunk((first, first_reply), profile, freeze_inputs=True)
|
||||
second = _put(
|
||||
supervisor.evidence,
|
||||
message_id="merge-user-2",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
content="第二问",
|
||||
conversation_id="growing-history",
|
||||
)
|
||||
second_reply = _put(
|
||||
supervisor.evidence,
|
||||
message_id="merge-assistant-2",
|
||||
role="assistant",
|
||||
written_by=WrittenBy.AGENT,
|
||||
content="第二答",
|
||||
conversation_id="growing-history",
|
||||
)
|
||||
|
||||
chunks = supervisor._prepare_conversation_chunks(
|
||||
supervisor.evidence.list_conversation(second.evidence_id), profile, None
|
||||
)
|
||||
merged_chunk = _merge_historical_chunks(chunks)[0]
|
||||
merged_task = supervisor._enqueue_chunk(merged_chunk, profile, freeze_inputs=True)
|
||||
|
||||
pending = supervisor.tasks.list(statuses=(MemoryTaskStatus.PENDING,))
|
||||
merged = next(task for task in supervisor.tasks.list() if task.task_id != old.task_id)
|
||||
assert supervisor.tasks.require(old.task_id).status.value == "cancelled"
|
||||
assert [task.task_id for task in pending] == [merged.task_id]
|
||||
assert {ref["evidence_id"] for ref in supervisor.tasks.inputs(merged.task_id)} == {
|
||||
first.evidence_id,
|
||||
first_reply.evidence_id,
|
||||
second.evidence_id,
|
||||
second_reply.evidence_id,
|
||||
}
|
||||
assert merged_task.task_id == merged.task_id
|
||||
|
||||
|
||||
def test_backfill_budget_yields_between_rounds_and_resumes_same_cursor(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
supervisor, records = setup(tmp_path, count=3)
|
||||
scheduled = []
|
||||
clock = iter((0.0, 0.0, 331.0, 331.0, 331.0, 662.0, 662.0, 662.0))
|
||||
supervisor._backfill_clock = lambda: next(clock)
|
||||
supervisor._backfill_round_budget_seconds = 330.0
|
||||
releases = []
|
||||
|
||||
async def schedule(work):
|
||||
assert supervisor._backfill_lock.locked()
|
||||
scheduled.extend(work)
|
||||
|
||||
original_sleep = asyncio.sleep
|
||||
|
||||
async def observed_sleep(delay):
|
||||
if delay == 0:
|
||||
releases.append(not supervisor._backfill_lock.locked())
|
||||
await original_sleep(0)
|
||||
|
||||
monkeypatch.setattr(supervisor, "_schedule_tasks_off_thread", schedule)
|
||||
monkeypatch.setattr("one_memory.memory_processing.asyncio.sleep", observed_sleep)
|
||||
try:
|
||||
await supervisor.backfill_off_thread()
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
assert releases == [True, True]
|
||||
assert len(scheduled) == len(records)
|
||||
assert len({work.task_id for work in scheduled}) == len(records)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_shutdown_waits_for_backfill_preparation_and_never_dispatches_tail(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
supervisor, _records = setup(tmp_path)
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
scheduled = []
|
||||
original = supervisor.evidence.list_conversation
|
||||
|
||||
def blocked(evidence_id):
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
return original(evidence_id)
|
||||
|
||||
monkeypatch.setattr(supervisor.evidence, "list_conversation", blocked)
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.extend(work)
|
||||
)
|
||||
task = asyncio.create_task(supervisor.backfill_off_thread())
|
||||
try:
|
||||
assert await asyncio.to_thread(entered.wait, 3)
|
||||
stopping = asyncio.create_task(supervisor.shutdown())
|
||||
await asyncio.sleep(0.03)
|
||||
assert not stopping.done()
|
||||
finally:
|
||||
release.set()
|
||||
await task
|
||||
await stopping
|
||||
assert scheduled == []
|
||||
assert supervisor._backfill_preparations == set()
|
||||
assert supervisor.tasks.list() == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("count", [0, 1, 8, 9, 18])
|
||||
def test_empty_and_batch_boundaries_preserve_all_inputs(tmp_path, monkeypatch, count):
|
||||
async def scenario():
|
||||
supervisor, records = setup(tmp_path, count=count)
|
||||
scheduled = []
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.extend(work)
|
||||
)
|
||||
active_calls = []
|
||||
active = supervisor.profiles.active
|
||||
|
||||
def counted(secrets):
|
||||
active_calls.append(True)
|
||||
return active(secrets)
|
||||
|
||||
monkeypatch.setattr(supervisor.profiles, "active", counted)
|
||||
try:
|
||||
await supervisor.backfill_off_thread()
|
||||
assert len(active_calls) == 1, "batch identity checks should not reaccess credentials"
|
||||
assert {supervisor.tasks.require(work.task_id).input_ref for work in scheduled} == {
|
||||
record.evidence_id for record in records
|
||||
}
|
||||
assert len(supervisor.tasks.list()) == count
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_empty_denied_batches_do_not_end_the_remaining_traversal(tmp_path, monkeypatch):
|
||||
async def scenario():
|
||||
supervisor, records = setup(tmp_path, count=19)
|
||||
scheduled = []
|
||||
original = supervisor.evidence.list_conversation
|
||||
earliest = records[0].evidence_id
|
||||
monkeypatch.setattr(
|
||||
supervisor.evidence,
|
||||
"list_conversation",
|
||||
lambda identity: original(identity) if identity == earliest else [],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.append(work)
|
||||
)
|
||||
try:
|
||||
await supervisor.backfill_off_thread()
|
||||
assert scheduled[:2] == [[], []]
|
||||
useful = [work for batch in scheduled for work in batch]
|
||||
assert len(useful) == 1
|
||||
assert supervisor.tasks.require(useful[0].task_id).input_ref == earliest
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_replacement_backfill_waits_for_cancelled_writer_and_remains_idempotent(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
async def scenario():
|
||||
supervisor, records = setup(tmp_path)
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
reads, scheduled = [], []
|
||||
original = supervisor.evidence.list_conversation
|
||||
|
||||
def blocked(evidence_id):
|
||||
reads.append(evidence_id)
|
||||
if len(reads) == 1:
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
return original(evidence_id)
|
||||
|
||||
monkeypatch.setattr(supervisor.evidence, "list_conversation", blocked)
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.extend(work)
|
||||
)
|
||||
old = asyncio.create_task(supervisor.backfill_off_thread())
|
||||
try:
|
||||
assert await asyncio.to_thread(entered.wait, 3)
|
||||
old.cancel()
|
||||
replacement = asyncio.create_task(supervisor.backfill_off_thread())
|
||||
await asyncio.sleep(0.03)
|
||||
assert not old.done() and not replacement.done()
|
||||
assert len(reads) == 1
|
||||
finally:
|
||||
release.set()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await old
|
||||
await replacement
|
||||
await supervisor.shutdown()
|
||||
assert len(supervisor.tasks.list()) == len(records)
|
||||
assert len({work.task_id for work in scheduled}) == len(scheduled) == len(records)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("unclaimable", ["completed", "deferred"])
|
||||
def test_unclaimable_first_seed_does_not_delay_first_useful_batch(
|
||||
tmp_path, monkeypatch, unclaimable
|
||||
):
|
||||
async def scenario():
|
||||
supervisor, records = setup(tmp_path)
|
||||
first = supervisor.prepare_conversation(records[-1])[0]
|
||||
if unclaimable == "completed":
|
||||
_, token = supervisor.tasks.claim(first.task_id)
|
||||
supervisor.tasks.complete(first.task_id, runner_token=token)
|
||||
else:
|
||||
supervisor.tasks.defer_outside_window(first.task_id)
|
||||
scheduled, reads = [], []
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
original = supervisor.evidence.list_conversation
|
||||
|
||||
def blocked(identity):
|
||||
reads.append(identity)
|
||||
if len(reads) == 3:
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
return original(identity)
|
||||
|
||||
monkeypatch.setattr(supervisor.evidence, "list_conversation", blocked)
|
||||
monkeypatch.setattr(
|
||||
supervisor, "_queue_validated_work", lambda work: scheduled.extend(work)
|
||||
)
|
||||
task = asyncio.create_task(supervisor.backfill_off_thread())
|
||||
try:
|
||||
assert await asyncio.to_thread(entered.wait, 3)
|
||||
assert len(scheduled) == 1
|
||||
assert scheduled[0].task_id != first.task_id
|
||||
finally:
|
||||
release.set()
|
||||
await task
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_cancelled_shutdown_waits_for_inflight_generator_and_preserves_cleanup(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
async def scenario():
|
||||
supervisor, _records = setup(tmp_path)
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
original = supervisor.evidence.list_conversation
|
||||
|
||||
def blocked(identity):
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
return original(identity)
|
||||
|
||||
monkeypatch.setattr(supervisor.evidence, "list_conversation", blocked)
|
||||
backfill = asyncio.create_task(supervisor.backfill_off_thread())
|
||||
try:
|
||||
assert await asyncio.to_thread(entered.wait, 3)
|
||||
stopping = asyncio.create_task(supervisor.shutdown())
|
||||
await asyncio.sleep(0.02)
|
||||
stopping.cancel()
|
||||
await asyncio.sleep(0.02)
|
||||
stopping.cancel()
|
||||
await asyncio.sleep(0.02)
|
||||
assert not stopping.done() and not backfill.done()
|
||||
assert len(supervisor._backfill_preparations) == 1
|
||||
finally:
|
||||
release.set()
|
||||
await asyncio.gather(stopping, backfill, return_exceptions=True)
|
||||
assert not backfill.cancelled() and backfill.exception() is None
|
||||
assert supervisor._backfill_preparations == set()
|
||||
assert supervisor.tasks.list() == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_exact_backfill_keeps_granted_prefix_when_same_conversation_grows(tmp_path):
|
||||
async def scenario():
|
||||
h = Harness(tmp_path)
|
||||
first = h.put("first", memory_content=True)
|
||||
second = h.put("second", memory_content=True)
|
||||
receipt = h.grant([first, second])
|
||||
later = h.put("later", memory_content=True)
|
||||
other = h.put("other", conversation="other", memory_content=True)
|
||||
supervisor = h.supervisor()
|
||||
try:
|
||||
await supervisor.backfill_off_thread(expected_permission_revision=receipt.revision)
|
||||
tasks = supervisor.tasks.list()
|
||||
# One task per turn; both granted turns, nothing after them.
|
||||
assert len(tasks) == 1
|
||||
assert {task.input_ref for task in tasks} == {second.evidence_id}
|
||||
refs = {
|
||||
item["evidence_id"]
|
||||
for task in tasks
|
||||
for item in supervisor.tasks.inputs(task.task_id)
|
||||
}
|
||||
assert {first.evidence_id, second.evidence_id} <= refs
|
||||
assert later.evidence_id not in refs
|
||||
assert other.evidence_id not in refs
|
||||
# The first turn may already be running; whatever reached the model was granted.
|
||||
assert all(
|
||||
{record.evidence_id for record in lease.records}
|
||||
<= {first.evidence_id, second.evidence_id}
|
||||
for lease, _data_dir in h.factory_calls
|
||||
)
|
||||
finally:
|
||||
await supervisor.shutdown()
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -1,460 +0,0 @@
|
||||
"""T04: what the person explicitly asks to remember is written through HY's import path.
|
||||
|
||||
Whole native HY client behind the production permission context; only the model
|
||||
endpoints are scripted through an httpx MockTransport, never a network.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import sqlite3
|
||||
from contextlib import asynccontextmanager, closing
|
||||
from datetime import UTC, datetime
|
||||
from uuid import uuid4
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import SecretStr
|
||||
|
||||
from one_memory import continuous_policy, ingest_journal
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
from one_memory.hy_memory_bridge import HyMemoryIngestResult, HyMemoryOutput
|
||||
from one_memory.hy_permission_context import permission_bridge_factory
|
||||
from one_memory.imports.models import EvidenceInput, EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.matters.selection import select_evidence
|
||||
from one_memory.memory_notes import SOURCE_ID
|
||||
from one_memory.model_profiles import ModelCapabilities, ModelProfileInput, ProviderKind
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
from tests.source_consent import confirm_model_content
|
||||
from tests.test_memory_processing import FakeBridge, _runner, _setup, _task
|
||||
|
||||
TEXT = "请记住:项目周会固定不排在周五。"
|
||||
|
||||
|
||||
def _vector(text: str) -> list[float]:
|
||||
digest = hashlib.sha256(text.encode()).digest()
|
||||
values = [float(byte) + 1.0 for byte in digest * 12]
|
||||
norm = sum(value * value for value in values) ** 0.5
|
||||
return [value / norm for value in values]
|
||||
|
||||
|
||||
class ScriptedModel:
|
||||
"""The model endpoints HY reaches. Reconcile says "nothing to change" unless told."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.reconcile_prompts: list[str] = []
|
||||
self.other_prompts: list[str] = []
|
||||
self.embeddings_down = False
|
||||
|
||||
def __call__(self, request: httpx.Request) -> httpx.Response:
|
||||
body = json.loads(request.content)
|
||||
if request.url.path.endswith("embeddings"):
|
||||
if self.embeddings_down:
|
||||
return httpx.Response(503, json={"error": "down"})
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"object": "list",
|
||||
"model": "test-embedding",
|
||||
"data": [
|
||||
{"index": i, "object": "embedding", "embedding": _vector(text)}
|
||||
for i, text in enumerate(body["input"])
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "total_tokens": 1},
|
||||
},
|
||||
)
|
||||
prompt = "\n".join(str(message.get("content")) for message in body["messages"])
|
||||
reconcile = "Existing memories" in prompt
|
||||
(self.reconcile_prompts if reconcile else self.other_prompts).append(prompt)
|
||||
# Extraction or reconcile: the model finds nothing worth writing.
|
||||
content = "[]" if reconcile else '{"memory": [], "identity": [], "intentions": []}'
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "scripted",
|
||||
"object": "chat.completion",
|
||||
"created": 1,
|
||||
"model": "test-generation",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"finish_reason": "stop",
|
||||
"message": {"role": "assistant", "content": content},
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def hy_calls(monkeypatch):
|
||||
from hy_memory.client import HyMemoryClient
|
||||
|
||||
calls: list[tuple[str, object]] = []
|
||||
original_add = HyMemoryClient.add
|
||||
original_extracted = HyMemoryClient.add_extracted
|
||||
|
||||
def add(self, data, **kwargs):
|
||||
calls.append(("add", [turn["content"] for turn in data]))
|
||||
return original_add(self, data, **kwargs)
|
||||
|
||||
def add_extracted(self, contents, **kwargs):
|
||||
raw = original_extracted(self, contents, **kwargs)
|
||||
calls.append(("add_extracted", {"contents": list(contents), "raw": raw}))
|
||||
return raw
|
||||
|
||||
monkeypatch.setattr(HyMemoryClient, "add", add)
|
||||
monkeypatch.setattr(HyMemoryClient, "add_extracted", add_extracted)
|
||||
return calls
|
||||
|
||||
|
||||
def _application(tmp_path):
|
||||
return create_app(
|
||||
Settings(
|
||||
data_dir=tmp_path / "data",
|
||||
source_home=tmp_path,
|
||||
api_token=SecretStr("t04"),
|
||||
imports_enabled=False,
|
||||
automatic_memory_processing=False,
|
||||
live_sync_enabled=False,
|
||||
),
|
||||
secret_store=InMemorySecretStore(),
|
||||
)
|
||||
|
||||
|
||||
async def _configure(app, model: ScriptedModel) -> None:
|
||||
profiles, secrets = app.state.model_profile_store, app.state.secret_store
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Scripted",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
base_url="http://127.0.0.1:1/v1",
|
||||
model="test-generation",
|
||||
embedding_provider_kind=ProviderKind.OLLAMA,
|
||||
embedding_base_url="http://127.0.0.1:2/v1",
|
||||
embedding_model="test-embedding",
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True, model_available=True, embedding_available=True
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
processor = app.state.memory_processing_supervisor
|
||||
processor.permission_bridge_factory = permission_bridge_factory(
|
||||
processor._bridge, http_transport=httpx.MockTransport(model)
|
||||
)
|
||||
|
||||
|
||||
async def _remember(app, client, text: str) -> dict:
|
||||
saved = (
|
||||
await client.post(
|
||||
"/api/v1/memory-notes",
|
||||
json={"request_id": str(uuid4()), "text": text, "scope": "all-apps"},
|
||||
)
|
||||
).json()
|
||||
await confirm_model_content(
|
||||
client,
|
||||
source_ids=("one-memory-manual",),
|
||||
evidence_ids=(saved["evidence_id"],),
|
||||
future_additions=False,
|
||||
)
|
||||
started = await client.post(f"/api/v1/memory-notes/{saved['evidence_id']}/process")
|
||||
assert started.status_code == 200, started.text
|
||||
await _wait_for_note_job(app, saved["evidence_id"])
|
||||
return (await client.get(f"/api/v1/memory-notes/{saved['evidence_id']}")).json()
|
||||
|
||||
|
||||
async def _wait_for_note_job(app, evidence_id: str, *, timeout: float = 10) -> str:
|
||||
"""Wait for the production durable note job, not the legacy worker map."""
|
||||
deadline = asyncio.get_running_loop().time() + timeout
|
||||
key = f"entry:{evidence_id}"
|
||||
while asyncio.get_running_loop().time() < deadline:
|
||||
with sqlite3.connect(app.state.evidence_store.database_path) as db:
|
||||
row = db.execute(
|
||||
"SELECT state FROM memory_write_jobs WHERE idempotency_key=?",
|
||||
(key,),
|
||||
).fetchone()
|
||||
if row is not None and row[0] not in {"pending", "running"}:
|
||||
return str(row[0])
|
||||
await asyncio.sleep(0.05)
|
||||
raise AssertionError(f"note job for {evidence_id} never settled")
|
||||
|
||||
|
||||
def _client(app):
|
||||
return AsyncClient(
|
||||
transport=ASGITransport(app=app),
|
||||
base_url="http://test",
|
||||
headers={"Authorization": "Bearer t04"},
|
||||
)
|
||||
|
||||
|
||||
def test_note_is_written_through_hy_import_even_when_the_model_would_extract_nothing(
|
||||
tmp_path, hy_calls
|
||||
):
|
||||
async def scenario():
|
||||
app, model = _application(tmp_path), ScriptedModel()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
await _configure(app, model)
|
||||
receipt = await _remember(app, client, TEXT)
|
||||
|
||||
assert receipt["state"] == "ready", (receipt, [name for name, _ in hy_calls])
|
||||
assert [name for name, _ in hy_calls] == ["add_extracted"]
|
||||
assert hy_calls[0][1]["contents"] == [TEXT]
|
||||
assert len(receipt["memory_ids"]) == 1
|
||||
memory = app.state.memory_governance_store.get(
|
||||
receipt["memory_ids"][0], user_id="local"
|
||||
)
|
||||
assert memory is not None and memory.content == TEXT
|
||||
assert memory.status == "confirmed" and memory.scopes == ("all-apps",)
|
||||
assert receipt["evidence_id"] in memory.evidence_ids
|
||||
assert memory.memory_id in hy_calls[0][1]["raw"]["memory_ids"]
|
||||
assert app.state.evidence_store.get(receipt["evidence_id"]).verbatim == TEXT
|
||||
# Still the person's own words: matter discovery keeps reading the note.
|
||||
selected = select_evidence(app.state.evidence_store.database_path)
|
||||
assert receipt["evidence_id"] in {record.evidence_id for record in selected.records}
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_enabled_continuous_policy_processes_a_saved_note_without_separate_consent(
|
||||
tmp_path, hy_calls
|
||||
):
|
||||
async def scenario():
|
||||
app, model = _application(tmp_path), ScriptedModel()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
await _configure(app, model)
|
||||
app.state.memory_note_store.intake = None
|
||||
profile = app.state.model_profile_store.active(app.state.secret_store)
|
||||
assert profile is not None
|
||||
with app.state.evidence_store._connect(writing=True) as db:
|
||||
ingest_journal.initialize(db)
|
||||
continuous_policy.initialize(db)
|
||||
continuous_policy.enable(
|
||||
db,
|
||||
profile=app.state.model_profile_store.snapshot(
|
||||
profile.profile_id, profile.revision
|
||||
),
|
||||
expected_revision=0,
|
||||
expected_legacy_revision=0,
|
||||
operation_id=str(uuid4()),
|
||||
backlog_mode="none",
|
||||
user_confirmed=True,
|
||||
)
|
||||
|
||||
saved = await client.post(
|
||||
"/api/v1/memory-notes",
|
||||
json={"request_id": str(uuid4()), "text": TEXT, "scope": "all-apps"},
|
||||
)
|
||||
assert saved.status_code == 201, saved.text
|
||||
processor = app.state.memory_processing_supervisor
|
||||
await asyncio.gather(*tuple(processor._workers.values()))
|
||||
receipt = (
|
||||
await client.get(f"/api/v1/memory-notes/{saved.json()['evidence_id']}")
|
||||
).json()
|
||||
|
||||
assert receipt["state"] == "ready"
|
||||
assert [name for name, _ in hy_calls] == ["add_extracted"]
|
||||
with app.state.evidence_store._connect() as db:
|
||||
assert (
|
||||
db.execute("SELECT COUNT(*) FROM model_content_permissions").fetchone()[0] == 0
|
||||
)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.xfail(
|
||||
strict=True,
|
||||
reason=(
|
||||
"T04 wait-user: a note's permission lease covers only that note's evidence, so HY "
|
||||
"cannot see the memory an earlier note produced and adds a second one. Letting it "
|
||||
"see earlier notes means sending them to the model with this one: a consent-scope "
|
||||
"decision. The bridge's no-ids fallback also cannot find it (decorated search "
|
||||
"content; the leased vector store has no keyword_search)."
|
||||
),
|
||||
)
|
||||
def test_identical_second_note_points_at_the_memory_hy_already_holds(tmp_path, hy_calls):
|
||||
async def scenario():
|
||||
app, model = _application(tmp_path), ScriptedModel()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
await _configure(app, model)
|
||||
first = await _remember(app, client, TEXT)
|
||||
assert first["state"] == "ready", first
|
||||
second = await _remember(app, client, TEXT)
|
||||
|
||||
# HY really deduplicated: reconcile saw the first memory and wrote nothing new.
|
||||
assert len(model.reconcile_prompts) == 1 and TEXT in model.reconcile_prompts[0]
|
||||
assert first["memory_ids"][0] in model.reconcile_prompts[0]
|
||||
assert not hy_calls[-1][1]["raw"].get("memory_ids")
|
||||
assert second["state"] == "ready", second
|
||||
assert second["memory_ids"] == first["memory_ids"]
|
||||
memories = app.state.memory_governance_store.list(user_id="local")
|
||||
assert [item.memory_id for item in memories] == first["memory_ids"]
|
||||
assert {first["evidence_id"], second["evidence_id"]} <= set(memories[0].evidence_ids)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_note_saved_before_t04_is_imported_too_and_keeps_all_apps_scope(tmp_path):
|
||||
"""Notes stored before T04 are not migrated; retrying one must still write it as said."""
|
||||
calls = []
|
||||
|
||||
class Bridge:
|
||||
async def ingest(self, request):
|
||||
pytest.fail("A note the person asked to keep must be imported, not re-extracted")
|
||||
|
||||
async def ingest_extracted(self, request, contents):
|
||||
calls.append(list(contents))
|
||||
return HyMemoryIngestResult(
|
||||
success=True,
|
||||
memories=(
|
||||
HyMemoryOutput(
|
||||
memory_id="legacy-note-memory",
|
||||
content=TEXT,
|
||||
owner="user",
|
||||
layer="L2_FACT",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
async def profile_memories(self, **kwargs):
|
||||
return ()
|
||||
|
||||
async def close(self):
|
||||
pass
|
||||
|
||||
async def scenario():
|
||||
app, model = _application(tmp_path), ScriptedModel()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
await _configure(app, model)
|
||||
request_id = str(uuid4())
|
||||
record = app.state.evidence_store.put(
|
||||
EvidenceInput(
|
||||
source_id=SOURCE_ID,
|
||||
conversation_id=f"note:{request_id}",
|
||||
message_id=request_id,
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
created_at=datetime.now(UTC),
|
||||
verbatim=TEXT,
|
||||
origin_meta={
|
||||
EvidenceOrigin.MANUAL_IMPORT: OriginMetadata(
|
||||
raw_meta={
|
||||
"input_mode": "direct_memory_note",
|
||||
"recall_scope": "all-apps",
|
||||
}
|
||||
)
|
||||
},
|
||||
)
|
||||
).record
|
||||
database = app.state.evidence_store.database_path
|
||||
with closing(sqlite3.connect(database)) as db, db:
|
||||
db.execute(
|
||||
"INSERT INTO source_grants VALUES ('local',?,?,'default',?,NULL)",
|
||||
(SOURCE_ID, str(database.parent), datetime.now(UTC).isoformat()),
|
||||
)
|
||||
await confirm_model_content(
|
||||
client,
|
||||
source_ids=(SOURCE_ID,),
|
||||
evidence_ids=(record.evidence_id,),
|
||||
future_additions=False,
|
||||
)
|
||||
processor = app.state.memory_processing_supervisor
|
||||
|
||||
@asynccontextmanager
|
||||
async def isolated(lease, _secret, _data):
|
||||
lease.recheck()
|
||||
yield Bridge()
|
||||
|
||||
processor.permission_bridge_factory = isolated
|
||||
app.state.memory_write_drain.stop()
|
||||
await app.state.memory_write_drain_task
|
||||
started = await client.post(f"/api/v1/memory-notes/{record.evidence_id}/process")
|
||||
assert started.status_code == 200, started.text
|
||||
assert await app.state.memory_write_drain.run_one()
|
||||
deadline = asyncio.get_running_loop().time() + 5
|
||||
while True:
|
||||
await asyncio.gather(*tuple(processor._workers.values()))
|
||||
receipt = (await client.get(f"/api/v1/memory-notes/{record.evidence_id}")).json()
|
||||
if receipt["state"] != "running" or asyncio.get_running_loop().time() >= deadline:
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
assert calls == [[TEXT]]
|
||||
assert receipt["state"] == "ready" and receipt["memory_ids"] == ["legacy-note-memory"]
|
||||
memory = app.state.memory_governance_store.get("legacy-note-memory", user_id="local")
|
||||
assert memory.scopes == ("all-apps",) and memory.content == TEXT
|
||||
assert record.evidence_id in memory.evidence_ids
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_model_outage_keeps_the_original_words_and_a_retry_writes_them(tmp_path, hy_calls):
|
||||
async def scenario():
|
||||
app, model = _application(tmp_path), ScriptedModel()
|
||||
async with app.router.lifespan_context(app), _client(app) as client:
|
||||
await _configure(app, model)
|
||||
model.embeddings_down = True
|
||||
failed = await _remember(app, client, TEXT)
|
||||
assert failed["state"] in {"waiting_for_model", "failed"}, failed
|
||||
assert failed["memory_ids"] == []
|
||||
assert app.state.evidence_store.get(failed["evidence_id"]).verbatim == TEXT
|
||||
assert app.state.memory_governance_store.list(user_id="local") == []
|
||||
|
||||
model.embeddings_down = False
|
||||
retried = await client.post(
|
||||
"/api/v1/memory-processing/retry-unknown", json={"confirm": True}
|
||||
)
|
||||
assert retried.status_code == 200, retried.text
|
||||
await _wait_for_note_job(app, failed["evidence_id"])
|
||||
receipt = (await client.get(f"/api/v1/memory-notes/{failed['evidence_id']}")).json()
|
||||
assert receipt["state"] == "ready", receipt
|
||||
assert hy_calls[-1][0] == "add_extracted" and hy_calls[-1][1]["contents"] == [TEXT]
|
||||
memory = app.state.memory_governance_store.get(
|
||||
receipt["memory_ids"][0], user_id="local"
|
||||
)
|
||||
assert memory.content == TEXT and failed["evidence_id"] in memory.evidence_ids
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("raw_meta", "imported"),
|
||||
[
|
||||
({"input_mode": "direct_memory_note", "recall_scope": "all-apps"}, True),
|
||||
({"matter_id": "matter-1"}, False),
|
||||
],
|
||||
ids=["memory-note", "matter-note"],
|
||||
)
|
||||
def test_only_a_note_saved_as_memory_skips_extraction(tmp_path, raw_meta, imported):
|
||||
"""Matter notes share the manual source but are context for a matter, not a memory."""
|
||||
stores = _setup(tmp_path)
|
||||
evidence, _profiles, tasks, governance, _secrets, profile = stores
|
||||
record = evidence.put(
|
||||
EvidenceInput(
|
||||
source_id=SOURCE_ID,
|
||||
conversation_id=f"note:{uuid4()}",
|
||||
message_id=str(uuid4()),
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
created_at=datetime.now(UTC),
|
||||
verbatim=TEXT,
|
||||
origin_meta={EvidenceOrigin.MANUAL_IMPORT: OriginMetadata(raw_meta=raw_meta)},
|
||||
)
|
||||
).record
|
||||
bridge = FakeBridge()
|
||||
|
||||
asyncio.run(_runner(tmp_path, stores, bridge).run(_task(tasks, profile, record).task_id))
|
||||
|
||||
assert len(bridge.requests) == 1
|
||||
assert bridge.extracted_contents == ([(TEXT,)] if imported else [])
|
||||
if imported:
|
||||
(memory,) = governance.list(user_id="local")
|
||||
assert memory.content == TEXT and record.evidence_id in memory.evidence_ids
|
||||
@@ -1,6 +1,5 @@
|
||||
import asyncio
|
||||
import sqlite3
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
@@ -9,9 +8,7 @@ from pydantic import SecretStr
|
||||
|
||||
from one_memory.api import create_app
|
||||
from one_memory.config import Settings
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError, HyMemoryIngestResult, HyMemoryOutput
|
||||
from one_memory.imports.models import WrittenBy
|
||||
from one_memory.model_profiles import ModelCapabilities, ModelProfileInput, ProviderKind
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
|
||||
|
||||
@@ -37,30 +34,6 @@ def client_for(app):
|
||||
)
|
||||
|
||||
|
||||
async def wait_for_note_job(app, evidence_id, *, timeout=10.0):
|
||||
"""Wait for the durable note job to leave pending/running.
|
||||
|
||||
The note now reaches HY through the shared durable queue
|
||||
(`entry:<evidence_id>`), so the drain - not the supervisor's in-memory
|
||||
workers - is what completes it. Waiting on `processor._workers` directly
|
||||
races that drain and can observe zero requests.
|
||||
"""
|
||||
import time as _time
|
||||
|
||||
deadline = _time.monotonic() + timeout
|
||||
key = f"entry:{evidence_id}"
|
||||
while _time.monotonic() < deadline:
|
||||
with sqlite3.connect(app.state.evidence_store.database_path) as db:
|
||||
row = db.execute(
|
||||
"SELECT state FROM memory_write_jobs WHERE idempotency_key=?",
|
||||
(key,),
|
||||
).fetchone()
|
||||
if row is not None and row[0] not in {"pending", "running"}:
|
||||
return row[0]
|
||||
await asyncio.sleep(0.05)
|
||||
raise AssertionError(f"note job for {evidence_id} never settled")
|
||||
|
||||
|
||||
def test_empty_library_can_save_a_note_without_a_matter_or_configured_model(tmp_path):
|
||||
async def scenario():
|
||||
app = application(tmp_path)
|
||||
@@ -153,12 +126,6 @@ def test_saved_note_evidence_returns_raw_and_consent_reference_from_same_record(
|
||||
assert value["evidence_id"] == saved["evidence_id"] and value["revision"] == 1
|
||||
assert value["role"] == "user" and value["written_by"] == "person"
|
||||
assert value["available"] is True and value["source_id"] == "one-memory-manual"
|
||||
from one_memory.model_content_permissions import model_content_evidence_snapshot
|
||||
|
||||
record = app.state.evidence_store.get(saved["evidence_id"])
|
||||
reference = model_content_evidence_snapshot(record).model_dump(mode="json")
|
||||
assert all(value[key] == expected for key, expected in reference.items())
|
||||
assert app.state.model_content_permissions.receipt(user_id="local") is None
|
||||
assert app.state.memory_task_store.list() == []
|
||||
assert (await client.get("/api/v1/memory-notes/missing/evidence")).status_code == 404
|
||||
app.state.source_grant_store.revoke(user_id="local", source_id="one-memory-manual")
|
||||
@@ -210,128 +177,3 @@ def test_revoked_note_source_is_not_silently_reauthorized(tmp_path):
|
||||
assert (await client.get("/api/v1/memory-notes")).json() == []
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fail", [False, True])
|
||||
def test_note_reuses_hy_queue_and_never_reports_saved_raw_text_as_extracted(tmp_path, fail):
|
||||
async def scenario():
|
||||
app = application(tmp_path)
|
||||
requests = []
|
||||
|
||||
class Bridge:
|
||||
async def ingest(self, request):
|
||||
pytest.fail("A note the person asked to keep must be imported, not re-extracted")
|
||||
|
||||
async def ingest_extracted(self, request, contents):
|
||||
assert list(contents) == [turn.content for turn in request.turns]
|
||||
requests.append(request)
|
||||
if fail:
|
||||
raise HyMemoryBridgeError("hy_memory_unavailable")
|
||||
return HyMemoryIngestResult(
|
||||
success=True,
|
||||
memories=(
|
||||
HyMemoryOutput(
|
||||
memory_id="note-memory",
|
||||
content="用户不在周五安排项目周会。",
|
||||
owner="user",
|
||||
layer="L2_FACT",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
async def profile_memories(self, **kwargs):
|
||||
return ()
|
||||
|
||||
async def close(self):
|
||||
pass
|
||||
|
||||
async with app.router.lifespan_context(app), client_for(app) as client:
|
||||
profiles, secrets = app.state.model_profile_store, app.state.secret_store
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(name="Test", provider_kind=ProviderKind.OLLAMA, model="fixture"),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True, model_available=True, embedding_available=True
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
processor = app.state.memory_processing_supervisor
|
||||
|
||||
@asynccontextmanager
|
||||
async def isolated_bridge(lease, _secret, _data):
|
||||
lease.recheck()
|
||||
yield Bridge()
|
||||
|
||||
processor.permission_bridge_factory = isolated_bridge
|
||||
response = await client.post(
|
||||
"/api/v1/memory-notes",
|
||||
json={
|
||||
"request_id": str(uuid4()),
|
||||
"text": "请记住:项目周会不排在周五。",
|
||||
"scope": "all-apps",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 201
|
||||
saved = response.json()
|
||||
assert saved["state"] == "waiting_for_permission"
|
||||
assert requests == []
|
||||
# Reproduce E5: the source is revoked while this unsent durable
|
||||
# note is parked, so the queue records a terminal cancellation.
|
||||
# Explicit processing after a fresh grant must safely revive it.
|
||||
with sqlite3.connect(app.state.evidence_store.database_path) as db:
|
||||
db.execute(
|
||||
"UPDATE memory_write_jobs SET state='cancelled',"
|
||||
"last_error='model_content_permission_revoked',hy_invoked=0 "
|
||||
"WHERE idempotency_key=?",
|
||||
(f"entry:{saved['evidence_id']}",),
|
||||
)
|
||||
preview = await client.post(
|
||||
"/api/v1/model-content-permission/preview",
|
||||
json={
|
||||
"profile_id": profile.profile_id,
|
||||
"profile_revision": profile.revision,
|
||||
"source_ids": ["one-memory-manual"],
|
||||
"evidence_ids": [saved["evidence_id"]],
|
||||
},
|
||||
)
|
||||
assert preview.status_code == 200
|
||||
consent = await client.post(
|
||||
"/api/v1/model-content-permission",
|
||||
json={
|
||||
"snapshot": preview.json(),
|
||||
"expected_revision": 0,
|
||||
"user_confirmed": True,
|
||||
},
|
||||
)
|
||||
assert consent.status_code == 200
|
||||
started = await client.post(f"/api/v1/memory-notes/{saved['evidence_id']}/process")
|
||||
assert started.status_code == 200
|
||||
# The note is completed by the shared durable queue now; wait on its
|
||||
# job rather than the supervisor's in-memory workers.
|
||||
await wait_for_note_job(app, saved["evidence_id"])
|
||||
receipt = (await client.get(f"/api/v1/memory-notes/{saved['evidence_id']}")).json()
|
||||
assert (
|
||||
len(requests) == 1
|
||||
and requests[0].turns[0].content == "请记住:项目周会不排在周五。"
|
||||
)
|
||||
assert app.state.evidence_store.count() == 1
|
||||
if fail:
|
||||
assert receipt["state"] == "waiting_for_model" and not receipt["memory_ids"]
|
||||
assert receipt["error"] == "hy_memory_unavailable"
|
||||
else:
|
||||
assert receipt["state"] == "ready" and receipt["memory_ids"] == ["note-memory"]
|
||||
memory = app.state.memory_governance_store.get("note-memory", user_id="local")
|
||||
assert memory.scopes == ("all-apps",)
|
||||
recall = await client.post("/api/v1/recall", json={"query": "周五 项目周会"})
|
||||
assert recall.status_code == 200
|
||||
assert any(hit["memory_id"] == "note-memory" for hit in recall.json()["hits"])
|
||||
await client.post(f"/api/v1/memory-notes/{saved['evidence_id']}/process", json={})
|
||||
await wait_for_note_job(app, saved["evidence_id"])
|
||||
assert len(requests) == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
@@ -315,6 +315,12 @@ def test_memory_task_logs_six_step_durations_without_content(tmp_path) -> None:
|
||||
events = [json.loads(line) for line in written.splitlines()]
|
||||
steps = [event for event in events if event["event"] == "memory_task.step_completed"]
|
||||
assert [event["details"]["phase"] for event in steps] == ["1", "2", "3", "4", "5", "6"]
|
||||
finished = next(event for event in events if event["event"] == "memory_task.finished")
|
||||
assert finished["details"] == {
|
||||
"attempt": 1,
|
||||
"pending_count": 0,
|
||||
"result_count": 1,
|
||||
}
|
||||
assert tasks.processing_metrics()["hy_calls"] == 1
|
||||
assert all(event["duration_ms"] >= 0 for event in steps)
|
||||
assert marker not in written
|
||||
@@ -2925,7 +2931,7 @@ def test_frozen_harness_refuses_new_processing_before_a_task_or_model_call(tmp_p
|
||||
supervisor._bridges[(profile.profile_id, profile.revision, None)] = bridge
|
||||
|
||||
assert supervisor._source_may_write([user]) is False
|
||||
with pytest.raises(HyMemoryBridgeError, match="model_content_source_disabled"):
|
||||
with pytest.raises(HyMemoryBridgeError, match="hy_memory_source_disabled"):
|
||||
asyncio.run(supervisor.process_records((user, reply)))
|
||||
assert asyncio.run(supervisor.enqueue_conversation_off_thread(user)) == []
|
||||
assert supervisor.prepare_backfill() == []
|
||||
@@ -2999,7 +3005,7 @@ def test_revoked_grant_refuses_new_processing(tmp_path) -> None:
|
||||
supervisor._bridges[(profile.profile_id, profile.revision, None)] = bridge
|
||||
|
||||
assert supervisor._source_may_write([user]) is False
|
||||
with pytest.raises(HyMemoryBridgeError, match="model_content_source_disabled"):
|
||||
with pytest.raises(HyMemoryBridgeError, match="hy_memory_source_disabled"):
|
||||
asyncio.run(supervisor.process_records((user, reply)))
|
||||
assert asyncio.run(supervisor.enqueue_conversation_off_thread(user)) == []
|
||||
assert supervisor.prepare_backfill() == []
|
||||
|
||||
@@ -13,7 +13,6 @@ def test_sync_config_defaults_off_and_survives_restart(tmp_path):
|
||||
|
||||
saved = json.loads((tmp_path / "config.json").read_text())
|
||||
assert saved == {
|
||||
"model_content_allowed": False,
|
||||
"schema_version": 1,
|
||||
"sources": {"cursor": {"sync_enabled": True}},
|
||||
}
|
||||
@@ -31,19 +30,27 @@ def test_sync_config_updates_one_source_without_losing_another(tmp_path):
|
||||
assert reopened.sync_enabled("claude-code")
|
||||
|
||||
|
||||
def test_model_content_permission_is_one_global_final_state(tmp_path):
|
||||
def test_legacy_model_content_permission_is_ignored_on_upgrade(tmp_path):
|
||||
(tmp_path / "config.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": 1,
|
||||
"model_content_allowed": False,
|
||||
"sources": {"cursor": {"sync_enabled": True}},
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
store = ConfigStore(tmp_path)
|
||||
assert not store.model_content_allowed()
|
||||
|
||||
store.set_model_content_allowed(True)
|
||||
store.set_sync("cursor", True)
|
||||
reopened = ConfigStore(tmp_path)
|
||||
assert reopened.model_content_allowed()
|
||||
assert reopened.sync_enabled("cursor")
|
||||
|
||||
reopened.set_model_content_allowed(False)
|
||||
assert not ConfigStore(tmp_path).model_content_allowed()
|
||||
assert ConfigStore(tmp_path).sync_enabled("cursor")
|
||||
assert store.sync_enabled("cursor")
|
||||
store.set_sync("codex", True)
|
||||
assert json.loads((tmp_path / "config.json").read_text()) == {
|
||||
"schema_version": 1,
|
||||
"sources": {
|
||||
"codex": {"sync_enabled": True},
|
||||
"cursor": {"sync_enabled": True},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def test_invalid_config_fails_loudly_instead_of_silently_disabling_sources(tmp_path):
|
||||
|
||||
@@ -1097,6 +1097,52 @@ def test_reconciled_absent_timeout_is_requeued_once_without_resetting_attempts(
|
||||
)
|
||||
|
||||
|
||||
def test_unresolved_unknown_is_automatically_retried_only_once(tmp_path) -> None:
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
queue = DurableWriteQueue(path)
|
||||
record = queue.submit(_request())
|
||||
with sqlite3.connect(path) as connection, connection:
|
||||
connection.execute(
|
||||
"UPDATE memory_write_jobs SET state='unknown',attempts=1,hy_invoked=1,"
|
||||
"last_error='model_content_outcome_unknown',updated_at=? WHERE job_id=?",
|
||||
("2026-01-01T00:00:00+00:00", record.job_id),
|
||||
)
|
||||
|
||||
cutoff = datetime(2026, 1, 1, 0, 5, tzinfo=UTC)
|
||||
assert queue.retry_first_unknown_after_reconcile(older_than=cutoff) == 1
|
||||
first_retry = queue.get(record.request_id)
|
||||
assert first_retry.state == PENDING
|
||||
with sqlite3.connect(path) as connection:
|
||||
assert connection.execute(
|
||||
"SELECT attempts FROM memory_write_jobs WHERE job_id=?",
|
||||
(record.job_id,),
|
||||
).fetchone()[0] == 1
|
||||
|
||||
drain = DurableWriteDrain(queue, execute=_never_execute, owner="retry-owner")
|
||||
assert drain.claim(limit=1) == (record.job_id,)
|
||||
assert queue.mark_invoked(record.job_id, owner="retry-owner")
|
||||
queue.settle(
|
||||
record.job_id,
|
||||
state=UNKNOWN,
|
||||
owner="retry-owner",
|
||||
error_code="model_content_outcome_unknown",
|
||||
hy_invoked=True,
|
||||
)
|
||||
|
||||
with sqlite3.connect(path) as connection:
|
||||
assert connection.execute(
|
||||
"SELECT attempts FROM memory_write_jobs WHERE job_id=?",
|
||||
(record.job_id,),
|
||||
).fetchone()[0] == 2
|
||||
assert (
|
||||
queue.retry_first_unknown_after_reconcile(
|
||||
older_than=datetime.now(UTC) + timedelta(minutes=1)
|
||||
)
|
||||
== 0
|
||||
)
|
||||
assert queue.get(record.request_id).state == UNKNOWN
|
||||
|
||||
|
||||
def test_drain_runs_unknown_reconciliation_during_busy_safe_maintenance(tmp_path) -> None:
|
||||
queue = DurableWriteQueue(tmp_path / "one-memory.sqlite3")
|
||||
calls = 0
|
||||
@@ -1116,6 +1162,20 @@ def test_drain_runs_unknown_reconciliation_during_busy_safe_maintenance(tmp_path
|
||||
assert calls == 1
|
||||
|
||||
|
||||
def test_unknown_reconciliation_runs_on_a_ten_second_ceiling(tmp_path) -> None:
|
||||
queue = DurableWriteQueue(tmp_path / "one-memory.sqlite3")
|
||||
|
||||
production = DurableWriteDrain(queue, execute=_never_execute)
|
||||
immediate = DurableWriteDrain(
|
||||
queue,
|
||||
execute=_never_execute,
|
||||
reap_interval=timedelta(0),
|
||||
)
|
||||
|
||||
assert production._unknown_reconcile_interval == timedelta(seconds=10)
|
||||
assert immediate._unknown_reconcile_interval == timedelta(0)
|
||||
|
||||
|
||||
def test_a_reclaimed_uninvoked_job_with_attempts_left_is_retryable(tmp_path) -> None:
|
||||
"""Recovery of a never-invoked lease is the retry path, not `unknown`.
|
||||
|
||||
@@ -2063,6 +2123,19 @@ def test_sqlite_claim_keeps_fifo_order_when_later_work_is_live(tmp_path) -> None
|
||||
assert drain.claim(limit=1) == (normal.job_id,)
|
||||
|
||||
|
||||
def test_unknown_source_head_does_not_block_later_pending_work(tmp_path) -> None:
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
queue = DurableWriteQueue(path)
|
||||
uncertain = queue.submit(_request(evidence=("e1",), key="unknown"))
|
||||
later = queue.submit(_request(evidence=("e2",), key="later"))
|
||||
queue.settle(uncertain.job_id, state=UNKNOWN, hy_invoked=True)
|
||||
|
||||
drain = DurableWriteDrain(queue, execute=_never_execute, owner="next-owner")
|
||||
|
||||
assert drain.claim(limit=1) == (later.job_id,)
|
||||
assert queue.get("unknown").state == UNKNOWN
|
||||
|
||||
|
||||
def test_sqlite_claim_orders_different_sources_by_enqueue_sequence(tmp_path) -> None:
|
||||
path = tmp_path / "one-memory.sqlite3"
|
||||
queue = DurableWriteQueue(path)
|
||||
@@ -3267,103 +3340,6 @@ def test_idle_maintenance_database_failure_is_sampled_without_database_error(
|
||||
assert events[-1]["error_type"] == "RuntimeError"
|
||||
|
||||
|
||||
@pytest.mark.slow
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.serial
|
||||
def test_no_model_permission_parks_without_sending(tmp_path) -> None:
|
||||
"""Model egress permission is separate from source read permission.
|
||||
|
||||
With a permission guard installed but no grant, the durable job must park
|
||||
in `waiting_for_permission` and never reach HY: the Evidence is committed
|
||||
locally, but no private verbatim is sent to a model. It must not read as
|
||||
`failed` (a user switch is not a pipeline break), and a later grant must be
|
||||
able to re-admit it.
|
||||
"""
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from one_memory.imports.models import (
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
OriginMetadata,
|
||||
WrittenBy,
|
||||
)
|
||||
from one_memory.imports.storage import EvidenceStore
|
||||
from one_memory.memory_governance import MemoryGovernanceStore
|
||||
from one_memory.memory_processing import MemoryProcessingSupervisor
|
||||
from one_memory.memory_tasks import MemoryTaskStore
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
from one_memory.model_profiles import (
|
||||
ModelCapabilities,
|
||||
ModelProfileInput,
|
||||
ModelProfileStore,
|
||||
ProviderKind,
|
||||
)
|
||||
from one_memory.secrets import InMemorySecretStore
|
||||
|
||||
database = tmp_path / "one-memory.sqlite3"
|
||||
secrets = InMemorySecretStore()
|
||||
store = EvidenceStore(database)
|
||||
record = store.put(
|
||||
EvidenceInput(
|
||||
source_id="cursor",
|
||||
conversation_id="c",
|
||||
message_id="m-private",
|
||||
role="user",
|
||||
written_by=WrittenBy.PERSON,
|
||||
verbatim="private verbatim that must not be sent",
|
||||
created_at=datetime.now(UTC),
|
||||
origin_meta={EvidenceOrigin.FILE: OriginMetadata()},
|
||||
)
|
||||
).record
|
||||
|
||||
profiles = ModelProfileStore(database)
|
||||
profile = profiles.create(
|
||||
ModelProfileInput(
|
||||
name="Local",
|
||||
provider_kind=ProviderKind.OLLAMA,
|
||||
model="hy-bot-phi4",
|
||||
embedding_model="nomic-embed-text",
|
||||
),
|
||||
secrets,
|
||||
)
|
||||
profiles.record_capabilities(
|
||||
profile.profile_id,
|
||||
expected_revision=profile.revision,
|
||||
capabilities=ModelCapabilities(
|
||||
endpoint_reachable=True,
|
||||
model_available=True,
|
||||
structured_output=True,
|
||||
embedding_available=True,
|
||||
),
|
||||
secrets=secrets,
|
||||
)
|
||||
profiles.activate(profile.profile_id, secrets)
|
||||
|
||||
config = ConfigStore(tmp_path)
|
||||
config.set_model_content_allowed(False)
|
||||
supervisor = MemoryProcessingSupervisor(
|
||||
tasks=MemoryTaskStore(database),
|
||||
profiles=profiles,
|
||||
secrets=secrets,
|
||||
evidence=store,
|
||||
governance=MemoryGovernanceStore(database),
|
||||
data_dir=tmp_path,
|
||||
model_content_config=config,
|
||||
)
|
||||
|
||||
queue = DurableWriteQueue(database)
|
||||
submitted = queue.submit(_request(evidence=(record.evidence_id,)))
|
||||
before = queue.get(submitted.request_id)
|
||||
assert before.state == "pending"
|
||||
_claim_through_drain(queue, submitted.job_id)
|
||||
|
||||
asyncio.run(execute_job(submitted.job_id, queue=queue, supervisor=supervisor, evidence=store))
|
||||
|
||||
settled = queue.get(submitted.request_id)
|
||||
assert settled.state == "waiting_for_permission", settled.state
|
||||
assert "permission" in settled.detail.lower()
|
||||
|
||||
|
||||
def test_an_unknown_is_retired_only_by_an_inquiry(tmp_path) -> None:
|
||||
"""`unknown` must be resolvable, but never by a blind retry.
|
||||
|
||||
|
||||
@@ -2103,7 +2103,47 @@ def test_migration_49_resets_only_cursor_history_when_conversation_time_is_missi
|
||||
|
||||
result = asyncio.run(migrate(SqliteMigrationStore(path), MIGRATIONS))
|
||||
|
||||
assert result.applied == (49, 50)
|
||||
assert result.applied == (49, 50, 51)
|
||||
with main_database(path).read() as connection:
|
||||
assert connection.execute("SELECT COUNT(*) FROM source_cursors").fetchone()[0] == 0
|
||||
assert connection.execute("SELECT COUNT(*) FROM live_source_cursors").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_upgrade_drops_obsolete_hy_node_permissions_without_touching_memories(
|
||||
tmp_path,
|
||||
) -> None:
|
||||
import sqlite3
|
||||
|
||||
from one_memory.memory_server.migration_runner import SqliteMigrationStore
|
||||
|
||||
path = tmp_path / "existing.sqlite3"
|
||||
asyncio.run(
|
||||
migrate(
|
||||
SqliteMigrationStore(path),
|
||||
tuple(migration for migration in MIGRATIONS if migration.version <= 50),
|
||||
)
|
||||
)
|
||||
with sqlite3.connect(path) as connection:
|
||||
connection.execute(
|
||||
"CREATE TABLE hy_node_permissions ("
|
||||
"node_id TEXT PRIMARY KEY, fingerprint TEXT NOT NULL, evidence_json TEXT NOT NULL)"
|
||||
)
|
||||
connection.execute(
|
||||
"INSERT INTO hy_node_permissions(node_id,fingerprint,evidence_json) "
|
||||
"VALUES ('legacy-node', ?, '[]')",
|
||||
("a" * 64,),
|
||||
)
|
||||
|
||||
result = asyncio.run(migrate(SqliteMigrationStore(path), MIGRATIONS))
|
||||
|
||||
assert result.applied == (51,)
|
||||
with sqlite3.connect(path) as connection:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in connection.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'"
|
||||
)
|
||||
}
|
||||
assert "hy_node_permissions" not in tables
|
||||
assert "governed_memories" in tables
|
||||
assert "evidence" in tables
|
||||
@@ -1070,6 +1070,49 @@ def test_confirmed_retry_requeues_task_only_unknown_without_a_durable_job(tmp_pa
|
||||
assert store.retry_unknown_confirmed() == 0
|
||||
|
||||
|
||||
def test_task_only_unknown_is_automatically_retried_only_once(tmp_path) -> None:
|
||||
store, profile = _profile_and_tasks(tmp_path)
|
||||
task = store.create(
|
||||
request_id="auto-retry-unknown",
|
||||
user_id="local",
|
||||
kind=MemoryTaskKind.EXTRACT,
|
||||
input_ref="orphan-evidence",
|
||||
profile=profile,
|
||||
)
|
||||
with sqlite3.connect(store.database_path) as connection, connection:
|
||||
connection.execute(
|
||||
"UPDATE memory_tasks SET status=?,attempts=1,last_error_code=?,updated_at=? "
|
||||
"WHERE task_id=?",
|
||||
(
|
||||
MemoryTaskStatus.UNKNOWN,
|
||||
"model_content_outcome_unknown",
|
||||
"2026-01-01T00:00:00+00:00",
|
||||
task.task_id,
|
||||
),
|
||||
)
|
||||
|
||||
cutoff = datetime(2026, 1, 1, 0, 5, tzinfo=UTC)
|
||||
assert store.retry_first_unknown_after_reconcile(older_than=cutoff) == 1
|
||||
retried = store.require(task.task_id)
|
||||
assert retried.status is MemoryTaskStatus.PENDING
|
||||
assert retried.attempts == 1
|
||||
|
||||
running, token = store.claim(task.task_id)
|
||||
assert running.attempts == 2
|
||||
store.mark_unknown(
|
||||
task.task_id,
|
||||
runner_token=token,
|
||||
error_code="model_content_outcome_unknown",
|
||||
)
|
||||
assert (
|
||||
store.retry_first_unknown_after_reconcile(
|
||||
older_than=datetime.now(UTC) + timedelta(minutes=1)
|
||||
)
|
||||
== 0
|
||||
)
|
||||
assert store.require(task.task_id).status is MemoryTaskStatus.UNKNOWN
|
||||
|
||||
|
||||
def test_orphan_recovery_preserves_active_running_tasks(tmp_path) -> None:
|
||||
store, profile = _profile_and_tasks(tmp_path)
|
||||
active = store.create(
|
||||
|
||||
@@ -1,303 +0,0 @@
|
||||
"""Bulk consent reads preserve exact identities and SQLite snapshot boundaries."""
|
||||
|
||||
import asyncio
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from one_memory.database_access import close_main_databases
|
||||
from one_memory.imports.models import EvidenceOrigin, OriginMetadata, WrittenBy
|
||||
from one_memory.model_content_permissions import (
|
||||
ModelContentPermissionError,
|
||||
ModelContentPermissionSnapshot,
|
||||
model_content_evidence_snapshot,
|
||||
)
|
||||
from one_memory.model_content_routes import model_content_router
|
||||
from tests.test_model_content_permissions import (
|
||||
SOURCE,
|
||||
USER,
|
||||
evidence_value,
|
||||
)
|
||||
from tests.test_model_content_permissions import (
|
||||
env as environment_fixture,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def env(tmp_path):
|
||||
return environment_fixture.__wrapped__(tmp_path)
|
||||
|
||||
|
||||
def many_records(environment, count):
|
||||
records = [environment.record]
|
||||
with environment.evidence._connect(writing=True) as connection:
|
||||
for index in range(1, count):
|
||||
value = evidence_value(str(index), text=f"Synthetic {index}: " + "x" * 128)
|
||||
if index % 2:
|
||||
value = value.model_copy(
|
||||
update={
|
||||
"role": "assistant",
|
||||
"written_by": WrittenBy.AGENT,
|
||||
"origin_meta": {
|
||||
EvidenceOrigin.FILE: OriginMetadata(raw_meta={"nested": [1, "text"]}),
|
||||
EvidenceOrigin.HOOK: OriginMetadata(source_version="fixture-v1"),
|
||||
},
|
||||
}
|
||||
)
|
||||
records.append(environment.evidence.put(value, connection=connection).record)
|
||||
return records
|
||||
|
||||
|
||||
def app_for(environment):
|
||||
app = FastAPI()
|
||||
app.state.model_profile_store = environment.profiles
|
||||
app.state.evidence_store = environment.evidence
|
||||
app.state.model_content_permissions = environment.permissions
|
||||
app.include_router(model_content_router([]))
|
||||
return app
|
||||
|
||||
|
||||
def preview(environment, **changes):
|
||||
async def request():
|
||||
async with AsyncClient(
|
||||
transport=ASGITransport(app=app_for(environment)), base_url="http://test"
|
||||
) as client:
|
||||
return await client.post(
|
||||
"/api/v1/model-content-permission/preview",
|
||||
json={
|
||||
"profile_id": environment.profile.profile_id,
|
||||
"profile_revision": environment.profile.revision,
|
||||
"source_ids": [SOURCE],
|
||||
"all_imported_from_selected_sources": True,
|
||||
**changes,
|
||||
},
|
||||
)
|
||||
|
||||
return asyncio.run(request())
|
||||
|
||||
|
||||
def observe_connections(monkeypatch):
|
||||
connects, statements = [], []
|
||||
original = sqlite3.connect
|
||||
|
||||
def observed(*args, **kwargs):
|
||||
connection = original(*args, **kwargs)
|
||||
connects.append(connection)
|
||||
connection.set_trace_callback(lambda sql: statements.append(sql))
|
||||
return connection
|
||||
|
||||
close_main_databases() # pooled connections would skip the patch
|
||||
|
||||
monkeypatch.setattr(sqlite3, "connect", observed)
|
||||
return connects, statements
|
||||
|
||||
|
||||
def test_preview_uses_bounded_bulk_reads_without_per_id_connections(env, monkeypatch):
|
||||
records = many_records(env, 1000)
|
||||
expected = env.snapshot(records=tuple(records))
|
||||
connects, statements = observe_connections(monkeypatch)
|
||||
response = preview(env)
|
||||
assert response.status_code == 200
|
||||
actual = ModelContentPermissionSnapshot.model_validate(response.json())
|
||||
assert actual == expected
|
||||
assert len(connects) <= 2, "preview still opens one connection per Evidence ID"
|
||||
selects = [sql for sql in statements if sql.lstrip().upper().startswith("SELECT")]
|
||||
assert len(selects) <= 20, "preview still rereads each record separately"
|
||||
assert env.permissions.receipt(user_id=USER) is None
|
||||
|
||||
|
||||
def test_hot_check_is_bulk_and_reuses_same_transaction_verified_refs(env, monkeypatch):
|
||||
records = many_records(env, 1000)
|
||||
receipt = env.permissions.grant(
|
||||
env.snapshot(records=tuple(records)), expected_revision=0, user_confirmed=True
|
||||
)
|
||||
connects, statements = observe_connections(monkeypatch)
|
||||
assert env.check(receipt, records=(records[0],)) == receipt
|
||||
assert len(connects) == 1
|
||||
selects = [sql for sql in statements if sql.lstrip().upper().startswith("SELECT")]
|
||||
assert len(selects) <= 20, (
|
||||
"checking one input still performs per-record SQL for the entire scope"
|
||||
)
|
||||
|
||||
|
||||
def test_all_source_preview_and_future_boundary_share_one_read_snapshot(env, monkeypatch):
|
||||
records = many_records(env, 4)
|
||||
expected = env.snapshot(records=tuple(records), future=True)
|
||||
source = env.permissions._source
|
||||
changed = False
|
||||
|
||||
def concurrent_insert(connection, user_id, source_id):
|
||||
nonlocal changed
|
||||
result = source(connection, user_id, source_id)
|
||||
if not changed:
|
||||
changed = True
|
||||
env.evidence.put(evidence_value("arrived-during-preview"))
|
||||
return result
|
||||
|
||||
monkeypatch.setattr(env.permissions, "_source", concurrent_insert)
|
||||
response = preview(env, include_future_additions=True)
|
||||
assert response.status_code == 200
|
||||
actual = ModelContentPermissionSnapshot.model_validate(response.json())
|
||||
assert changed and actual == expected, "preview mixed old IDs with a newer future watermark"
|
||||
receipt = env.permissions.grant(actual, expected_revision=0, user_confirmed=True)
|
||||
assert receipt.snapshot == expected
|
||||
added = env.evidence.put(evidence_value("arrived-during-preview")).record
|
||||
assert added.evidence_id not in {ref.evidence_id for ref in actual.evidence}
|
||||
assert env.check(receipt, records=(added,)) == receipt
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mutation", ["body", "author", "origins", "delete"])
|
||||
def test_hot_check_still_rejects_changed_unrequested_scope_record(env, mutation):
|
||||
records = many_records(env, 4)
|
||||
receipt = env.permissions.grant(
|
||||
env.snapshot(records=tuple(records)), expected_revision=0, user_confirmed=True
|
||||
)
|
||||
other = records[-1]
|
||||
with sqlite3.connect(env.database) as db:
|
||||
sql = {
|
||||
"body": (
|
||||
"UPDATE evidence SET verbatim='changed without revision or hash' "
|
||||
"WHERE evidence_id=?"
|
||||
),
|
||||
"author": "UPDATE evidence SET written_by='unknown' WHERE evidence_id=?",
|
||||
"origins": "UPDATE evidence_origins SET source_version='changed' WHERE evidence_id=?",
|
||||
"delete": "DELETE FROM evidence WHERE evidence_id=?",
|
||||
}[mutation]
|
||||
db.execute(sql, (other.evidence_id,))
|
||||
with pytest.raises(ModelContentPermissionError, match="evidence_changed"):
|
||||
env.check(receipt, records=(records[0],))
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"selection,code",
|
||||
[
|
||||
("missing", "evidence_not_found"),
|
||||
("other_user", "evidence_not_found"),
|
||||
("other_source", "invalid_content_scope"),
|
||||
("duplicate", "invalid_content_scope"),
|
||||
("ambiguous", "ambiguous_content_scope"),
|
||||
],
|
||||
)
|
||||
def test_bulk_preview_keeps_exact_selection_failures(env, selection, code):
|
||||
other_user = env.evidence.put(evidence_value("other-user", user_id="other")).record
|
||||
other_source = env.evidence.put(evidence_value("other-source", source_id="cursor")).record
|
||||
identifiers = {
|
||||
"missing": ["not-present"],
|
||||
"other_user": [other_user.evidence_id],
|
||||
"other_source": [other_source.evidence_id],
|
||||
"duplicate": [env.record.evidence_id, env.record.evidence_id],
|
||||
"ambiguous": [env.record.evidence_id],
|
||||
}[selection]
|
||||
response = preview(
|
||||
env, evidence_ids=identifiers, all_imported_from_selected_sources=selection == "ambiguous"
|
||||
)
|
||||
assert response.status_code == 409
|
||||
assert response.json()["detail"] == f"model_content_{code}"
|
||||
assert env.permissions.receipt(user_id=USER) is None
|
||||
|
||||
|
||||
def test_bulk_snapshot_preserves_every_existing_hash_field(env):
|
||||
records = many_records(env, 8)
|
||||
expected = tuple(
|
||||
sorted(
|
||||
(model_content_evidence_snapshot(record) for record in records),
|
||||
key=lambda ref: ref.evidence_id,
|
||||
)
|
||||
)
|
||||
actual = ModelContentPermissionSnapshot.model_validate(preview(env).json())
|
||||
assert actual.evidence == expected
|
||||
assert not any(record.verbatim in actual.model_dump_json() for record in records)
|
||||
|
||||
|
||||
def test_identifier_selection_does_not_mix_a_newer_future_watermark(env, monkeypatch):
|
||||
records = many_records(env, 4)
|
||||
expected = env.snapshot(records=tuple(records), future=True)
|
||||
original_connect = sqlite3.connect
|
||||
inserted = False
|
||||
|
||||
class ConcurrentInsert(sqlite3.Connection):
|
||||
def execute(self, sql, parameters=()):
|
||||
nonlocal inserted
|
||||
cursor = super().execute(sql, parameters)
|
||||
if sql.startswith("SELECT evidence_id FROM evidence WHERE user_id=") and not inserted:
|
||||
inserted = True
|
||||
env.evidence.put(evidence_value("after-identifier-selection"))
|
||||
return cursor
|
||||
|
||||
def connect(*args, **kwargs):
|
||||
return original_connect(*args, **kwargs, factory=ConcurrentInsert)
|
||||
|
||||
close_main_databases() # pooled connections would skip the patch
|
||||
|
||||
monkeypatch.setattr(sqlite3, "connect", connect)
|
||||
response = preview(env, include_future_additions=True)
|
||||
assert inserted and response.status_code == 200
|
||||
snapshot = ModelContentPermissionSnapshot.model_validate(response.json())
|
||||
assert snapshot == expected
|
||||
receipt = env.permissions.grant(snapshot, expected_revision=0, user_confirmed=True)
|
||||
assert receipt.snapshot == expected
|
||||
added = env.evidence.put(evidence_value("after-identifier-selection")).record
|
||||
assert added.evidence_id not in {ref.evidence_id for ref in snapshot.evidence}
|
||||
assert env.check(receipt, records=(added,)) == receipt
|
||||
|
||||
|
||||
def test_bulk_batches_keep_one_snapshot_and_next_check_observes_an_edit(env, monkeypatch):
|
||||
records = many_records(env, 1001)
|
||||
snapshot = env.snapshot(records=tuple(records))
|
||||
receipt = env.permissions.grant(snapshot, expected_revision=0, user_confirmed=True)
|
||||
# Large grants now build a bounded scope seal. Invalidate it explicitly to
|
||||
# exercise the full-snapshot fallback and its single read transaction.
|
||||
with sqlite3.connect(env.database) as db:
|
||||
db.execute(
|
||||
"UPDATE model_content_scope_seals SET valid=0 WHERE user_id=? AND revision=?",
|
||||
(receipt.user_id, receipt.revision),
|
||||
)
|
||||
ordered = sorted(records, key=lambda record: record.evidence_id)
|
||||
later = ordered[-1]
|
||||
original_batch = env.permissions._evidence_batch
|
||||
calls = []
|
||||
|
||||
def edit_after_first_batch(connection, identifiers):
|
||||
yield from original_batch(connection, identifiers)
|
||||
calls.append(len(identifiers))
|
||||
if len(calls) == 1:
|
||||
with sqlite3.connect(env.database) as db:
|
||||
db.execute(
|
||||
"UPDATE evidence SET verbatim='edited but same stored hash' "
|
||||
"WHERE evidence_id=?",
|
||||
(later.evidence_id,),
|
||||
)
|
||||
|
||||
with monkeypatch.context() as scoped:
|
||||
scoped.setattr(env.permissions, "_evidence_batch", edit_after_first_batch)
|
||||
assert env.check(receipt, records=(ordered[0],)) == receipt
|
||||
assert calls == [500, 500, 1]
|
||||
with pytest.raises(ModelContentPermissionError, match="evidence_changed"):
|
||||
env.check(receipt, records=(ordered[0],))
|
||||
|
||||
|
||||
def test_same_transaction_reuse_does_not_trust_forged_request_metadata(env):
|
||||
receipt = env.grant()
|
||||
forged = env.record.model_copy(update={"written_by": WrittenBy.AGENT})
|
||||
with pytest.raises(ModelContentPermissionError, match="evidence_changed"):
|
||||
env.check(receipt, records=(forged,))
|
||||
with pytest.raises(ModelContentPermissionError, match="invalid_content_scope"):
|
||||
env.check(receipt, records=(env.record, env.record))
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"column,value",
|
||||
[
|
||||
("raw_meta_json", "not-json"),
|
||||
("origin", "unknown-origin"),
|
||||
("source_version", "x" * 129),
|
||||
],
|
||||
)
|
||||
def test_bulk_corrupt_origins_fail_closed(env, column, value):
|
||||
receipt = env.grant()
|
||||
with sqlite3.connect(env.database) as db:
|
||||
db.execute(f"UPDATE evidence_origins SET {column}=?", (value,))
|
||||
with pytest.raises(ModelContentPermissionError, match="evidence_changed"):
|
||||
env.check(receipt)
|
||||
Loaded 100 of 200 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user