Compare commits
48
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e026438ced | ||
|
|
f3834df2dc | ||
|
|
dc740604d2 | ||
|
|
55bf946bcc | ||
|
|
14fcc6fe38 | ||
|
|
1ef0766891 | ||
|
|
3370a715dd | ||
|
|
638d1b7bde | ||
|
|
d6890793ec | ||
|
|
32bf05f821 | ||
|
|
b6e6a3fb43 | ||
|
|
4e44f59026 | ||
|
|
35bd56ddd0 | ||
|
|
2cea39da4f | ||
|
|
059e3a4903 | ||
|
|
0a9b10dbfc | ||
|
|
3a74a33456 | ||
|
|
514bf5c47b | ||
|
|
4bc918af7a | ||
|
|
75163a0d1e | ||
|
|
09e4bc8a30 | ||
|
|
4dbf7b6ed2 | ||
|
|
8f1c4045e9 | ||
|
|
8918098fa8 | ||
|
|
0afe787c95 | ||
|
|
67ae43f920 | ||
|
|
3ce89ea560 | ||
|
|
7b7fd217a8 | ||
|
|
7efe054e83 | ||
|
|
fdbcd9934d | ||
|
|
3171007b3c | ||
|
|
a04fcb50d5 | ||
|
|
a28ade2faf | ||
|
|
1c424928a2 | ||
|
|
688878171e | ||
|
|
a3f09f4cac | ||
|
|
0c59413ad8 | ||
|
|
625f877b43 | ||
|
|
1608a4cfc2 | ||
|
|
20b92d8a89 | ||
|
|
2a4c5e38e7 | ||
|
|
c891a1fd1d | ||
|
|
7741e17f09 | ||
|
|
2a4dc2ffa2 | ||
|
|
61b5ef30a9 | ||
|
|
f79770cdb0 | ||
|
|
9ba602ffd2 | ||
|
|
842fbe6044 |
No files matched your search
@@ -102,3 +102,65 @@ jobs:
|
||||
if-no-files-found: error
|
||||
compression-level: 0
|
||||
retention-days: 14
|
||||
|
||||
publish-draft-release:
|
||||
if: ${{ inputs.run_lifecycle && startsWith(github.ref, 'refs/tags/v') }}
|
||||
needs: matrix
|
||||
runs-on: ubuntu-24.04
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
|
||||
with:
|
||||
persist-credentials: false
|
||||
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
|
||||
with:
|
||||
pattern: memhub-runtime-linux-*
|
||||
path: memhub/release-assets
|
||||
merge-multiple: true
|
||||
- name: Verify tag and Linux release assets
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version=$(node -p 'require("./package.json").version')
|
||||
node scripts/verify-release-version.mjs \
|
||||
--channel stable \
|
||||
--version "$version" \
|
||||
--build-number "$(node -p 'require("./package.json").release.build_number')" \
|
||||
--tag "$GITHUB_REF_NAME"
|
||||
for arch in x64 arm64; do
|
||||
archive="release-assets/memhub-runtime-v${version}-linux-${arch}.tar.gz"
|
||||
test -f "$archive"
|
||||
test -f "$archive.sha256"
|
||||
(
|
||||
cd release-assets
|
||||
sha256sum -c "$(basename "$archive.sha256")"
|
||||
)
|
||||
done
|
||||
- name: Upload Linux assets to a draft GitHub Release
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if gh release view "$GITHUB_REF_NAME" \
|
||||
--repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
|
||||
test "$(
|
||||
gh release view "$GITHUB_REF_NAME" \
|
||||
--repo "$GITHUB_REPOSITORY" \
|
||||
--json isDraft \
|
||||
--jq .isDraft
|
||||
)" = "true"
|
||||
else
|
||||
gh release create "$GITHUB_REF_NAME" \
|
||||
--repo "$GITHUB_REPOSITORY" \
|
||||
--verify-tag \
|
||||
--draft \
|
||||
--generate-notes \
|
||||
--title "$GITHUB_REF_NAME"
|
||||
fi
|
||||
gh release upload "$GITHUB_REF_NAME" \
|
||||
release-assets/memhub-runtime-v*-linux-*.tar.gz \
|
||||
release-assets/memhub-runtime-v*-linux-*.tar.gz.sha256 \
|
||||
--repo "$GITHUB_REPOSITORY" \
|
||||
--clobber
|
||||
+64
-2
@@ -2,6 +2,45 @@
|
||||
|
||||
Hy Memory and MemHub are versioned and released independently.
|
||||
|
||||
## Hy Memory 2.0.3 — 2026-10-17
|
||||
|
||||
- Added an optional runtime HTTP transport factory so embedded products can
|
||||
enforce request budgets without wrapping or restricting SDK local-store
|
||||
access.
|
||||
- Published
|
||||
[`hy-memory==2.0.3`](https://pypi.org/project/hy-memory/2.0.3/) with
|
||||
unchanged memory extraction, reconciliation, storage, and Coding/Working
|
||||
behavior.
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Fixed governed-memory revisions and repeated evidence links failing when the
|
||||
visibility dirty-marker trigger encounters the same memory twice. Startup
|
||||
now replaces the affected legacy trigger without changing schema versions
|
||||
or governed and visibility data.
|
||||
- Serialized each SDK SQLite store's shared connection lifecycle so concurrent
|
||||
history/cache operations cannot race with `close()` and crash the process.
|
||||
- Scoped Coding reconciliation targets to the current HY authority,
|
||||
conversation, and workspace, preventing near-duplicate sessions from
|
||||
mutating an experience or working fact owned by another evidence context.
|
||||
- Kept post-write authority verification failures available for read-only
|
||||
reconciliation while preventing automatic or generic confirmed retries from
|
||||
invoking the model again. Automatic reconciliation skips those sticky rows
|
||||
before applying its batch limit, and linked durable jobs exclusively own
|
||||
retry scheduling for their memory tasks.
|
||||
- Clarified that `add(metadata=...)` carries legacy SDK processing options,
|
||||
while persistent caller business context belongs in `meta`.
|
||||
- Added an opt-in `get(..., include_raw=True)` path for reading `L1_RAW`
|
||||
memories, including memories written in lite mode. The default remains
|
||||
hidden for backward compatibility.
|
||||
- Removed MemHub's task-scoped HyPermission store wrappers and global
|
||||
model-content consent switch. The trusted embedded Hy Memory SDK now uses
|
||||
its native local stores directly.
|
||||
- Existing installations ignore the retired config flag, readmit previously
|
||||
parked pre-invocation work, and drop only the obsolete
|
||||
`hy_node_permissions` proof table. Evidence, governed memories, model
|
||||
profiles, SDK stores, and source/host grants are preserved.
|
||||
|
||||
## Hy Memory 2.0.2 — 2026-10-17
|
||||
|
||||
- Published
|
||||
@@ -16,11 +55,34 @@ Hy Memory and MemHub are versioned and released independently.
|
||||
Because PyPI release metadata is immutable, the next SDK patch release must
|
||||
carry the updated license metadata; `2.0.1` retains its published metadata.
|
||||
|
||||
## MemHub 1.0.1 — Unreleased
|
||||
## MemHub 1.0.3 — Unreleased
|
||||
|
||||
- Updated the source version to `1.0.1` (build 5).
|
||||
- Updated the source version to `1.0.3` (build 7).
|
||||
- Pinned the public SDK to `hy-memory==2.0.3`.
|
||||
- Removed the HyPermission store wrappers and global model-content consent
|
||||
switch; the trusted embedded SDK now uses native local stores.
|
||||
- Prevented uncertain writes from blocking later source work and added one
|
||||
reconciled automatic retry after ten seconds.
|
||||
- Added product-facing memory pipeline counters and the log troubleshooting
|
||||
runbook.
|
||||
- Allowed a verified packaged Desktop Runtime from a different commit to take
|
||||
over an equal-build install atomically while preserving downgrade and
|
||||
tamper refusal.
|
||||
|
||||
## MemHub 1.0.2 — 2026-10-17
|
||||
|
||||
- Updated the source version to `1.0.2` (build 6).
|
||||
- Pinned the public SDK to `hy-memory==2.0.2`.
|
||||
- Pinned `onnxruntime==1.23.2` to retain macOS x64 packaging support.
|
||||
- Fixed relaunch and upgrade failures when a prior managed Runtime exists.
|
||||
- Embedded canonical build identity when an external packaging pipeline omits
|
||||
release environment fields.
|
||||
- Added automatic Linux Runtime upload to tagged Draft Releases.
|
||||
|
||||
## MemHub 1.0.1 — Superseded before release
|
||||
|
||||
- The candidate tag was not published as a GitHub Release; `1.0.2` replaces it
|
||||
after the managed-Runtime relaunch fix.
|
||||
|
||||
## MemHub 1.0.0
|
||||
|
||||
|
||||
@@ -32,7 +32,9 @@ English | [中文](README.zh.md)
|
||||
|
||||
## Latest News
|
||||
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.2`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.2/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** quantized **FP8** (**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**) and **GGUF** (**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**) versions of **Hy-Memory-30B-A3B** on Hugging Face!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **Released [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** and **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**, with a signed macOS desktop app and verified macOS/Linux Runtimes!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.4`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.4/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** on Hugging Face along with its [technical report](https://arxiv.org/abs/2610.22196) on attributed reinforcement learning for long-term memory writing!
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **Released** **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)** (submitted to **ICLR 2027**).
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **Accepted** at **EMNLP 2026**: **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)**!
|
||||
@@ -63,13 +65,13 @@ Welcome to the **Hy Memory Research Family**, covering cognitive chat memory, lo
|
||||
### 1. Installation
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.2 # core (includes ChromaDB and Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.2" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.2" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.2" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.2" # + MySQL cache
|
||||
pip install "hy-memory[rerank]==2.0.2" # + local multilingual cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.2" # everything
|
||||
pip install hy-memory==2.0.4 # core (includes ChromaDB and Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.4" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.4" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.4" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.4" # + MySQL cache
|
||||
pip install "hy-memory[rerank]==2.0.4" # + local multilingual cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.4" # everything
|
||||
```
|
||||
|
||||
### 2. Configuration
|
||||
|
||||
+10
-8
@@ -32,7 +32,9 @@
|
||||
|
||||
## 最新动态
|
||||
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.2`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.2/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **Hy-Memory-30B-A3B** 的 **FP8**(**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**)与 **GGUF**(**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**)量化版本模型!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **正式发布 [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** 与 **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**,提供已签名的 macOS 桌面应用及经过验证的 macOS/Linux Runtime!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.4`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.4/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
|
||||
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** 长期记忆写入模型及[技术报告](https://arxiv.org/abs/2610.22196)!
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **基准与方法发布** 推出长周期与交错式编程记忆评测基准 **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)**(已投稿 **ICLR 2027**)。
|
||||
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **论文录用** 双系统认知记忆架构论文 **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)** 被 **EMNLP 2026** 接收!
|
||||
@@ -63,13 +65,13 @@
|
||||
### 1. 安装
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.2 # 核心(含 ChromaDB 与 Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.2" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.2" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.2" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.2" # + MySQL 缓存
|
||||
pip install "hy-memory[rerank]==2.0.2" # + 本地多语言 cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.2" # 全部
|
||||
pip install hy-memory==2.0.4 # 核心(含 ChromaDB 与 Kuzu)
|
||||
pip install "hy-memory[qdrant]==2.0.4" # + Qdrant
|
||||
pip install "hy-memory[faiss]==2.0.4" # + FAISS
|
||||
pip install "hy-memory[graph]==2.0.4" # + Neo4j
|
||||
pip install "hy-memory[mysql]==2.0.4" # + MySQL 缓存
|
||||
pip install "hy-memory[rerank]==2.0.4" # + 本地多语言 cross-encoder rerank
|
||||
pip install "hy-memory[all]==2.0.4" # 全部
|
||||
```
|
||||
|
||||
### 2. 配置
|
||||
|
||||
@@ -152,7 +152,7 @@ client = HyMemoryClient.from_config({
|
||||
安装可选依赖:
|
||||
|
||||
```bash
|
||||
pip install 'hy-memory[rerank]==2.0.1'
|
||||
pip install 'hy-memory[rerank]==2.0.4'
|
||||
```
|
||||
|
||||
启用后,`HyMemoryClient(...)` 构造阶段会完成模型下载/读取、device 放置和一次 dummy
|
||||
|
||||
+2
-3
@@ -37,8 +37,7 @@ do not belong in the public source tree.
|
||||
- Root GitHub workflows run SDK and MemHub checks from their respective working
|
||||
directories. Native lifecycle workflows produce reviewable Actions artifacts;
|
||||
tag releases remain fail-closed on public identity, signing and notarization.
|
||||
- Hy Memory is published as `hy-memory==2.0.2`. MemHub pins that public
|
||||
distribution; its selected npm package is `memhub-cli`, which is not yet
|
||||
published.
|
||||
- Hy Memory is published as `hy-memory==2.0.4`. MemHub pins that public
|
||||
distribution; its selected npm package is `memhub-cli` 1.0.5.
|
||||
- Hy-Memory source uses Apache-2.0 at the repository root. Third-party
|
||||
components retain their own licenses and attribution notices.
|
||||
+21
-6
@@ -5,14 +5,14 @@
|
||||
## 安装与快速开始
|
||||
|
||||
```bash
|
||||
pip install hy-memory==2.0.2
|
||||
pip install hy-memory==2.0.4
|
||||
|
||||
# 可选后端
|
||||
pip install 'hy-memory[qdrant]==2.0.1'
|
||||
pip install 'hy-memory[faiss]==2.0.1'
|
||||
pip install 'hy-memory[graph]==2.0.1' # Neo4j;Kuzu 已在核心依赖中
|
||||
pip install 'hy-memory[mysql]==2.0.1'
|
||||
pip install 'hy-memory[rerank]==2.0.1' # 本地多语言 cross-encoder rerank
|
||||
pip install 'hy-memory[qdrant]==2.0.4'
|
||||
pip install 'hy-memory[faiss]==2.0.4'
|
||||
pip install 'hy-memory[graph]==2.0.4' # Neo4j;Kuzu 已在核心依赖中
|
||||
pip install 'hy-memory[mysql]==2.0.4'
|
||||
pip install 'hy-memory[rerank]==2.0.4' # 本地多语言 cross-encoder rerank
|
||||
```
|
||||
|
||||
默认 LLM 和 Embedding 均使用 OpenAI-compatible 接口。使用 OpenAI 时可只设置:
|
||||
@@ -106,6 +106,15 @@ result = client.add(
|
||||
)
|
||||
```
|
||||
|
||||
`metadata` 与 `meta` 不是同一字段:
|
||||
|
||||
- `metadata` 是兼容旧调用方保留的 SDK 处理选项,会进入内部
|
||||
`WriteRequest.extra`,不保证持久化或随查询结果返回。
|
||||
- `meta` 是调用方的业务上下文,会持久化并在 `search()`、`get()` 等结果中返回。
|
||||
同一记忆被多次合并时,各次 `meta` 会按写入顺序保存在列表中。
|
||||
|
||||
需要以后读回的数据必须放在 `meta`,不要放在 `metadata`。
|
||||
|
||||
`scene` 的公开取值:
|
||||
|
||||
- `None`:普通文本直接走 chat;messages 含 tool 时执行快速场景判断。
|
||||
@@ -263,6 +272,8 @@ catalog 也可以在没有基础 L0 时形成只读的合成画像。
|
||||
|
||||
```python
|
||||
memory = client.get("memory_id")
|
||||
# lite 模式写入的是 L1_RAW;按 ID 读取时必须显式选择返回原文。
|
||||
raw_memory = client.get("memory_id", include_raw=True)
|
||||
result = client.update("memory_id", "更新后的内容")
|
||||
result = client.delete("memory_id")
|
||||
result = client.delete_all(user_id="user_001")
|
||||
@@ -282,6 +293,10 @@ result = client.clone_user("user_001", "user_002", agent_id=None)
|
||||
对应异步方法为 `async_get()`、`async_update()`、`async_delete()`、
|
||||
`async_delete_all()`、`async_list_memories()`、`async_clone_user()`。
|
||||
|
||||
`get()` / `async_get()` 默认继续隐藏 `L1_RAW`,因此不会改变现有调用方的原文访问
|
||||
边界。只有显式传入 `include_raw=True` 才返回该层;这也是 lite 模式按 ID 读取
|
||||
写入内容及其 `meta` 的方式。MemHub 的 HTTP 读取接口不开放此参数。
|
||||
|
||||
普通 `list_memories()` 返回 `vdb` 桶,并在 Graph 有结果时附带 `graph`;传入
|
||||
`workspace_id` 时返回生产力记忆的 `scene="productivity"` 和 `experiences`。
|
||||
|
||||
|
||||
@@ -22,7 +22,8 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
# ================================================================
|
||||
# 数据结构
|
||||
# ================================================================@dataclass
|
||||
# ================================================================
|
||||
@dataclass
|
||||
class SessionSummaryResult:
|
||||
"""Session 摘要结果 (V2)"""
|
||||
success: bool
|
||||
|
||||
@@ -39,6 +39,7 @@ import logging
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
import uuid
|
||||
|
||||
@@ -252,6 +253,17 @@ def _series_episode_id(
|
||||
return str(uuid.uuid5(uuid.NAMESPACE_URL, "hy-memory:series-episode:" + raw))
|
||||
|
||||
|
||||
def _series_memory_at(times: List[int]) -> Optional[datetime]:
|
||||
"""Earliest member timestamp as the same local wall clock it came from.
|
||||
|
||||
``_node_timestamp`` produces ``datetime.timestamp()`` values. Reading them
|
||||
back with ``utcfromtimestamp`` turns them into a naive *UTC* datetime, which
|
||||
the vector payload later re-interprets as local time and shifts by the
|
||||
machine's UTC offset.
|
||||
"""
|
||||
return datetime.fromtimestamp(min(times)) if times else None
|
||||
|
||||
|
||||
def _node_timestamp(node: Any) -> Optional[int]:
|
||||
value = getattr(node, "memory_at", None) or getattr(node, "gmt_created", None)
|
||||
if value is None:
|
||||
@@ -974,8 +986,6 @@ async def finalize_series_episode_containers(
|
||||
Stored text is only a title plus a structural intro. Exact L2 bodies remain in
|
||||
their canonical nodes and are referenced by ID from dated timeline segments.
|
||||
"""
|
||||
from datetime import datetime
|
||||
|
||||
from ..models.memory import MemoryLayer, MemoryNode, MemoryStatus, SourceType
|
||||
from ..pipelines._retrieval.session_episode import (
|
||||
add_parent_episode_membership,
|
||||
@@ -1061,7 +1071,7 @@ async def finalize_series_episode_containers(
|
||||
segment["memory_at"] for segment in plan["timeline"]
|
||||
if segment.get("memory_at") is not None
|
||||
]
|
||||
memory_at = datetime.utcfromtimestamp(min(times)) if times else None
|
||||
memory_at = _series_memory_at(times)
|
||||
content = f"{plan['title']}. {plan['intro']}".strip()
|
||||
series = MemoryNode(
|
||||
node_id=series_id, user_id=user_id,
|
||||
|
||||
@@ -1123,6 +1123,7 @@ class Extractor:
|
||||
max_tokens=self._llm_config.agent_max_tokens,
|
||||
tools=None,
|
||||
tool_choice=None,
|
||||
audit_stage="extract",
|
||||
)
|
||||
self._call_count += 1
|
||||
total_tokens += response.tokens_used
|
||||
@@ -1233,6 +1234,7 @@ class Extractor:
|
||||
max_tokens=self._llm_config.agent_max_tokens,
|
||||
tools=None,
|
||||
tool_choice=None,
|
||||
audit_stage="extract",
|
||||
)
|
||||
self._call_count += 1
|
||||
total_tokens += response.tokens_used
|
||||
|
||||
@@ -34,6 +34,7 @@ import os
|
||||
import weakref
|
||||
|
||||
from ..config import MemoryConfig
|
||||
from ..utils.model_call_context import model_call_stage
|
||||
from ..core.workload import llm_slot
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -348,6 +349,7 @@ class LLMProvider:
|
||||
stop: List[str] = None,
|
||||
tools: Optional[List[Dict[str, Any]]] = None,
|
||||
tool_choice: Optional[Any] = None,
|
||||
audit_stage: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> LLMResponse:
|
||||
"""
|
||||
@@ -386,15 +388,16 @@ class LLMProvider:
|
||||
len(prompt), _content_fingerprint(prompt),
|
||||
)
|
||||
|
||||
response = await self._call_llm(
|
||||
prompt=prompt,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
stop=stop,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
**kwargs
|
||||
)
|
||||
with model_call_stage(audit_stage):
|
||||
response = await self._call_llm(
|
||||
prompt=prompt,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
stop=stop,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
self._total_calls += 1
|
||||
self._total_tokens += response.tokens_used
|
||||
@@ -496,11 +499,21 @@ class LLMProvider:
|
||||
clients = self._clients.setdefault(loop, {})
|
||||
client = clients.get(client_key)
|
||||
if client is None:
|
||||
client_options: Dict[str, Any] = {
|
||||
"api_key": base_api_key,
|
||||
"base_url": base_url,
|
||||
"timeout": self._llm_config.timeout,
|
||||
"max_retries": 0,
|
||||
}
|
||||
transport_factory = getattr(self.config, "http_transport_factory", None)
|
||||
if callable(transport_factory):
|
||||
import httpx
|
||||
|
||||
client_options["http_client"] = httpx.AsyncClient(
|
||||
transport=transport_factory()
|
||||
)
|
||||
client = AsyncOpenAI(
|
||||
api_key=base_api_key,
|
||||
base_url=base_url,
|
||||
timeout=self._llm_config.timeout,
|
||||
max_retries=0,
|
||||
**client_options,
|
||||
)
|
||||
clients[client_key] = client
|
||||
|
||||
@@ -585,6 +598,7 @@ class LLMProvider:
|
||||
temperature: Optional[float] = None,
|
||||
tools: Optional[List[Dict[str, Any]]] = None,
|
||||
tool_choice: Optional[Any] = None,
|
||||
audit_stage: Optional[str] = None,
|
||||
) -> LLMResponse:
|
||||
"""
|
||||
多轮 messages 调用(支持 tool results 回传)。
|
||||
@@ -612,13 +626,14 @@ class LLMProvider:
|
||||
f"msgs={len(messages)} tools={len(tools) if tools else 0}"
|
||||
)
|
||||
|
||||
response = await self._chat_completions(
|
||||
messages,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
)
|
||||
with model_call_stage(audit_stage):
|
||||
response = await self._chat_completions(
|
||||
messages,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
tools=tools,
|
||||
tool_choice=tool_choice,
|
||||
)
|
||||
|
||||
self._total_calls += 1
|
||||
self._total_tokens += response.tokens_used
|
||||
|
||||
@@ -669,6 +669,7 @@ class MemoryReconciler:
|
||||
response = await llm.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=500,
|
||||
audit_stage="reconcile",
|
||||
)
|
||||
text = response.content.strip()
|
||||
if "```json" in text:
|
||||
@@ -1383,6 +1384,7 @@ class MemoryReconciler:
|
||||
response = await llm.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=dyn_max_tokens,
|
||||
audit_stage="reconcile",
|
||||
)
|
||||
except Exception as e:
|
||||
# LLM 彻底不可用(provider 已按 max_retries 重试过,网关 500 /
|
||||
|
||||
+170
-78
@@ -30,6 +30,7 @@ import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
@@ -377,6 +378,14 @@ class _LoopThread:
|
||||
return await await_concurrent_future_guarded(future, self._loop)
|
||||
|
||||
def stop(self):
|
||||
if self._loop.is_closed():
|
||||
return
|
||||
if threading.current_thread() is self._thread:
|
||||
# Called from inside the loop: stopping and joining ourselves would
|
||||
# deadlock. Just ask run_forever() to return.
|
||||
self._loop.stop()
|
||||
return
|
||||
|
||||
def _stop():
|
||||
if self._lag_handle is not None:
|
||||
self._lag_handle.cancel()
|
||||
@@ -386,6 +395,8 @@ class _LoopThread:
|
||||
|
||||
self._loop.call_soon_threadsafe(_stop)
|
||||
self._thread.join(timeout=5)
|
||||
if not self._thread.is_alive() and not self._loop.is_closed():
|
||||
self._loop.close()
|
||||
|
||||
|
||||
class HyMemoryClient:
|
||||
@@ -1213,6 +1224,8 @@ class HyMemoryClient:
|
||||
target_query: str,
|
||||
*,
|
||||
user_ids: List[str],
|
||||
agent_ids: Optional[List[str]] = None,
|
||||
session_ids: Optional[List[str]] = None,
|
||||
workspace_id: Optional[str],
|
||||
branch: Optional[str],
|
||||
limit: int,
|
||||
@@ -1346,6 +1359,7 @@ class HyMemoryClient:
|
||||
try:
|
||||
fact_hits = await _store.search_facts(
|
||||
embedding, user_id=primary_user,
|
||||
agent_ids=agent_ids, session_ids=session_ids,
|
||||
workspace_id=workspace_id, branch=branch,
|
||||
top=max(limit, 5),
|
||||
# 词面通道 + 把下面那道闸门下移到重排之前:先砍掉过不了闸的候选再重排,
|
||||
@@ -1425,6 +1439,8 @@ class HyMemoryClient:
|
||||
hits = await _store.search_by_query_embedding(
|
||||
embedding,
|
||||
user_id=primary_user,
|
||||
agent_ids=agent_ids,
|
||||
session_ids=session_ids,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
top=max(limit * 4, 8), # 过采样为 dedup 留余量
|
||||
@@ -1597,7 +1613,9 @@ class HyMemoryClient:
|
||||
session_id: Session ID(三级隔离 key)
|
||||
source: 调用来源,例如 openclaw / cursor / codex;与 agent_id 正交。
|
||||
写入前执行 strip().lower(),默认 "default"。
|
||||
metadata: 自定义元数据
|
||||
metadata: 旧版 SDK 处理选项,会并入内部 WriteRequest.extra。
|
||||
不保证持久化或随查询返回;调用方需要保存并读回的业务字段
|
||||
请使用 meta。
|
||||
memory_at: 记忆时间戳(不传则用当前时间),用于导入历史记忆
|
||||
scene: 显式指定写入路由,跳过 LLM 自动判定。
|
||||
- None(默认):含 tool 时 LLM 判 chat/working/coding;
|
||||
@@ -1623,8 +1641,8 @@ class HyMemoryClient:
|
||||
branch: Coding memory 路径专用。分支名。同 workspace_id。chat 路径忽略。
|
||||
meta: 调用方自定义业务 meta,随本次产出的记忆存储、查询时原样返回。
|
||||
不 embed、不参与召回、不作过滤条件。
|
||||
与 metadata 不同:metadata 会并入 extra(含 SDK 内部控制键),
|
||||
meta 独立存放,只做透传。
|
||||
与 metadata 不同:metadata 是旧版内部处理选项,
|
||||
meta 独立持久化,只做业务透传。
|
||||
语义为「该次 write 时刻的上下文」,故存为 list——同一节点被
|
||||
多次写入合并时会追加而非覆盖,详见 get() 返回的 meta 字段。
|
||||
|
||||
@@ -1791,17 +1809,26 @@ class HyMemoryClient:
|
||||
query_embedding=query_embedding,
|
||||
))
|
||||
|
||||
def get(self, memory_id: str) -> Optional[Dict[str, Any]]:
|
||||
def get(
|
||||
self,
|
||||
memory_id: str,
|
||||
*,
|
||||
include_raw: bool = False,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
获取单条记忆。
|
||||
|
||||
Args:
|
||||
memory_id: 记忆 ID
|
||||
include_raw: 是否允许返回 L1_RAW 原始记忆。默认 False,保持原有隐藏行为;
|
||||
lite 模式的记忆均为 L1_RAW,需显式设为 True 才能按 ID 读取。
|
||||
|
||||
Returns:
|
||||
{"memory_id", "content", "layer", "request_id", "elapsed_ms"} 或 None(不存在时)
|
||||
"""
|
||||
return self._loop_thread.run(self.async_get(memory_id))
|
||||
return self._loop_thread.run(
|
||||
self.async_get(memory_id, include_raw=include_raw)
|
||||
)
|
||||
|
||||
def update(
|
||||
self,
|
||||
@@ -2136,14 +2163,33 @@ class HyMemoryClient:
|
||||
from .metrics import MetricsCollector
|
||||
return await MetricsCollector.get().get_snapshot(minutes=minutes)
|
||||
|
||||
def _stop_owned_loops(self) -> None:
|
||||
"""Stop every event-loop thread this client owns.
|
||||
|
||||
Solo clients create both a write loop and a search loop. Stopping only
|
||||
the write loop left the search thread (and a self-wakeup timer every
|
||||
50ms) alive for the process lifetime. Shared runtimes own their loops
|
||||
and manage them through ``SharedRuntime.aclose()``.
|
||||
"""
|
||||
if not getattr(self, "_owns_runtime", False):
|
||||
return
|
||||
seen = set()
|
||||
for loop_thread in (
|
||||
getattr(self, "_loop_thread", None),
|
||||
getattr(self, "_search_loop_thread", None),
|
||||
):
|
||||
if loop_thread is None or id(loop_thread) in seen:
|
||||
continue
|
||||
seen.add(id(loop_thread))
|
||||
loop_thread.stop()
|
||||
|
||||
def close(self):
|
||||
"""关闭客户端,释放所有资源"""
|
||||
if self._registry is not None:
|
||||
self._loop_thread.run(self._close_async())
|
||||
# solo 模式:本 client 拥有 loop_thread,可以安全 stop
|
||||
# shared 模式:loop_thread 由 runtime 管,留给 runtime.aclose() 关
|
||||
if hasattr(self, '_loop_thread') and self._owns_runtime:
|
||||
self._loop_thread.stop()
|
||||
try:
|
||||
if self._registry is not None:
|
||||
self._loop_thread.run(self._close_async())
|
||||
finally:
|
||||
self._stop_owned_loops()
|
||||
|
||||
def history(self, memory_id: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
@@ -2846,6 +2892,81 @@ class HyMemoryClient:
|
||||
|
||||
return resp
|
||||
|
||||
@staticmethod
|
||||
def _public_search_views(
|
||||
result_memories: List[Dict[str, Any]],
|
||||
*,
|
||||
wanted_layers: Optional[set],
|
||||
lite_mode: bool,
|
||||
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
|
||||
"""Build the public search items and the private dedup view together.
|
||||
|
||||
The dedup marker must come from the same source element as its public
|
||||
item. Building the dedup view with a second ``zip(result_memories,
|
||||
all_mems)`` is wrong as soon as one result is dropped by the layer
|
||||
filter: every later kind shifts by one, and a canonical_state can enter
|
||||
fuzzy dedup and be shadowed.
|
||||
"""
|
||||
from .models.memory import MemoryLayer
|
||||
|
||||
all_mems: List[Dict[str, Any]] = []
|
||||
dedup_mems: List[Dict[str, Any]] = []
|
||||
for mem in result_memories:
|
||||
if wanted_layers is not None and mem.get("layer", "") not in wanted_layers:
|
||||
continue
|
||||
if lite_mode and mem.get("layer", "") != MemoryLayer.L1_RAW.value:
|
||||
continue
|
||||
score = mem.get("score", 0.0)
|
||||
item = {
|
||||
"memory_id": mem.get("memory_id", ""),
|
||||
"content": mem.get("content", ""),
|
||||
"score": round(score, 4),
|
||||
"layer": mem.get("layer", ""),
|
||||
# 多 user 检索时调用方需据此追溯归属(R1)
|
||||
"user_id": mem.get("user_id", "") or "",
|
||||
"agent_id": mem.get("agent_id", "") or "",
|
||||
"source": mem.get("source", "default") or "default",
|
||||
"retrieval_source": mem.get("retrieval_source", "") or "",
|
||||
"owner": mem.get("owner"),
|
||||
"speculate": mem.get("speculate"),
|
||||
# 不透出 source_raw_memory_id:它指向 L1_RAW 节点,而 raw 对外
|
||||
# 读路径一律不可见,该 id 到手也查不到东西(按 ID 取返回"不存在"),
|
||||
# 留着只会让调用方以为能据此追溯原文。
|
||||
# 节点上仍保留该字段,内部溯源与 clone 重映射不受影响。
|
||||
"tags": mem.get("tags") or [],
|
||||
"memory_at": mem.get("memory_at"),
|
||||
"gmt_created": mem.get("gmt_created"),
|
||||
# 调用方自定义 meta(透传负载)。由 reader 从 custom["meta"] 取出,
|
||||
# 未实现该字段的 reader 下为空列表。
|
||||
"meta": mem.get("meta") or [],
|
||||
"_schema_channel": bool(mem.get("_schema_channel")),
|
||||
}
|
||||
if mem.get("schema_type"):
|
||||
item["schema_type"] = mem["schema_type"]
|
||||
if mem.get("workspace_id"):
|
||||
item["workspace_id"] = mem["workspace_id"]
|
||||
if mem.get("branch"):
|
||||
item["branch"] = mem["branch"]
|
||||
if mem.get("evolution_chain"):
|
||||
item["evolution_chain"] = mem["evolution_chain"]
|
||||
if mem.get("capsule_memories") is not None:
|
||||
item["capsule_memories"] = mem["capsule_memories"]
|
||||
if mem.get("episode_id"):
|
||||
item["episode_id"] = mem["episode_id"]
|
||||
if mem.get("member_ids") is not None:
|
||||
item["member_ids"] = mem["member_ids"]
|
||||
if mem.get("timeline") is not None:
|
||||
item["timeline"] = mem["timeline"]
|
||||
if mem.get("rerank_score") is not None:
|
||||
item["rerank_score"] = mem["rerank_score"]
|
||||
all_mems.append(item)
|
||||
|
||||
dedup_item = dict(item)
|
||||
dedup_item["_memory_kind"] = mem.get("_memory_kind", "")
|
||||
dedup_mems.append(dedup_item)
|
||||
|
||||
return all_mems, dedup_mems
|
||||
|
||||
def _spawn_search_dedup(
|
||||
self,
|
||||
all_mems: List[Dict[str, Any]],
|
||||
@@ -3390,6 +3511,8 @@ class HyMemoryClient:
|
||||
return await self._search_coding(
|
||||
effective_target_query,
|
||||
user_ids=effective_user_ids,
|
||||
agent_ids=effective_agent_ids,
|
||||
session_ids=effective_session_ids,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
limit=limit,
|
||||
@@ -3479,62 +3602,11 @@ class HyMemoryClient:
|
||||
if effective_search_layers else None
|
||||
)
|
||||
|
||||
_all_mems = []
|
||||
for mem in result.memories:
|
||||
if _wanted_layers is not None and mem.get("layer", "") not in _wanted_layers:
|
||||
continue
|
||||
if lite_mode and mem.get("layer", "") != MemoryLayer.L1_RAW.value:
|
||||
continue
|
||||
score = mem.get("score", 0.0)
|
||||
item = {
|
||||
"memory_id": mem.get("memory_id", ""),
|
||||
"content": mem.get("content", ""),
|
||||
"score": round(score, 4),
|
||||
"layer": mem.get("layer", ""),
|
||||
# 多 user 检索时调用方需据此追溯归属(R1)
|
||||
"user_id": mem.get("user_id", "") or "",
|
||||
"agent_id": mem.get("agent_id", "") or "",
|
||||
"source": mem.get("source", "default") or "default",
|
||||
"retrieval_source": mem.get("retrieval_source", "") or "",
|
||||
"owner": mem.get("owner"),
|
||||
"speculate": mem.get("speculate"),
|
||||
# 不透出 source_raw_memory_id:它指向 L1_RAW 节点,而 raw 对外
|
||||
# 读路径一律不可见,该 id 到手也查不到东西(按 ID 取返回"不存在"),
|
||||
# 留着只会让调用方以为能据此追溯原文。
|
||||
# 节点上仍保留该字段,内部溯源与 clone 重映射不受影响。
|
||||
"tags": mem.get("tags") or [],
|
||||
"memory_at": mem.get("memory_at"),
|
||||
"gmt_created": mem.get("gmt_created"),
|
||||
# 调用方自定义 meta(透传负载)。由 reader 从 custom["meta"] 取出,
|
||||
# 未实现该字段的 reader 下为空列表。
|
||||
"meta": mem.get("meta") or [],
|
||||
"_schema_channel": bool(mem.get("_schema_channel")),
|
||||
}
|
||||
if mem.get("schema_type"):
|
||||
item["schema_type"] = mem["schema_type"]
|
||||
if mem.get("workspace_id"):
|
||||
item["workspace_id"] = mem["workspace_id"]
|
||||
if mem.get("branch"):
|
||||
item["branch"] = mem["branch"]
|
||||
if mem.get("evolution_chain"):
|
||||
item["evolution_chain"] = mem["evolution_chain"]
|
||||
if mem.get("capsule_memories") is not None:
|
||||
item["capsule_memories"] = mem["capsule_memories"]
|
||||
if mem.get("episode_id"):
|
||||
item["episode_id"] = mem["episode_id"]
|
||||
if mem.get("member_ids") is not None:
|
||||
item["member_ids"] = mem["member_ids"]
|
||||
if mem.get("timeline") is not None:
|
||||
item["timeline"] = mem["timeline"]
|
||||
if mem.get("rerank_score") is not None:
|
||||
item["rerank_score"] = mem["rerank_score"]
|
||||
_all_mems.append(item)
|
||||
|
||||
_dedup_mems = []
|
||||
for source, public in zip(result.memories, _all_mems):
|
||||
copy = dict(public)
|
||||
copy["_memory_kind"] = source.get("_memory_kind", "")
|
||||
_dedup_mems.append(copy)
|
||||
_all_mems, _dedup_mems = self._public_search_views(
|
||||
result.memories,
|
||||
wanted_layers=_wanted_layers,
|
||||
lite_mode=lite_mode,
|
||||
)
|
||||
|
||||
# search 链路去重:仅对 L2_FACT + L4_IDENTITY(重复重灾区)判重删库。
|
||||
# L0 及其他层不参与。fire-and-forget,不阻塞本次返回(清理面向后续查询)。
|
||||
@@ -3650,12 +3722,21 @@ class HyMemoryClient:
|
||||
# ================================================================
|
||||
|
||||
@_ensure_internal_loop
|
||||
async def async_get(self, memory_id: str) -> Optional[Dict[str, Any]]:
|
||||
async def async_get(
|
||||
self,
|
||||
memory_id: str,
|
||||
*,
|
||||
include_raw: bool = False,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
获取单条记忆(异步)。
|
||||
|
||||
VDB + Graph 双路并行查询,哪个有结果返回哪个。
|
||||
|
||||
Args:
|
||||
memory_id: 记忆 ID
|
||||
include_raw: 是否允许返回 L1_RAW 原始记忆,默认 False。
|
||||
|
||||
Returns:
|
||||
{"memory_id", "content", "layer", "source", "request_id", "elapsed_ms"} 或 None
|
||||
"""
|
||||
@@ -3683,17 +3764,24 @@ class HyMemoryClient:
|
||||
|
||||
vdb_node, graph_node = await _asyncio_get.gather(_get_vdb(), _get_graph())
|
||||
|
||||
# L1_RAW 不对外可见:按 ID 直取走主键查询、绕过向量库的 layer filter,
|
||||
# 故在此统一挡掉。async_get 是 GET /v1/memories/{id} 与
|
||||
# /v1/memories/batch 两个对外接口的共同必经处,而内部写链路不经过它
|
||||
# (writer 写后校验直接调 vector_store.get_by_id),挡在此处不影响内部。
|
||||
# L1_RAW 默认不对外可见:按 ID 直取走主键查询、绕过向量库的 layer
|
||||
# filter,故在此统一挡掉。SDK 调用方可通过 include_raw=True 显式读取;
|
||||
# HTTP 接口不传该参数,继续保持隐藏。内部写链路不经过此方法。
|
||||
#
|
||||
# 置于 gather 之后、埋点之前:命中 raw 仍照常落 GET 埋点(保留可观测性,
|
||||
# 便于发现有调用方拿 raw id 试探),只是不把内容返回给调用方。
|
||||
# 对调用方的表现与"该 id 不存在"完全一致,不额外暴露"存在但被挡"的信息。
|
||||
if vdb_node is not None and vdb_node.layer == MemoryLayer.L1_RAW:
|
||||
if (
|
||||
not include_raw
|
||||
and vdb_node is not None
|
||||
and vdb_node.layer == MemoryLayer.L1_RAW
|
||||
):
|
||||
vdb_node = None
|
||||
if graph_node is not None and getattr(graph_node, "layer", None) == MemoryLayer.L1_RAW:
|
||||
if (
|
||||
not include_raw
|
||||
and graph_node is not None
|
||||
and getattr(graph_node, "layer", None) == MemoryLayer.L1_RAW
|
||||
):
|
||||
graph_node = None
|
||||
|
||||
elapsed = round((time.perf_counter() - t0) * 1000, 2)
|
||||
@@ -6069,9 +6157,10 @@ class HyMemoryClient:
|
||||
effective_agent_ids = agent_ids if agent_ids is not None else []
|
||||
|
||||
if not effective_agent_ids:
|
||||
# 查所有 agent 的历史:用 "user_id:" 前缀匹配
|
||||
isolation_key = self._build_isolation_key(uid, "default")
|
||||
return await self._history_store.get_recent(isolation_key, limit=limit)
|
||||
# 查所有 agent 的历史:isolation_key 前缀匹配。
|
||||
# 不能退回 "u:default" —— add() 的默认 agent 是 "default_agent",
|
||||
# 默认用法下会查空。
|
||||
return await self._history_store.get_recent_for_user(uid, limit=limit)
|
||||
else:
|
||||
# 查指定 agent 的历史
|
||||
all_results = []
|
||||
@@ -6097,4 +6186,7 @@ class HyMemoryClient:
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *args):
|
||||
await self._close_async()
|
||||
try:
|
||||
await self._close_async()
|
||||
finally:
|
||||
self._stop_owned_loops()
|
||||
@@ -919,6 +919,7 @@ class CodingMemoryExtractor:
|
||||
resp = await self.llm_provider.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=self.max_tokens,
|
||||
audit_stage="coding_extract",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"[coding-extract] LLM call failed: {e}")
|
||||
|
||||
@@ -466,7 +466,13 @@ class WorkingFactReconciler:
|
||||
return [], []
|
||||
|
||||
candidates = await self._retrieve_candidates(
|
||||
drafts, user_id=user_id, source=source,
|
||||
drafts,
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
source=source,
|
||||
)
|
||||
|
||||
try:
|
||||
@@ -499,7 +505,15 @@ class WorkingFactReconciler:
|
||||
# ------------------------------------------------------------
|
||||
|
||||
async def _retrieve_candidates(
|
||||
self, drafts: List[WorkingFactDraft], *, user_id: str, source: str
|
||||
self,
|
||||
drafts: List[WorkingFactDraft],
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
session_id: Optional[str],
|
||||
workspace_id: Optional[str],
|
||||
branch: Optional[str],
|
||||
source: str,
|
||||
) -> List[List[WorkingFact]]:
|
||||
"""每条 draft 用 statement 召回同 user 的 top-N active 事实。
|
||||
|
||||
@@ -512,6 +526,10 @@ class WorkingFactReconciler:
|
||||
关掉时整份召回原样交给 LLM。适用于 subject 不是稳定实体名的数据集(自由文本、
|
||||
位置标签),那里精确字符串相等会把同实体判成不同主体,反而拦掉真实更新。
|
||||
"""
|
||||
# Empty/legacy session metadata remains readable, but cannot prove
|
||||
# conversation authority for a destructive SUPERSEDE.
|
||||
if not session_id:
|
||||
return [[] for _ in drafts]
|
||||
try:
|
||||
embeddings = await self.embed_service.embed_batch(
|
||||
[d.statement for d in drafts]
|
||||
@@ -525,6 +543,10 @@ class WorkingFactReconciler:
|
||||
try:
|
||||
hits = await self.store.search_facts(
|
||||
emb, user_id=user_id, top=self.candidates_per_draft,
|
||||
agent_id=agent_id,
|
||||
session_id=session_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
# 写入侧同样接词面通道:候选选错了 LLM 就没有机会判对。这里不传
|
||||
# min_score——写入侧要的是"有哪些可能的前身",让 LLM 看更宽的名单,
|
||||
# 阈值过滤是读取侧的事。
|
||||
@@ -571,6 +593,7 @@ class WorkingFactReconciler:
|
||||
resp = await self.llm_provider.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=self.max_tokens,
|
||||
audit_stage="coding_fact_reconcile",
|
||||
)
|
||||
return self._parse_ops(resp.content)
|
||||
|
||||
@@ -637,6 +660,7 @@ class WorkingFactReconciler:
|
||||
resp = await self.llm_provider.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=_CHAIN_SUMMARY_MAX_TOKENS,
|
||||
audit_stage="coding_fact_summary",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"[fact-reconcile] chain summary failed: {e}")
|
||||
@@ -792,6 +816,15 @@ class WorkingFactReconciler:
|
||||
old = await self.store.get_fact(old_id, user_id=user_id)
|
||||
if old is None:
|
||||
continue
|
||||
if (
|
||||
old.agent_id != agent_id
|
||||
or old.session_id != session_id
|
||||
or old.workspace_id != workspace_id
|
||||
):
|
||||
logger.warning(
|
||||
f"[fact-reconcile] refused cross-authority SUPERSEDE: {old.fact_id}"
|
||||
)
|
||||
continue
|
||||
# 闸门在候选阶段已经拦过一道,这里再拦一次:候选是按 statement 召回的,
|
||||
# 而 LLM 只能覆盖候选名单里的 id,理论上到不了这儿。真到了就说明前面
|
||||
# 哪一环漏了,宁可让这次覆盖不生效——多留一条旧事实是可恢复的,删掉
|
||||
|
||||
@@ -29,6 +29,9 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
_JUDGE_MAX_TURNS = 20
|
||||
_JUDGE_HEAD_TURNS = 5
|
||||
_JUDGE_USER_CONTENT_LIMIT = 1_000
|
||||
_JUDGE_USER_CONTENT_HEAD = 200
|
||||
_JUDGE_USER_CONTENT_TAIL = 200
|
||||
_JUDGE_MAX_QUERY_CHARS = 300
|
||||
_JUDGE_MAX_QUERIES = 20
|
||||
|
||||
@@ -186,6 +189,7 @@ async def classify_messages_scene(
|
||||
prompt=prompt,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
audit_stage="coding_judge",
|
||||
)
|
||||
_elapsed = round((time.perf_counter() - _t0) * 1000, 2)
|
||||
result = _parse_scene_json(resp.content)
|
||||
@@ -631,9 +635,13 @@ def _format_turns(summary: List[Dict[str, Any]]) -> str:
|
||||
rows = []
|
||||
for t in summary:
|
||||
user = t.get("user", "") or ""
|
||||
# 单 turn 用户文本截到 200 字符
|
||||
if len(user) > 200:
|
||||
user = user[:200] + "..."
|
||||
if len(user) > _JUDGE_USER_CONTENT_LIMIT:
|
||||
omitted = len(user) - _JUDGE_USER_CONTENT_HEAD - _JUDGE_USER_CONTENT_TAIL
|
||||
user = (
|
||||
user[:_JUDGE_USER_CONTENT_HEAD]
|
||||
+ f"\n[...{omitted} chars omitted...]\n"
|
||||
+ user[-_JUDGE_USER_CONTENT_TAIL:]
|
||||
)
|
||||
tools = t.get("tools", []) or []
|
||||
rows.append(
|
||||
f"- turn {t.get('turn', '?')}: user={user!r}, tools={tools}"
|
||||
|
||||
@@ -240,7 +240,7 @@ class CodingMemoryReconciler:
|
||||
)
|
||||
|
||||
# 3. 校验 + 执行
|
||||
executed = await self._execute(ops, drafts)
|
||||
executed = await self._execute(ops, drafts, candidates_per_draft)
|
||||
logger.info(
|
||||
f"[coding-reconcile] drafts={len(drafts)} ops={len(executed)} "
|
||||
+ ", ".join(f"{a}={sum(1 for o in executed if o.action == a)}"
|
||||
@@ -263,6 +263,12 @@ class CodingMemoryReconciler:
|
||||
"""
|
||||
result: List[List[CodingMemory]] = []
|
||||
for d in drafts:
|
||||
# A coding mutation must be attributable to one conversation.
|
||||
# Empty/legacy session metadata stays readable through ordinary
|
||||
# search, but it cannot authorize UPDATE/DELETE.
|
||||
if not d.session_id:
|
||||
result.append([])
|
||||
continue
|
||||
try:
|
||||
# task 为主入口,search_keys 为附加入口(去重、去空)
|
||||
queries = [d.task] + [k for k in (d.search_keys or []) if k and k != d.task]
|
||||
@@ -274,6 +280,8 @@ class CodingMemoryReconciler:
|
||||
hits = await self.store.search_by_query_embedding(
|
||||
emb,
|
||||
user_id=d.user_id,
|
||||
agent_id=d.agent_id,
|
||||
session_id=d.session_id,
|
||||
workspace_id=d.workspace_id,
|
||||
branch=d.branch,
|
||||
top=self.candidates_per_draft * 4,
|
||||
@@ -310,6 +318,7 @@ class CodingMemoryReconciler:
|
||||
resp = await self.llm_provider.complete(
|
||||
prompt=prompt,
|
||||
max_tokens=self.max_tokens,
|
||||
audit_stage="coding_reconcile",
|
||||
)
|
||||
return self._parse_ops(resp.content), prompt, resp.content, resp
|
||||
|
||||
@@ -414,6 +423,7 @@ class CodingMemoryReconciler:
|
||||
self,
|
||||
ops: List[ReconcileOp],
|
||||
drafts: List[CodingMemoryDraft],
|
||||
candidates_per_draft: List[List[CodingMemory]],
|
||||
) -> List[ReconcileOp]:
|
||||
"""
|
||||
校验 + 执行 ops。返回成功执行的 ops(filter 掉无效 op)。
|
||||
@@ -453,6 +463,11 @@ class CodingMemoryReconciler:
|
||||
if not op.target_experience_id:
|
||||
logger.warning(f"[coding-reconcile] drop UPDATE missing target: {op.to_dict()}")
|
||||
continue
|
||||
if not self._target_was_offered(op, candidates_per_draft):
|
||||
logger.warning(
|
||||
f"[coding-reconcile] drop UPDATE outside authority scope: {op.to_dict()}"
|
||||
)
|
||||
continue
|
||||
if op.draft_idx in consumed:
|
||||
logger.warning(f"[coding-reconcile] drop UPDATE on consumed draft: {op.to_dict()}")
|
||||
continue
|
||||
@@ -475,6 +490,11 @@ class CodingMemoryReconciler:
|
||||
if not op.target_experience_id:
|
||||
logger.warning(f"[coding-reconcile] drop DELETE missing target: {op.to_dict()}")
|
||||
continue
|
||||
if not self._target_was_offered(op, candidates_per_draft):
|
||||
logger.warning(
|
||||
f"[coding-reconcile] drop DELETE outside authority scope: {op.to_dict()}"
|
||||
)
|
||||
continue
|
||||
# DELETE 需要知道 user_id;通过 draft_idx(如有)或第一个 draft 推断
|
||||
user_id = self._infer_user_id(op, drafts)
|
||||
if not user_id:
|
||||
@@ -511,6 +531,19 @@ class CodingMemoryReconciler:
|
||||
|
||||
return executed
|
||||
|
||||
@staticmethod
|
||||
def _target_was_offered(
|
||||
op: ReconcileOp,
|
||||
candidates_per_draft: List[List[CodingMemory]],
|
||||
) -> bool:
|
||||
"""Only mutate a target frozen into this draft's scoped candidate set."""
|
||||
if op.draft_idx is None or not (0 <= op.draft_idx < len(candidates_per_draft)):
|
||||
return False
|
||||
return any(
|
||||
candidate.experience_id == op.target_experience_id
|
||||
for candidate in candidates_per_draft[op.draft_idx]
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _infer_user_id(
|
||||
op: ReconcileOp, drafts: List[CodingMemoryDraft]
|
||||
|
||||
+98
-15
@@ -209,6 +209,14 @@ def _scope_where(
|
||||
return sql, params
|
||||
|
||||
|
||||
def _in_clause(column: str, values: Optional[List[str]]) -> Tuple[str, List[Any]]:
|
||||
"""SQL ``column IN (...)`` + parameters; empty values mean no condition."""
|
||||
if not values:
|
||||
return "", []
|
||||
placeholders = ",".join("?" for _ in values)
|
||||
return f"{column} IN ({placeholders})", list(values)
|
||||
|
||||
|
||||
def _scope_matches(
|
||||
row_workspace_id: Optional[str],
|
||||
row_branch: Optional[str],
|
||||
@@ -529,10 +537,20 @@ class CodingMemoryStore:
|
||||
"""覆盖式更新一条 coding memory。"""
|
||||
# 拿旧 keys 长度,先删旧 VDB rows
|
||||
prev = await self.get_by_id(memory.experience_id, user_id=memory.user_id)
|
||||
prev_key_count = (1 + len(prev.search_keys)) if prev else 0
|
||||
if prev is None:
|
||||
raise ValueError("coding_memory_update_target_missing")
|
||||
if (
|
||||
not prev.session_id
|
||||
or not memory.session_id
|
||||
or prev.agent_id != memory.agent_id
|
||||
or prev.session_id != memory.session_id
|
||||
or prev.workspace_id != memory.workspace_id
|
||||
):
|
||||
raise ValueError("coding_memory_update_authority_mismatch")
|
||||
prev_key_count = 1 + len(prev.search_keys)
|
||||
|
||||
memory.updated_at = datetime.utcnow()
|
||||
if memory.created_at is None and prev is not None:
|
||||
if memory.created_at is None:
|
||||
memory.created_at = prev.created_at
|
||||
await _run_in_pool(self._upsert_sql_sync, memory)
|
||||
# 删旧 VDB
|
||||
@@ -671,6 +689,10 @@ class CodingMemoryStore:
|
||||
query_embedding: List[float],
|
||||
user_id: str,
|
||||
*,
|
||||
agent_id: Optional[str] = None,
|
||||
session_id: Optional[str] = None,
|
||||
agent_ids: Optional[List[str]] = None,
|
||||
session_ids: Optional[List[str]] = None,
|
||||
workspace_id: Optional[str] = None,
|
||||
branch: Optional[str] = None,
|
||||
top: int = 40,
|
||||
@@ -688,21 +710,34 @@ class CodingMemoryStore:
|
||||
},
|
||||
...
|
||||
]
|
||||
workspace_id / branch 是作用域过滤,规则见 _scope_where:不传则纯语义召回
|
||||
(行为与加过滤前完全一致);传了则只保留 SQLite 里落在作用域内的经验,并按
|
||||
作用域的选择率放大向量取数,避免"先取 top-N 再过滤"把召回吃掉。
|
||||
agent_id / session_id(或 agent_ids / session_ids)是写授权作用域,
|
||||
传入时精确匹配;workspace_id / branch 的规则见 _scope_where。
|
||||
不传任何作用域则维持纯语义召回;传了则只保留 SQLite 里落在作用域内的经验,
|
||||
并按选择率放大向量取数,避免"先取 top-N 再过滤"把召回吃掉。
|
||||
"""
|
||||
if self._vec is None:
|
||||
return []
|
||||
|
||||
effective_agent_ids = list(agent_ids) if agent_ids else (
|
||||
[agent_id] if agent_id else []
|
||||
)
|
||||
effective_session_ids = list(session_ids) if session_ids else (
|
||||
[session_id] if session_id else []
|
||||
)
|
||||
allowed: Optional[Set[str]] = None
|
||||
if workspace_id:
|
||||
if effective_agent_ids or effective_session_ids or workspace_id:
|
||||
allowed, total = await _run_in_pool(
|
||||
self._scoped_experience_ids_sync, user_id, workspace_id, branch
|
||||
self._authority_scoped_experience_ids_sync,
|
||||
user_id,
|
||||
effective_agent_ids,
|
||||
effective_session_ids,
|
||||
workspace_id,
|
||||
branch,
|
||||
)
|
||||
if not allowed:
|
||||
logger.info(
|
||||
f"[coding-store] no experience in scope: "
|
||||
f"agent_id={agent_id!r} session_id={session_id!r} "
|
||||
f"workspace_id={workspace_id!r} branch={branch!r} "
|
||||
f"(user has {total}); skip vdb search"
|
||||
)
|
||||
@@ -778,6 +813,10 @@ class CodingMemoryStore:
|
||||
query_embedding: List[float],
|
||||
user_id: str,
|
||||
*,
|
||||
agent_id: Optional[str] = None,
|
||||
session_id: Optional[str] = None,
|
||||
agent_ids: Optional[List[str]] = None,
|
||||
session_ids: Optional[List[str]] = None,
|
||||
workspace_id: Optional[str] = None,
|
||||
branch: Optional[str] = None,
|
||||
top: int = 5,
|
||||
@@ -791,8 +830,9 @@ class CodingMemoryStore:
|
||||
SUPERSEDE 时向量行已被删掉,理论上召不到 superseded;这里再按 SQLite 的
|
||||
status 过一道,避免两边不一致时把过期结论放出去。
|
||||
|
||||
workspace_id / branch 是作用域过滤,规则见 _scope_where,语义与经验层一致:
|
||||
不传不过滤,传了按 SQLite 里的值筛,并按选择率放大向量取数。
|
||||
agent_id / session_id(或 agent_ids / session_ids)是写授权作用域,
|
||||
传入时精确匹配;workspace_id / branch 的规则见 _scope_where。
|
||||
不传任何作用域则不过滤,传了按 SQLite 里的值筛,并按选择率放大向量取数。
|
||||
|
||||
向量库按 top 的 4 倍过采样:向量行和 SQLite 行不是一个事务里写的,中间挂掉
|
||||
就会留下"向量还在、SQLite 没了"的孤儿,软删时删向量失败也会留下 superseded
|
||||
@@ -808,14 +848,27 @@ class CodingMemoryStore:
|
||||
if self._fact_vec is None:
|
||||
return []
|
||||
|
||||
effective_agent_ids = list(agent_ids) if agent_ids else (
|
||||
[agent_id] if agent_id else []
|
||||
)
|
||||
effective_session_ids = list(session_ids) if session_ids else (
|
||||
[session_id] if session_id else []
|
||||
)
|
||||
fetch = top * 4
|
||||
if workspace_id:
|
||||
if effective_agent_ids or effective_session_ids or workspace_id:
|
||||
scoped, total = await _run_in_pool(
|
||||
self._count_active_facts_sync, user_id, workspace_id, branch
|
||||
self._count_active_facts_sync,
|
||||
user_id,
|
||||
effective_agent_ids,
|
||||
effective_session_ids,
|
||||
workspace_id,
|
||||
branch,
|
||||
)
|
||||
if not scoped:
|
||||
logger.info(
|
||||
f"[coding-store] no active fact in scope: "
|
||||
f"agent_ids={effective_agent_ids!r} "
|
||||
f"session_ids={effective_session_ids!r} "
|
||||
f"workspace_id={workspace_id!r} branch={branch!r} "
|
||||
f"(user has {total}); skip fact vdb search"
|
||||
)
|
||||
@@ -852,6 +905,8 @@ class CodingMemoryStore:
|
||||
alive = [
|
||||
(f, scores[f.fact_id]) for f in facts
|
||||
if f.status == "active"
|
||||
and (not effective_agent_ids or f.agent_id in effective_agent_ids)
|
||||
and (not effective_session_ids or f.session_id in effective_session_ids)
|
||||
and _scope_matches(f.workspace_id, f.branch, workspace_id, branch)
|
||||
]
|
||||
kept = [(f, s) for f, s in alive if s >= min_score]
|
||||
@@ -1050,14 +1105,28 @@ class CodingMemoryStore:
|
||||
by_id = {r["experience_id"]: _row_to_memory(r) for r in rows}
|
||||
return [by_id[eid] for eid in experience_ids if eid in by_id]
|
||||
|
||||
def _scoped_experience_ids_sync(
|
||||
self, user_id: str, workspace_id: str, branch: Optional[str]
|
||||
def _authority_scoped_experience_ids_sync(
|
||||
self,
|
||||
user_id: str,
|
||||
agent_ids: Optional[List[str]],
|
||||
session_ids: Optional[List[str]],
|
||||
workspace_id: Optional[str],
|
||||
branch: Optional[str],
|
||||
) -> Tuple[Set[str], int]:
|
||||
"""返回 (落在作用域内的 experience_id 集合, 该 user 的经验总数)。
|
||||
"""返回 (落在授权作用域内的 experience_id 集合, 该 user 的经验总数)。
|
||||
|
||||
一次查询同时拿到过滤结果和选择率分母:前者供向量命中做交集,后者定放大倍数。
|
||||
agent_ids / session_ids 为空表示该维度不筛;非空按 IN 精确匹配。
|
||||
"""
|
||||
scope_sql, scope_params = _scope_where(workspace_id, branch)
|
||||
agent_sql, agent_params = _in_clause("agent_id", agent_ids)
|
||||
if agent_sql:
|
||||
scope_sql = f"({agent_sql}) AND ({scope_sql})"
|
||||
scope_params = [*agent_params, *scope_params]
|
||||
session_sql, session_params = _in_clause("session_id", session_ids)
|
||||
if session_sql:
|
||||
scope_sql = f"({session_sql}) AND ({scope_sql})"
|
||||
scope_params = [*session_params, *scope_params]
|
||||
with self._lock:
|
||||
assert self._conn is not None
|
||||
cur = self._conn.execute(
|
||||
@@ -1069,13 +1138,27 @@ class CodingMemoryStore:
|
||||
return {r["experience_id"] for r in rows if r["in_scope"]}, len(rows)
|
||||
|
||||
def _count_active_facts_sync(
|
||||
self, user_id: str, workspace_id: str, branch: Optional[str]
|
||||
self,
|
||||
user_id: str,
|
||||
agent_ids: Optional[List[str]],
|
||||
session_ids: Optional[List[str]],
|
||||
workspace_id: Optional[str],
|
||||
branch: Optional[str],
|
||||
) -> Tuple[int, int]:
|
||||
"""返回 (作用域内 active 事实数, 该 user 的 active 事实总数)。
|
||||
|
||||
事实层的过滤在 get_facts 拿到对象之后用 _scope_matches 做,这里只要选择率。
|
||||
agent_ids / session_ids 为空表示该维度不筛;非空按 IN 精确匹配。
|
||||
"""
|
||||
scope_sql, scope_params = _scope_where(workspace_id, branch)
|
||||
agent_sql, agent_params = _in_clause("agent_id", agent_ids)
|
||||
if agent_sql:
|
||||
scope_sql = f"({agent_sql}) AND ({scope_sql})"
|
||||
scope_params = [*agent_params, *scope_params]
|
||||
session_sql, session_params = _in_clause("session_id", session_ids)
|
||||
if session_sql:
|
||||
scope_sql = f"({session_sql}) AND ({scope_sql})"
|
||||
scope_params = [*session_params, *scope_params]
|
||||
with self._lock:
|
||||
assert self._conn is not None
|
||||
row = self._conn.execute(
|
||||
|
||||
+8
-1
@@ -6,7 +6,7 @@ Agent Memory - 配置管理
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional, Dict, Any, List
|
||||
from typing import Optional, Dict, Any, Callable, List
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -1252,6 +1252,13 @@ class MemoryConfig:
|
||||
enable_agent: bool = False
|
||||
debug: bool = False
|
||||
metrics_enabled: bool = True
|
||||
# Runtime-only HTTP transport factory. MemHub uses this for request-cost
|
||||
# budgeting without restricting the embedded SDK's local store access.
|
||||
http_transport_factory: Optional[Callable[[], Any]] = field(
|
||||
default=None,
|
||||
repr=False,
|
||||
compare=False,
|
||||
)
|
||||
|
||||
def __post_init__(self):
|
||||
# enable_graph 跟随 mode 走:mode=ultra 时自动启用,无需独立环境变量
|
||||
|
||||
@@ -356,11 +356,21 @@ class EmbedService:
|
||||
key = self._client_cache_key(base_url, api_key, timeout)
|
||||
client = self._client_cache.get(key)
|
||||
if client is None:
|
||||
client_options = {
|
||||
"api_key": api_key,
|
||||
"base_url": base_url or None,
|
||||
"timeout": timeout,
|
||||
"max_retries": 0,
|
||||
}
|
||||
transport_factory = getattr(self.config, "http_transport_factory", None)
|
||||
if callable(transport_factory):
|
||||
import httpx
|
||||
|
||||
client_options["http_client"] = httpx.AsyncClient(
|
||||
transport=transport_factory()
|
||||
)
|
||||
client = AsyncOpenAI(
|
||||
api_key=api_key,
|
||||
base_url=base_url or None,
|
||||
timeout=timeout,
|
||||
max_retries=0,
|
||||
**client_options,
|
||||
)
|
||||
self._client_cache[key] = client
|
||||
return client
|
||||
|
||||
@@ -464,24 +464,67 @@ class MysqlCache(CacheBase):
|
||||
# ================================================================
|
||||
|
||||
async def store_metrics_minute(self, minute_ts: str, data: dict) -> None:
|
||||
import aiomysql
|
||||
|
||||
try:
|
||||
now = datetime.now().isoformat()
|
||||
# 尝试 UPSERT
|
||||
existing = await self._fetchone(
|
||||
"SELECT data FROM system_metrics WHERE minute_ts = %s", (minute_ts,)
|
||||
)
|
||||
if existing:
|
||||
old_data = json.loads(existing["data"])
|
||||
merged = self._merge_metric_buckets(old_data, data)
|
||||
await self._execute(
|
||||
"UPDATE system_metrics SET data = %s, created_at = %s WHERE minute_ts = %s",
|
||||
(json.dumps(merged, default=str), now, minute_ts),
|
||||
)
|
||||
else:
|
||||
await self._execute(
|
||||
"INSERT INTO system_metrics (minute_ts, data, created_at) VALUES (%s, %s, %s)",
|
||||
(minute_ts, json.dumps(data, default=str), now),
|
||||
)
|
||||
# Read-merge-write must be one transaction: two instances flushing
|
||||
# the same minute used to overwrite each other's counters, and a
|
||||
# concurrent insert could lose the whole batch to a duplicate key.
|
||||
for attempt in range(2):
|
||||
try:
|
||||
async with self._pool.acquire() as conn:
|
||||
await conn.begin()
|
||||
try:
|
||||
async with conn.cursor(aiomysql.DictCursor) as cur:
|
||||
await cur.execute(
|
||||
"SELECT data FROM system_metrics "
|
||||
"WHERE minute_ts = %s FOR UPDATE",
|
||||
(minute_ts,),
|
||||
)
|
||||
row = await cur.fetchone()
|
||||
if row is not None:
|
||||
old_raw = (
|
||||
row["data"]
|
||||
if isinstance(row, dict)
|
||||
else row[0]
|
||||
)
|
||||
merged = self._merge_metric_buckets(
|
||||
json.loads(old_raw), data,
|
||||
)
|
||||
await cur.execute(
|
||||
"UPDATE system_metrics SET data = %s, "
|
||||
"created_at = %s WHERE minute_ts = %s",
|
||||
(
|
||||
json.dumps(merged, default=str),
|
||||
now,
|
||||
minute_ts,
|
||||
),
|
||||
)
|
||||
else:
|
||||
await cur.execute(
|
||||
"INSERT INTO system_metrics "
|
||||
"(minute_ts, data, created_at) "
|
||||
"VALUES (%s, %s, %s)",
|
||||
(
|
||||
minute_ts,
|
||||
json.dumps(data, default=str),
|
||||
now,
|
||||
),
|
||||
)
|
||||
await conn.commit()
|
||||
return
|
||||
except Exception:
|
||||
await conn.rollback()
|
||||
raise
|
||||
except (aiomysql.IntegrityError, aiomysql.OperationalError) as error:
|
||||
code = error.args[0] if error.args else None
|
||||
deadlock = code in (1205, 1213)
|
||||
if attempt or (isinstance(error, aiomysql.OperationalError) and not deadlock):
|
||||
raise
|
||||
# A competing writer inserted the row, or InnoDB chose us
|
||||
# as the deadlock victim. Retry so this batch merges.
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.warning(f"store_metrics_minute failed: {e}")
|
||||
|
||||
|
||||
+136
-125
@@ -7,7 +7,7 @@ Agent Memory V2 - SqliteCache
|
||||
设计模式跟随 history_store.py:
|
||||
- 标准库 sqlite3 + _run_in_sqlite_pool()
|
||||
- WAL 模式 + check_same_thread=False
|
||||
- threading.Lock 保护写操作
|
||||
- 每实例 threading.RLock 保护共享连接的全部操作与关闭
|
||||
|
||||
三张表:
|
||||
- memory_operations / pipeline_logs / system_metrics: 审计与观测
|
||||
@@ -122,7 +122,10 @@ class SqliteCache(CacheBase):
|
||||
def __init__(self, config: MemoryConfig):
|
||||
self.config = config
|
||||
self._conn: Optional[sqlite3.Connection] = None
|
||||
self._lock = threading.Lock()
|
||||
# One connection is shared by the executor workers. Use the same lock
|
||||
# for reads, writes, initialization and close so close() cannot free
|
||||
# the connection while another worker is inside sqlite3.
|
||||
self._lock = threading.RLock()
|
||||
|
||||
# 读取 db_path
|
||||
cache_cfg = getattr(config, "cache", None)
|
||||
@@ -144,35 +147,37 @@ class SqliteCache(CacheBase):
|
||||
logger.debug(f"SqliteCache initialized: {self._db_path}")
|
||||
|
||||
def _init_sync(self) -> None:
|
||||
db_dir = os.path.dirname(self._db_path)
|
||||
if db_dir:
|
||||
os.makedirs(db_dir, exist_ok=True)
|
||||
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
|
||||
self._conn.row_factory = sqlite3.Row
|
||||
self._conn.execute("PRAGMA journal_mode=WAL;")
|
||||
self._conn.executescript(_CREATE_TABLES_SQL)
|
||||
self._migrate_pipeline_logs_columns()
|
||||
self._conn.commit()
|
||||
with self._lock:
|
||||
db_dir = os.path.dirname(self._db_path)
|
||||
if db_dir:
|
||||
os.makedirs(db_dir, exist_ok=True)
|
||||
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
|
||||
self._conn.row_factory = sqlite3.Row
|
||||
self._conn.execute("PRAGMA journal_mode=WAL;")
|
||||
self._conn.executescript(_CREATE_TABLES_SQL)
|
||||
self._migrate_pipeline_logs_columns()
|
||||
self._conn.commit()
|
||||
|
||||
def _migrate_pipeline_logs_columns(self) -> None:
|
||||
"""给旧库的 pipeline_logs_v2 补 namespace/session/event time 列。"""
|
||||
try:
|
||||
cols = {
|
||||
row["name"]
|
||||
for row in self._conn.execute("PRAGMA table_info(pipeline_logs_v2)")
|
||||
}
|
||||
_text_cols = ("namespace", "session_id")
|
||||
for col in _text_cols:
|
||||
if col not in cols:
|
||||
with self._lock:
|
||||
try:
|
||||
cols = {
|
||||
row["name"]
|
||||
for row in self._conn.execute("PRAGMA table_info(pipeline_logs_v2)")
|
||||
}
|
||||
_text_cols = ("namespace", "session_id")
|
||||
for col in _text_cols:
|
||||
if col not in cols:
|
||||
self._conn.execute(
|
||||
f"ALTER TABLE pipeline_logs_v2 ADD COLUMN {col} TEXT NOT NULL DEFAULT ''"
|
||||
)
|
||||
if "event_at_ms" not in cols:
|
||||
self._conn.execute(
|
||||
f"ALTER TABLE pipeline_logs_v2 ADD COLUMN {col} TEXT NOT NULL DEFAULT ''"
|
||||
"ALTER TABLE pipeline_logs_v2 ADD COLUMN event_at_ms INTEGER NOT NULL DEFAULT 0"
|
||||
)
|
||||
if "event_at_ms" not in cols:
|
||||
self._conn.execute(
|
||||
"ALTER TABLE pipeline_logs_v2 ADD COLUMN event_at_ms INTEGER NOT NULL DEFAULT 0"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug(f"[sqlite] pipeline_logs_v2 column migration skipped: {e}")
|
||||
except Exception as e:
|
||||
logger.debug(f"[sqlite] pipeline_logs_v2 column migration skipped: {e}")
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._conn is not None:
|
||||
@@ -180,9 +185,10 @@ class SqliteCache(CacheBase):
|
||||
logger.info("SqliteCache closed")
|
||||
|
||||
def _close_sync(self) -> None:
|
||||
if self._conn:
|
||||
self._conn.close()
|
||||
self._conn = None
|
||||
with self._lock:
|
||||
if self._conn:
|
||||
self._conn.close()
|
||||
self._conn = None
|
||||
|
||||
# ================================================================
|
||||
# 统计
|
||||
@@ -255,39 +261,40 @@ class SqliteCache(CacheBase):
|
||||
"""
|
||||
try:
|
||||
def _query():
|
||||
conditions = []
|
||||
params = []
|
||||
if request_id:
|
||||
conditions.append("request_id = ?")
|
||||
params.append(request_id)
|
||||
if memory_id:
|
||||
conditions.append("memory_id = ?")
|
||||
params.append(memory_id)
|
||||
if user_id:
|
||||
conditions.append("user_id = ?")
|
||||
params.append(user_id)
|
||||
with self._lock:
|
||||
conditions = []
|
||||
params = []
|
||||
if request_id:
|
||||
conditions.append("request_id = ?")
|
||||
params.append(request_id)
|
||||
if memory_id:
|
||||
conditions.append("memory_id = ?")
|
||||
params.append(memory_id)
|
||||
if user_id:
|
||||
conditions.append("user_id = ?")
|
||||
params.append(user_id)
|
||||
|
||||
if not conditions:
|
||||
return []
|
||||
if not conditions:
|
||||
return []
|
||||
|
||||
where = " AND ".join(conditions)
|
||||
rows = self._conn.execute(
|
||||
f"SELECT * FROM memory_operations_v2 WHERE {where} "
|
||||
f"ORDER BY id DESC LIMIT ?",
|
||||
params + [limit],
|
||||
).fetchall()
|
||||
result = []
|
||||
for r in rows:
|
||||
row = dict(r)
|
||||
# 反序列化 supersedes JSON 字符串 → list
|
||||
import json as _json
|
||||
raw_sup = row.get("supersedes", "")
|
||||
try:
|
||||
row["supersedes"] = _json.loads(raw_sup) if raw_sup else []
|
||||
except Exception:
|
||||
row["supersedes"] = []
|
||||
result.append(row)
|
||||
return result
|
||||
where = " AND ".join(conditions)
|
||||
rows = self._conn.execute(
|
||||
f"SELECT * FROM memory_operations_v2 WHERE {where} "
|
||||
f"ORDER BY id DESC LIMIT ?",
|
||||
params + [limit],
|
||||
).fetchall()
|
||||
result = []
|
||||
for r in rows:
|
||||
row = dict(r)
|
||||
# 反序列化 supersedes JSON 字符串 → list
|
||||
import json as _json
|
||||
raw_sup = row.get("supersedes", "")
|
||||
try:
|
||||
row["supersedes"] = _json.loads(raw_sup) if raw_sup else []
|
||||
except Exception:
|
||||
row["supersedes"] = []
|
||||
result.append(row)
|
||||
return result
|
||||
|
||||
return await _run_in_sqlite_pool(_query)
|
||||
except Exception as e:
|
||||
@@ -369,37 +376,38 @@ class SqliteCache(CacheBase):
|
||||
"""
|
||||
try:
|
||||
def _query():
|
||||
conditions = []
|
||||
params = []
|
||||
if request_id:
|
||||
conditions.append("request_id = ?")
|
||||
params.append(request_id)
|
||||
if user_id:
|
||||
conditions.append("user_id = ?")
|
||||
params.append(user_id)
|
||||
if step:
|
||||
conditions.append("step = ?")
|
||||
params.append(step)
|
||||
with self._lock:
|
||||
conditions = []
|
||||
params = []
|
||||
if request_id:
|
||||
conditions.append("request_id = ?")
|
||||
params.append(request_id)
|
||||
if user_id:
|
||||
conditions.append("user_id = ?")
|
||||
params.append(user_id)
|
||||
if step:
|
||||
conditions.append("step = ?")
|
||||
params.append(step)
|
||||
|
||||
if not conditions:
|
||||
return []
|
||||
if not conditions:
|
||||
return []
|
||||
|
||||
where = " AND ".join(conditions)
|
||||
rows = self._conn.execute(
|
||||
f"SELECT * FROM pipeline_logs_v2 WHERE {where} "
|
||||
f"ORDER BY id ASC LIMIT ?",
|
||||
params + [limit],
|
||||
).fetchall()
|
||||
out = [dict(r) for r in rows]
|
||||
# 派生可读的 event_at(ISO ms)供查询接口直接用;event_at_ms 为 0 时留空
|
||||
from datetime import datetime as _dt
|
||||
for _r in out:
|
||||
_ms = _r.get("event_at_ms") or 0
|
||||
_r["event_at"] = (
|
||||
_dt.fromtimestamp(_ms / 1000.0).isoformat(timespec="milliseconds")
|
||||
if _ms else ""
|
||||
)
|
||||
return out
|
||||
where = " AND ".join(conditions)
|
||||
rows = self._conn.execute(
|
||||
f"SELECT * FROM pipeline_logs_v2 WHERE {where} "
|
||||
f"ORDER BY id ASC LIMIT ?",
|
||||
params + [limit],
|
||||
).fetchall()
|
||||
out = [dict(r) for r in rows]
|
||||
# 派生可读的 event_at(ISO ms)供查询接口直接用;event_at_ms 为 0 时留空
|
||||
from datetime import datetime as _dt
|
||||
for _r in out:
|
||||
_ms = _r.get("event_at_ms") or 0
|
||||
_r["event_at"] = (
|
||||
_dt.fromtimestamp(_ms / 1000.0).isoformat(timespec="milliseconds")
|
||||
if _ms else ""
|
||||
)
|
||||
return out
|
||||
|
||||
return await _run_in_sqlite_pool(_query)
|
||||
except Exception as e:
|
||||
@@ -414,26 +422,27 @@ class SqliteCache(CacheBase):
|
||||
"""存储一个分钟桶的增量指标数据(UPSERT: 同一分钟多次写入会合并)"""
|
||||
try:
|
||||
def _store():
|
||||
now = datetime.now().isoformat()
|
||||
# UPSERT: 如果同一分钟已存在,合并数据
|
||||
existing = self._conn.execute(
|
||||
"SELECT data FROM system_metrics WHERE minute_ts = ?",
|
||||
(minute_ts,),
|
||||
).fetchone()
|
||||
if existing:
|
||||
import json as _json
|
||||
old_data = _json.loads(existing["data"])
|
||||
merged = self._merge_metric_buckets(old_data, data)
|
||||
self._conn.execute(
|
||||
"UPDATE system_metrics SET data = ?, created_at = ? WHERE minute_ts = ?",
|
||||
(_json.dumps(merged, default=str), now, minute_ts),
|
||||
)
|
||||
else:
|
||||
self._conn.execute(
|
||||
"INSERT INTO system_metrics (minute_ts, data, created_at) VALUES (?, ?, ?)",
|
||||
(minute_ts, json.dumps(data, default=str), now),
|
||||
)
|
||||
self._conn.commit()
|
||||
with self._lock:
|
||||
now = datetime.now().isoformat()
|
||||
# UPSERT: 如果同一分钟已存在,合并数据
|
||||
existing = self._conn.execute(
|
||||
"SELECT data FROM system_metrics WHERE minute_ts = ?",
|
||||
(minute_ts,),
|
||||
).fetchone()
|
||||
if existing:
|
||||
import json as _json
|
||||
old_data = _json.loads(existing["data"])
|
||||
merged = self._merge_metric_buckets(old_data, data)
|
||||
self._conn.execute(
|
||||
"UPDATE system_metrics SET data = ?, created_at = ? WHERE minute_ts = ?",
|
||||
(_json.dumps(merged, default=str), now, minute_ts),
|
||||
)
|
||||
else:
|
||||
self._conn.execute(
|
||||
"INSERT INTO system_metrics (minute_ts, data, created_at) VALUES (?, ?, ?)",
|
||||
(minute_ts, json.dumps(data, default=str), now),
|
||||
)
|
||||
self._conn.commit()
|
||||
await _run_in_sqlite_pool(_store)
|
||||
except Exception as e:
|
||||
logger.warning(f"store_metrics_minute failed: {e}")
|
||||
@@ -442,17 +451,18 @@ class SqliteCache(CacheBase):
|
||||
"""读取指定时间范围内的分钟指标数据"""
|
||||
try:
|
||||
def _load():
|
||||
rows = self._conn.execute(
|
||||
"SELECT data FROM system_metrics WHERE minute_ts >= ? AND minute_ts <= ? ORDER BY minute_ts",
|
||||
(start_ts, end_ts),
|
||||
).fetchall()
|
||||
results = []
|
||||
for row in rows:
|
||||
try:
|
||||
results.append(json.loads(row["data"]))
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
return results
|
||||
with self._lock:
|
||||
rows = self._conn.execute(
|
||||
"SELECT data FROM system_metrics WHERE minute_ts >= ? AND minute_ts <= ? ORDER BY minute_ts",
|
||||
(start_ts, end_ts),
|
||||
).fetchall()
|
||||
results = []
|
||||
for row in rows:
|
||||
try:
|
||||
results.append(json.loads(row["data"]))
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
return results
|
||||
return await _run_in_sqlite_pool(_load)
|
||||
except Exception as e:
|
||||
logger.warning(f"load_metrics_range failed: {e}")
|
||||
@@ -462,11 +472,12 @@ class SqliteCache(CacheBase):
|
||||
"""删除指定时间之前的 metrics 数据"""
|
||||
try:
|
||||
def _cleanup():
|
||||
self._conn.execute(
|
||||
"DELETE FROM system_metrics WHERE minute_ts < ?",
|
||||
(before_ts,),
|
||||
)
|
||||
self._conn.commit()
|
||||
with self._lock:
|
||||
self._conn.execute(
|
||||
"DELETE FROM system_metrics WHERE minute_ts < ?",
|
||||
(before_ts,),
|
||||
)
|
||||
self._conn.commit()
|
||||
await _run_in_sqlite_pool(_cleanup)
|
||||
except Exception as e:
|
||||
logger.warning(f"cleanup_old_metrics failed: {e}")
|
||||
|
||||
+113
-69
@@ -16,6 +16,7 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import sqlite3
|
||||
import threading
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, List, Any, Optional
|
||||
|
||||
@@ -65,6 +66,11 @@ def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _escape_like(value: str) -> str:
|
||||
"""Escape LIKE wildcards so a user id is matched literally."""
|
||||
return value.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
|
||||
|
||||
class HistoryStore:
|
||||
"""
|
||||
SQLite 历史存储 - 记录所有内存操作的审计日志。
|
||||
@@ -89,6 +95,10 @@ class HistoryStore:
|
||||
"""
|
||||
self._db_path = config.history.db_path
|
||||
self._conn: Optional[sqlite3.Connection] = None
|
||||
# sqlite3.Connection.close() must never race with an execute() on the
|
||||
# same connection. The lock also serializes operations on this shared
|
||||
# connection; separate HistoryStore instances remain independent.
|
||||
self._conn_lock = threading.RLock()
|
||||
|
||||
async def initialize(self) -> None:
|
||||
"""创建表和索引(如果不存在)"""
|
||||
@@ -97,20 +107,21 @@ class HistoryStore:
|
||||
|
||||
def _init_sync(self) -> None:
|
||||
"""同步初始化:创建目录、连接数据库、建表建索引"""
|
||||
# 确保目录存在
|
||||
db_dir = os.path.dirname(self._db_path)
|
||||
if db_dir:
|
||||
os.makedirs(db_dir, exist_ok=True)
|
||||
with self._conn_lock:
|
||||
# 确保目录存在
|
||||
db_dir = os.path.dirname(self._db_path)
|
||||
if db_dir:
|
||||
os.makedirs(db_dir, exist_ok=True)
|
||||
|
||||
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
|
||||
self._conn.row_factory = sqlite3.Row
|
||||
# 开启 WAL 模式,提高并发读写性能
|
||||
self._conn.execute("PRAGMA journal_mode=WAL;")
|
||||
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
|
||||
self._conn.row_factory = sqlite3.Row
|
||||
# 开启 WAL 模式,提高并发读写性能
|
||||
self._conn.execute("PRAGMA journal_mode=WAL;")
|
||||
|
||||
self._conn.execute(_CREATE_TABLE_SQL)
|
||||
for idx_sql in _CREATE_INDEXES_SQL:
|
||||
self._conn.execute(idx_sql)
|
||||
self._conn.commit()
|
||||
self._conn.execute(_CREATE_TABLE_SQL)
|
||||
for idx_sql in _CREATE_INDEXES_SQL:
|
||||
self._conn.execute(idx_sql)
|
||||
self._conn.commit()
|
||||
|
||||
# ================================================================
|
||||
# 记录操作
|
||||
@@ -218,13 +229,14 @@ class HistoryStore:
|
||||
return await _run_in_sqlite_pool(self._get_history_sync, memory_id)
|
||||
|
||||
def _get_history_sync(self, memory_id: str) -> List[Dict[str, Any]]:
|
||||
if self._conn is None:
|
||||
return []
|
||||
cursor = self._conn.execute(
|
||||
"SELECT * FROM memory_history WHERE memory_id = ? ORDER BY created_at ASC",
|
||||
(memory_id,),
|
||||
)
|
||||
return [dict(row) for row in cursor.fetchall()]
|
||||
with self._conn_lock:
|
||||
if self._conn is None:
|
||||
return []
|
||||
cursor = self._conn.execute(
|
||||
"SELECT * FROM memory_history WHERE memory_id = ? ORDER BY created_at ASC",
|
||||
(memory_id,),
|
||||
)
|
||||
return [dict(row) for row in cursor.fetchall()]
|
||||
|
||||
async def get_recent(
|
||||
self,
|
||||
@@ -244,14 +256,43 @@ class HistoryStore:
|
||||
return await _run_in_sqlite_pool(self._get_recent_sync, isolation_key, limit)
|
||||
|
||||
def _get_recent_sync(self, isolation_key: str, limit: int) -> List[Dict[str, Any]]:
|
||||
if self._conn is None:
|
||||
return []
|
||||
cursor = self._conn.execute(
|
||||
"SELECT * FROM memory_history WHERE isolation_key = ? "
|
||||
"ORDER BY created_at DESC LIMIT ?",
|
||||
(isolation_key, limit),
|
||||
with self._conn_lock:
|
||||
if self._conn is None:
|
||||
return []
|
||||
cursor = self._conn.execute(
|
||||
"SELECT * FROM memory_history WHERE isolation_key = ? "
|
||||
"ORDER BY created_at DESC LIMIT ?",
|
||||
(isolation_key, limit),
|
||||
)
|
||||
return [dict(row) for row in cursor.fetchall()]
|
||||
|
||||
async def get_recent_for_user(
|
||||
self,
|
||||
user_id: str,
|
||||
limit: int = 50,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""获取某 user 下所有 agent 的最近操作。
|
||||
|
||||
isolation_key 形如 ``"{user_id}:{agent_id}"``。user_id 里的 LIKE
|
||||
通配符会被转义,保证只匹配这个 user,不会漏到 ``u1`` 之外。
|
||||
"""
|
||||
prefix = _escape_like(f"{user_id}:") + "%"
|
||||
return await _run_in_sqlite_pool(
|
||||
self._get_recent_prefix_sync, prefix, limit,
|
||||
)
|
||||
return [dict(row) for row in cursor.fetchall()]
|
||||
|
||||
def _get_recent_prefix_sync(
|
||||
self, pattern: str, limit: int,
|
||||
) -> List[Dict[str, Any]]:
|
||||
with self._conn_lock:
|
||||
if self._conn is None:
|
||||
return []
|
||||
cursor = self._conn.execute(
|
||||
"SELECT * FROM memory_history WHERE isolation_key LIKE ? ESCAPE '\\' "
|
||||
"ORDER BY created_at DESC LIMIT ?",
|
||||
(pattern, limit),
|
||||
)
|
||||
return [dict(row) for row in cursor.fetchall()]
|
||||
|
||||
async def count(self, isolation_key: str = "") -> int:
|
||||
"""
|
||||
@@ -266,16 +307,17 @@ class HistoryStore:
|
||||
return await _run_in_sqlite_pool(self._count_sync, isolation_key)
|
||||
|
||||
def _count_sync(self, isolation_key: str) -> int:
|
||||
if self._conn is None:
|
||||
return 0
|
||||
if isolation_key:
|
||||
cursor = self._conn.execute(
|
||||
"SELECT COUNT(*) FROM memory_history WHERE isolation_key = ?",
|
||||
(isolation_key,),
|
||||
)
|
||||
else:
|
||||
cursor = self._conn.execute("SELECT COUNT(*) FROM memory_history")
|
||||
return cursor.fetchone()[0]
|
||||
with self._conn_lock:
|
||||
if self._conn is None:
|
||||
return 0
|
||||
if isolation_key:
|
||||
cursor = self._conn.execute(
|
||||
"SELECT COUNT(*) FROM memory_history WHERE isolation_key = ?",
|
||||
(isolation_key,),
|
||||
)
|
||||
else:
|
||||
cursor = self._conn.execute("SELECT COUNT(*) FROM memory_history")
|
||||
return cursor.fetchone()[0]
|
||||
|
||||
async def close(self) -> None:
|
||||
"""关闭数据库连接"""
|
||||
@@ -284,9 +326,10 @@ class HistoryStore:
|
||||
logger.info("[HistoryStore] closed")
|
||||
|
||||
def _close_sync(self) -> None:
|
||||
if self._conn:
|
||||
self._conn.close()
|
||||
self._conn = None
|
||||
with self._conn_lock:
|
||||
if self._conn:
|
||||
self._conn.close()
|
||||
self._conn = None
|
||||
|
||||
# ================================================================
|
||||
# 内部
|
||||
@@ -339,33 +382,34 @@ class HistoryStore:
|
||||
role: str = "",
|
||||
extra: Optional[Dict[str, Any]] = None,
|
||||
) -> int:
|
||||
if self._conn is None:
|
||||
return -1
|
||||
now = _now_iso()
|
||||
extra_json = json.dumps(extra or {}, ensure_ascii=False)
|
||||
cursor = self._conn.execute(
|
||||
"""INSERT INTO memory_history
|
||||
(memory_id, isolation_key, event,
|
||||
old_memory, new_memory, old_status, new_status,
|
||||
change_reason, layer, actor_id, role, extra,
|
||||
created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
memory_id,
|
||||
isolation_key,
|
||||
event,
|
||||
old_memory,
|
||||
new_memory,
|
||||
old_status,
|
||||
new_status,
|
||||
change_reason,
|
||||
layer,
|
||||
actor_id,
|
||||
role,
|
||||
extra_json,
|
||||
now,
|
||||
now,
|
||||
),
|
||||
)
|
||||
self._conn.commit()
|
||||
return cursor.lastrowid
|
||||
with self._conn_lock:
|
||||
if self._conn is None:
|
||||
return -1
|
||||
now = _now_iso()
|
||||
extra_json = json.dumps(extra or {}, ensure_ascii=False)
|
||||
cursor = self._conn.execute(
|
||||
"""INSERT INTO memory_history
|
||||
(memory_id, isolation_key, event,
|
||||
old_memory, new_memory, old_status, new_status,
|
||||
change_reason, layer, actor_id, role, extra,
|
||||
created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
memory_id,
|
||||
isolation_key,
|
||||
event,
|
||||
old_memory,
|
||||
new_memory,
|
||||
old_status,
|
||||
new_status,
|
||||
change_reason,
|
||||
layer,
|
||||
actor_id,
|
||||
role,
|
||||
extra_json,
|
||||
now,
|
||||
now,
|
||||
),
|
||||
)
|
||||
self._conn.commit()
|
||||
return cursor.lastrowid
|
||||
@@ -462,14 +462,39 @@ class FaissVectorStore(VectorStoreBase):
|
||||
return None
|
||||
|
||||
async def get_embeddings(self, node_ids: List[str]) -> Dict[str, List[float]]:
|
||||
"""批量取向量(in-memory _data 直接读 embedding),供去重算 cosine。"""
|
||||
out: Dict[str, List[float]] = {}
|
||||
for nid in node_ids:
|
||||
point_id = self._node_id_to_point_id(nid)
|
||||
entry = self._data.get(point_id)
|
||||
if entry and entry.get("embedding"):
|
||||
out[nid] = list(entry["embedding"])
|
||||
return out
|
||||
"""批量取向量,供去重 / episode 摘要计算 cosine。
|
||||
|
||||
``_data`` 只保存 payload 与 FAISS 内部下标;普通 upsert 不额外存一份
|
||||
embedding,向量本体在 FAISS index 里。这里按当前有效下标 reconstruct,
|
||||
并用 ``_idx_to_id`` 校验,避免读到已被替换的旧下标。
|
||||
"""
|
||||
|
||||
def _get() -> Dict[str, List[float]]:
|
||||
out: Dict[str, List[float]] = {}
|
||||
with self._lock:
|
||||
for nid in node_ids:
|
||||
point_id = self._node_id_to_point_id(nid)
|
||||
entry = self._data.get(point_id)
|
||||
if entry is None:
|
||||
continue
|
||||
embedding = entry.get("embedding")
|
||||
if embedding:
|
||||
out[nid] = list(embedding)
|
||||
continue
|
||||
idx = entry.get("idx")
|
||||
if (
|
||||
idx is None
|
||||
or self._index is None
|
||||
or self._idx_to_id.get(int(idx)) != point_id
|
||||
):
|
||||
continue
|
||||
try:
|
||||
out[nid] = self._index.reconstruct(int(idx)).tolist()
|
||||
except Exception:
|
||||
continue
|
||||
return out
|
||||
|
||||
return await _run_in_vdb_pool(_get)
|
||||
|
||||
# ================================================================
|
||||
# 删除
|
||||
|
||||
@@ -10,8 +10,8 @@ HY Memory - Intention (L7) 召回 + 惰性过期转换 + 复本判定
|
||||
节点的 layer 改写为 l2_fact(持久化 update_payload),并从意图结果里剔除——
|
||||
过期意图变成普通历史事实,下次走 normal 召回;
|
||||
- **判重(重点)**:对幸存意图按下述规则分组,同组只留一条本次返回:
|
||||
同 user_id + 同一天 memory_at + valid_until 相同(都为 None 或都相等)
|
||||
+ cosine 相似度 ≥ MEMORY_DEDUP_THRESHOLD (默认 0.95)
|
||||
同 user_id + 同 agent_id + 同一天 memory_at + valid_until 相同
|
||||
(都为 None 或都相等)+ cosine 相似度 ≥ MEMORY_DEDUP_THRESHOLD (默认 0.95)
|
||||
→ 视为同一个意图的复本(extractor 复读机 / 多 session 重复表达)。
|
||||
保留组里 `gmt_created` 最早的(老的先来先得,稳定 id);
|
||||
- **异步删库**:其余复本 fire-and-forget 从 VDB 删除,不影响本次调用 RT。
|
||||
@@ -112,11 +112,19 @@ def _partition_duplicates(
|
||||
if len(hits) < 2:
|
||||
return list(hits), []
|
||||
|
||||
# 1. 按 (memory_at day, valid_until day) 分桶
|
||||
buckets: Dict[Tuple[Any, Any], List[int]] = {}
|
||||
# 1. 按 (user_id, agent_id, memory_at day, valid_until day) 分桶。
|
||||
# user_id / agent_id 必须进键:多用户检索时不同用户的同一天意图可能
|
||||
# 文本、日期都相同,但它们是两个独立的租户数据;只按日期分桶会把其中
|
||||
# 一个判成复本并硬删除。
|
||||
buckets: Dict[Tuple[Any, Any, Any, Any], List[int]] = {}
|
||||
for idx, h in enumerate(hits):
|
||||
node = h.get("node")
|
||||
key = (_memory_at_day(node), _valid_until_key(node))
|
||||
key = (
|
||||
getattr(node, "user_id", None) if node else None,
|
||||
getattr(node, "agent_id", None) if node else None,
|
||||
_memory_at_day(node),
|
||||
_valid_until_key(node),
|
||||
)
|
||||
buckets.setdefault(key, []).append(idx)
|
||||
|
||||
def _gmt(h: Dict[str, Any]) -> float:
|
||||
@@ -229,6 +237,8 @@ async def recall_intentions(
|
||||
*,
|
||||
user_ids: Optional[List[str]] = None,
|
||||
agent_ids: Optional[List[str]] = None,
|
||||
isolation_key: str = "",
|
||||
isolation_keys: Optional[List[str]] = None,
|
||||
limit: int = 10,
|
||||
now: Optional[datetime] = None,
|
||||
stats_out: Optional[Dict[str, Any]] = None,
|
||||
@@ -243,6 +253,9 @@ async def recall_intentions(
|
||||
query_embedding: 查询向量
|
||||
user_ids: 用户隔离(单/多用户)
|
||||
agent_ids: 可选 agent 过滤
|
||||
isolation_key / isolation_keys: 精确的 (user, agent, session) 隔离键。
|
||||
与其它召回通道一致:调用方在按 session 过滤时只生成这两项,
|
||||
user_ids / agent_ids 为 None;不把它们透传下去就会退化成全库检索。
|
||||
limit: 召回上限(返回给上游的条数)
|
||||
now: 当前时间(默认 datetime.now(),测试可注入)
|
||||
stats_out: 可选 dict,本函数把分段耗时与数量**原地回填**进去,供调用方落
|
||||
@@ -283,6 +296,8 @@ async def recall_intentions(
|
||||
query_embedding=query_embedding,
|
||||
user_ids=user_ids or None,
|
||||
agent_ids=agent_ids,
|
||||
isolation_key=isolation_key,
|
||||
isolation_keys=isolation_keys,
|
||||
layers=[MemoryLayer.L7_INTENTION],
|
||||
limit=fetch_limit,
|
||||
score_threshold=0.0, # intention 不设语义门槛
|
||||
|
||||
@@ -1840,6 +1840,8 @@ class BaseReadPipeline(ReadPipeline):
|
||||
query_embedding,
|
||||
user_ids=isolation_params.get("user_ids"),
|
||||
agent_ids=isolation_params.get("agent_ids"),
|
||||
isolation_key=isolation_params.get("isolation_key") or "",
|
||||
isolation_keys=isolation_params.get("isolation_keys"),
|
||||
limit=request.intention_limit,
|
||||
stats_out=_intent_stats,
|
||||
include_sources=request.include_sources,
|
||||
|
||||
@@ -968,7 +968,7 @@ class MemoryWriter(WritePipeline):
|
||||
async def _run():
|
||||
try:
|
||||
nid = await summarize_closed_episode(
|
||||
vector_store=self.vector_store,
|
||||
vector_store=await self._get_vector_store(),
|
||||
embed_service=self.embed_service,
|
||||
llm_provider=self.llm_provider,
|
||||
episode_id=episode_id,
|
||||
|
||||
+19
-6
@@ -29,7 +29,7 @@ import logging
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import sys
|
||||
import threading
|
||||
from datetime import datetime
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from typing import Any, Dict, Optional
|
||||
@@ -495,6 +495,16 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
|
||||
_json_response(self, 200, result)
|
||||
|
||||
|
||||
def _request_shutdown(server: ThreadingHTTPServer) -> None:
|
||||
"""Ask ``serve_forever`` to stop without blocking the calling thread.
|
||||
|
||||
``BaseServer.shutdown()`` waits for the ``serve_forever`` loop to exit. A
|
||||
signal handler runs on the very thread executing that loop, so calling it
|
||||
there deadlocks forever. Hand the call to a helper thread instead.
|
||||
"""
|
||||
threading.Thread(target=server.shutdown, daemon=True).start()
|
||||
|
||||
|
||||
def run_server(port: int = 19527, host: str = "127.0.0.1"):
|
||||
"""Start the HTTP server (blocking)."""
|
||||
# Configure logging — basicConfig for root; hy_memory level set after client init
|
||||
@@ -523,13 +533,12 @@ def run_server(port: int = 19527, host: str = "127.0.0.1"):
|
||||
logger.info(f"[server] Listening on http://{host}:{port}")
|
||||
logger.info(f"[server] Health check: http://{host}:{port}/healthz")
|
||||
|
||||
# Graceful shutdown
|
||||
# Graceful shutdown. The signal handler only asks a helper thread to stop
|
||||
# the loop; cleanup runs after serve_forever() returns so it cannot race
|
||||
# the request threads or deadlock the serving thread.
|
||||
def _shutdown(signum, frame):
|
||||
logger.info("[server] Shutting down...")
|
||||
server.shutdown()
|
||||
if _client is not None:
|
||||
_client.close()
|
||||
sys.exit(0)
|
||||
_request_shutdown(server)
|
||||
|
||||
signal.signal(signal.SIGINT, _shutdown)
|
||||
signal.signal(signal.SIGTERM, _shutdown)
|
||||
@@ -538,6 +547,10 @@ def run_server(port: int = 19527, host: str = "127.0.0.1"):
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
_shutdown(None, None)
|
||||
finally:
|
||||
server.server_close()
|
||||
if _client is not None:
|
||||
_client.close()
|
||||
|
||||
|
||||
def main():
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Context propagated to host-provided HTTP transports for cost accounting."""
|
||||
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar, Token
|
||||
|
||||
_model_call_stage: ContextVar[str] = ContextVar(
|
||||
"hy_memory_model_call_stage",
|
||||
default="other",
|
||||
)
|
||||
|
||||
|
||||
def current_model_call_stage() -> str:
|
||||
return _model_call_stage.get()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def model_call_stage(stage: str | None) -> Iterator[None]:
|
||||
normalized = (stage or "other").strip().lower() or "other"
|
||||
token: Token[str] = _model_call_stage.set(normalized)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_model_call_stage.reset(token)
|
||||
@@ -18,12 +18,32 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, List, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: A namespace is a single path component, not a path. `\w` keeps Unicode
|
||||
#: names working while excluding separators, drive letters and NUL.
|
||||
_SAFE_SUBDIR = re.compile(r"[\w.-]{1,64}", re.UNICODE)
|
||||
|
||||
|
||||
def _safe_subdir(value: str) -> Optional[str]:
|
||||
"""Return ``value`` if it is safe as one directory name, else ``None``.
|
||||
|
||||
``user_id`` is caller-controlled. Without this check a value such as
|
||||
``"/tmp/evil__u1"`` makes ``Path(base) / "/tmp/evil"`` discard ``base``
|
||||
entirely; ``"../.."`` escapes it the same way.
|
||||
"""
|
||||
candidate = (value or "").strip()
|
||||
if not candidate or candidate in {".", ".."}:
|
||||
return None
|
||||
if not _SAFE_SUBDIR.fullmatch(candidate):
|
||||
return None
|
||||
return candidate
|
||||
|
||||
# 兼容旧 import
|
||||
TraceFileWriter = None # set after class definition
|
||||
|
||||
@@ -40,7 +60,7 @@ class PipelineLogWriter:
|
||||
|
||||
def __init__(self, log_dir: str, default_subdir: str = "default"):
|
||||
self._log_dir = log_dir
|
||||
self._default_subdir = default_subdir or "default"
|
||||
self._default_subdir = _safe_subdir(default_subdir) or "default"
|
||||
try:
|
||||
(Path(self._log_dir) / self._default_subdir).mkdir(parents=True, exist_ok=True)
|
||||
except Exception:
|
||||
@@ -49,7 +69,7 @@ class PipelineLogWriter:
|
||||
def _resolve_subdir(self, user_id: str) -> str:
|
||||
"""从 user_id 的 "{prefix}__{user_id}" 前缀取 prefix 作为子目录;无前缀则回落。"""
|
||||
if user_id and "__" in user_id:
|
||||
prefix = user_id.split("__", 1)[0].strip()
|
||||
prefix = _safe_subdir(user_id.split("__", 1)[0])
|
||||
if prefix:
|
||||
return prefix
|
||||
return self._default_subdir
|
||||
@@ -80,7 +100,11 @@ class PipelineLogWriter:
|
||||
split_namespace_user,
|
||||
)
|
||||
|
||||
tokens = (prompt_tokens or 0) + (completion_tokens or 0) + (total_tokens or 0)
|
||||
# Providers define total_tokens as prompt+completion. Adding all
|
||||
# three fields doubled every cost estimate in this JSONL.
|
||||
tokens = (total_tokens or 0) or (
|
||||
(prompt_tokens or 0) + (completion_tokens or 0)
|
||||
)
|
||||
namespace, uid = split_namespace_user(user_id or "")
|
||||
if not is_llm_content_logging_enabled():
|
||||
prompt = redact_content(prompt)
|
||||
@@ -111,7 +135,14 @@ class PipelineLogWriter:
|
||||
line = json.dumps(record, ensure_ascii=False, default=str)
|
||||
date_str = datetime.now().strftime("%Y-%m-%d")
|
||||
subdir = self._resolve_subdir(user_id)
|
||||
log_dir = Path(self._log_dir) / subdir
|
||||
base_dir = Path(self._log_dir).resolve()
|
||||
log_dir = (base_dir / subdir).resolve()
|
||||
# Defence in depth: even if a future change lets a suspicious
|
||||
# component through, never write outside the configured root.
|
||||
try:
|
||||
log_dir.relative_to(base_dir)
|
||||
except ValueError:
|
||||
log_dir = base_dir / self._default_subdir
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_file = log_dir / f"{date_str}.log"
|
||||
with open(log_file, "a", encoding="utf-8") as f:
|
||||
|
||||
+10
-7
@@ -15,12 +15,12 @@ write、search,并把搜索结果格式化为可交给模型的 memory text ch
|
||||
|
||||
```text
|
||||
distribution: hy-memory
|
||||
version: 2.0.2
|
||||
version: 2.0.3
|
||||
mode: 按运行时配置选择
|
||||
reader: hybrid
|
||||
```
|
||||
|
||||
仓库 pin、lockfile 和运行时版本守卫均固定到公开的 `hy-memory==2.0.2`。
|
||||
仓库 pin、lockfile 和运行时版本守卫均固定到公开的 `hy-memory==2.0.3`。
|
||||
MemHub 在本地 Runtime 内嵌入 SDK,但不绕过自身的来源授权、模型授权、Evidence、
|
||||
Governance 和撤权边界。
|
||||
|
||||
@@ -305,9 +305,9 @@ Runtime 应长期复用 client,不为每次请求重新初始化数据库和
|
||||
|
||||
```text
|
||||
Evidence
|
||||
-> ModelProcessingPermissionLease
|
||||
-> HyPermissionTransport
|
||||
-> HyMemoryBridge.ingest / ingest_extracted / search_ids
|
||||
-> HyMemoryBridge
|
||||
-> embedded Hy Memory SDK
|
||||
-> native Vector / Coding / Graph stores
|
||||
-> write result verification
|
||||
-> Governance
|
||||
-> Recall Broker / UI
|
||||
@@ -316,8 +316,11 @@ Evidence
|
||||
不要在新业务模块里直接创建 `HyMemoryClient`,否则会绕过:
|
||||
|
||||
- 来源读取授权;
|
||||
- 模型内容处理授权;
|
||||
- 发送前与响应后的撤权复核;
|
||||
- 共享 Runtime、模型配置、API Key 安全存储与请求预算;
|
||||
- Evidence 引用与作者归属;
|
||||
- 写后回读验证;
|
||||
- Governance 状态、版本、更正和删除级联。
|
||||
|
||||
Hy Memory SDK 是 MemHub 的内置可信组件,可自由访问和管理本机的全部
|
||||
Hy Memory 数据。来源授权仍决定哪些宿主数据可以进入本机;宿主 Recall Grant
|
||||
仍决定哪些外部 AI 应用可以读取治理后的记忆。这两类边界不限制内置 SDK 的本地 CRUD。
|
||||
+22
-8
@@ -1,18 +1,30 @@
|
||||
# MemHub / One Memory
|
||||
# MemHub
|
||||
|
||||
MemHub 是一个本地优先的共享记忆产品:把用户授权的多个 AI 产品中的历史对话和新增内容
|
||||
汇总到同一份记忆,并让这些 AI 在独立授权后复用它。Hy Memory 是 MemHub 使用的 Python
|
||||
SDK 和记忆引擎。One Memory / `one-memory` 是历史模块、数据目录、服务和兼容命令名称。
|
||||
MemHub是一个本地优先的共享记忆产品。它把用户明确授权的AI对话和编码记录汇总到一份本地记忆中,再由用户分别授权给支持的AI工具调用。你可以继续使用熟悉的客户端,同时保留跨工具延续的上下文。Hy Memory是MemHub使用的Python SDK和记忆引擎。
|
||||
|
||||
# 💡❤️ MemHub 刚刚开源,欢迎一起优化共建! ❤️💡
|
||||
|
||||
**# 💭 MemHub处于preview阶段,当前公开版本可以下载和体验,部分平台制品、安装路径与宿主体验仍在持续优化验证。安装前请先查看Release里的实际制品和版本说明。**
|
||||
|
||||
> 🙋 如果你遇到问题/有建议:欢迎提交Issue,并附上MemHub版本、操作系统与芯片架构、复现步骤、预期结果、实际结果和经过脱敏的日志。请不要上传APIKey、真实对话、个人路径或其他敏感信息。
|
||||
|
||||
> 我们会公开跟进可复现的问题,持续发布修复,并根据真实反馈完善安装、同步和召回体验。
|
||||
|
||||
**# 感谢每一条反馈,期待你和我们一起打磨MemHub的体验 :)**
|
||||
|
||||
> 💗 MemHub has just been open-sourced and is under active development. The current public release is available for download andevaluation. Some platform packages, installation paths, and host integrations are still being completed and verified. Check the assets
|
||||
and release notes before installing.
|
||||
If you run into a problem, please open an Issue with the version, platform,reproduction steps, and sanitized logs, We wilreviewreports, publish fixes, and keep the support matrix up to date.
|
||||
|
||||
## 发行与平台合同
|
||||
|
||||
公共 Python SDK 为 `hy-memory==2.0.2`。MemHub 源码版本为 `1.0.1`
|
||||
(build 5),GitHub 仓库为
|
||||
公共 Python SDK 为 `hy-memory==2.0.4`。MemHub 源码版本为 `1.0.4`
|
||||
(build 13),GitHub 仓库为
|
||||
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
|
||||
`memhub-cli`。当前代码架构见
|
||||
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
|
||||
|
||||
`v1.0.1` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
|
||||
`v1.0.4` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
|
||||
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
|
||||
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
|
||||
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
|
||||
@@ -42,6 +54,8 @@ Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命
|
||||
|
||||
架构详情见 [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) 和
|
||||
[`docs/contracts/runtime-unification.md`](docs/contracts/runtime-unification.md)。
|
||||
产品、测试和一线支持排查导入与记忆沉淀时,使用
|
||||
[`docs/PRODUCT_LOG_TROUBLESHOOTING.md`](docs/PRODUCT_LOG_TROUBLESHOOTING.md)。
|
||||
|
||||
## 开发环境
|
||||
|
||||
@@ -99,7 +113,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
|
||||
|
||||
## Hy Memory
|
||||
|
||||
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.2`。SDK 配置、权限边界及 search
|
||||
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.4`。SDK 配置、权限边界及 search
|
||||
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
|
||||
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@
|
||||
"dataLicense": "CC0-1.0",
|
||||
"SPDXID": "SPDXRef-DOCUMENT",
|
||||
"name": "one-memory",
|
||||
"documentNamespace": "https://one-memory.local/spdx/2026-10-15T17:26:47Z",
|
||||
"documentNamespace": "https://one-memory.local/spdx/2026-10-17T13:52:48Z",
|
||||
"creationInfo": {
|
||||
"created": "2026-10-15T17:26:47Z",
|
||||
"created": "2026-10-17T13:52:48Z",
|
||||
"creators": [
|
||||
"Tool: one-memory-generate-sbom"
|
||||
]
|
||||
@@ -2335,7 +2335,7 @@
|
||||
{
|
||||
"SPDXID": "SPDXRef-Package-hy-memory",
|
||||
"name": "hy-memory",
|
||||
"versionInfo": "2.0.2",
|
||||
"versionInfo": "2.0.4",
|
||||
"downloadLocation": "NOASSERTION",
|
||||
"filesAnalyzed": false,
|
||||
"licenseConcluded": "Apache-2.0",
|
||||
|
||||
@@ -2,14 +2,14 @@
|
||||
|
||||
MemHub itself is licensed under Apache-2.0 (see `LICENSE`).
|
||||
It bundles the runtime dependencies listed below.
|
||||
Generated from the installed dependency tree on 2026-10-15T17:26:47Z
|
||||
Generated from the installed dependency tree on 2026-10-17T13:52:48Z
|
||||
by `python3 scripts/generate_sbom.py`. Do not edit by hand.
|
||||
|
||||
Per-package license texts are not reproduced here yet; the license
|
||||
column is filled in as part of release review, and the SBOM records
|
||||
`NOASSERTION` until then rather than guessing.
|
||||
|
||||
## hy-memory 2.0.2
|
||||
## hy-memory 2.0.4
|
||||
|
||||
License: Apache-2.0
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@one-memory/cli",
|
||||
"version": "1.0.1",
|
||||
"version": "1.0.4",
|
||||
"description": "MemHub bootstrap CLI for installing and managing the local-first shared memory runtime",
|
||||
"license": "Apache-2.0",
|
||||
"repository": {
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
* It is generated rather than hand-written so the two cannot drift: this reads
|
||||
* the real package.json and only strips what is provably not needed at runtime.
|
||||
*/
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { dirname, resolve } from "node:path";
|
||||
@@ -20,12 +19,14 @@ import { fileURLToPath } from "node:url";
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
const root = resolve(here, "..");
|
||||
const repositoryRoot = resolve(root, "../..");
|
||||
const require = createRequire(import.meta.url);
|
||||
const { resolveBuildVersion } = require("../../../scripts/release-version.cjs");
|
||||
const releaseVersion = resolveBuildVersion(process.env, repositoryRoot);
|
||||
const { assertSemver } = require("../../../scripts/release-version.cjs");
|
||||
const pkgPath = resolve(root, "package.json");
|
||||
const pkg = JSON.parse(readFileSync(pkgPath, "utf8"));
|
||||
const publishVersion = assertSemver(
|
||||
process.env.MEMHUB_NPM_VERSION?.trim() || pkg.version,
|
||||
"MEMHUB_NPM_VERSION",
|
||||
);
|
||||
const publishName = process.env.MEMHUB_NPM_PACKAGE?.trim();
|
||||
if (!publishName || !/^(?:@[a-z0-9._-]+\/)?[a-z0-9._-]+$/.test(publishName)) {
|
||||
throw new Error(
|
||||
@@ -53,7 +54,7 @@ mkdirSync(outDir, { recursive: true });
|
||||
|
||||
const published = { ...pkg };
|
||||
published.name = publishName;
|
||||
published.version = releaseVersion;
|
||||
published.version = publishVersion;
|
||||
for (const section of ["dependencies", "devDependencies", "peerDependencies"]) {
|
||||
if (!published[section]) continue;
|
||||
published[section] = Object.fromEntries(
|
||||
|
||||
@@ -32,7 +32,6 @@ const REQUIRED_COMMAND_SURFACE = [
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"models",
|
||||
@@ -77,7 +76,7 @@ const REQUIRED_MARKERS = [
|
||||
|
||||
/**
|
||||
* Sub-commands the UI-write parity work added. A package can declare the
|
||||
* parent command (`permission`, `models`) while still missing the new verb, so
|
||||
* parent command (`models`) while still missing the new verb, so
|
||||
* checking command names alone would pass a stale bundle. The marker is the
|
||||
* exact literal the catalog usage string contains.
|
||||
*/
|
||||
@@ -88,9 +87,6 @@ const REQUIRED_SUBCOMMANDS = [
|
||||
{ command: "import-file", marker: "memhub import-file discard" },
|
||||
{ command: "notes", marker: "memhub notes add" },
|
||||
{ command: "notes", marker: "memhub notes process" },
|
||||
{ command: "permission", marker: "memhub permission status" },
|
||||
{ command: "permission", marker: "memhub permission allow" },
|
||||
{ command: "permission", marker: "memhub permission deny" },
|
||||
{ command: "extraction", marker: "memhub extraction set" },
|
||||
{ command: "graph", marker: "memhub graph width" },
|
||||
{ command: "models", marker: "memhub models edit" },
|
||||
@@ -153,8 +149,6 @@ const STOPPED_DESTRUCTIVE_PROBES = [
|
||||
"--confirm",
|
||||
],
|
||||
},
|
||||
{ name: "permission_allow", args: ["permission", "allow"] },
|
||||
{ name: "permission_deny", args: ["permission", "deny"] },
|
||||
{
|
||||
name: "extraction_set_apply",
|
||||
args: ["extraction", "set", "--window", "6m", "--confirm"],
|
||||
|
||||
+49
-84
@@ -4,7 +4,11 @@ import { createReadStream } from "node:fs";
|
||||
import { access } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import type { HealthResponse, ImportJob } from "@one-memory/contracts";
|
||||
import {
|
||||
type HealthResponse,
|
||||
type ImportJob,
|
||||
isHostId,
|
||||
} from "@one-memory/contracts";
|
||||
import {
|
||||
type AcquiredRuntimeDistribution,
|
||||
acquireRuntimeDistribution,
|
||||
@@ -81,6 +85,7 @@ import {
|
||||
readMemoryWrite,
|
||||
readScanRoots,
|
||||
removeScanRoot,
|
||||
resetOnboarding,
|
||||
submitBackup,
|
||||
submitMemoryWrite,
|
||||
submitScan,
|
||||
@@ -190,13 +195,6 @@ import {
|
||||
renderMemoryDetailText,
|
||||
renderMemoryPageText,
|
||||
} from "./memory-report.js";
|
||||
import {
|
||||
type ModelContentPermissionDocument,
|
||||
modelContentPermissionExitCode,
|
||||
readModelContentPermission,
|
||||
renderModelContentPermissionText,
|
||||
setModelContentPermission,
|
||||
} from "./model-content-permission-flow.js";
|
||||
import {
|
||||
executeModelProbe,
|
||||
inspectModelProbePlan,
|
||||
@@ -217,12 +215,10 @@ import {
|
||||
executeModelProfileActivate,
|
||||
executeModelProfileEdit,
|
||||
type ModelWriteDocument,
|
||||
type ModelWritePlan,
|
||||
modelWriteExitCode,
|
||||
planModelProfileActivate,
|
||||
planModelProfileEdit,
|
||||
renderModelWriteText,
|
||||
validateModelWritePlan,
|
||||
} from "./model-write-flows.js";
|
||||
import { claimPlanAttempt } from "./operation-journal.js";
|
||||
import {
|
||||
@@ -339,7 +335,7 @@ import {
|
||||
renderVersion,
|
||||
renderVersionDocument,
|
||||
} from "./usage.js";
|
||||
import { CLI_VERSION, PRODUCT } from "./version.js";
|
||||
import { CLI_VERSION, PRODUCT, RUNTIME_RELEASE_VERSION } from "./version.js";
|
||||
import { runWatchLoop, type WatchEvent, type WatchRead } from "./watch-loop.js";
|
||||
|
||||
export interface RunControls {
|
||||
@@ -425,7 +421,8 @@ async function selectInstallSource(
|
||||
throw new Error("release_manifest_url_unavailable");
|
||||
}
|
||||
const manifestUrl =
|
||||
explicitManifestUrl ?? template.replaceAll("{version}", CLI_VERSION);
|
||||
explicitManifestUrl ??
|
||||
template.replaceAll("{version}", RUNTIME_RELEASE_VERSION);
|
||||
const allowedHosts = parseAllowedReleaseHosts(
|
||||
environment.MEMHUB_RELEASE_ALLOWED_HOSTS,
|
||||
);
|
||||
@@ -470,7 +467,7 @@ async function resolveRemoteInstallPreview(
|
||||
): Promise<ResolvedRuntimeRelease> {
|
||||
return (controls.resolveRuntimeRelease ?? resolveRuntimeRelease)({
|
||||
manifestUrl: selection.manifestUrl,
|
||||
expectedVersion: CLI_VERSION,
|
||||
expectedVersion: RUNTIME_RELEASE_VERSION,
|
||||
channel: selection.channel,
|
||||
platform: process.platform,
|
||||
arch: process.arch,
|
||||
@@ -484,7 +481,7 @@ async function acquireRemoteInstallSource(
|
||||
): Promise<AcquiredRuntimeDistribution> {
|
||||
return (controls.acquireRuntimeDistribution ?? acquireRuntimeDistribution)({
|
||||
manifestUrl: selection.manifestUrl,
|
||||
expectedVersion: CLI_VERSION,
|
||||
expectedVersion: RUNTIME_RELEASE_VERSION,
|
||||
channel: selection.channel,
|
||||
platform: process.platform,
|
||||
arch: process.arch,
|
||||
@@ -697,6 +694,36 @@ async function runCommand(
|
||||
// `path list` and `adapter list` are read-only: they have no plan to confirm,
|
||||
// so they are answered here rather than falling into the two-phase branch
|
||||
// (which would reject them for having nothing to apply).
|
||||
// Back to the cold-start screens on purpose. Deleting the app cannot do this
|
||||
// (the completion flag lives with the memories, by design), and deleting the
|
||||
// data to get there would cost the user everything they have stored.
|
||||
if (named === "onboarding" && arguments_.includes("reset")) {
|
||||
if (
|
||||
arguments_.filter((item) => item === "--json").length > 1 ||
|
||||
arguments_.filter((item) => item !== "--json").join(" ") !==
|
||||
"onboarding reset"
|
||||
) {
|
||||
return usage();
|
||||
}
|
||||
const attachment = await attachToRuntime(
|
||||
resolveRuntimeCliConfig(environment, process.argv[1]).dataDir,
|
||||
);
|
||||
if (attachment === null) {
|
||||
return runtimeUnavailable(descriptor, asJson);
|
||||
}
|
||||
const state = await resetOnboarding(attachment);
|
||||
if (state === null) {
|
||||
return reject(descriptor, asJson, "引导状态端点不可用。");
|
||||
}
|
||||
if (asJson) {
|
||||
process.stdout.write(`${JSON.stringify(state)}\n`);
|
||||
return EXIT_OK;
|
||||
}
|
||||
process.stdout.write(
|
||||
`引导流程已回到首次运行(status=${state.status}),记忆数据未改动。\n`,
|
||||
);
|
||||
return EXIT_OK;
|
||||
}
|
||||
if (named === "path" && arguments_.includes("list")) {
|
||||
if (
|
||||
arguments_.filter((item) => item === "--json").length > 1 ||
|
||||
@@ -2462,45 +2489,6 @@ async function runCommand(
|
||||
);
|
||||
return uiWriteExitCode(document);
|
||||
}
|
||||
if (named === "permission") {
|
||||
const parsed = parsePermissionCommand(arguments_.slice(1));
|
||||
if (!parsed) return usage();
|
||||
const config = resolveRuntimeCliConfig(environment, process.argv[1]);
|
||||
const runtime = await inspectRuntimeStatus({
|
||||
dataDir: config.dataDir,
|
||||
probeIdentity,
|
||||
requireServiceReceipt: true,
|
||||
});
|
||||
const attached =
|
||||
runtime.state === "running"
|
||||
? await attachClients(config.dataDir)
|
||||
: { reader: null, operations: null, failure: runtime.code };
|
||||
if (
|
||||
runtime.state !== "running" ||
|
||||
(parsed === "status" ? !attached.reader : !attached.operations)
|
||||
) {
|
||||
const document: ModelContentPermissionDocument = {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
state: "unknown",
|
||||
code:
|
||||
runtime.state !== "running"
|
||||
? runtime.code
|
||||
: (attached.failure ?? "runtime_attach_failed"),
|
||||
};
|
||||
emitModelContentPermissionDocument(document, asJson);
|
||||
return runtime.state === "stopped" ? 3 : 1;
|
||||
}
|
||||
const document =
|
||||
parsed === "status"
|
||||
? await readModelContentPermission(attached.reader!)
|
||||
: await setModelContentPermission(
|
||||
attached.operations!,
|
||||
parsed === "allow",
|
||||
);
|
||||
emitModelContentPermissionDocument(document, asJson);
|
||||
return modelContentPermissionExitCode(document);
|
||||
}
|
||||
if (named === "notes" || named === "extraction" || named === "graph") {
|
||||
const parsed = parseExtendedWriteCommand(
|
||||
named as "notes" | "extraction" | "graph",
|
||||
@@ -5124,18 +5112,6 @@ function parseHeadlessCommand(
|
||||
: null;
|
||||
}
|
||||
|
||||
function parsePermissionCommand(
|
||||
rest: readonly string[],
|
||||
): "status" | "allow" | "deny" | null {
|
||||
if (rest.filter((item) => item === "--json").length > 1) return null;
|
||||
const tokens = rest.filter((item) => item !== "--json");
|
||||
const action = tokens[0];
|
||||
return tokens.length === 1 &&
|
||||
(action === "status" || action === "allow" || action === "deny")
|
||||
? action
|
||||
: null;
|
||||
}
|
||||
|
||||
function unwrapPlan(value: unknown): unknown {
|
||||
if (
|
||||
value &&
|
||||
@@ -5184,10 +5160,10 @@ type HostCommand =
|
||||
args: unknown[];
|
||||
};
|
||||
|
||||
// `deepseek-harness:web` is a real id — that host is configured per profile —
|
||||
// and `host list` prints it. Rejecting it here would mean listing a host the
|
||||
// CLI then refuses to act on.
|
||||
const HOST_ID = /^[a-z0-9][a-z0-9-]{0,31}(:[a-z0-9][a-z0-9-]{0,31})?$/;
|
||||
// Host ids are read with the same grammar as the permission layer and the
|
||||
// grant store (`@one-memory/contracts`): `deepseek-harness:web` is a real id —
|
||||
// that host is configured per profile — and `host list` prints it, so rejecting
|
||||
// it here would mean listing a host the CLI then refuses to act on.
|
||||
const FINGERPRINT = /^[0-9a-f]{64}$/;
|
||||
|
||||
/** Host subcommands.
|
||||
@@ -5210,7 +5186,7 @@ function parseHostCommand(rest: readonly string[]): HostCommand | null {
|
||||
const token = tokens[index];
|
||||
if (token === "--host") {
|
||||
const value = tokens[index + 1];
|
||||
if (!value || !HOST_ID.test(value)) return null;
|
||||
if (!value || !isHostId(value)) return null;
|
||||
if (hostId !== undefined) return null;
|
||||
hostId = value;
|
||||
index += 1;
|
||||
@@ -5243,7 +5219,7 @@ function parseHostCommand(rest: readonly string[]): HostCommand | null {
|
||||
qualifier !== undefined &&
|
||||
(verb === "status" || verb === "install" || verb === "uninstall")
|
||||
) {
|
||||
if (!HOST_ID.test(qualifier)) return null;
|
||||
if (!isHostId(qualifier)) return null;
|
||||
hostId = qualifier;
|
||||
// Drop the consumed host positional so the verb/qualifier checks below see
|
||||
// the same shape as the flag form.
|
||||
@@ -5259,7 +5235,7 @@ function parseHostCommand(rest: readonly string[]): HostCommand | null {
|
||||
positional[2] !== undefined
|
||||
) {
|
||||
const trailing = positional[2];
|
||||
if (!HOST_ID.test(trailing)) return null;
|
||||
if (!isHostId(trailing)) return null;
|
||||
hostId = trailing;
|
||||
positional.splice(2, 1);
|
||||
verb = positional[0];
|
||||
@@ -5323,17 +5299,6 @@ function emitModelReportDocument(
|
||||
);
|
||||
}
|
||||
|
||||
function emitModelContentPermissionDocument(
|
||||
document: ModelContentPermissionDocument,
|
||||
asJson: boolean,
|
||||
): void {
|
||||
process.stdout.write(
|
||||
asJson
|
||||
? `${JSON.stringify(document)}\n`
|
||||
: renderModelContentPermissionText(document),
|
||||
);
|
||||
}
|
||||
|
||||
function emitSourceSyncDocument(
|
||||
document: SourceSyncFlowDocument,
|
||||
asJson: boolean,
|
||||
@@ -6560,7 +6525,7 @@ function parseInstallOptions(
|
||||
...(releaseManifestUrl ? { releaseManifestUrl } : {}),
|
||||
releaseChannel:
|
||||
releaseChannel ??
|
||||
(CLI_VERSION.includes("-") ? "prerelease" : "stable"),
|
||||
(RUNTIME_RELEASE_VERSION.includes("-") ? "prerelease" : "stable"),
|
||||
installRoot: resolvedInstallRoot,
|
||||
home: resolvedHome,
|
||||
uid: uid ?? process.getuid?.(),
|
||||
|
||||
@@ -493,14 +493,6 @@ export const COMMANDS: readonly CommandDescriptor[] = [
|
||||
],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
|
||||
},
|
||||
{
|
||||
name: "permission",
|
||||
summary: "读取或直接设置全局模型内容外发许可。",
|
||||
usage:
|
||||
"memhub permission status [--json]\n memhub permission allow [--json]\n memhub permission deny [--json]",
|
||||
options: [JSON_OPTION],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
|
||||
},
|
||||
{
|
||||
name: "extraction",
|
||||
summary: "设置提取窗口;默认只出计划,--confirm 后才写入。",
|
||||
@@ -659,6 +651,14 @@ export const COMMANDS: readonly CommandDescriptor[] = [
|
||||
],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
|
||||
},
|
||||
{
|
||||
name: "onboarding",
|
||||
summary:
|
||||
"onboarding reset 回到首次运行的引导流程。只清除引导完成状态:不删记忆数据,也不需要卸载重装。",
|
||||
usage: "memhub onboarding reset [--json]",
|
||||
options: [JSON_OPTION],
|
||||
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE],
|
||||
},
|
||||
{
|
||||
name: "help",
|
||||
summary: "显示总帮助或单个命令的用法。",
|
||||
|
||||
@@ -81,7 +81,6 @@ const COMMANDS = [
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"host",
|
||||
@@ -93,6 +92,7 @@ const COMMANDS = [
|
||||
"uninstall",
|
||||
"upgrade",
|
||||
"delete-data",
|
||||
"onboarding",
|
||||
"help",
|
||||
"version",
|
||||
// A6 product commands. Appended, not inserted: the catalog is a stable
|
||||
@@ -259,7 +259,14 @@ describe("CLI-22 shared output contract", () => {
|
||||
|
||||
describe("argument validation", () => {
|
||||
it("fails unknown commands without printing data or running anything", async () => {
|
||||
for (const argv of [["unknown"], ["unknown", "--help"], ["--typo"]]) {
|
||||
for (const argv of [
|
||||
["unknown"],
|
||||
["unknown", "--help"],
|
||||
["--typo"],
|
||||
["permission", "status"],
|
||||
["permission", "allow"],
|
||||
["permission", "deny"],
|
||||
]) {
|
||||
const streams = captureStreams();
|
||||
try {
|
||||
expect(await run(argv, {})).toBe(2);
|
||||
|
||||
@@ -384,6 +384,101 @@ describe("unified context search", () => {
|
||||
expect(streams.stdout()).not.toContain("must-not-print");
|
||||
});
|
||||
|
||||
it("counts characters as code points so an emoji hit stays grounded", async () => {
|
||||
// Core reports used_characters with Python's len(), which counts code
|
||||
// points: this text is 10 characters there and 12 UTF-16 units here.
|
||||
const text = "消费保 📍 扩展 📡";
|
||||
expect(text.length).toBe(12);
|
||||
expect(Array.from(text).length).toBe(10);
|
||||
transport.post.mockResolvedValue({
|
||||
status: 200,
|
||||
data: {
|
||||
...RESPONSE,
|
||||
paths: {
|
||||
...RESPONSE.paths,
|
||||
memory: {
|
||||
...RESPONSE.paths.memory,
|
||||
status: "empty",
|
||||
returned_count: 0,
|
||||
},
|
||||
},
|
||||
items: [
|
||||
{
|
||||
...(RESPONSE.items[0] as Record<string, unknown>),
|
||||
messages: [
|
||||
{
|
||||
...(
|
||||
RESPONSE.items[0] as {
|
||||
messages: Array<Record<string, unknown>>;
|
||||
}
|
||||
).messages[0],
|
||||
text,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
used_characters: 10,
|
||||
},
|
||||
});
|
||||
const streams = captureStreams();
|
||||
try {
|
||||
expect(await run(["recall", "test", "--json"])).toBe(0);
|
||||
} finally {
|
||||
streams.restore();
|
||||
}
|
||||
expect(JSON.parse(streams.stdout())).toMatchObject({
|
||||
state: "grounded",
|
||||
used_characters: 10,
|
||||
items: [{ kind: "raw", messages: [{ text }] }],
|
||||
});
|
||||
});
|
||||
|
||||
it("still rejects a response whose character total does not match", async () => {
|
||||
// The same text counted as UTF-16 units is exactly what Core does not mean,
|
||||
// so the cross-check must keep failing instead of accepting both totals.
|
||||
const text = "消费保 📍 扩展 📡";
|
||||
transport.post.mockResolvedValue({
|
||||
status: 200,
|
||||
data: {
|
||||
...RESPONSE,
|
||||
paths: {
|
||||
...RESPONSE.paths,
|
||||
memory: {
|
||||
...RESPONSE.paths.memory,
|
||||
status: "empty",
|
||||
returned_count: 0,
|
||||
},
|
||||
},
|
||||
items: [
|
||||
{
|
||||
...(RESPONSE.items[0] as Record<string, unknown>),
|
||||
messages: [
|
||||
{
|
||||
...(
|
||||
RESPONSE.items[0] as {
|
||||
messages: Array<Record<string, unknown>>;
|
||||
}
|
||||
).messages[0],
|
||||
text,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
used_characters: 12,
|
||||
},
|
||||
});
|
||||
const streams = captureStreams();
|
||||
try {
|
||||
expect(await run(["recall", "test", "--json"])).toBe(1);
|
||||
} finally {
|
||||
streams.restore();
|
||||
}
|
||||
expect(JSON.parse(streams.stdout())).toMatchObject({
|
||||
state: "invalid",
|
||||
code: "response_invalid",
|
||||
});
|
||||
});
|
||||
|
||||
it("reports endpoint denial without exposing the response body", async () => {
|
||||
transport.post.mockResolvedValue({
|
||||
status: 403,
|
||||
|
||||
@@ -181,9 +181,9 @@ function validateContextResponse(
|
||||
items.push(item);
|
||||
characters +=
|
||||
item.kind === "memory"
|
||||
? String(item.content).length
|
||||
? characterCount(item.content)
|
||||
: (item.messages as Array<Record<string, unknown>>).reduce(
|
||||
(sum, message) => sum + String(message.text).length,
|
||||
(sum, message) => sum + characterCount(message.text),
|
||||
0,
|
||||
);
|
||||
}
|
||||
@@ -330,6 +330,17 @@ function record(value: unknown): Record<string, unknown> | null {
|
||||
: null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count characters the way Core does. Python's len() counts Unicode code
|
||||
* points; String.prototype.length counts UTF-16 code units, so an emoji outside
|
||||
* the BMP is one character to Core and two here. Comparing the two totals made
|
||||
* every response containing an emoji fail the used_characters cross-check and
|
||||
* surface as response_invalid.
|
||||
*/
|
||||
function characterCount(value: unknown): number {
|
||||
return Array.from(String(value)).length;
|
||||
}
|
||||
|
||||
function boundedText(
|
||||
value: unknown,
|
||||
maximum: number,
|
||||
|
||||
@@ -8,7 +8,6 @@ import axios from "axios";
|
||||
*/
|
||||
const READ_ONLY_PATHS = [
|
||||
"/api/v1/call-events",
|
||||
"/api/v1/model-content-permission",
|
||||
"/api/v1/home-summary",
|
||||
"/api/v1/import-jobs",
|
||||
"/api/v1/live-sync",
|
||||
@@ -398,6 +397,41 @@ export async function readMemoryWrite(
|
||||
}
|
||||
}
|
||||
|
||||
export interface OnboardingState {
|
||||
status: string;
|
||||
selected_source_ids: string[];
|
||||
started_at: string | null;
|
||||
completed_at: string | null;
|
||||
}
|
||||
|
||||
/** Send onboarding back to its first-run state.
|
||||
*
|
||||
* Returns null when the endpoint is not there, so an older runtime degrades to
|
||||
* "not available" instead of failing the command. Only the completion flag
|
||||
* moves: memories, sources and settings live in other tables, which is what
|
||||
* makes this the honest way to see the cold-start screens again.
|
||||
*/
|
||||
export async function resetOnboarding(
|
||||
attachment: RuntimeAttachmentLike,
|
||||
): Promise<OnboardingState | null> {
|
||||
try {
|
||||
const response = await axios.post<OnboardingState>(
|
||||
`${baseUrlOf(attachment)}/api/v1/onboarding/reset`,
|
||||
undefined,
|
||||
{
|
||||
headers: { Authorization: `Bearer ${attachment.apiToken}` },
|
||||
timeout: 15_000,
|
||||
proxy: false,
|
||||
maxRedirects: 0,
|
||||
maxContentLength: MAX_RESPONSE_BYTES,
|
||||
},
|
||||
);
|
||||
return response.data;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function readScanRoots(
|
||||
attachment: RuntimeAttachmentLike,
|
||||
): Promise<{
|
||||
|
||||
@@ -30,7 +30,6 @@ const IMPORT_FILE =
|
||||
const MEMORY_NOTES_PATH = "/api/v1/memory-notes";
|
||||
const MEMORY_NOTE_PROCESS =
|
||||
/^\/api\/v1\/memory-notes\/[A-Za-z0-9][\w.:-]{0,127}\/process$/;
|
||||
const MODEL_CONTENT_PERMISSION_PATH = "/api/v1/model-content-permission";
|
||||
const EXTRACTION_SETTINGS_PATH = "/api/v1/extraction-settings";
|
||||
const GRAPH_WIDTH_PATH = "/api/v1/graph-width";
|
||||
const MANUAL_EXTRACT_PATH = "/api/v1/memory-processing/manual-extract";
|
||||
@@ -134,7 +133,6 @@ export function createCoreOperations(attachment: {
|
||||
!IMPORT_FILE_COMMIT.test(path) &&
|
||||
path !== MEMORY_NOTES_PATH &&
|
||||
!MEMORY_NOTE_PROCESS.test(path) &&
|
||||
path !== MODEL_CONTENT_PERMISSION_PATH &&
|
||||
path !== GRAPH_WIDTH_PATH &&
|
||||
path !== MANUAL_EXTRACT_PATH &&
|
||||
path !== ONBOARDING_COMPLETE_PATH &&
|
||||
|
||||
@@ -259,7 +259,7 @@ describe("A2 dual-mode init", () => {
|
||||
expect(streams.value()).toContain("处理:已开启");
|
||||
});
|
||||
|
||||
it("continues the TTY wizard through model probe and content permission", async () => {
|
||||
it("continues the TTY wizard through model setup and probe", async () => {
|
||||
const profileId = "22222222-2222-4222-8222-222222222222";
|
||||
const profile = {
|
||||
profile_id: profileId,
|
||||
@@ -361,12 +361,6 @@ describe("A2 dual-mode init", () => {
|
||||
};
|
||||
}
|
||||
if (url.endsWith("/activate")) return { status: 200, data: profile };
|
||||
if (url === "/api/v1/model-content-permission") {
|
||||
return {
|
||||
status: 200,
|
||||
data: { allowed: body.allowed },
|
||||
};
|
||||
}
|
||||
if (url === "/api/v1/onboarding/complete") {
|
||||
return {
|
||||
status: 200,
|
||||
@@ -383,7 +377,7 @@ describe("A2 dual-mode init", () => {
|
||||
},
|
||||
);
|
||||
const prompt = extendedPrompter(
|
||||
["", "", "", "", "", "", "", "", "n"],
|
||||
["", "", "", "", "", "", "", "n"],
|
||||
"sk-never-print-this",
|
||||
);
|
||||
const streams = captureStdout();
|
||||
@@ -402,7 +396,6 @@ describe("A2 dual-mode init", () => {
|
||||
streams.restore();
|
||||
}
|
||||
expect(streams.value()).toContain("已接入来源:codex");
|
||||
expect(prompt.written.join("\n")).toContain("模型内容外发:已允许");
|
||||
expect(streams.value()).toContain("首次设置完成。");
|
||||
expect(`${streams.value()}\n${prompt.written.join("\n")}`).not.toContain(
|
||||
"sk-never-print-this",
|
||||
@@ -412,10 +405,10 @@ describe("A2 dual-mode init", () => {
|
||||
expect.objectContaining({ api_key: "sk-never-print-this" }),
|
||||
undefined,
|
||||
);
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
expect(transport.post).not.toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
{ allowed: true },
|
||||
undefined,
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v1/onboarding/complete",
|
||||
|
||||
@@ -132,12 +132,6 @@ function clients(options: { failedProbes?: number } = {}) {
|
||||
if (path.endsWith("/activate")) {
|
||||
return { state: "committed" as const, data: profile(true) };
|
||||
}
|
||||
if (path === "/api/v1/model-content-permission") {
|
||||
return {
|
||||
state: "committed" as const,
|
||||
data: { allowed: body.allowed },
|
||||
};
|
||||
}
|
||||
return {
|
||||
state: "rejected" as const,
|
||||
code: "unexpected_path",
|
||||
@@ -160,8 +154,8 @@ function clients(options: { failedProbes?: number } = {}) {
|
||||
}
|
||||
|
||||
describe("interactive model setup", () => {
|
||||
it("uses App defaults, keeps the key out of output, and allows model content", async () => {
|
||||
const prompt = prompter(["", "", "", "", "", ""]);
|
||||
it("uses App defaults and keeps the key out of output", async () => {
|
||||
const prompt = prompter(["", "", ""]);
|
||||
const api = clients();
|
||||
|
||||
const result = await runInteractiveModelSetup(prompt.value, api, {
|
||||
@@ -170,10 +164,7 @@ describe("interactive model setup", () => {
|
||||
installHosts: false,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: true,
|
||||
});
|
||||
expect(result).toMatchObject({ state: "configured" });
|
||||
expect(api.post).toHaveBeenCalledWith(
|
||||
"/api/v1/model-profiles",
|
||||
expect.objectContaining({
|
||||
@@ -184,9 +175,10 @@ describe("interactive model setup", () => {
|
||||
embedding_provider_kind: "local_onnx",
|
||||
}),
|
||||
);
|
||||
expect(api.post).toHaveBeenCalledWith("/api/v1/model-content-permission", {
|
||||
allowed: true,
|
||||
});
|
||||
expect(api.post).not.toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
expect.anything(),
|
||||
);
|
||||
expect(prompt.written.join("\n")).not.toContain("sk-do-not-print");
|
||||
});
|
||||
|
||||
@@ -205,7 +197,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("removes an unverified profile before skipping after a failed probe", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "0"]);
|
||||
const prompt = prompter(["", "", "", "0"]);
|
||||
const api = clients();
|
||||
api.post.mockImplementation(
|
||||
async (path: string, body: Record<string, unknown>) => {
|
||||
@@ -280,7 +272,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("cleans a failed probe and lets the user re-enter the model settings", async () => {
|
||||
const prompt = prompter(["", "", "", "", "", "", "", "", "", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", "", "", ""]);
|
||||
const api = clients({ failedProbes: 1 });
|
||||
|
||||
const result = await runInteractiveModelSetup(prompt.value, api, {
|
||||
@@ -289,10 +281,7 @@ describe("interactive model setup", () => {
|
||||
installHosts: false,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: true,
|
||||
});
|
||||
expect(result).toMatchObject({ state: "configured" });
|
||||
expect(api.remove).toHaveBeenCalledTimes(1);
|
||||
expect(
|
||||
api.post.mock.calls.filter(([path]) => path === "/api/v1/model-profiles"),
|
||||
@@ -305,7 +294,7 @@ describe("interactive model setup", () => {
|
||||
const prompt = prompter(["", "", ""]);
|
||||
prompt.value.ask = async () => {
|
||||
asked += 1;
|
||||
if (asked === 4) throw new Error("interrupted");
|
||||
if (asked === 3) throw new Error("interrupted");
|
||||
return "";
|
||||
};
|
||||
const api = clients();
|
||||
@@ -322,7 +311,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("installs selected detected hosts through preview and fingerprint apply", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -360,7 +349,6 @@ describe("interactive model setup", () => {
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: ["codex"],
|
||||
});
|
||||
expect(read).toHaveBeenNthCalledWith(
|
||||
@@ -373,7 +361,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("lists and installs only hosts that are actually present", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -420,7 +408,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("completes the wizard and keeps onboarding reachable when one host install fails", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -469,7 +457,6 @@ describe("interactive model setup", () => {
|
||||
// calling the install a failure.
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: ["codex"],
|
||||
});
|
||||
const summary = prompt.written.join("\n");
|
||||
@@ -479,7 +466,7 @@ describe("interactive model setup", () => {
|
||||
});
|
||||
|
||||
it("tells the user to install App-only hosts in the App instead of calling them failed", async () => {
|
||||
const prompt = prompter(["", "", "", "n", "", ""]);
|
||||
const prompt = prompter(["", "", "", "", ""]);
|
||||
const api = clients();
|
||||
const read = vi
|
||||
.fn()
|
||||
@@ -506,7 +493,6 @@ describe("interactive model setup", () => {
|
||||
// only the App can install, so nothing counts as "installed".
|
||||
expect(result).toMatchObject({
|
||||
state: "configured",
|
||||
model_content_allowed: false,
|
||||
installed_hosts: [],
|
||||
});
|
||||
const summary = prompt.written.join("\n");
|
||||
|
||||
@@ -11,7 +11,6 @@ import {
|
||||
} from "./headless-flows.js";
|
||||
import { approvalFingerprint, listHosts, readHost } from "./host-flow.js";
|
||||
import { parseSelection, type WizardPrompter } from "./init-wizard.js";
|
||||
import { setModelContentPermission } from "./model-content-permission-flow.js";
|
||||
import { executeModelProbe, planModelProbe } from "./model-probe-flow.js";
|
||||
import {
|
||||
executeModelProfileActivate,
|
||||
@@ -122,7 +121,6 @@ export type InteractiveModelSetupState =
|
||||
| {
|
||||
state: "configured";
|
||||
profile: ModelProfile;
|
||||
model_content_allowed: boolean;
|
||||
installed_hosts: string[];
|
||||
}
|
||||
| { state: "skipped" }
|
||||
@@ -289,15 +287,11 @@ async function installHosts(
|
||||
function writeSummary(
|
||||
prompter: WizardPrompter,
|
||||
profile: ModelProfile,
|
||||
modelContentAllowed: boolean,
|
||||
installedHosts: readonly string[],
|
||||
failedHosts: readonly { hostId: string; label: string }[],
|
||||
): void {
|
||||
prompter.write("配置摘要:");
|
||||
prompter.write(` 模型:${profile.name} / ${profile.model}`);
|
||||
prompter.write(
|
||||
` 模型内容外发:${modelContentAllowed ? "已允许" : "已禁止"}`,
|
||||
);
|
||||
prompter.write(
|
||||
` 应用接入:${installedHosts.length > 0 ? installedHosts.join("、") : "已跳过"}`,
|
||||
);
|
||||
@@ -353,14 +347,6 @@ export async function runInteractiveModelSetup(
|
||||
prompter.write("未填写 API Key。输入 back 返回,或重新选择模型服务。");
|
||||
continue;
|
||||
}
|
||||
const permissionAnswer = (
|
||||
await prompter.ask("允许将对话内容发送给模型以整理记忆?[Y/n]: ")
|
||||
)
|
||||
.trim()
|
||||
.toLowerCase();
|
||||
const modelContentAllowed = !["n", "no", "否", "0"].includes(
|
||||
permissionAnswer,
|
||||
);
|
||||
const hostApi = options.hostApi ?? { list: listHosts, read: readHost };
|
||||
const selectedHosts = options.installHosts
|
||||
? await chooseDetectedHosts(prompter, options.hostHome, hostApi)
|
||||
@@ -452,13 +438,6 @@ export async function runInteractiveModelSetup(
|
||||
return { state: "failed", code: "model_activate_failed" };
|
||||
}
|
||||
|
||||
const permission = await setModelContentPermission(
|
||||
clients.operations,
|
||||
modelContentAllowed,
|
||||
);
|
||||
if (permission.state !== "ok") {
|
||||
return { state: "failed", code: "model_content_permission_set_failed" };
|
||||
}
|
||||
const { installed, failed } = await installHosts(
|
||||
options.dataDir,
|
||||
selectedHosts,
|
||||
@@ -467,7 +446,6 @@ export async function runInteractiveModelSetup(
|
||||
writeSummary(
|
||||
prompter,
|
||||
profile,
|
||||
modelContentAllowed,
|
||||
installed,
|
||||
failed.map((hostId) => ({
|
||||
hostId,
|
||||
@@ -477,7 +455,6 @@ export async function runInteractiveModelSetup(
|
||||
return {
|
||||
state: "configured",
|
||||
profile,
|
||||
model_content_allowed: modelContentAllowed,
|
||||
installed_hosts: installed,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -180,5 +180,8 @@ describe("import-job write allow-list", () => {
|
||||
await expect(
|
||||
operations.post("/api/v1/import-jobs/job-1/delete", {}),
|
||||
).rejects.toThrow("core_operation_path_not_allowed");
|
||||
await expect(
|
||||
operations.post("/api/v1/model-content-permission", { allowed: true }),
|
||||
).rejects.toThrow("core_operation_path_not_allowed");
|
||||
});
|
||||
});
|
||||
@@ -1,154 +0,0 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { run } from "./cli.js";
|
||||
|
||||
const transport = vi.hoisted(() => ({
|
||||
get: vi.fn(),
|
||||
post: vi.fn(),
|
||||
put: vi.fn(),
|
||||
patch: vi.fn(),
|
||||
delete: vi.fn(),
|
||||
}));
|
||||
vi.mock("axios", () => ({
|
||||
default: { create: vi.fn(() => transport), get: vi.fn() },
|
||||
}));
|
||||
|
||||
const runtime = vi.hoisted(() => ({
|
||||
inspectRuntimeStatus: vi.fn(),
|
||||
discoverRuntimeAttachment: vi.fn(),
|
||||
}));
|
||||
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
|
||||
...runtime,
|
||||
}));
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
runtime.inspectRuntimeStatus.mockResolvedValue({
|
||||
state: "running",
|
||||
code: "runtime_verified",
|
||||
instanceId: "instance-1",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
version: "1.2.3",
|
||||
});
|
||||
runtime.discoverRuntimeAttachment.mockResolvedValue({
|
||||
instanceId: "instance-1",
|
||||
apiToken: "synthetic-runtime-token",
|
||||
host: "127.0.0.1",
|
||||
port: 8765,
|
||||
servicePid: 4242,
|
||||
health: {
|
||||
status: "ok",
|
||||
service: "one-memory-core",
|
||||
version: "1.2.3",
|
||||
instance_id: "instance-1",
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
describe("model content permission CLI", () => {
|
||||
it("documents only final-state permission and manual extraction commands", async () => {
|
||||
const permission = await capturedRun(["help", "permission"]);
|
||||
expect(permission.code).toBe(0);
|
||||
expect(permission.stdout).toContain("memhub permission allow");
|
||||
expect(permission.stdout).toContain("memhub permission deny");
|
||||
expect(permission.stdout).not.toMatch(
|
||||
/\b(preview|confirm|process|revoke)\b/,
|
||||
);
|
||||
expect(permission.stdout).not.toContain("revision");
|
||||
expect(permission.stdout).not.toContain("receipt");
|
||||
|
||||
const processing = await capturedRun(["help", "processing"]);
|
||||
expect(processing.code).toBe(0);
|
||||
expect(processing.stdout).toContain("memhub processing extract");
|
||||
expect(processing.stdout).not.toMatch(/\b(plan|apply|receipt)\b/);
|
||||
expect(processing.stdout).not.toContain("backlog");
|
||||
});
|
||||
|
||||
it("reads the final state", async () => {
|
||||
transport.get.mockResolvedValue({ status: 200, data: { allowed: true } });
|
||||
|
||||
const result = await capturedRun(["permission", "status", "--json"]);
|
||||
|
||||
expect(result.code).toBe(0);
|
||||
expect(JSON.parse(result.stdout)).toMatchObject({
|
||||
state: "ok",
|
||||
allowed: true,
|
||||
});
|
||||
expect(transport.get).toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["allow", true],
|
||||
["deny", false],
|
||||
] as const)("sets %s directly with only allowed", async (verb, allowed) => {
|
||||
transport.post.mockResolvedValue({ status: 200, data: { allowed } });
|
||||
|
||||
const result = await capturedRun(["permission", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(0);
|
||||
expect(JSON.parse(result.stdout)).toMatchObject({ state: "ok", allowed });
|
||||
expect(transport.post).toHaveBeenCalledWith(
|
||||
"/api/v1/model-content-permission",
|
||||
{ allowed },
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
|
||||
it.each(["preview", "confirm", "process", "revoke"])(
|
||||
"rejects removed %s commands before runtime access",
|
||||
async (verb) => {
|
||||
const result = await capturedRun(["permission", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(2);
|
||||
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
|
||||
expect(transport.get).not.toHaveBeenCalled();
|
||||
expect(transport.post).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["plan", "apply", "receipt"])(
|
||||
"rejects removed processing %s commands before runtime access",
|
||||
async (verb) => {
|
||||
const result = await capturedRun(["processing", verb, "--json"]);
|
||||
|
||||
expect(result.code).toBe(2);
|
||||
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
|
||||
expect(transport.get).not.toHaveBeenCalled();
|
||||
expect(transport.post).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
async function capturedRun(arguments_: string[]) {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const stdoutSpy = vi
|
||||
.spyOn(process.stdout, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stdout += String(chunk);
|
||||
return true;
|
||||
});
|
||||
const stderrSpy = vi
|
||||
.spyOn(process.stderr, "write")
|
||||
.mockImplementation((chunk) => {
|
||||
stderr += String(chunk);
|
||||
return true;
|
||||
});
|
||||
try {
|
||||
return {
|
||||
code: await run(arguments_, {
|
||||
ONE_MEMORY_DATA_DIR: "/synthetic/data",
|
||||
}),
|
||||
stdout,
|
||||
stderr,
|
||||
};
|
||||
} finally {
|
||||
stdoutSpy.mockRestore();
|
||||
stderrSpy.mockRestore();
|
||||
}
|
||||
}
|
||||
@@ -1,66 +0,0 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import type { CoreReader } from "./core-client.js";
|
||||
import type { CoreOperations } from "./core-operations.js";
|
||||
import {
|
||||
readModelContentPermission,
|
||||
setModelContentPermission,
|
||||
} from "./model-content-permission-flow.js";
|
||||
|
||||
describe("model content permission flow", () => {
|
||||
it("reads the single final-state boolean", async () => {
|
||||
const reader = vi.fn().mockResolvedValue({
|
||||
state: "ok",
|
||||
data: { allowed: true },
|
||||
});
|
||||
|
||||
await expect(
|
||||
readModelContentPermission(reader as CoreReader),
|
||||
).resolves.toMatchObject({ state: "ok", allowed: true });
|
||||
expect(reader).toHaveBeenCalledWith("/api/v1/model-content-permission");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["allow", true],
|
||||
["deny", false],
|
||||
] as const)(
|
||||
"sets %s directly without revision or receipt fields",
|
||||
async (_, allowed) => {
|
||||
const post = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed },
|
||||
});
|
||||
|
||||
await expect(
|
||||
setModelContentPermission({ post } as CoreOperations, allowed),
|
||||
).resolves.toMatchObject({ state: "ok", allowed });
|
||||
expect(post).toHaveBeenCalledWith("/api/v1/model-content-permission", {
|
||||
allowed,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("does not accept a malformed or mismatched success response", async () => {
|
||||
const malformed = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed: true, revision: 1 },
|
||||
});
|
||||
await expect(
|
||||
setModelContentPermission({ post: malformed } as CoreOperations, true),
|
||||
).resolves.toMatchObject({
|
||||
state: "unknown",
|
||||
code: "model_content_permission_response_invalid",
|
||||
});
|
||||
|
||||
const mismatched = vi.fn().mockResolvedValue({
|
||||
state: "committed",
|
||||
data: { allowed: false },
|
||||
});
|
||||
await expect(
|
||||
setModelContentPermission({ post: mismatched } as CoreOperations, true),
|
||||
).resolves.toMatchObject({
|
||||
state: "unknown",
|
||||
code: "model_content_permission_state_mismatch",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,96 +0,0 @@
|
||||
import type { CoreReader } from "./core-client.js";
|
||||
import type { CoreOperations } from "./core-operations.js";
|
||||
import { PRODUCT } from "./version.js";
|
||||
|
||||
export type ModelContentPermissionDocument =
|
||||
| {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
state: "ok";
|
||||
allowed: boolean;
|
||||
}
|
||||
| {
|
||||
schema_version: 1;
|
||||
product: string;
|
||||
state: "rejected" | "unknown";
|
||||
code: string;
|
||||
};
|
||||
|
||||
export async function readModelContentPermission(
|
||||
reader: CoreReader,
|
||||
): Promise<ModelContentPermissionDocument> {
|
||||
const outcome = await reader<unknown>("/api/v1/model-content-permission");
|
||||
if (outcome.state !== "ok") {
|
||||
return failure("unknown", outcome.code);
|
||||
}
|
||||
return permissionState(
|
||||
outcome.data,
|
||||
"model_content_permission_response_invalid",
|
||||
);
|
||||
}
|
||||
|
||||
export async function setModelContentPermission(
|
||||
operations: CoreOperations,
|
||||
allowed: boolean,
|
||||
): Promise<ModelContentPermissionDocument> {
|
||||
const outcome = await operations.post<unknown>(
|
||||
"/api/v1/model-content-permission",
|
||||
{ allowed },
|
||||
);
|
||||
if (outcome.state === "rejected") {
|
||||
return failure("rejected", outcome.code);
|
||||
}
|
||||
if (outcome.state === "unknown") {
|
||||
return failure("unknown", outcome.code);
|
||||
}
|
||||
const document = permissionState(
|
||||
outcome.data,
|
||||
"model_content_permission_response_invalid",
|
||||
);
|
||||
return document.state === "ok" && document.allowed !== allowed
|
||||
? failure("unknown", "model_content_permission_state_mismatch")
|
||||
: document;
|
||||
}
|
||||
|
||||
export function modelContentPermissionExitCode(
|
||||
document: ModelContentPermissionDocument,
|
||||
): number {
|
||||
return document.state === "ok" ? 0 : 1;
|
||||
}
|
||||
|
||||
export function renderModelContentPermissionText(
|
||||
document: ModelContentPermissionDocument,
|
||||
): string {
|
||||
if (document.state === "ok") {
|
||||
return `模型内容外发:${document.allowed ? "允许" : "禁止"}\n`;
|
||||
}
|
||||
return `模型内容权限未更新:${document.state}(${document.code})\n`;
|
||||
}
|
||||
|
||||
function permissionState(
|
||||
value: unknown,
|
||||
invalidCode: string,
|
||||
): ModelContentPermissionDocument {
|
||||
if (
|
||||
value === null ||
|
||||
typeof value !== "object" ||
|
||||
Array.isArray(value) ||
|
||||
Object.keys(value).length !== 1 ||
|
||||
typeof (value as Record<string, unknown>).allowed !== "boolean"
|
||||
) {
|
||||
return failure("unknown", invalidCode);
|
||||
}
|
||||
return {
|
||||
schema_version: 1,
|
||||
product: PRODUCT,
|
||||
state: "ok",
|
||||
allowed: (value as { allowed: boolean }).allowed,
|
||||
};
|
||||
}
|
||||
|
||||
function failure(
|
||||
state: "rejected" | "unknown",
|
||||
code: string,
|
||||
): ModelContentPermissionDocument {
|
||||
return { schema_version: 1, product: PRODUCT, state, code };
|
||||
}
|
||||
@@ -137,8 +137,8 @@ describe("package matrix harness", () => {
|
||||
};
|
||||
expect(document.state).toBe("verified");
|
||||
const results = document.commands.stopped_destructive.results;
|
||||
// 13 destructive commands, each unknown/runtime_not_running with exit 3.
|
||||
expect(Object.keys(results)).toHaveLength(13);
|
||||
// 11 destructive commands, each unknown/runtime_not_running with exit 3.
|
||||
expect(Object.keys(results)).toHaveLength(11);
|
||||
for (const [name, result] of Object.entries(results)) {
|
||||
expect(result, `${name} did not fail closed`).toEqual({
|
||||
exit: 3,
|
||||
@@ -162,7 +162,7 @@ describe("package matrix harness", () => {
|
||||
const shim = [
|
||||
"#!/usr/bin/env node",
|
||||
"const a=process.argv.slice(2);",
|
||||
"if(['calls','import-file','notes','permission','extraction','graph','models'].includes(a[0])||(a[0]==='processing'&&a[1]==='extract')){",
|
||||
"if(['calls','import-file','notes','extraction','graph','models'].includes(a[0])||(a[0]==='processing'&&a[1]==='extract')){",
|
||||
' process.stdout.write(\'{"schema_version":1,"product":"memhub","state":"committed","operation_id":"x","receipt":{}}\\n\');',
|
||||
" process.exit(0);",
|
||||
"}",
|
||||
@@ -188,7 +188,7 @@ describe("package matrix harness", () => {
|
||||
expect(failure?.state).toBe("failed");
|
||||
expect(
|
||||
failure?.failures.filter((item) => item.includes("did not fail closed")),
|
||||
).toHaveLength(13);
|
||||
).toHaveLength(11);
|
||||
});
|
||||
|
||||
it("can require an expected bundle SHA and reports bundle metadata", {
|
||||
@@ -280,9 +280,8 @@ describe("package matrix harness", () => {
|
||||
});
|
||||
|
||||
it("rejects a catalog that declares the parent command but not the new verb", async () => {
|
||||
// A bundle can still declare `permission`/`models` while missing
|
||||
// `permission allow` or `models activate`. Names alone would pass it; the
|
||||
// usage marker must not.
|
||||
// A bundle can still declare `models` while missing `models activate`.
|
||||
// Names alone would pass it; the usage marker must not.
|
||||
const declared = [
|
||||
"install",
|
||||
"init",
|
||||
@@ -292,7 +291,6 @@ describe("package matrix harness", () => {
|
||||
"calls",
|
||||
"import-file",
|
||||
"notes",
|
||||
"permission",
|
||||
"extraction",
|
||||
"graph",
|
||||
"models",
|
||||
@@ -342,9 +340,6 @@ describe("package matrix harness", () => {
|
||||
},
|
||||
};
|
||||
const failures = harness.validateCatalog(catalog);
|
||||
expect(failures).toContain(
|
||||
'command "permission" is missing usage "memhub permission allow"',
|
||||
);
|
||||
expect(failures).toContain(
|
||||
'command "models" is missing usage "memhub models activate"',
|
||||
);
|
||||
|
||||
@@ -151,9 +151,6 @@ describe("memhub short-command contract", () => {
|
||||
"memhub import-file discard",
|
||||
"memhub notes add",
|
||||
"memhub notes process",
|
||||
"memhub permission status",
|
||||
"memhub permission allow",
|
||||
"memhub permission deny",
|
||||
"memhub models setup",
|
||||
"memhub models activate",
|
||||
"memhub models edit",
|
||||
@@ -163,6 +160,9 @@ describe("memhub short-command contract", () => {
|
||||
]) {
|
||||
expect(flattened).toContain(shortForm);
|
||||
}
|
||||
expect(COMMANDS.some((command) => command.name === "permission")).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("ships a memhub bin entry pointing at a real bundle path", () => {
|
||||
|
||||
@@ -48,7 +48,11 @@ describe("publish manifest is production-safe", () => {
|
||||
execFileSync("node", ["scripts/prepare-publish-manifest.mjs"], {
|
||||
cwd: root,
|
||||
stdio: "pipe",
|
||||
env: { ...process.env, MEMHUB_NPM_PACKAGE: "@example/memhub-cli" },
|
||||
env: {
|
||||
...process.env,
|
||||
MEMHUB_NPM_PACKAGE: "@example/memhub-cli",
|
||||
MEMHUB_NPM_VERSION: "1.0.5",
|
||||
},
|
||||
});
|
||||
const published = JSON.parse(
|
||||
readFileSync(path.join(root, "dist-publish", "package.json"), "utf8"),
|
||||
@@ -56,6 +60,7 @@ describe("publish manifest is production-safe", () => {
|
||||
expect(published.dependencies).toBeUndefined();
|
||||
expect(published.private).toBe(false);
|
||||
expect(published.name).toBe("@example/memhub-cli");
|
||||
expect(published.version).toBe("1.0.5");
|
||||
expect(published.bin).toMatchObject({
|
||||
memhub: "dist/one-memory.cjs",
|
||||
"memhub-cli": "dist/one-memory.cjs",
|
||||
|
||||
@@ -840,6 +840,7 @@ describe("CLI-21 memory reads", () => {
|
||||
expect(isReadOnlyPath("/api/v1/memories/graph")).toBe(false);
|
||||
expect(isReadOnlyPath("/api/v1/memories/ask")).toBe(false);
|
||||
expect(isReadOnlyPath("/api/v1/memories/m1")).toBe(true);
|
||||
expect(isReadOnlyPath("/api/v1/model-content-permission")).toBe(false);
|
||||
expect(
|
||||
isReadOnlyPath(
|
||||
"/api/v1/memory-mutations/123e4567-e89b-12d3-a456-426614174000",
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
declare const __ONE_MEMORY_VERSION__: string;
|
||||
declare const __ONE_MEMORY_BUILD_NUMBER__: string;
|
||||
declare const __MEMHUB_DEFAULT_RUNTIME_VERSION__: string;
|
||||
|
||||
const SEMVER =
|
||||
/^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$/;
|
||||
|
||||
const embeddedVersion =
|
||||
typeof __ONE_MEMORY_VERSION__ === "string"
|
||||
@@ -9,13 +13,18 @@ const embeddedBuildNumber =
|
||||
typeof __ONE_MEMORY_BUILD_NUMBER__ === "string"
|
||||
? __ONE_MEMORY_BUILD_NUMBER__.trim()
|
||||
: "";
|
||||
const embeddedRuntimeVersion =
|
||||
typeof __MEMHUB_DEFAULT_RUNTIME_VERSION__ === "string"
|
||||
? __MEMHUB_DEFAULT_RUNTIME_VERSION__.trim()
|
||||
: "";
|
||||
|
||||
export const CLI_VERSION =
|
||||
/^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$/.test(
|
||||
embeddedVersion,
|
||||
)
|
||||
? embeddedVersion
|
||||
: "development";
|
||||
export const CLI_VERSION = SEMVER.test(embeddedVersion)
|
||||
? embeddedVersion
|
||||
: "development";
|
||||
|
||||
export const RUNTIME_RELEASE_VERSION = SEMVER.test(embeddedRuntimeVersion)
|
||||
? embeddedRuntimeVersion
|
||||
: CLI_VERSION;
|
||||
|
||||
export const CLI_BUILD_NUMBER = /^\d+$/.test(embeddedBuildNumber)
|
||||
? Number(embeddedBuildNumber)
|
||||
|
||||
@@ -2,11 +2,22 @@ import { createRequire } from "node:module";
|
||||
import { defineConfig } from "tsup";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const { resolveBuildVersion } =
|
||||
const { assertSemver, resolveBuildVersion } =
|
||||
require("../../scripts/release-version.cjs") as {
|
||||
assertSemver(value: string, label?: string): string;
|
||||
resolveBuildVersion(environment?: NodeJS.ProcessEnv, root?: string): string;
|
||||
};
|
||||
const releaseVersion = resolveBuildVersion(process.env);
|
||||
const packageVersion = (require("./package.json") as { version: string })
|
||||
.version;
|
||||
const cliVersion = assertSemver(
|
||||
process.env.MEMHUB_NPM_VERSION?.trim() || packageVersion,
|
||||
"MEMHUB_NPM_VERSION",
|
||||
);
|
||||
const runtimeReleaseVersion = assertSemver(
|
||||
process.env.MEMHUB_DEFAULT_RUNTIME_VERSION?.trim() || releaseVersion,
|
||||
"MEMHUB_DEFAULT_RUNTIME_VERSION",
|
||||
);
|
||||
|
||||
export default defineConfig({
|
||||
entry: { "one-memory": "src/entry.ts" },
|
||||
@@ -15,7 +26,7 @@ export default defineConfig({
|
||||
outDir: "dist",
|
||||
clean: true,
|
||||
define: {
|
||||
__ONE_MEMORY_VERSION__: JSON.stringify(releaseVersion),
|
||||
__ONE_MEMORY_VERSION__: JSON.stringify(cliVersion),
|
||||
__ONE_MEMORY_BUILD_NUMBER__: JSON.stringify(
|
||||
process.env.ONE_MEMORY_BUILD_NUMBER ?? "unknown",
|
||||
),
|
||||
@@ -25,11 +36,15 @@ export default defineConfig({
|
||||
__MEMHUB_DEFAULT_RELEASE_MANIFEST_URL_TEMPLATE__: JSON.stringify(
|
||||
process.env.MEMHUB_DEFAULT_RELEASE_MANIFEST_URL_TEMPLATE ?? "",
|
||||
),
|
||||
__MEMHUB_DEFAULT_RUNTIME_VERSION__: JSON.stringify(runtimeReleaseVersion),
|
||||
},
|
||||
// The shared operations package ships TypeScript sources, so it must be
|
||||
// inlined into the CJS bundle the same way runtime-control is; otherwise the
|
||||
// emitted bundle tries to `require` a `.js` file that only exists as `.ts`.
|
||||
// `@one-memory/contracts` is the same kind of package and is imported for a
|
||||
// value (the host-id grammar), so it has to be inlined too.
|
||||
noExternal: [
|
||||
"@one-memory/contracts",
|
||||
"@one-memory/operations",
|
||||
"@one-memory/runtime-control",
|
||||
"axios",
|
||||
|
||||
@@ -11,6 +11,7 @@ export default defineConfig({
|
||||
__ONE_MEMORY_BUILD_NUMBER__: JSON.stringify("unknown"),
|
||||
__ONE_MEMORY_COMMIT__: JSON.stringify("unknown"),
|
||||
__MEMHUB_DEFAULT_RELEASE_MANIFEST_URL_TEMPLATE__: JSON.stringify(""),
|
||||
__MEMHUB_DEFAULT_RUNTIME_VERSION__: JSON.stringify(version),
|
||||
},
|
||||
test: {
|
||||
testTimeout: 120_000,
|
||||
|
||||
@@ -11,7 +11,7 @@ folder_layout = layout == 'folder'
|
||||
datas = [('src/one_memory/embedding_assets', 'one_memory/embedding_assets')]
|
||||
binaries = []
|
||||
# Loaded lazily by the API; static dependency analysis cannot discover it.
|
||||
hiddenimports = ['one_memory.hy_permission_context']
|
||||
hiddenimports = []
|
||||
tmp_ret = collect_all('hy_memory')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('chromadb')
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "uv_build"
|
||||
|
||||
[project]
|
||||
name = "one-memory"
|
||||
version = "1.0.1"
|
||||
version = "1.0.4"
|
||||
description = "MemHub local-first memory core"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
@@ -12,7 +12,7 @@ dependencies = [
|
||||
"fastapi>=0.141.1",
|
||||
"filelock>=3.32.4",
|
||||
# Public SDK version validated against MemHub's permission and recall contracts.
|
||||
"hy-memory==2.0.2",
|
||||
"hy-memory==2.0.4",
|
||||
# Retrieval's derived vector index directly consumes these locked versions.
|
||||
"chromadb==1.5.9",
|
||||
"numpy==2.5.2",
|
||||
|
||||
@@ -20,7 +20,6 @@ from copy import copy
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from enum import StrEnum
|
||||
from functools import partial
|
||||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, cast
|
||||
from uuid import UUID
|
||||
@@ -101,7 +100,13 @@ from one_memory.imports.access import (
|
||||
SourceAccessError,
|
||||
root_identity_matches,
|
||||
)
|
||||
from one_memory.imports.adapters import SourceAdapter, SyncCompleteness
|
||||
from one_memory.imports.adapters import (
|
||||
ScanFailure,
|
||||
ScannedEvidence,
|
||||
ScanOptions,
|
||||
SourceAdapter,
|
||||
SyncCompleteness,
|
||||
)
|
||||
from one_memory.imports.candidates import (
|
||||
SourceCandidateError,
|
||||
SourceCandidateSelection,
|
||||
@@ -112,7 +117,11 @@ from one_memory.imports.connections import (
|
||||
ConnectionIntentReceipt,
|
||||
SourceConnectionStore,
|
||||
)
|
||||
from one_memory.imports.grants import SourceGrantReceipt, SourceGrantStore
|
||||
from one_memory.imports.grants import (
|
||||
SourceAuthorizationState,
|
||||
SourceGrantReceipt,
|
||||
SourceGrantStore,
|
||||
)
|
||||
from one_memory.imports.host_packages import CapabilityState, HostPackageCapabilities
|
||||
from one_memory.imports.jobs import (
|
||||
ImportIssueRecord,
|
||||
@@ -164,7 +173,6 @@ from one_memory.memory_library import (
|
||||
)
|
||||
from one_memory.memory_notes import MemoryNoteStore, memory_note_router
|
||||
from one_memory.memory_processing import (
|
||||
MemoryBridgeProtocol,
|
||||
MemoryProcessingSupervisor,
|
||||
_conversation_chunks,
|
||||
memory_task_timeout_seconds,
|
||||
@@ -177,11 +185,14 @@ from one_memory.memory_queue import (
|
||||
resolve_extraction_state,
|
||||
)
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
from one_memory.memory_tasks import MemoryTask, MemoryTaskStatus, MemoryTaskStore
|
||||
from one_memory.model_content_routes import model_content_router
|
||||
from one_memory.memory_tasks import (
|
||||
NON_RETRYABLE_POST_WRITE_CODES,
|
||||
MemoryTask,
|
||||
MemoryTaskStatus,
|
||||
MemoryTaskStore,
|
||||
)
|
||||
from one_memory.model_presets import seed_preset_profile
|
||||
from one_memory.model_probe import ModelProber, ModelProbeResult
|
||||
from one_memory.model_processing_permission_guard import ModelProcessingPermissionLease
|
||||
from one_memory.model_profiles import (
|
||||
ActiveModelProfileError,
|
||||
ModelCreationOperationConflictError,
|
||||
@@ -194,6 +205,7 @@ from one_memory.model_profiles import (
|
||||
ModelProfileStore,
|
||||
ModelProfileUnavailableError,
|
||||
)
|
||||
from one_memory.model_request_audit import AuditedModelTransport, ModelRequestAuditStore
|
||||
from one_memory.model_request_budget import BudgetedModelTransport, ModelRequestBudget
|
||||
from one_memory.onboarding import OnboardingState, OnboardingStore
|
||||
from one_memory.recall import GovernedRecallTarget, RecallReceipt, RecallRequest
|
||||
@@ -285,6 +297,9 @@ class SourceResponse(BaseModel):
|
||||
candidate_revision: int = 0
|
||||
candidate_change_allowed: bool = False
|
||||
candidate_change_reason: str | None = None
|
||||
#: Why `authorized` is false, so the UI can ask for the one thing that
|
||||
#: would fix it instead of offering every recovery at once.
|
||||
authorization_state: SourceAuthorizationState = SourceAuthorizationState.OK
|
||||
|
||||
|
||||
class PreviewSourceCandidateRequest(BaseModel):
|
||||
@@ -317,6 +332,22 @@ class SourceSyncState(BaseModel):
|
||||
sync_enabled: bool
|
||||
|
||||
|
||||
class SedimentationRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
enabled: StrictBool
|
||||
|
||||
|
||||
class SedimentationState(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
enabled: bool
|
||||
#: What the switch means for the model bill and for what still runs. Read
|
||||
#: by the app so the setting can say so without hardcoding the wording.
|
||||
pending_retained: bool = True
|
||||
imports_unaffected: bool = True
|
||||
|
||||
|
||||
class DetectionResponse(BaseModel):
|
||||
source_id: str
|
||||
detected: bool
|
||||
@@ -343,7 +374,7 @@ class ConnectSourcesRequest(BaseModel):
|
||||
import_upload_id: UUID | None = None
|
||||
|
||||
|
||||
ONBOARDING_LARGE_HISTORY_SESSIONS = 200
|
||||
ONBOARDING_LARGE_HISTORY_USER_TURNS = 200
|
||||
ONBOARDING_HISTORY_WINDOW = timedelta(days=3)
|
||||
|
||||
|
||||
@@ -818,6 +849,13 @@ def create_app(
|
||||
evidence=app.state.evidence_store,
|
||||
)
|
||||
app.state.memory_write_queue = memory_write_queue
|
||||
# Builds through 1.0.2 parked pre-invocation jobs behind the
|
||||
# retired model-content consent bit. The embedded SDK is trusted
|
||||
# now, so make those durable jobs claimable on the first startup.
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.readmit_parked,
|
||||
states=("waiting_for_permission",),
|
||||
)
|
||||
repaired_identity_failures = await asyncio.to_thread(
|
||||
memory_write_queue.repair_pre_send_identity_failures
|
||||
)
|
||||
@@ -883,6 +921,7 @@ def create_app(
|
||||
)
|
||||
app.state.import_job_store = ImportJobStore(database_path)
|
||||
app.state.onboarding_store = OnboardingStore(database_path)
|
||||
app.state.history_import_gate_released = False
|
||||
app.state.extraction_settings_store = ExtractionSettingsStore(database_path)
|
||||
app.state.source_grant_store = SourceGrantStore(
|
||||
database_path,
|
||||
@@ -905,6 +944,7 @@ def create_app(
|
||||
app.state.source_permission_locks = {
|
||||
source_id: asyncio.Lock() for source_id in app.state.source_catalog
|
||||
}
|
||||
app.state.sedimentation_lock = asyncio.Lock()
|
||||
_follow_workbuddy_home(app, runtime_settings.source_home)
|
||||
app.state.source_detection = {}
|
||||
app.state.secret_store = secret_store or KeyringSecretStore()
|
||||
@@ -960,7 +1000,7 @@ def create_app(
|
||||
# Not fatal: without the tables every read checks exactly, as
|
||||
# it does today.
|
||||
app.state.memory_visibility_error = str(error)
|
||||
app.state.model_request_budget = (
|
||||
model_request_budget = (
|
||||
ModelRequestBudget(
|
||||
database_path,
|
||||
request_limit=runtime_settings.model_request_limit,
|
||||
@@ -970,6 +1010,8 @@ def create_app(
|
||||
if runtime_settings.model_request_limit is not None
|
||||
else None
|
||||
)
|
||||
app.state.model_request_budget = model_request_budget
|
||||
app.state.model_request_audit_store = ModelRequestAuditStore(database_path)
|
||||
app.state.memory_answerer = MemoryAnswerer(
|
||||
governance=app.state.memory_governance_store,
|
||||
filter_memories=app.state.source_read_access.memories,
|
||||
@@ -1008,6 +1050,23 @@ def create_app(
|
||||
governance=app.state.memory_governance_store,
|
||||
data_dir=runtime_settings.data_dir,
|
||||
extraction_settings=app.state.extraction_settings_store,
|
||||
bridge_factory=lambda profile, api_key, data_dir: HyMemoryBridge(
|
||||
profile=profile,
|
||||
api_key=api_key,
|
||||
data_dir=data_dir,
|
||||
http_transport_factory=lambda: AuditedModelTransport(
|
||||
app.state.model_request_audit_store,
|
||||
profile_id=profile.profile_id,
|
||||
profile_revision=profile.revision,
|
||||
provider_kind=str(profile.provider_kind),
|
||||
model=profile.model,
|
||||
inner=(
|
||||
BudgetedModelTransport(model_request_budget)
|
||||
if model_request_budget is not None
|
||||
else None
|
||||
),
|
||||
),
|
||||
),
|
||||
# `None` means "whatever the active profile allows". Pinned to
|
||||
# 1 in `a59f4b5` as temporary load-shedding for an acceptance
|
||||
# package, and left pinned after it: a hardcoded 1 is rule 9's
|
||||
@@ -1016,12 +1075,9 @@ def create_app(
|
||||
# profile concurrency 4, recall latency is unchanged while
|
||||
# precipitation goes 4x.
|
||||
max_concurrent_tasks=runtime_settings.precipitation_max_concurrent_tasks,
|
||||
model_content_config=app.state.config_store,
|
||||
permission_bridge_factory=lambda lease, secret, directory: _permission_context(
|
||||
app, lease, secret, directory
|
||||
),
|
||||
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
|
||||
service_levels=app.state.service_levels,
|
||||
sedimentation_enabled=lambda: _config_store(app).sedimentation_enabled(),
|
||||
)
|
||||
# C2 owns the single production budget (handle 5s + reserve 250ms,
|
||||
# total 5.25s, lease 10s, renew 1s, claim 250ms). The composition
|
||||
@@ -1031,26 +1087,41 @@ def create_app(
|
||||
drain_budget = production_drain_budget()
|
||||
|
||||
async def reconcile_unknown_writes() -> None:
|
||||
await app.state.memory_processing_supervisor.reconcile_unknown_outputs()
|
||||
unknown = await asyncio.to_thread(memory_write_queue.unknown_jobs, limit=128)
|
||||
await app.state.memory_processing_supervisor.reconcile_unknown_outputs(
|
||||
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
|
||||
)
|
||||
unknown = await asyncio.to_thread(
|
||||
memory_write_queue.unknown_jobs,
|
||||
limit=128,
|
||||
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
|
||||
)
|
||||
for job in unknown:
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.resolve_unknown_from_linked_output,
|
||||
job.job_id,
|
||||
)
|
||||
await asyncio.to_thread(
|
||||
memory_write_queue.retry_timed_out_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(minutes=5),
|
||||
memory_write_queue.retry_first_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(seconds=10),
|
||||
limit=32,
|
||||
)
|
||||
await asyncio.to_thread(
|
||||
app.state.memory_task_store.retry_first_unknown_after_reconcile,
|
||||
older_than=datetime.now(UTC) - timedelta(seconds=10),
|
||||
limit=128,
|
||||
)
|
||||
|
||||
async def admit_hy_invocation(
|
||||
source_id: str,
|
||||
mark_invoked: Callable[[], Awaitable[bool]],
|
||||
) -> bool:
|
||||
"""Serialize pause with the last boundary before an HY call."""
|
||||
lock = app.state.source_permission_locks.setdefault(source_id, asyncio.Lock())
|
||||
async with lock:
|
||||
source_lock = app.state.source_permission_locks.setdefault(
|
||||
source_id, asyncio.Lock()
|
||||
)
|
||||
async with app.state.sedimentation_lock, source_lock:
|
||||
if not _config_store(app).sedimentation_enabled():
|
||||
raise RuntimeError("model_sedimentation_disabled")
|
||||
if not _source_writes_on(app, source_id):
|
||||
raise RuntimeError("model_content_source_disabled")
|
||||
return await mark_invoked()
|
||||
@@ -1088,8 +1159,18 @@ def create_app(
|
||||
+ 10
|
||||
),
|
||||
reconcile_unknown=reconcile_unknown_writes,
|
||||
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
|
||||
source_claim_enabled=lambda source_id: _config_store(app).sync_enabled(source_id),
|
||||
source_writes_permitted=lambda source_id: _sedimentation_claim_enabled(
|
||||
app, source_id
|
||||
),
|
||||
# The claim gate has to be the same question the write gate
|
||||
# asks, not a second answer to it. It used to be
|
||||
# `sync_enabled`, the per-app toggle, and `one-memory-manual` is
|
||||
# not a connected app: it has no entry in the configuration
|
||||
# document, so that predicate answers False for it while the
|
||||
# write gate admits it by its grant. Every manual note's job was
|
||||
# therefore skipped on every drain tick — the note stayed
|
||||
# "saved, waiting" with no task behind it, forever.
|
||||
source_claim_enabled=lambda source_id: _sedimentation_claim_enabled(app, source_id),
|
||||
)
|
||||
app.state.memory_write_drain = memory_write_drain
|
||||
app.state.memory_write_drain_task = watched_task(
|
||||
@@ -1118,10 +1199,7 @@ def create_app(
|
||||
for chunk in _conversation_chunks(conversation, max_turns=12, completed_only=True):
|
||||
if (
|
||||
record not in chunk
|
||||
or not any(
|
||||
item.role == "user" and item.written_by.value == "person"
|
||||
for item in chunk
|
||||
)
|
||||
or not any(item.role == "user" for item in chunk)
|
||||
or not any(item.role == "assistant" for item in chunk)
|
||||
):
|
||||
continue
|
||||
@@ -1134,7 +1212,6 @@ def create_app(
|
||||
app.state.matter_discovery.processor = app.state.memory_processing_supervisor
|
||||
app.state.matter_discovery.write_intake = app.state.memory_write_intake
|
||||
app.state.matter_discovery.write_queue = memory_write_queue
|
||||
app.state.processing_coordinator = None
|
||||
app.state.memory_note_store = MemoryNoteStore(
|
||||
app.state.evidence_store,
|
||||
app.state.memory_processing_supervisor,
|
||||
@@ -1446,7 +1523,6 @@ def create_app(
|
||||
app.include_router(matter_router(protected))
|
||||
app.include_router(memory_note_router(protected))
|
||||
app.include_router(file_import_router(protected))
|
||||
app.include_router(model_content_router(protected))
|
||||
app.include_router(context_router(protected))
|
||||
|
||||
def require_workbuddy_boundary(request: Request) -> None:
|
||||
@@ -1609,7 +1685,7 @@ def create_app(
|
||||
# --- Reader-only internal endpoints. Registered here, before the public
|
||||
# routes below; no early return, so every route after this still registers.
|
||||
|
||||
@app.get("/internal/semantic-ids")
|
||||
@app.get("/internal/semantic-ids", dependencies=protected)
|
||||
async def internal_semantic_ids(
|
||||
query: str = Query(min_length=1, max_length=2_000),
|
||||
limit: int = Query(default=12, ge=1, le=20),
|
||||
@@ -1621,7 +1697,7 @@ def create_app(
|
||||
return {"ids": []}
|
||||
return {"ids": [memory_id for memory_id, _score in hits]}
|
||||
|
||||
@app.post("/internal/call-event")
|
||||
@app.post("/internal/call-event", dependencies=protected)
|
||||
async def internal_call_event(request: Request) -> dict[str, bool]:
|
||||
"""Side effects the Reader cannot write: it opens with query_only."""
|
||||
try:
|
||||
@@ -1640,7 +1716,7 @@ def create_app(
|
||||
return {"recorded": False}
|
||||
return {"recorded": bool(recorded)}
|
||||
|
||||
@app.get("/internal/graph-snapshot")
|
||||
@app.get("/internal/graph-snapshot", dependencies=protected)
|
||||
async def internal_graph_snapshot(
|
||||
limit: int = Query(default=160, ge=1, le=300),
|
||||
) -> dict[str, object]:
|
||||
@@ -1736,6 +1812,30 @@ def create_app(
|
||||
"events": events,
|
||||
}
|
||||
|
||||
@app.get("/api/v1/diagnostics/model-usage", dependencies=protected)
|
||||
def diagnostic_model_usage(
|
||||
since: datetime | None = Query(default=None), # noqa: B008
|
||||
request_id: str | None = Query( # noqa: B008
|
||||
default=None, min_length=1, max_length=512
|
||||
),
|
||||
limit: int = Query(default=200, ge=1, le=2_000), # noqa: B008
|
||||
) -> dict[str, object]:
|
||||
"""Return content-free precipitation usage for provider reconciliation."""
|
||||
|
||||
normalized_since = (
|
||||
(since if since.tzinfo is not None else since.replace(tzinfo=UTC))
|
||||
if since is not None
|
||||
else None
|
||||
)
|
||||
report = app.state.model_request_audit_store.report(
|
||||
since=normalized_since.astimezone(UTC).isoformat()
|
||||
if normalized_since is not None
|
||||
else None,
|
||||
request_id=request_id,
|
||||
limit=limit,
|
||||
)
|
||||
return {"schema_version": 1, **report}
|
||||
|
||||
@app.get(
|
||||
"/api/v1/runtime/identity",
|
||||
response_model=HealthResponse,
|
||||
@@ -1873,6 +1973,20 @@ def create_app(
|
||||
"""Record UI completion only; finishing or skipping is never model consent."""
|
||||
return await asyncio.to_thread(_onboarding_store(app).complete)
|
||||
|
||||
@app.post(
|
||||
"/api/v1/onboarding/reset",
|
||||
response_model=OnboardingState,
|
||||
dependencies=protected,
|
||||
)
|
||||
async def reset_onboarding() -> OnboardingState:
|
||||
"""Go back to the first-run screens. Only the completion flag moves.
|
||||
|
||||
Memories, sources and settings live in other tables and are untouched,
|
||||
which is the point: someone who wants to see the cold-start flow again
|
||||
should not have to delete anything to do it.
|
||||
"""
|
||||
return await asyncio.to_thread(_onboarding_store(app).reset)
|
||||
|
||||
@app.get(
|
||||
"/api/v1/sources",
|
||||
response_model=list[SourceResponse],
|
||||
@@ -1910,6 +2024,11 @@ def create_app(
|
||||
candidate_change_reason=app.state.source_candidate_store.change_reason(
|
||||
registration
|
||||
),
|
||||
authorization_state=grant_store.authorization_state(
|
||||
user_id="local",
|
||||
source_id=source_id,
|
||||
candidate=registration.candidate.path,
|
||||
),
|
||||
)
|
||||
for source_id, registration in _catalog(app).items()
|
||||
]
|
||||
@@ -2221,24 +2340,39 @@ def create_app(
|
||||
raise HTTPException(status_code=409, detail=error.code) from None
|
||||
if grant is None:
|
||||
raise HTTPException(status_code=409, detail="grant_required")
|
||||
await _prepare_source_resume(source_id)
|
||||
try:
|
||||
await _prepare_source_resume(source_id)
|
||||
except BaseException as error:
|
||||
raise _sync_refused("sync_prepare_failed") from error
|
||||
else:
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
try:
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
except BaseException as error:
|
||||
raise _sync_refused("sync_prepare_failed") from error
|
||||
try:
|
||||
await asyncio.to_thread(_config_store(app).set_sync, source_id, request.enabled)
|
||||
except BaseException:
|
||||
except BaseException as error:
|
||||
if request.enabled:
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
else:
|
||||
await _prepare_source_resume(source_id)
|
||||
await _activate_source_resume(source_id)
|
||||
await _refresh_live_watch(app)
|
||||
raise
|
||||
raise _sync_refused("sync_config_write_failed") from error
|
||||
try:
|
||||
if request.enabled:
|
||||
await _activate_source_resume(source_id)
|
||||
await _refresh_live_watch(app)
|
||||
try:
|
||||
await _activate_source_resume(source_id)
|
||||
except BaseException as error:
|
||||
raise _sync_refused("sync_resume_failed") from error
|
||||
try:
|
||||
await _refresh_live_watch(app)
|
||||
except BaseException as error:
|
||||
raise _sync_refused("sync_watcher_failed") from error
|
||||
except BaseException:
|
||||
# Rolls back the desired state and settles the watcher exactly as
|
||||
# before; the only change is that the raise above now carries a
|
||||
# code saying which stage failed.
|
||||
await asyncio.to_thread(_config_store(app).set_sync, source_id, previous)
|
||||
if request.enabled:
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
@@ -2328,7 +2462,7 @@ def create_app(
|
||||
await asyncio.to_thread(
|
||||
app.state.memory_task_store.park_source_for_permission,
|
||||
source_id=source_id,
|
||||
error_code="model_content_permission_required",
|
||||
error_code="hy_memory_source_revoked",
|
||||
)
|
||||
await _live_sync(app).stop_source(source_id)
|
||||
_supervisor(app).unregister_adapter(source_id)
|
||||
@@ -2369,6 +2503,8 @@ def create_app(
|
||||
async def connect_sources(
|
||||
request: ConnectSourcesRequest,
|
||||
) -> ConnectSourcesResponse:
|
||||
if request.onboarding:
|
||||
app.state.history_import_gate_released = False
|
||||
source_ids = tuple(dict.fromkeys(request.source_ids))
|
||||
registrations = [_require_registration(app, source_id) for source_id in source_ids]
|
||||
if request.request_id is None:
|
||||
@@ -2668,6 +2804,12 @@ def create_app(
|
||||
# D75: work queued for the previous model continues with this one.
|
||||
# Only tasks that never reached the model are re-bound.
|
||||
await asyncio.to_thread(app.state.memory_task_store.rebind_unstarted_profile, profile)
|
||||
# A missing profile is an event-driven wait, not a provider outage.
|
||||
# Release those durable rows only now that activation has changed the
|
||||
# condition; the ordinary drain poll will claim them without a restart.
|
||||
await asyncio.to_thread(
|
||||
app.state.memory_write_queue.readmit_model_profile_waiters
|
||||
)
|
||||
# Not awaited: the catch-up reads every conversation ever imported, and
|
||||
# the answer to "which profile is active" does not depend on it.
|
||||
_start_backfill(app)
|
||||
@@ -3534,6 +3676,48 @@ def create_app(
|
||||
def put_extraction_settings(request: ExtractionWindowUpdate) -> ExtractionSettings:
|
||||
return _memory_processing_supervisor(app).set_extraction_window(request.window_preset)
|
||||
|
||||
@app.get(
|
||||
"/api/v1/sedimentation",
|
||||
response_model=SedimentationState,
|
||||
dependencies=protected,
|
||||
)
|
||||
def get_sedimentation() -> SedimentationState:
|
||||
"""Whether the model may still turn conversations into memories."""
|
||||
return SedimentationState(enabled=_config_store(app).sedimentation_enabled())
|
||||
|
||||
@app.put(
|
||||
"/api/v1/sedimentation",
|
||||
response_model=SedimentationState,
|
||||
dependencies=protected,
|
||||
)
|
||||
async def put_sedimentation(request: SedimentationRequest) -> SedimentationState:
|
||||
"""Start or stop model-driven sedimentation, keeping the configuration.
|
||||
|
||||
Stopping is the user's cost control: nothing further goes to the model
|
||||
and already queued work waits rather than failing, so starting again
|
||||
resumes it. Importing conversations is not affected either way.
|
||||
"""
|
||||
store = _config_store(app)
|
||||
changed = False
|
||||
async with app.state.sedimentation_lock:
|
||||
previous = store.sedimentation_enabled()
|
||||
if previous is not request.enabled:
|
||||
await asyncio.to_thread(store.set_sedimentation_enabled, request.enabled)
|
||||
changed = True
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="memory.sedimentation",
|
||||
event="memory_sedimentation.toggled",
|
||||
outcome="succeeded",
|
||||
details={"enabled": request.enabled},
|
||||
)
|
||||
if changed and request.enabled:
|
||||
# Work parked by the stop is picked up again rather than waiting
|
||||
# for the next natural dispatch, but not on this request's time:
|
||||
# see _resume_sedimentation.
|
||||
_resume_sedimentation(app)
|
||||
return SedimentationState(enabled=store.sedimentation_enabled())
|
||||
|
||||
@app.post(
|
||||
"/api/v1/memory-processing/manual-extract",
|
||||
response_model=ManualExtractReceipt,
|
||||
@@ -3894,12 +4078,8 @@ async def _authorize_answer_generation(
|
||||
# The explicit question is this operation's input, not a standing permission for history.
|
||||
if not question.strip() or not grounding:
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
if not await asyncio.to_thread(_config_store(app).model_content_allowed):
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
|
||||
def check() -> None:
|
||||
if not _config_store(app).model_content_allowed():
|
||||
raise MemoryAnswerError("memory_answer_generation_not_authorized")
|
||||
if _model_profile_store(app).active_identity() != (
|
||||
profile.profile_id,
|
||||
profile.revision,
|
||||
@@ -3923,29 +4103,6 @@ async def _authorize_answer_generation(
|
||||
return recheck
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _permission_context(
|
||||
app: FastAPI,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
secret: str | None,
|
||||
directory: Path,
|
||||
) -> AsyncIterator[MemoryBridgeProtocol]:
|
||||
module = await asyncio.to_thread(import_module, "one_memory.hy_permission_context")
|
||||
budget = cast(ModelRequestBudget | None, app.state.model_request_budget)
|
||||
source_id = lease.records[0].source_id if lease.records else "manual"
|
||||
factory = module.permission_bridge_factory(
|
||||
lambda profile, api_key, data_dir: _memory_processing_supervisor(app)._bridge(
|
||||
profile,
|
||||
api_key,
|
||||
data_dir,
|
||||
scope_key=source_id,
|
||||
),
|
||||
**({"http_transport": BudgetedModelTransport(budget)} if budget else {}),
|
||||
)
|
||||
async with factory(lease, secret, directory) as bridge:
|
||||
yield cast(MemoryBridgeProtocol, bridge)
|
||||
|
||||
|
||||
class _LazyASGIApp:
|
||||
"""Compatibility ASGI app that is built only when a server imports it.
|
||||
|
||||
@@ -3977,6 +4134,16 @@ class _LazyASGIApp:
|
||||
app = _LazyASGIApp()
|
||||
|
||||
|
||||
def _sync_refused(code: str) -> HTTPException:
|
||||
"""A sync refusal that says which stage refused.
|
||||
|
||||
Turning the switch on takes four steps: authorize, prepare, write, resume.
|
||||
The UI used to be told only that the switch had sprung back, and every
|
||||
stage needs a different fix, so each one gets its own code.
|
||||
"""
|
||||
return HTTPException(status_code=409, detail=code)
|
||||
|
||||
|
||||
def _granted_roots(
|
||||
source_ids: Iterable[str],
|
||||
grant_store: SourceGrantStore,
|
||||
@@ -4028,6 +4195,7 @@ def _source_response(
|
||||
totals: tuple[int, int],
|
||||
*,
|
||||
candidate_change_reason: str | None = None,
|
||||
authorization_state: SourceAuthorizationState = SourceAuthorizationState.OK,
|
||||
) -> SourceResponse:
|
||||
location = inspect_source_location(
|
||||
registration.descriptor.source_id, registration.candidate.path
|
||||
@@ -4240,7 +4408,7 @@ async def _onboarding_history_after(
|
||||
home: Path,
|
||||
now: datetime | None = None,
|
||||
) -> datetime | None:
|
||||
"""Return one shared cutoff when any selected harness has large history.
|
||||
"""Return one shared cutoff when any selected harness has large, old history.
|
||||
|
||||
The cutoff is decided before any history job is created, then persisted on
|
||||
every accepted job. This is deliberately global: independently limiting
|
||||
@@ -4249,6 +4417,8 @@ async def _onboarding_history_after(
|
||||
"""
|
||||
if not request.onboarding:
|
||||
return None
|
||||
anchor = now or datetime.now(UTC)
|
||||
cutoff = anchor - ONBOARDING_HISTORY_WINDOW
|
||||
revisions = request.source_candidate_revisions or {}
|
||||
for registration in registrations:
|
||||
source_id = registration.descriptor.source_id
|
||||
@@ -4276,29 +4446,52 @@ async def _onboarding_history_after(
|
||||
home=home,
|
||||
)
|
||||
adapter = registration.create_adapter(grant)
|
||||
count_sessions = cast(
|
||||
Callable[..., Awaitable[int]] | None,
|
||||
getattr(adapter, "count_sessions", None),
|
||||
)
|
||||
if count_sessions is not None:
|
||||
session_count = await count_sessions(limit=ONBOARDING_LARGE_HISTORY_SESSIONS + 1)
|
||||
else:
|
||||
pending = _install_state(adapter)
|
||||
if pending is None:
|
||||
continue
|
||||
session_count = (await pending).session_count
|
||||
if await _has_large_old_history(adapter, cutoff=cutoff):
|
||||
return cutoff
|
||||
except (OSError, SourceAccessError):
|
||||
# A supported source whose metadata count cannot be completed is
|
||||
# treated conservatively. The later connection still owns the
|
||||
# precise authorization error, while onboarding never falls back
|
||||
# to an unbounded history import.
|
||||
session_count = ONBOARDING_LARGE_HISTORY_SESSIONS + 1
|
||||
if session_count > ONBOARDING_LARGE_HISTORY_SESSIONS:
|
||||
anchor = now or datetime.now(UTC)
|
||||
return anchor - ONBOARDING_HISTORY_WINDOW
|
||||
return cutoff
|
||||
return None
|
||||
|
||||
|
||||
async def _has_large_old_history(adapter: SourceAdapter, *, cutoff: datetime) -> bool:
|
||||
"""Probe one authorized Harness with the same parsers the import will use.
|
||||
|
||||
Stop as soon as both policy facts are proven: more than 200 genuine person
|
||||
turns and at least one record older than the three-day cutoff. Counting
|
||||
parsed Evidence instead of transcript files prevents one giant coding
|
||||
session from bypassing the limit, while reusing the adapters keeps host
|
||||
envelopes and tool-shaped user records out of the count.
|
||||
"""
|
||||
|
||||
user_turns = 0
|
||||
has_old_data = False
|
||||
async for event in adapter.scan(ScanOptions()):
|
||||
if isinstance(event, ScanFailure):
|
||||
# A partial probe must never turn into an unbounded first import.
|
||||
return True
|
||||
if not isinstance(event, ScannedEvidence):
|
||||
continue
|
||||
observed = event.evidence.created_at
|
||||
if observed is None and event.cursor.observed_modified_ns > 0:
|
||||
observed = datetime.fromtimestamp(
|
||||
event.cursor.observed_modified_ns / 1_000_000_000,
|
||||
tz=UTC,
|
||||
)
|
||||
if observed is not None:
|
||||
if observed.tzinfo is None:
|
||||
observed = observed.replace(tzinfo=UTC)
|
||||
has_old_data = has_old_data or observed.astimezone(UTC) < cutoff
|
||||
if event.evidence.role == "user":
|
||||
user_turns += 1
|
||||
if user_turns > ONBOARDING_LARGE_HISTORY_USER_TURNS and has_old_data:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _install_state(adapter: object) -> Awaitable[SourceInstallState] | None:
|
||||
"""Ask an adapter for the two-level answer when it has one.
|
||||
|
||||
@@ -4590,7 +4783,7 @@ async def _keep_loop_heartbeat(watchdog: LoopWatchdog) -> None:
|
||||
async def _emit_resource_snapshots(interval_seconds: float = 60.0) -> None:
|
||||
while True:
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="core.resources",
|
||||
event="core.resource_snapshot",
|
||||
outcome="observed",
|
||||
@@ -4615,6 +4808,33 @@ def _start_backfill(app: FastAPI) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _resume_sedimentation(app: FastAPI) -> None:
|
||||
"""Pick the work parked by a stop back up, off the caller's clock.
|
||||
|
||||
Turning the switch back on is a decision, not a job the caller waits for:
|
||||
the choice is already stored, and what follows it is maintenance over every
|
||||
unfinished task followed by a scan of the whole queue. On a real backlog
|
||||
that runs for many minutes, so awaiting it here left the PUT unanswered
|
||||
until the client gave up — the renderer showed "启用未能确认" for a switch
|
||||
that had in fact been saved, and the switch only appeared to turn on once
|
||||
the follow-up read came back. Measured: one toggle, twenty-plus minutes,
|
||||
no response, client timeout at thirty seconds.
|
||||
|
||||
Repeated toggles reuse the running walk; it reads the switch as it goes, so
|
||||
a stop part-way through is honoured on the next batch instead of needing a
|
||||
cancel.
|
||||
"""
|
||||
running = cast("asyncio.Task[None] | None", getattr(app.state, "sedimentation_resume", None))
|
||||
if running is not None and not running.done():
|
||||
return
|
||||
app.state.sedimentation_resume = watched_task(
|
||||
asyncio.get_running_loop(),
|
||||
lambda: _memory_processing_supervisor(app).resume_allowed(),
|
||||
component="memory.sedimentation",
|
||||
name="one-memory-sedimentation-resume",
|
||||
)
|
||||
|
||||
|
||||
def _start_visibility_recompute(app: FastAPI) -> None:
|
||||
"""Keep the stored visibility conclusions current, in the background.
|
||||
|
||||
@@ -4723,8 +4943,6 @@ async def _run_backfill(app: FastAPI) -> None:
|
||||
app.state.memory_backfill_error = None
|
||||
if not app.state.settings.automatic_memory_processing:
|
||||
return
|
||||
if not await asyncio.to_thread(_config_store(app).model_content_allowed):
|
||||
return
|
||||
try:
|
||||
queue = app.state.memory_write_queue
|
||||
intake = app.state.memory_write_intake
|
||||
@@ -4860,6 +5078,52 @@ def _source_writes_on(app: FastAPI, source_id: str) -> bool:
|
||||
return _source_processing_enabled(app, source_id)
|
||||
|
||||
|
||||
def _sedimentation_claim_enabled(app: FastAPI, source_id: str) -> bool:
|
||||
"""Claim queued model work only when policy, source and model all admit it."""
|
||||
|
||||
return (
|
||||
_config_store(app).sedimentation_enabled()
|
||||
and _model_profile_store(app).active_identity() is not None
|
||||
and _history_import_gate_released(app)
|
||||
and _source_writes_on(app, source_id)
|
||||
)
|
||||
|
||||
|
||||
def _history_import_gate_released(app: FastAPI) -> bool:
|
||||
"""Hold first sedimentation until every accepted history import is terminal."""
|
||||
|
||||
if getattr(app.state, "history_import_gate_released", False):
|
||||
return True
|
||||
onboarding = getattr(app.state, "onboarding_store", None)
|
||||
jobs = getattr(app.state, "import_job_store", None)
|
||||
if onboarding is None or jobs is None:
|
||||
return True
|
||||
state = onboarding.get()
|
||||
if state.status != "importing" or not state.selected_source_ids:
|
||||
app.state.history_import_gate_released = True
|
||||
return True
|
||||
selected = set(state.selected_source_ids)
|
||||
matching = [
|
||||
job
|
||||
for job in jobs.list_jobs()
|
||||
if job.operation == "history"
|
||||
and job.source_id in selected
|
||||
and (state.started_at is None or job.created_at >= state.started_at)
|
||||
]
|
||||
if {job.source_id for job in matching} != selected:
|
||||
return False
|
||||
active = {
|
||||
ImportStatus.PENDING,
|
||||
ImportStatus.RUNNING,
|
||||
ImportStatus.PAUSED,
|
||||
ImportStatus.INTERRUPTED,
|
||||
}
|
||||
released = all(job.status not in active for job in matching)
|
||||
if released:
|
||||
app.state.history_import_gate_released = True
|
||||
return released
|
||||
|
||||
|
||||
def _model_prober(app: FastAPI) -> ModelProber:
|
||||
return cast(ModelProber, app.state.model_prober)
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import hmac
|
||||
import ipaddress
|
||||
from collections.abc import Callable
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
@@ -35,7 +36,15 @@ def is_loopback_origin(origin: str) -> bool:
|
||||
host = parsed.hostname
|
||||
if not host:
|
||||
return False
|
||||
return host in LOOPBACK_HOSTNAMES or host.startswith("127.")
|
||||
if host in LOOPBACK_HOSTNAMES:
|
||||
return True
|
||||
# A prefix check is not enough: "127.attacker.example" is a perfectly valid
|
||||
# DNS name that resolves wherever its owner wants (including 127.0.0.1 via
|
||||
# DNS rebinding). Parse the address and ask the standard library.
|
||||
try:
|
||||
return ipaddress.ip_address(host).is_loopback
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def local_api_guard(
|
||||
|
||||
@@ -175,6 +175,22 @@ async def in_interactive_read[T](
|
||||
return await in_read_pool(interactive_read_pool(), work, *args, **kwargs)
|
||||
|
||||
|
||||
def _retrieve[T](future: "asyncio.Future[T]") -> None:
|
||||
"""Mark a shielded read's outcome as seen.
|
||||
|
||||
`in_read_pool` shields the wrapped worker, so a caller that gives up and
|
||||
cancels leaves the read running to completion. When that read then fails,
|
||||
nothing awaits the future any more: asyncio reports the orphan at GC as
|
||||
"Future exception was never retrieved", the loop handler turns it into a
|
||||
`core.loop.task_exception` error diagnostic, and a lane that degraded on
|
||||
its own budget -- correctly, and already reported -- looks like a crashed
|
||||
task on top of it. Retrieving here keeps the diagnostics honest. Awaiting
|
||||
still raises for callers that are still there, so no failure is hidden.
|
||||
"""
|
||||
if not future.cancelled():
|
||||
future.exception()
|
||||
|
||||
|
||||
async def in_read_pool[T](
|
||||
pool: ThreadPoolExecutor,
|
||||
work: Callable[..., T],
|
||||
@@ -193,7 +209,9 @@ async def in_read_pool[T](
|
||||
_interactive_reads -= 1
|
||||
|
||||
worker.add_done_callback(released)
|
||||
return await asyncio.shield(asyncio.wrap_future(worker))
|
||||
pending = asyncio.wrap_future(worker)
|
||||
pending.add_done_callback(_retrieve)
|
||||
return await asyncio.shield(pending)
|
||||
|
||||
|
||||
async def wait_for_interactive_reads() -> None:
|
||||
|
||||
@@ -76,8 +76,7 @@ def _memory(db: sqlite3.Connection, reference: CallReference) -> SourceRecordRef
|
||||
if row["status"] == "deleted" or row["deleted_at"] is not None:
|
||||
return _hidden(reference)
|
||||
|
||||
# Same local revocation/ownership predicate as SourceReadAccess.memories,
|
||||
# evaluated BEFORE freshness (which can read support text), in this snapshot.
|
||||
# Same local revocation/ownership predicate as SourceReadAccess.memories.
|
||||
# Its public methods open their own connections, so cannot be used here.
|
||||
access = db.execute(
|
||||
"SELECT EXISTS (SELECT 1 FROM governed_memory_evidence "
|
||||
@@ -121,8 +120,8 @@ def resolve_call_references(database_path: Path, event: CallEvent) -> list[Sourc
|
||||
|
||||
Local-owner only: event host attribution is NOT authorization. Explicit local
|
||||
source revocations deny access; absent grants retain legacy local semantics.
|
||||
A memory needs at least one accessible active citation. No proof preserves
|
||||
shared freshness compatibility, not a claim of verified historical freshness.
|
||||
A memory needs at least one accessible active citation. Evidence freshness
|
||||
proofs are retired and do not affect availability.
|
||||
|
||||
Any read/parse/budget failure raises a fixed whole-request error, never a
|
||||
partial result or per-item deletion. The route owner must map this separately
|
||||
|
||||
@@ -266,7 +266,13 @@ def serve(*, maintenance: bool = False, reader: bool = False) -> None:
|
||||
)
|
||||
application = create_reader_app(reader_settings)
|
||||
print(f"reader_port:{port}", file=sys.stderr, flush=True)
|
||||
uvicorn.run(application, host=settings.host, port=port, reload=False)
|
||||
uvicorn.run(
|
||||
application,
|
||||
host=settings.host,
|
||||
port=port,
|
||||
reload=False,
|
||||
access_log=False,
|
||||
)
|
||||
return
|
||||
|
||||
from one_memory.api import create_app
|
||||
@@ -297,9 +303,21 @@ def serve(*, maintenance: bool = False, reader: bool = False) -> None:
|
||||
)
|
||||
if maintenance:
|
||||
with CoreRuntime(settings, publish_receipt=False):
|
||||
uvicorn.run(application, host=settings.host, port=settings.port, reload=False)
|
||||
uvicorn.run(
|
||||
application,
|
||||
host=settings.host,
|
||||
port=settings.port,
|
||||
reload=False,
|
||||
access_log=False,
|
||||
)
|
||||
else:
|
||||
uvicorn.run(application, host=settings.host, port=settings.port, reload=False)
|
||||
uvicorn.run(
|
||||
application,
|
||||
host=settings.host,
|
||||
port=settings.port,
|
||||
reload=False,
|
||||
access_log=False,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -196,7 +196,7 @@ def read_raw_context(
|
||||
if _prefers_declared_facts(request.query)
|
||||
else "1"
|
||||
)
|
||||
person_authored = "CASE WHEN e.written_by='person' THEN 1 ELSE 0 END"
|
||||
person_authored = "CASE WHEN e.role='user' THEN 1 ELSE 0 END"
|
||||
query_echo = (
|
||||
"CASE WHEN trim(e.verbatim)=? OR trim(e.verbatim)=? THEN 0 ELSE 1 END"
|
||||
if _prefers_declared_facts(request.query)
|
||||
@@ -470,9 +470,8 @@ def revalidate_raw_context(
|
||||
)
|
||||
except (ValueError, TypeError):
|
||||
break
|
||||
if (row["role"], row["written_by"], created) != (
|
||||
if (row["role"], created) != (
|
||||
message.role,
|
||||
message.written_by,
|
||||
message.created_at,
|
||||
):
|
||||
break
|
||||
|
||||
@@ -66,12 +66,12 @@ def _field(text: str) -> ResumeField | None:
|
||||
return None
|
||||
|
||||
|
||||
def _raw_class(written_by: str, evidence_id: str, accepted: frozenset[str]) -> EvidenceClass:
|
||||
if written_by == "person":
|
||||
def _raw_class(role: str, evidence_id: str, accepted: frozenset[str]) -> EvidenceClass:
|
||||
if role == "user":
|
||||
return "person_statement"
|
||||
if written_by == "agent":
|
||||
if role == "assistant":
|
||||
return "agent_report"
|
||||
if written_by == "tool" and evidence_id in accepted:
|
||||
if role == "tool" and evidence_id in accepted:
|
||||
return "system_acceptance"
|
||||
return "tool_output"
|
||||
|
||||
@@ -118,7 +118,7 @@ def assemble_resume_packet(
|
||||
field=_field(message.text) or "unclassified",
|
||||
text=message.text,
|
||||
evidence_class=_raw_class(
|
||||
message.written_by,
|
||||
message.role,
|
||||
message.evidence_id,
|
||||
verified_system_evidence_ids,
|
||||
),
|
||||
|
||||
@@ -345,7 +345,7 @@ def _lexical_raw_rows(
|
||||
if context_raw._prefers_declared_facts(query.query)
|
||||
else "1"
|
||||
)
|
||||
person_authored = "CASE WHEN e.written_by='person' THEN 1 ELSE 0 END"
|
||||
person_authored = "CASE WHEN e.role='user' THEN 1 ELSE 0 END"
|
||||
query_echo = (
|
||||
"CASE WHEN trim(e.verbatim)=? OR trim(e.verbatim)=? THEN 0 ELSE 1 END"
|
||||
if context_raw._prefers_declared_facts(query.query)
|
||||
@@ -507,7 +507,7 @@ def _raw_item(row: sqlite3.Row, start1: int, matched_by: tuple[str, ...]) -> Raw
|
||||
"source_id": row["source_id"],
|
||||
"conversation_id": row["conversation_id"],
|
||||
"role": row["role"],
|
||||
"written_by": "person" if row["written_by"] == "person" else "agent",
|
||||
"written_by": "person" if row["role"] == "user" else "agent",
|
||||
"event_time": event_time.isoformat() if event_time else None,
|
||||
"spans": [{"start_cp": start, "end_cp": start + len(text), "text": text}],
|
||||
"matched_by": list(matched_by),
|
||||
|
||||
@@ -1039,7 +1039,7 @@ def close_main_databases() -> None:
|
||||
def log_main_database_stats(moment: str) -> None:
|
||||
"""One line in core.log: how many connections are open and who waited."""
|
||||
stats = main_database_stats()
|
||||
logger.info(
|
||||
logger.debug(
|
||||
"main_database_connections moment=%s %s",
|
||||
moment,
|
||||
" ".join(f"{key}={value}" for key, value in stats.items()),
|
||||
|
||||
@@ -19,6 +19,7 @@ from __future__ import annotations
|
||||
|
||||
import atexit
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
@@ -139,6 +140,10 @@ _DETAIL_KEYS = frozenset(
|
||||
"accounted_ms",
|
||||
"unaccounted_ms",
|
||||
"conversation_hash",
|
||||
# The sedimentation switch: a bounded boolean, the same shape as
|
||||
# `spawned` and `index_complete`. Left out of this list it did not
|
||||
# drop a log line, it turned the switch's own request into a 500.
|
||||
"enabled",
|
||||
}
|
||||
)
|
||||
_trace_id: ContextVar[str | None] = ContextVar("one_memory_trace_id", default=None)
|
||||
@@ -649,12 +654,24 @@ def diagnostic_event(**values: object) -> None:
|
||||
active = _active
|
||||
if active is None:
|
||||
return
|
||||
if values.get("level") == "debug" and not logging.getLogger().isEnabledFor(
|
||||
logging.DEBUG
|
||||
):
|
||||
return
|
||||
try:
|
||||
active.write(**values) # type: ignore[arg-type]
|
||||
except OSError:
|
||||
# Diagnostics must not become a new product outage when a disk is full
|
||||
# or the log directory is temporarily unavailable.
|
||||
return
|
||||
except ValueError:
|
||||
# Same promise, same reason, for a rejected payload instead of a failed
|
||||
# write. An unlisted `details` key once turned every external recall
|
||||
# into a 500 and then killed the process, and the sedimentation switch
|
||||
# into a permanent failure. The payload itself is still wrong and is
|
||||
# still caught in CI by tests/test_diagnostic_payload_keys.py; what
|
||||
# must not happen is the caller paying for it.
|
||||
return
|
||||
|
||||
|
||||
def current_session_id() -> str | None:
|
||||
|
||||
@@ -21,7 +21,6 @@ from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception, elapsed_ms
|
||||
from one_memory.hy_lifecycle import HyLifecycle, HyLifecycleUnreadable, read_hy_lifecycle
|
||||
from one_memory.hy_permission_store import native_store_read_slot
|
||||
from one_memory.hy_presence import HyPresence, confirm_deleted
|
||||
from one_memory.hy_runtime_config import (
|
||||
HY_MEMORY_MODE,
|
||||
@@ -33,6 +32,7 @@ from one_memory.hy_runtime_config import (
|
||||
)
|
||||
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
|
||||
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind, attach_disable_thinking
|
||||
from one_memory.model_request_audit import audit_precipitation_request
|
||||
from one_memory.semantic_deadline import SemanticMemoryHit
|
||||
|
||||
#: D102: how relevant HY has to call a hit before it counts. Raised from 0.25
|
||||
@@ -143,6 +143,22 @@ class HyMemoryBridgeError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES = 512
|
||||
|
||||
|
||||
def coding_tool_result_max_bytes() -> int:
|
||||
"""Return the per-tool payload ceiling used by MemHub's HY client."""
|
||||
|
||||
raw = os.environ.get("MEMORY_CODING_TOOL_RESULT_MAX_BYTES")
|
||||
if raw is None:
|
||||
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError:
|
||||
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
return value if value > 0 else MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
|
||||
|
||||
|
||||
def _exclusive[**P, T](
|
||||
method: Callable[P, Coroutine[Any, Any, T]],
|
||||
) -> Callable[P, Coroutine[Any, Any, T]]:
|
||||
@@ -376,11 +392,13 @@ class HyMemoryBridge:
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
client: HyMemoryClientProtocol | None = None,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
) -> None:
|
||||
self.profile = profile
|
||||
self.api_key = api_key
|
||||
self.data_dir = data_dir
|
||||
self._client = client
|
||||
self._http_transport_factory = http_transport_factory
|
||||
self._client_lock = asyncio.Lock()
|
||||
self._write_lock = asyncio.Lock()
|
||||
self._last_bindings: HyWriteBindings | None = None
|
||||
@@ -412,8 +430,9 @@ class HyMemoryBridge:
|
||||
|
||||
@_exclusive
|
||||
async def ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
|
||||
async with self._write_lock:
|
||||
return await self._ingest(request)
|
||||
with audit_precipitation_request(request.request_id):
|
||||
async with self._write_lock:
|
||||
return await self._ingest(request)
|
||||
|
||||
async def _ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
|
||||
started = monotonic()
|
||||
@@ -427,6 +446,11 @@ class HyMemoryBridge:
|
||||
)
|
||||
client = await self._get_client()
|
||||
guards = getattr(client, "_one_memory_write_guards", ())
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
reconcile_guards = tuple(guard for guard in guards if isinstance(guard, HyReconcileGuard))
|
||||
for guard in reconcile_guards:
|
||||
guard.begin_write()
|
||||
before = [len(guard.failures) for guard in guards]
|
||||
bindings = HyWriteBindings(request.source_context) if request.source_context else None
|
||||
if bindings:
|
||||
@@ -500,6 +524,12 @@ class HyMemoryBridge:
|
||||
if len(guard.failures) > count:
|
||||
raise HyMemoryBridgeError(guard.failures[count])
|
||||
result = await self._result(client, request, raw, expected_contents=None, bindings=bindings)
|
||||
result = await _with_native_mutation_delta(
|
||||
client,
|
||||
result,
|
||||
raw,
|
||||
reconcile_guards,
|
||||
)
|
||||
if bindings:
|
||||
outputs = []
|
||||
for memory in result.memories:
|
||||
@@ -529,8 +559,9 @@ class HyMemoryBridge:
|
||||
request: HyMemoryIngestRequest,
|
||||
contents: Sequence[str],
|
||||
) -> HyMemoryIngestResult:
|
||||
async with self._write_lock:
|
||||
return await self._ingest_extracted(request, contents)
|
||||
with audit_precipitation_request(request.request_id):
|
||||
async with self._write_lock:
|
||||
return await self._ingest_extracted(request, contents)
|
||||
|
||||
async def _ingest_extracted(
|
||||
self,
|
||||
@@ -553,6 +584,8 @@ class HyMemoryBridge:
|
||||
from one_memory.hy_response_guard import HyReconcileGuard
|
||||
|
||||
report_guards = tuple(guard for guard in guards if isinstance(guard, HyReconcileGuard))
|
||||
for guard in report_guards:
|
||||
guard.begin_write()
|
||||
expected_contents = frozenset(content.strip() for content in contents if content.strip())
|
||||
for guard in report_guards:
|
||||
guard.imported_contents = expected_contents
|
||||
@@ -599,6 +632,12 @@ class HyMemoryBridge:
|
||||
raw,
|
||||
expected_contents=expected_contents,
|
||||
)
|
||||
result = await _with_native_mutation_delta(
|
||||
client,
|
||||
result,
|
||||
raw,
|
||||
report_guards,
|
||||
)
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
component="hy.bridge",
|
||||
@@ -944,27 +983,29 @@ class HyMemoryBridge:
|
||||
caller could only rank by position — which forced semantic hits to be
|
||||
placed as a block ahead of everything the keyword lane found. The
|
||||
score is what lets the two lanes be ranked against each other.
|
||||
|
||||
Native search runs without a process-local write gate. MemHub expects
|
||||
concurrent read during precipitation; store work is a short burst after
|
||||
network-bound embed/LLM steps, and refusing reads with
|
||||
``hy_memory_store_busy`` only pushed callers onto weaker fallbacks.
|
||||
"""
|
||||
async with native_store_read_slot() as admitted:
|
||||
if not admitted:
|
||||
raise HyMemoryBridgeError("hy_memory_store_busy")
|
||||
client = await self._get_client()
|
||||
try:
|
||||
result = await _joined_thread(
|
||||
client.search,
|
||||
query,
|
||||
user_ids=[user_id],
|
||||
agent_ids=[] if agent_id == "*" else [agent_id],
|
||||
limit=limit,
|
||||
min_score=hy_relevance_floor(),
|
||||
schema_min_score=hy_relevance_floor(),
|
||||
schema_limit=limit,
|
||||
intention_limit=limit,
|
||||
scene="normal",
|
||||
reader=HY_MEMORY_READER,
|
||||
)
|
||||
except Exception as error:
|
||||
raise HyMemoryBridgeError("hy_memory_search_failed") from error
|
||||
client = await self._get_client()
|
||||
try:
|
||||
result = await _joined_thread(
|
||||
client.search,
|
||||
query,
|
||||
user_ids=[user_id],
|
||||
agent_ids=[] if agent_id == "*" else [agent_id],
|
||||
limit=limit,
|
||||
min_score=hy_relevance_floor(),
|
||||
schema_min_score=hy_relevance_floor(),
|
||||
schema_limit=limit,
|
||||
intention_limit=limit,
|
||||
scene="normal",
|
||||
reader=HY_MEMORY_READER,
|
||||
)
|
||||
except Exception as error:
|
||||
raise HyMemoryBridgeError("hy_memory_search_failed") from error
|
||||
hits = _replied_memories(result, "hy_memory_search_shape_unknown")
|
||||
found: list[SemanticMemoryHit] = []
|
||||
seen: set[str] = set()
|
||||
@@ -1087,6 +1128,7 @@ class HyMemoryBridge:
|
||||
self.profile,
|
||||
api_key=self.api_key,
|
||||
data_dir=self.data_dir,
|
||||
http_transport_factory=self._http_transport_factory,
|
||||
)
|
||||
|
||||
await _joined_thread(initialize)
|
||||
@@ -1113,7 +1155,7 @@ class HyMemoryBridge:
|
||||
)
|
||||
raise HyMemoryBridgeError("hy_memory_init_failed") from error
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="hy.client",
|
||||
event="hy.client_initialization.completed",
|
||||
outcome="succeeded",
|
||||
@@ -1136,6 +1178,7 @@ def create_hy_memory_client(
|
||||
*,
|
||||
api_key: str | None,
|
||||
data_dir: Path,
|
||||
http_transport_factory: Callable[[], Any] | None = None,
|
||||
) -> HyMemoryClientProtocol:
|
||||
configure_hy_memory_environment(data_dir)
|
||||
config = build_hy_memory_config(
|
||||
@@ -1143,6 +1186,7 @@ def create_hy_memory_client(
|
||||
api_key=api_key,
|
||||
data_dir=data_dir,
|
||||
)
|
||||
config.http_transport_factory = http_transport_factory
|
||||
reattach_legacy_vector_collection(profile, config)
|
||||
mismatch = inspect_graph_width_mismatch(config)
|
||||
if mismatch is not None:
|
||||
@@ -1287,6 +1331,72 @@ async def _coding_result(
|
||||
return HyMemoryIngestResult(success=True, memories=tuple(outputs))
|
||||
|
||||
|
||||
async def _with_native_mutation_delta(
|
||||
client: HyMemoryClientProtocol,
|
||||
result: HyMemoryIngestResult,
|
||||
raw: Mapping[str, object],
|
||||
reconcile_guards: Sequence[Any],
|
||||
) -> HyMemoryIngestResult:
|
||||
"""Map native SDK mutations back into MemHub governance.
|
||||
|
||||
Store permission wrappers used to discover this by intercepting every
|
||||
native CRUD call. The embedded SDK now owns unrestricted local CRUD, so
|
||||
chat targets come from the reconcile observer and coding targets come from
|
||||
the SDK's public result document.
|
||||
"""
|
||||
|
||||
retired = set(result.retired_memory_ids)
|
||||
superseded = {key: tuple(values) for key, values in result.superseded_memory_ids.items()}
|
||||
chat_targets = {
|
||||
key
|
||||
for guard in reconcile_guards
|
||||
for key in getattr(guard, "mutation_target_ids", ())
|
||||
if isinstance(key, str) and key
|
||||
}
|
||||
|
||||
operations = raw.get("ops")
|
||||
for item in operations if isinstance(operations, list) else ():
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
if item.get("action") == "DELETE":
|
||||
key = item.get("target_experience_id")
|
||||
if isinstance(key, str) and key:
|
||||
retired.add(key)
|
||||
|
||||
fact_operations = raw.get("fact_ops")
|
||||
for item in fact_operations if isinstance(fact_operations, list) else ():
|
||||
if not isinstance(item, Mapping) or item.get("action") != "SUPERSEDE":
|
||||
continue
|
||||
old, new = item.get("fact_id"), item.get("superseded_by")
|
||||
if isinstance(old, str) and old:
|
||||
retired.add(old)
|
||||
if isinstance(new, str) and new:
|
||||
superseded[old] = (new,)
|
||||
|
||||
for key in chat_targets:
|
||||
record = await _joined_thread(_read_hy_record, client, key)
|
||||
if record is None:
|
||||
retired.add(key)
|
||||
continue
|
||||
status = str(record.get("status", "")).lower()
|
||||
if status in {"deleted", "shadow", "superseded"}:
|
||||
retired.add(key)
|
||||
replacement = record.get("superseded_by")
|
||||
if isinstance(replacement, str) and replacement:
|
||||
superseded[key] = (replacement,)
|
||||
elif isinstance(replacement, list):
|
||||
values = tuple(value for value in replacement if isinstance(value, str) and value)
|
||||
if values:
|
||||
superseded[key] = values
|
||||
|
||||
return result.model_copy(
|
||||
update={
|
||||
"retired_memory_ids": tuple(sorted(retired)),
|
||||
"superseded_memory_ids": superseded,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _read_hy_record(
|
||||
client: HyMemoryClientProtocol,
|
||||
memory_id: str,
|
||||
@@ -1603,6 +1713,7 @@ def build_hy_memory_config(
|
||||
config.history.db_path = str(memory_root / "history.sqlite3")
|
||||
config.coding.enable = True
|
||||
config.coding.db_path = str(memory_root / "coding.sqlite3")
|
||||
config.coding.tool_result_max_bytes = coding_tool_result_max_bytes()
|
||||
config.recall.episode_recall_enabled = True
|
||||
config.recall.episode_summary_enabled = False
|
||||
config.recall.dynamic_episode_enabled = True
|
||||
|
||||
@@ -1,607 +0,0 @@
|
||||
"""Operation-bound read/write capability for the pinned native HY coding store.
|
||||
|
||||
HY's coding route writes two layers, experiences and working facts. Rows this
|
||||
lease cannot attest stay in the same backend but are invisible here, so they
|
||||
never reach a coding prompt and cannot be updated, superseded or linked.
|
||||
|
||||
HY's reconcilers catch store exceptions and carry on, so a denial latches and
|
||||
recheck() raises it again at finish; a swallowed denial still fails the task.
|
||||
Any native method not implemented here is denied rather than passed through.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from collections.abc import Iterator, Mapping, Sequence
|
||||
from copy import deepcopy
|
||||
from hashlib import sha256
|
||||
from types import MappingProxyType
|
||||
from typing import Any, NoReturn
|
||||
|
||||
from hy_memory.coding.types import CodingMemory, WorkingFact # type: ignore[import-untyped]
|
||||
|
||||
from one_memory.hy_permission_store import HyNodePermission
|
||||
from one_memory.model_processing_permission_guard import (
|
||||
ModelProcessingPermissionLease,
|
||||
permission_error,
|
||||
)
|
||||
|
||||
_NATIVE_WRITES = frozenset(
|
||||
{
|
||||
"insert",
|
||||
"update",
|
||||
"delete",
|
||||
"insert_fact",
|
||||
"supersede_fact",
|
||||
"set_fact_derived_experiences",
|
||||
"set_experience_source_facts",
|
||||
# Trace rows are raw conversation: writes must be gated like any other.
|
||||
"append_trace",
|
||||
}
|
||||
)
|
||||
|
||||
_TIMESTAMPS = frozenset({"created_at", "updated_at"})
|
||||
|
||||
# SDK 2.0.2 reads `config` off the coding store while routing a coding
|
||||
# turn. The wrapper must answer it, but the native `MemoryConfig` carries
|
||||
# credentials (llm/embedder api keys, graph and cache passwords) and is a live
|
||||
# handle into the backend, so it is never handed out. Only this redacted
|
||||
# projection is exposed: credential-bearing keys collapse to None.
|
||||
_CONFIG_REDACTED_KEYS = frozenset(
|
||||
{
|
||||
"api_key",
|
||||
"password",
|
||||
"username",
|
||||
"mysql_password",
|
||||
"mysql_user",
|
||||
"extra_headers",
|
||||
"extra_body",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class _RedactedConfig(Mapping[str, Any]):
|
||||
"""Read-only config view: readable by key and attribute, never the native one.
|
||||
|
||||
The SDK reaches config values both ways, so this keeps attribute access
|
||||
working. It is deliberately not the native MemoryConfig: it exposes only
|
||||
plain copied values, so no backend handle, connection or callable escapes.
|
||||
"""
|
||||
|
||||
__slots__ = ("_values",)
|
||||
|
||||
def __init__(self, values: Mapping[str, Any]) -> None:
|
||||
object.__setattr__(self, "_values", MappingProxyType(dict(values)))
|
||||
|
||||
def __getitem__(self, key: str) -> Any:
|
||||
return self._values[key]
|
||||
|
||||
def __iter__(self) -> Iterator[str]:
|
||||
return iter(self._values)
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self._values)
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
values = object.__getattribute__(self, "_values")
|
||||
try:
|
||||
return values[name]
|
||||
except KeyError:
|
||||
raise AttributeError(name) from None
|
||||
|
||||
def __setattr__(self, name: str, value: Any) -> NoReturn:
|
||||
raise AttributeError("redacted config is read-only")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"_RedactedConfig({sorted(self._values)})"
|
||||
|
||||
|
||||
def _redact_config(value: Any, depth: int = 0) -> Any:
|
||||
"""Copy config data into read-only views, collapsing credentials to None.
|
||||
|
||||
The SDK reads this both by attribute and by key, so each config section
|
||||
becomes a mapping-like view rather than a plain dict: attribute access stays
|
||||
available while the native object itself is never handed out.
|
||||
"""
|
||||
if depth > 4:
|
||||
return None
|
||||
if value is None or isinstance(value, (str, int, float, bool)):
|
||||
return value
|
||||
if isinstance(value, Mapping):
|
||||
return _RedactedConfig(
|
||||
{
|
||||
key: None if key in _CONFIG_REDACTED_KEYS else _redact_config(item, depth + 1)
|
||||
for key, item in value.items()
|
||||
}
|
||||
)
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [_redact_config(item, depth + 1) for item in value]
|
||||
if hasattr(value, "__dict__") and type(value).__module__.startswith("hy_memory"):
|
||||
return _redact_config(vars(value), depth + 1)
|
||||
return None
|
||||
|
||||
|
||||
def _config_projection(native: Any) -> Any:
|
||||
try:
|
||||
return _redact_config(native.config)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def hy_coding_fingerprint(row: CodingMemory | WorkingFact) -> str:
|
||||
"""Hash the complete native row as read back, including links and timestamps."""
|
||||
if type(row) is CodingMemory:
|
||||
kind = "experience"
|
||||
elif type(row) is WorkingFact:
|
||||
kind = "fact"
|
||||
else:
|
||||
raise permission_error("hy_coding_store_row_shape")
|
||||
payload = {"kind": kind, **row.to_dict()}
|
||||
return sha256(
|
||||
json.dumps(payload, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _row_id(row: CodingMemory | WorkingFact) -> str:
|
||||
return str(row.experience_id if type(row) is CodingMemory else row.fact_id)
|
||||
|
||||
|
||||
class HyPermissionCodingStore:
|
||||
"""Closed, task-exclusive adapter; initialize/close never own the native store.
|
||||
|
||||
node_permissions is the same attested ledger the chat vector store uses;
|
||||
coding rows are keyed by experience or fact id. written_nodes carries the
|
||||
whole lease footprint for each row written here, as the chat store does.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
native: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
*,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
node_permissions: Mapping[str, HyNodePermission] | None = None,
|
||||
) -> None:
|
||||
self._native = native
|
||||
self._lease = lease
|
||||
self._agent_id = agent_id
|
||||
self._session_id = session_id
|
||||
self._user_id = lease.user_id
|
||||
self._failure: str | None = None
|
||||
self._closed = False
|
||||
self._lock = asyncio.Lock()
|
||||
self._deleted: set[str] = set()
|
||||
self._written: dict[str, HyNodePermission] = {}
|
||||
self._superseded: dict[str, str] = {}
|
||||
self._evidence_ids = tuple(record.evidence_id for record in lease.records)
|
||||
self._permissions = {
|
||||
key: proof
|
||||
for key, proof in (node_permissions or {}).items()
|
||||
if set(proof.evidence_ids) <= set(self._evidence_ids)
|
||||
}
|
||||
self.recheck()
|
||||
if not agent_id or agent_id == "*" or not session_id or not self._evidence_ids:
|
||||
self._deny("scope")
|
||||
|
||||
def __getattr__(self, name: str) -> NoReturn:
|
||||
if name.startswith("__"):
|
||||
raise AttributeError(name)
|
||||
self._deny("unsupported_operation")
|
||||
|
||||
@property
|
||||
def written_nodes(self) -> Mapping[str, HyNodePermission]:
|
||||
self.recheck()
|
||||
return MappingProxyType(dict(self._written))
|
||||
|
||||
@property
|
||||
def config(self) -> Any:
|
||||
"""Read-only, redacted view of the native coding config.
|
||||
|
||||
SDK 2.0.2 reads `config` when it routes a coding turn. Answering
|
||||
it is safe only as a projection: the native MemoryConfig is a live
|
||||
backend handle and carries credentials, so callers get plain values
|
||||
with every credential-bearing key collapsed to None. This carries no
|
||||
identity, permission or tenant state and is not the native object.
|
||||
"""
|
||||
self._assert_open()
|
||||
return _config_projection(self._native)
|
||||
|
||||
@property
|
||||
def retired_ids(self) -> frozenset[str]:
|
||||
"""Experiences HY deleted in this operation."""
|
||||
self.recheck()
|
||||
return frozenset(self._deleted)
|
||||
|
||||
@property
|
||||
def superseded_ids(self) -> Mapping[str, tuple[str, ...]]:
|
||||
"""Facts HY superseded in this operation, with the fact that replaced each.
|
||||
|
||||
A successor linked before it existed must have been written here by
|
||||
now; otherwise the old fact points at a row nobody attested.
|
||||
"""
|
||||
self.recheck()
|
||||
if any(new not in self._written for new in self._superseded.values()):
|
||||
self._deny("link_denied")
|
||||
return MappingProxyType({old: (new,) for old, new in self._superseded.items()})
|
||||
|
||||
def recheck(self) -> None:
|
||||
self._assert_open()
|
||||
try:
|
||||
self._lease.recheck()
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
self._failure = (
|
||||
code
|
||||
if code.startswith("model_content_") and len(code) <= 100
|
||||
else "model_content_hy_coding_store_lease_failed"
|
||||
)
|
||||
raise permission_error(self._failure) from None
|
||||
|
||||
def _assert_open(self) -> None:
|
||||
if self._closed:
|
||||
self._deny("closed")
|
||||
if self._failure:
|
||||
raise permission_error(self._failure)
|
||||
|
||||
async def close(self) -> None:
|
||||
async with self._lock:
|
||||
self._closed = True
|
||||
self._permissions.clear()
|
||||
self._written.clear()
|
||||
|
||||
def _deny(self, suffix: str) -> NoReturn:
|
||||
self._failure = self._failure or f"model_content_hy_coding_store_{suffix}"
|
||||
raise permission_error(self._failure)
|
||||
|
||||
def _scope(self, user_id: str | None) -> None:
|
||||
if user_id not in (None, self._user_id):
|
||||
self._deny("scope")
|
||||
|
||||
async def _invoke(self, method: str, *args: Any, **kwargs: Any) -> Any:
|
||||
writing = method in _NATIVE_WRITES
|
||||
self.recheck() if writing else self._assert_open()
|
||||
try:
|
||||
result = await getattr(self._native, method)(*args, **kwargs)
|
||||
except Exception:
|
||||
self._deny("backend_failed")
|
||||
self.recheck() if writing else self._assert_open()
|
||||
return result
|
||||
|
||||
def _visible(self, row: Any, expected: Any) -> bool:
|
||||
if row is None:
|
||||
return False
|
||||
if type(row) is not expected:
|
||||
self._deny("row_shape")
|
||||
key = _row_id(row)
|
||||
proof = self._permissions.get(key)
|
||||
if proof is None or key in self._deleted or row.user_id != self._user_id:
|
||||
return False
|
||||
if hy_coding_fingerprint(row) != proof.fingerprint:
|
||||
self._deny("row_changed")
|
||||
return True
|
||||
|
||||
async def _experience(self, key: str) -> CodingMemory | None:
|
||||
if key not in self._permissions or key in self._deleted:
|
||||
return None
|
||||
row = await self._invoke("get_by_id", key, user_id=self._user_id)
|
||||
return deepcopy(row) if self._visible(row, CodingMemory) else None
|
||||
|
||||
async def _fact(self, key: str) -> WorkingFact | None:
|
||||
if key not in self._permissions or key in self._deleted:
|
||||
return None
|
||||
rows = await self._invoke("get_facts", [key], user_id=self._user_id)
|
||||
row = rows[0] if rows else None
|
||||
return deepcopy(row) if self._visible(row, WorkingFact) else None
|
||||
|
||||
def _new_row_in_scope(self, row: CodingMemory | WorkingFact) -> bool:
|
||||
return bool(
|
||||
row.user_id == self._user_id
|
||||
and row.agent_id == self._agent_id
|
||||
and row.session_id == self._session_id
|
||||
)
|
||||
|
||||
async def _capture(
|
||||
self, key: str, kind: Any, expected: CodingMemory | WorkingFact | None = None
|
||||
) -> None:
|
||||
row: Any
|
||||
if kind is CodingMemory:
|
||||
row = await self._invoke("get_by_id", key, user_id=self._user_id)
|
||||
else:
|
||||
rows = await self._invoke("get_facts", [key], user_id=self._user_id)
|
||||
row = rows[0] if rows else None
|
||||
if type(row) is not kind or row.user_id != self._user_id:
|
||||
self._deny("write_mismatch")
|
||||
if expected is not None:
|
||||
stored, planned = row.to_dict(), expected.to_dict()
|
||||
for field in _TIMESTAMPS:
|
||||
stored.pop(field, None)
|
||||
planned.pop(field, None)
|
||||
if stored != planned:
|
||||
self._deny("write_mismatch")
|
||||
proof = HyNodePermission(hy_coding_fingerprint(row), self._evidence_ids)
|
||||
self._permissions[key] = proof
|
||||
self._written[key] = proof
|
||||
|
||||
async def _plan_new(self, row: CodingMemory | WorkingFact, kind: Any) -> None:
|
||||
if type(row) is not kind or not self._new_row_in_scope(row):
|
||||
self._deny("scope")
|
||||
key = _row_id(row)
|
||||
if not key or key in self._permissions or key in self._deleted:
|
||||
self._deny("scope")
|
||||
if kind is CodingMemory:
|
||||
existing = await self._invoke("get_by_id", key)
|
||||
else:
|
||||
existing = next(iter(await self._invoke("get_facts", [key])), None)
|
||||
if existing is not None:
|
||||
self._deny("scope")
|
||||
|
||||
# Reads
|
||||
|
||||
async def get_by_id(
|
||||
self, experience_id: str, user_id: str | None = None
|
||||
) -> CodingMemory | None:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
return await self._experience(experience_id)
|
||||
|
||||
async def get_many(
|
||||
self, experience_ids: list[str], user_id: str | None = None
|
||||
) -> list[CodingMemory]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
wanted = [
|
||||
key
|
||||
for key in dict.fromkeys(experience_ids)
|
||||
if key in self._permissions and key not in self._deleted
|
||||
]
|
||||
if not wanted:
|
||||
return []
|
||||
rows = await self._invoke("get_many", wanted, user_id=self._user_id)
|
||||
return [deepcopy(row) for row in rows if self._visible(row, CodingMemory)]
|
||||
|
||||
async def search_by_query_embedding(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
user_id: str,
|
||||
*,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
top: int = 40,
|
||||
) -> list[dict[str, Any]]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
hits = await self._invoke(
|
||||
"search_by_query_embedding",
|
||||
deepcopy(query_embedding),
|
||||
self._user_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
top=top * 4,
|
||||
)
|
||||
result = []
|
||||
for hit in hits:
|
||||
key = hit.get("experience_id")
|
||||
if not isinstance(key, str) or await self._experience(key) is None:
|
||||
continue
|
||||
result.append(
|
||||
{
|
||||
"experience_id": key,
|
||||
"score": float(hit.get("score", 0.0)),
|
||||
"matched_key_kind": hit.get("matched_key_kind"),
|
||||
"matched_key_text": hit.get("matched_key_text"),
|
||||
}
|
||||
)
|
||||
return result[:top]
|
||||
|
||||
async def get_fact(self, fact_id: str, user_id: str | None = None) -> WorkingFact | None:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
return await self._fact(fact_id)
|
||||
|
||||
async def get_facts(self, fact_ids: list[str], user_id: str | None = None) -> list[WorkingFact]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
found = [await self._fact(key) for key in dict.fromkeys(fact_ids)]
|
||||
return [row for row in found if row is not None]
|
||||
|
||||
async def search_facts(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
user_id: str,
|
||||
*,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
top: int = 5,
|
||||
query_text: str | None = None,
|
||||
min_score: float = 0.0,
|
||||
) -> list[tuple[WorkingFact, float]]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
hits = await self._invoke(
|
||||
"search_facts",
|
||||
deepcopy(query_embedding),
|
||||
self._user_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
top=top * 4,
|
||||
query_text=query_text,
|
||||
min_score=min_score,
|
||||
)
|
||||
result = []
|
||||
for row, score in hits:
|
||||
if self._visible(row, WorkingFact):
|
||||
result.append((deepcopy(row), float(score)))
|
||||
return result[:top]
|
||||
|
||||
async def get_fact_ancestors(
|
||||
self, fact_id: str, *, user_id: str | None = None, max_depth: int = 8
|
||||
) -> list[WorkingFact]:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
if await self._fact(fact_id) is None:
|
||||
return []
|
||||
rows = await self._invoke(
|
||||
"get_fact_ancestors", fact_id, user_id=self._user_id, max_depth=max_depth
|
||||
)
|
||||
# Oldest first; the chain is only as long as its nearest unapproved link.
|
||||
kept: list[WorkingFact] = []
|
||||
for row in reversed(rows):
|
||||
if not self._visible(row, WorkingFact):
|
||||
break
|
||||
kept.append(deepcopy(row))
|
||||
return list(reversed(kept))
|
||||
|
||||
# Writes
|
||||
|
||||
async def insert(self, memory: CodingMemory) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
await self._plan_new(memory, CodingMemory)
|
||||
await self._invoke("insert", memory)
|
||||
await self._capture(memory.experience_id, CodingMemory, memory)
|
||||
|
||||
async def update(self, memory: CodingMemory) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if type(memory) is not CodingMemory or not self._new_row_in_scope(memory):
|
||||
self._deny("scope")
|
||||
if await self._experience(memory.experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
await self._invoke("update", memory)
|
||||
await self._capture(memory.experience_id, CodingMemory, memory)
|
||||
|
||||
async def delete(self, experience_id: str, user_id: str) -> bool:
|
||||
async with self._lock:
|
||||
self._scope(user_id)
|
||||
if await self._experience(experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
if await self._invoke("delete", experience_id, user_id=self._user_id) is not True:
|
||||
self._deny("write_failed")
|
||||
if await self._invoke("get_by_id", experience_id) is not None:
|
||||
self._deny("write_mismatch")
|
||||
self._deleted.add(experience_id)
|
||||
self._permissions.pop(experience_id, None)
|
||||
self._written.pop(experience_id, None)
|
||||
return True
|
||||
|
||||
async def insert_fact(self, fact: WorkingFact) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
await self._plan_new(fact, WorkingFact)
|
||||
await self._invoke("insert_fact", fact)
|
||||
await self._capture(fact.fact_id, WorkingFact, fact)
|
||||
|
||||
async def supersede_fact(self, old_fact_id: str, new_fact_id: str) -> bool:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._fact(old_fact_id) is None or old_fact_id == new_fact_id:
|
||||
self._deny("target_denied")
|
||||
# HY links the old fact before it inserts the new one. A successor
|
||||
# that does not exist yet may only become a row this lease writes;
|
||||
# an existing row it cannot attest stays out of reach.
|
||||
if await self._fact(new_fact_id) is None:
|
||||
if not new_fact_id or new_fact_id in self._deleted:
|
||||
self._deny("target_denied")
|
||||
if next(iter(await self._invoke("get_facts", [new_fact_id])), None) is not None:
|
||||
self._deny("target_denied")
|
||||
changed = await self._invoke("supersede_fact", old_fact_id, new_fact_id)
|
||||
await self._capture(old_fact_id, WorkingFact)
|
||||
if changed:
|
||||
self._superseded[old_fact_id] = new_fact_id
|
||||
return bool(changed)
|
||||
|
||||
async def set_fact_derived_experiences(self, fact_id: str, experience_ids: list[str]) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._fact(fact_id) is None:
|
||||
self._deny("target_denied")
|
||||
for key in experience_ids:
|
||||
if await self._experience(key) is None:
|
||||
self._deny("link_denied")
|
||||
await self._invoke("set_fact_derived_experiences", fact_id, list(experience_ids))
|
||||
await self._capture(fact_id, WorkingFact)
|
||||
|
||||
async def set_experience_source_facts(self, experience_id: str, fact_ids: list[str]) -> None:
|
||||
async with self._lock:
|
||||
self.recheck()
|
||||
if await self._experience(experience_id) is None:
|
||||
self._deny("target_denied")
|
||||
for key in fact_ids:
|
||||
if await self._fact(key) is None:
|
||||
self._deny("link_denied")
|
||||
await self._invoke("set_experience_source_facts", experience_id, list(fact_ids))
|
||||
await self._capture(experience_id, CodingMemory)
|
||||
|
||||
async def get_recent_trace_messages(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
max_user_turns: int = 5,
|
||||
) -> list[Any]:
|
||||
"""Read recent raw trace rows, still bounded by this lease.
|
||||
|
||||
Public SDK 2.0.2 asks for recent history before a coding write. The native call is
|
||||
scoped to the lease's own user and agent, and rows this lease cannot
|
||||
attest are filtered out, so unapproved conversations never reach a
|
||||
coding prompt through this path.
|
||||
"""
|
||||
self._scope(user_id)
|
||||
if agent_id not in (self._agent_id, "*"):
|
||||
self._deny("scope")
|
||||
rows = await self._invoke(
|
||||
"get_recent_trace_messages",
|
||||
user_id=self._user_id,
|
||||
agent_id=self._agent_id,
|
||||
max_user_turns=max_user_turns,
|
||||
)
|
||||
return [row for row in (rows or []) if self._trace_visible(row)]
|
||||
|
||||
async def append_trace(
|
||||
self,
|
||||
messages: Sequence[Any],
|
||||
*,
|
||||
request_id: str,
|
||||
user_id: str,
|
||||
agent_id: str = "default_agent",
|
||||
session_id: str | None = None,
|
||||
workspace_id: str | None = None,
|
||||
branch: str | None = None,
|
||||
) -> None:
|
||||
"""Append raw trace rows for this lease only.
|
||||
|
||||
Trace rows are raw conversation, so this is a write path: the lease is
|
||||
rechecked around the call and every message must belong to the leased
|
||||
user and session. Nothing outside the lease may be recorded.
|
||||
"""
|
||||
self._scope(user_id)
|
||||
if agent_id not in (self._agent_id, "*"):
|
||||
self._deny("scope")
|
||||
if session_id not in (None, self._session_id):
|
||||
self._deny("scope")
|
||||
for row in messages or []:
|
||||
if not self._trace_visible(row):
|
||||
self._deny("scope")
|
||||
await self._invoke(
|
||||
"append_trace",
|
||||
list(messages or []),
|
||||
request_id=request_id,
|
||||
user_id=self._user_id,
|
||||
agent_id=self._agent_id,
|
||||
session_id=self._session_id,
|
||||
workspace_id=workspace_id,
|
||||
branch=branch,
|
||||
)
|
||||
|
||||
def _trace_visible(self, row: Any) -> bool:
|
||||
"""A trace row may be read or written only inside this lease's scope."""
|
||||
if row is None:
|
||||
return False
|
||||
owner = getattr(row, "user_id", None)
|
||||
session = getattr(row, "session_id", None)
|
||||
return not (
|
||||
(owner is not None and owner != self._user_id)
|
||||
or (session is not None and session != self._session_id)
|
||||
)
|
||||
@@ -1,582 +0,0 @@
|
||||
"""Task-exclusive HY wiring; provenance metadata stays beside the existing Core DB."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from collections.abc import AsyncIterator, Callable
|
||||
from contextlib import asynccontextmanager, nullcontext, suppress
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
import httpx
|
||||
|
||||
from one_memory.database import take_the_write_lock_first
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_memory_bridge import (
|
||||
HyMemoryBridge,
|
||||
HyMemoryBridgeError,
|
||||
HyMemoryIngestRequest,
|
||||
HyMemoryIngestResult,
|
||||
HyMemoryOutput,
|
||||
LocalHyEmbedService,
|
||||
_joined_thread,
|
||||
)
|
||||
from one_memory.hy_permission_coding_store import HyPermissionCodingStore
|
||||
from one_memory.hy_permission_store import (
|
||||
HyNodePermission,
|
||||
HyPermissionGraphStore,
|
||||
HyPermissionVectorStore,
|
||||
)
|
||||
from one_memory.hy_permission_transport import HyPermissionTransport, HyPermissionTransportError
|
||||
from one_memory.hy_response_guard import HyResponseGuard
|
||||
from one_memory.hy_step_timing import HyStepTimer
|
||||
from one_memory.memory_server.migrations import hy_node_permissions_ddl
|
||||
from one_memory.memory_tasks import evidence_ref
|
||||
from one_memory.model_processing_permission_guard import ModelProcessingPermissionLease
|
||||
|
||||
|
||||
class _NativeProofs:
|
||||
def __init__(self, database: Path) -> None:
|
||||
self.database = database
|
||||
self._loaded_states: dict[str, tuple[int, str]] | None = None
|
||||
with main_database(database).write() as db, db:
|
||||
# Schema owned by `memory_server.migrations` (migration 26).
|
||||
db.execute(hy_node_permissions_ddl("sqlite"))
|
||||
|
||||
def load(self, lease: ModelProcessingPermissionLease) -> dict[str, HyNodePermission]:
|
||||
allowed = {record.evidence_id: evidence_ref(record) for record in lease.records}
|
||||
with main_database(self.database).read() as db:
|
||||
rows = db.execute(
|
||||
"SELECT p.node_id,p.fingerprint,p.evidence_json,g.revision,g.status "
|
||||
"FROM hy_node_permissions p LEFT JOIN governed_memories g ON g.memory_id=p.node_id "
|
||||
# Superseded nodes stay attested: every later head links back to them.
|
||||
"WHERE g.status IS NULL OR g.status IN ('confirmed','superseded')"
|
||||
).fetchall()
|
||||
proofs = {}
|
||||
states = {}
|
||||
for identifier, fingerprint, evidence_json, revision, status in rows:
|
||||
refs = json.loads(evidence_json)
|
||||
if (
|
||||
not isinstance(refs, list)
|
||||
or not refs
|
||||
or not all(isinstance(ref, dict) for ref in refs)
|
||||
):
|
||||
continue
|
||||
if all(allowed.get(ref.get("evidence_id")) == ref for ref in refs):
|
||||
proofs[identifier] = HyNodePermission(
|
||||
fingerprint, tuple(ref["evidence_id"] for ref in refs)
|
||||
)
|
||||
if revision is not None:
|
||||
states[identifier] = (revision, status)
|
||||
self._loaded_states = states
|
||||
return proofs
|
||||
|
||||
def guard(self, values: dict[str, HyNodePermission]) -> Callable[[], None]:
|
||||
identifiers = tuple(values)
|
||||
|
||||
def state() -> dict[str, tuple[int, str]]:
|
||||
found: dict[str, tuple[int, str]] = {}
|
||||
with main_database(self.database).read() as db:
|
||||
for start in range(0, len(identifiers), 900):
|
||||
batch = identifiers[start : start + 900]
|
||||
placeholders = ",".join("?" for _ in batch)
|
||||
for identifier, revision, status in db.execute(
|
||||
"SELECT memory_id,revision,status FROM governed_memories "
|
||||
f"WHERE memory_id IN ({placeholders})",
|
||||
batch,
|
||||
):
|
||||
found[identifier] = (revision, status)
|
||||
return found
|
||||
|
||||
if self._loaded_states is None:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
before = dict(self._loaded_states)
|
||||
|
||||
def check() -> None:
|
||||
if state() != before:
|
||||
raise HyMemoryBridgeError("model_content_context_changed")
|
||||
|
||||
check()
|
||||
return check
|
||||
|
||||
def save(
|
||||
self, lease: ModelProcessingPermissionLease, values: dict[str, HyNodePermission]
|
||||
) -> None:
|
||||
refs = {record.evidence_id: evidence_ref(record) for record in lease.records}
|
||||
with main_database(self.database).write(autocommit=True) as db:
|
||||
take_the_write_lock_first(db)
|
||||
with db:
|
||||
lease.recheck()
|
||||
db.executemany(
|
||||
"INSERT INTO hy_node_permissions VALUES (?,?,?) ON CONFLICT(node_id) "
|
||||
"DO UPDATE SET fingerprint=excluded.fingerprint,"
|
||||
"evidence_json=excluded.evidence_json",
|
||||
[
|
||||
(
|
||||
identifier,
|
||||
proof.fingerprint,
|
||||
json.dumps([refs[key] for key in proof.evidence_ids]),
|
||||
)
|
||||
for identifier, proof in values.items()
|
||||
],
|
||||
)
|
||||
|
||||
def refresh(
|
||||
self, lease: ModelProcessingPermissionLease, node_id: str
|
||||
) -> HyNodePermission | None:
|
||||
del lease
|
||||
with main_database(self.database).read() as db:
|
||||
row = db.execute(
|
||||
"SELECT fingerprint,evidence_json FROM hy_node_permissions WHERE node_id=?",
|
||||
(node_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
fingerprint = str(row["fingerprint"])
|
||||
refs = json.loads(row["evidence_json"])
|
||||
if (
|
||||
len(fingerprint) != 64
|
||||
or any(value not in "0123456789abcdef" for value in fingerprint)
|
||||
or not isinstance(refs, list)
|
||||
or not refs
|
||||
or not all(isinstance(ref, dict) for ref in refs)
|
||||
):
|
||||
return None
|
||||
evidence_ids = tuple(ref.get("evidence_id") for ref in refs)
|
||||
if any(not isinstance(value, str) or not value for value in evidence_ids):
|
||||
return None
|
||||
return HyNodePermission(fingerprint, evidence_ids)
|
||||
|
||||
|
||||
class _BoundLease:
|
||||
def __init__(
|
||||
self,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proof_check: Callable[[], None],
|
||||
step_timer: HyStepTimer | None = None,
|
||||
) -> None:
|
||||
self.lease = lease
|
||||
self.step_timer = step_timer
|
||||
self.user_id = lease.user_id
|
||||
self.profile = lease.profile
|
||||
self.task_id = lease.task_id
|
||||
self.records = lease.records
|
||||
self.source_context = lease.source_context
|
||||
self.proof_check = proof_check
|
||||
self.vector: HyPermissionVectorStore | None = None
|
||||
self.coding: HyPermissionCodingStore | None = None
|
||||
|
||||
def recheck(self) -> None:
|
||||
timer = self.step_timer
|
||||
with timer.step("permission_check") if timer is not None else nullcontext():
|
||||
self.lease.recheck()
|
||||
self.proof_check()
|
||||
if self.vector is not None:
|
||||
self.vector.recheck()
|
||||
if self.coding is not None:
|
||||
self.coding.recheck()
|
||||
|
||||
|
||||
class _Wiring:
|
||||
def __init__(
|
||||
self,
|
||||
client: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proofs_store: _NativeProofs,
|
||||
proofs: dict[str, HyNodePermission],
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
transport: httpx.AsyncBaseTransport | None,
|
||||
proof_check: Callable[[], None],
|
||||
step_timer: HyStepTimer,
|
||||
) -> None:
|
||||
self.client, self.lease, self.proofs_store, self.proofs = (
|
||||
client,
|
||||
lease,
|
||||
proofs_store,
|
||||
proofs,
|
||||
)
|
||||
self.step_timer = step_timer
|
||||
self.agent_id, self.session_id, self.http_transport = agent_id, session_id, transport
|
||||
self.cross_evidence_scope = False
|
||||
self.proof_check = proof_check
|
||||
self.saved: list[tuple[Any, str, Any]] = []
|
||||
self.transport: HyPermissionTransport
|
||||
self.vector: HyPermissionVectorStore
|
||||
self.graph: HyPermissionGraphStore | None = None
|
||||
self.coding: HyPermissionCodingStore
|
||||
self.active = False
|
||||
|
||||
def replace(self, target: Any, name: str, value: Any) -> None:
|
||||
self.saved.append((target, name, getattr(target, name)))
|
||||
setattr(target, name, value)
|
||||
|
||||
async def enter(self) -> None:
|
||||
client, registry = self.client, self.client._registry
|
||||
config = client._config
|
||||
if config.llm.enable_summary or config.llm.enable_query_rewrite:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
bound = _BoundLease(self.lease, self.proof_check, self.step_timer)
|
||||
self.transport = await HyPermissionTransport(
|
||||
bound, http_transport=self.http_transport, step_timer=self.step_timer
|
||||
).__aenter__()
|
||||
self.active = True
|
||||
try:
|
||||
for guard in getattr(client, "_one_memory_write_guards", ()):
|
||||
if hasattr(type(guard), "step_timer"):
|
||||
self.replace(guard, "step_timer", self.step_timer)
|
||||
system2 = await registry.get_system2_writer()
|
||||
writers = (await registry.get_writer(), system2._writer)
|
||||
native_vector = client._vector_store
|
||||
store_lease = _BoundLease(self.lease, self.proof_check, self.step_timer)
|
||||
self.vector = HyPermissionVectorStore(
|
||||
native_vector,
|
||||
cast(ModelProcessingPermissionLease, store_lease),
|
||||
agent_id=self.agent_id,
|
||||
session_id=self.session_id,
|
||||
cross_evidence_scope=self.cross_evidence_scope,
|
||||
node_permissions=self.proofs,
|
||||
refresh_permission=lambda node_id: asyncio.to_thread(
|
||||
self.proofs_store.refresh, self.lease, node_id
|
||||
),
|
||||
step_timer=self.step_timer,
|
||||
)
|
||||
bound.vector = self.vector
|
||||
embed = client._embed_service
|
||||
if isinstance(embed, LocalHyEmbedService):
|
||||
self.transport.bind_local_embed_service(embed)
|
||||
else:
|
||||
self.transport.bind_embed_service(embed)
|
||||
for writer in writers:
|
||||
provider = writer.mem_agent.llm_provider
|
||||
self.transport.bind_provider(
|
||||
provider.provider if isinstance(provider, HyResponseGuard) else provider
|
||||
)
|
||||
reconciler = writer.reconciler
|
||||
provider = reconciler._get_llm()
|
||||
self.transport.bind_provider(
|
||||
provider.provider if isinstance(provider, HyResponseGuard) else provider
|
||||
)
|
||||
self.replace(writer, "_external_vector_store", self.vector)
|
||||
self.replace(writer, "_vector_store", self.vector)
|
||||
self.replace(writer, "_vector_store_initialized", True)
|
||||
self.replace(writer, "_cache", None)
|
||||
self.replace(client, "_vector_store", self.vector)
|
||||
native_graph = client._graph_store
|
||||
if self.cross_evidence_scope and native_graph is not None:
|
||||
self.graph = HyPermissionGraphStore(native_graph, self.proof_check)
|
||||
self.replace(client, "_graph_store", self.graph)
|
||||
else:
|
||||
self.replace(client, "_graph_store", None)
|
||||
self.replace(registry, "_shared_vector_store", self.vector)
|
||||
self.replace(
|
||||
registry,
|
||||
"_shared_graph_store",
|
||||
self.graph if self.graph is not None else None,
|
||||
)
|
||||
self.replace(registry, "_shared_cache", None)
|
||||
self.replace(system2, "_vector_store", self.vector)
|
||||
self.replace(
|
||||
system2,
|
||||
"_graph_store",
|
||||
self.graph if self.graph is not None else None,
|
||||
)
|
||||
self.replace(system2, "_cache", None)
|
||||
# Result fallback may call search; rebuild only its non-generative reader.
|
||||
self.replace(registry, "_reader", None)
|
||||
self.replace(registry, "_readers", {})
|
||||
await self._enter_coding(bound, embed)
|
||||
self.transport.assert_allowed()
|
||||
except BaseException:
|
||||
await self.exit()
|
||||
raise
|
||||
|
||||
async def _enter_coding(self, bound: _BoundLease, embed: Any) -> None:
|
||||
"""HY judges and writes the coding route itself; every seam it uses is bound here."""
|
||||
client = self.client
|
||||
# Created now, so the judge and every coding stage share one bound provider.
|
||||
writer = await client._get_coding_writer()
|
||||
stages = (writer, writer.extractor, writer.reconciler, writer.fact_reconciler)
|
||||
native_store = client._coding_store
|
||||
if any(
|
||||
stage.store is not native_store
|
||||
for stage in (writer, writer.reconciler, writer.fact_reconciler)
|
||||
) or any(
|
||||
item.embed_service is not embed
|
||||
for item in (native_store, writer, writer.reconciler, writer.fact_reconciler)
|
||||
):
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
providers = {
|
||||
id(p): p for p in (client._coding_llm_provider, *(s.llm_provider for s in stages))
|
||||
}
|
||||
for provider in providers.values():
|
||||
self.transport.bind_provider(provider)
|
||||
self.coding = HyPermissionCodingStore(
|
||||
native_store,
|
||||
cast(
|
||||
ModelProcessingPermissionLease,
|
||||
_BoundLease(self.lease, self.proof_check, self.step_timer),
|
||||
),
|
||||
agent_id=self.agent_id,
|
||||
session_id=self.session_id,
|
||||
node_permissions=self.proofs,
|
||||
)
|
||||
bound.coding = self.coding
|
||||
for stage in (writer, writer.reconciler, writer.fact_reconciler):
|
||||
self.replace(stage, "store", self.coding)
|
||||
self.replace(client, "_coding_store", self.coding)
|
||||
for stage in (writer, writer.extractor, writer.reconciler):
|
||||
self.replace(stage, "cache", None)
|
||||
# The judge logs its prompt through the client's own cache.
|
||||
self.replace(client, "_cache", None)
|
||||
|
||||
async def finish(self) -> dict[str, HyNodePermission]:
|
||||
self.transport.assert_allowed()
|
||||
self.vector.recheck()
|
||||
self.coding.recheck()
|
||||
return {**self.vector.written_nodes, **self.coding.written_nodes}
|
||||
|
||||
async def retired(self) -> tuple[tuple[str, ...], dict[str, tuple[str, ...]]]:
|
||||
self.transport.assert_allowed()
|
||||
return (
|
||||
tuple(sorted(self.vector.retired_node_ids | self.coding.retired_ids)),
|
||||
{**self.vector.superseded_nodes, **self.coding.superseded_ids},
|
||||
)
|
||||
|
||||
async def exit(self) -> None:
|
||||
if not self.active:
|
||||
return
|
||||
self.active = False
|
||||
try:
|
||||
await self.transport.__aexit__(None, None, None)
|
||||
finally:
|
||||
for target, name, value in reversed(self.saved):
|
||||
setattr(target, name, value)
|
||||
self.saved.clear()
|
||||
if hasattr(self, "vector"):
|
||||
await self.vector.close()
|
||||
if self.graph is not None:
|
||||
await self.graph.close()
|
||||
if hasattr(self, "coding"):
|
||||
await self.coding.close()
|
||||
|
||||
|
||||
class _LeasedBridge:
|
||||
def __init__(
|
||||
self,
|
||||
bridge: HyMemoryBridge,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
proofs: _NativeProofs,
|
||||
transport: httpx.AsyncBaseTransport | None,
|
||||
) -> None:
|
||||
self.bridge, self.lease, self.proofs, self.transport = bridge, lease, proofs, transport
|
||||
self.wiring: _Wiring | None = None
|
||||
self.finished = False
|
||||
self.step_timer = HyStepTimer()
|
||||
self.request_id: str | None = None
|
||||
|
||||
def _step(self, name: str) -> Any:
|
||||
return self.step_timer.step(name)
|
||||
|
||||
async def _recheck(self) -> None:
|
||||
def recheck() -> None:
|
||||
with self._step("permission_check"):
|
||||
self.lease.recheck()
|
||||
|
||||
await _joined_thread(recheck)
|
||||
|
||||
async def _start(self, request: HyMemoryIngestRequest) -> None:
|
||||
if self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_reused")
|
||||
if request.user_id != self.lease.user_id or set(request.evidence_ids) - {
|
||||
item.evidence_id for item in self.lease.records
|
||||
}:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._start_operation(
|
||||
user_id=request.user_id,
|
||||
agent_id=request.agent_id,
|
||||
request_id=request.request_id,
|
||||
session_id=request.session_id,
|
||||
)
|
||||
|
||||
async def _start_operation(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
agent_id: str,
|
||||
request_id: str,
|
||||
session_id: str,
|
||||
cross_evidence_scope: bool = False,
|
||||
) -> None:
|
||||
if self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_reused")
|
||||
if user_id != self.lease.user_id:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
self.request_id = request_id
|
||||
await self._recheck()
|
||||
client = cast(Any, await self.bridge._get_client())
|
||||
with self._step("permission_store_read"):
|
||||
known = await asyncio.to_thread(self.proofs.load, self.lease)
|
||||
proof_check = await asyncio.to_thread(self.proofs.guard, known)
|
||||
self.wiring = _Wiring(
|
||||
client,
|
||||
self.lease,
|
||||
self.proofs,
|
||||
known,
|
||||
agent_id,
|
||||
session_id,
|
||||
self.transport,
|
||||
proof_check,
|
||||
self.step_timer,
|
||||
)
|
||||
self.wiring.cross_evidence_scope = cross_evidence_scope
|
||||
await _joined_thread(client._loop_thread.run, self.wiring.enter())
|
||||
|
||||
async def list_agent_ids(self, *, user_id: str, limit: int = 1_000) -> tuple[str, ...]:
|
||||
if user_id != self.lease.user_id or self.wiring is not None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._recheck()
|
||||
result = await self.bridge.list_agent_ids(user_id=user_id, limit=limit)
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def digest(self, *, user_id: str, agent_id: str) -> dict[str, object]:
|
||||
await self._start_operation(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
request_id=f"digest:{self.lease.task_id}:{agent_id}",
|
||||
session_id=f"digest:{self.lease.task_id}",
|
||||
cross_evidence_scope=True,
|
||||
)
|
||||
try:
|
||||
result = dict(await self.bridge.digest(user_id=user_id, agent_id=agent_id))
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
proofs = await self._check()
|
||||
with self._step("permission_store_write"):
|
||||
await asyncio.to_thread(self.proofs.save, self.lease, proofs)
|
||||
await self.close()
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def graph_snapshot(self, *, user_id: str, agent_id: str, limit: int = 300) -> Any:
|
||||
await self._start_operation(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
request_id=f"digest-graph:{self.lease.task_id}:{agent_id}",
|
||||
session_id=f"digest:{self.lease.task_id}",
|
||||
cross_evidence_scope=True,
|
||||
)
|
||||
try:
|
||||
result = await self.bridge.graph_snapshot(
|
||||
user_id=user_id,
|
||||
agent_id=agent_id,
|
||||
limit=limit,
|
||||
)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
await self._check()
|
||||
await self.close()
|
||||
await self._recheck()
|
||||
return result
|
||||
|
||||
async def ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
|
||||
await self._start(request)
|
||||
try:
|
||||
result = await self.bridge.ingest(request)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
return await self._with_retired(result)
|
||||
|
||||
async def ingest_extracted(
|
||||
self, request: HyMemoryIngestRequest, contents: tuple[str, ...]
|
||||
) -> HyMemoryIngestResult:
|
||||
await self._start(request)
|
||||
try:
|
||||
result = await self.bridge.ingest_extracted(request, contents)
|
||||
except HyMemoryBridgeError:
|
||||
await self._check()
|
||||
raise
|
||||
return await self._with_retired(result)
|
||||
|
||||
async def _with_retired(self, result: HyMemoryIngestResult) -> HyMemoryIngestResult:
|
||||
await self._check()
|
||||
if self.wiring is None:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
try:
|
||||
retired, superseded = await _joined_thread(
|
||||
self.wiring.client._loop_thread.run, self.wiring.retired()
|
||||
)
|
||||
except HyPermissionTransportError as error:
|
||||
await self._recheck()
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
return result.model_copy(
|
||||
update={"retired_memory_ids": retired, "superseded_memory_ids": superseded}
|
||||
)
|
||||
|
||||
async def _check(self) -> dict[str, HyNodePermission]:
|
||||
if self.wiring is None or self.finished:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
await self._recheck()
|
||||
try:
|
||||
return await _joined_thread(self.wiring.client._loop_thread.run, self.wiring.finish())
|
||||
except HyPermissionTransportError as error:
|
||||
await self._recheck()
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
|
||||
async def profile_memories(self, *, user_id: str, agent_id: str) -> tuple[HyMemoryOutput, ...]:
|
||||
if self.wiring is None or user_id != self.lease.user_id or agent_id != self.wiring.agent_id:
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
result = await self.bridge.profile_memories(user_id=user_id, agent_id=agent_id)
|
||||
proofs = await self._check()
|
||||
with self._step("permission_store_write"):
|
||||
await asyncio.to_thread(self.proofs.save, self.lease, proofs)
|
||||
# SDK context closes before governance can update a previously known memory.
|
||||
await self.close()
|
||||
return result
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.wiring is not None and not self.finished:
|
||||
self.finished = True
|
||||
try:
|
||||
await _joined_thread(self.wiring.client._loop_thread.run, self.wiring.exit())
|
||||
finally:
|
||||
self.step_timer.emit(request_id=self.request_id, task_id=self.lease.task_id)
|
||||
|
||||
|
||||
def permission_bridge_factory(
|
||||
resolve_bridge: Callable[..., Any],
|
||||
*,
|
||||
http_transport: httpx.AsyncBaseTransport | None = None,
|
||||
) -> Callable[..., Any]:
|
||||
@asynccontextmanager
|
||||
async def factory(
|
||||
lease: ModelProcessingPermissionLease, secret: str | None, data_dir: Path
|
||||
) -> AsyncIterator[Any]:
|
||||
bridge = resolve_bridge(lease.profile, secret, data_dir)
|
||||
if not isinstance(bridge, HyMemoryBridge):
|
||||
raise HyMemoryBridgeError("model_content_context_unverified")
|
||||
proofs = _NativeProofs(data_dir / "one-memory.sqlite3")
|
||||
async with bridge.exclusive_access():
|
||||
await bridge.prepare()
|
||||
bound = _LeasedBridge(bridge, lease, proofs, http_transport)
|
||||
try:
|
||||
yield bound
|
||||
except BaseException as error:
|
||||
with suppress(Exception):
|
||||
await bound.close()
|
||||
if isinstance(error, HyPermissionTransportError):
|
||||
await _joined_thread(lease.recheck)
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
raise
|
||||
else:
|
||||
try:
|
||||
await bound.close()
|
||||
except HyPermissionTransportError as error:
|
||||
raise HyMemoryBridgeError(error.code) from None
|
||||
|
||||
return factory
|
||||
@@ -1,880 +0,0 @@
|
||||
"""Operation-bound read/write capability for the pinned native HY chat vector store.
|
||||
|
||||
No semantic storage or extraction lives here. Pre-existing nodes require a full
|
||||
native-node attestation with complete approved Evidence provenance; known content
|
||||
alone cannot authorize tags, custom fields, vectors or links. Unknown nodes stay
|
||||
in the same HY backend but are not exposed to this operation.
|
||||
|
||||
The caller must exclusively own the native backend's writes during execution:
|
||||
its API has no compare-and-swap, and a swallowed get_by_id error can look missing.
|
||||
This component cannot make that check/upsert interval atomic. Factory integration
|
||||
must replace client/registry/both writers/cached-reader references, bind physical
|
||||
LLM/embedding sends to the lease AND this denial latch, and guard publication.
|
||||
Coding rows go through hy_permission_coding_store instead; graph, entities,
|
||||
digest, sweeper and summary remain unsupported.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import math
|
||||
import struct
|
||||
import weakref
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable, Mapping
|
||||
from contextlib import asynccontextmanager, nullcontext
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from hashlib import sha256
|
||||
from types import MappingProxyType
|
||||
from typing import Any, NoReturn
|
||||
|
||||
from hy_memory.models.memory import ( # type: ignore[import-untyped]
|
||||
MemoryLayer,
|
||||
MemoryNode,
|
||||
MemoryStatus,
|
||||
)
|
||||
|
||||
from one_memory.hy_step_timing import HyStepTimer
|
||||
from one_memory.model_processing_permission_guard import (
|
||||
ModelProcessingPermissionLease,
|
||||
permission_error,
|
||||
)
|
||||
|
||||
_WRITE_LOCKS: weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
|
||||
weakref.WeakKeyDictionary()
|
||||
)
|
||||
|
||||
|
||||
def _native_write_lock() -> asyncio.Lock:
|
||||
loop = asyncio.get_running_loop()
|
||||
lock = _WRITE_LOCKS.get(loop)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
_WRITE_LOCKS[loop] = lock
|
||||
return lock
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def native_store_read_slot() -> AsyncIterator[bool]:
|
||||
"""Admit one native HY read only while no write is active.
|
||||
|
||||
Chroma's native reader may retain the GIL after an asyncio timeout. Running
|
||||
it concurrently with HY's multi-step store writes has deadlocked the Core
|
||||
under sustained imports. Reads therefore share the physical store lock,
|
||||
but never wait behind a writer: host recall can immediately use the local
|
||||
governance ledger instead.
|
||||
"""
|
||||
lock = _native_write_lock()
|
||||
if lock.locked():
|
||||
yield False
|
||||
return
|
||||
await lock.acquire()
|
||||
try:
|
||||
yield True
|
||||
finally:
|
||||
lock.release()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HyNodePermission:
|
||||
"""Trusted caller's full-node attestation, not permission inferred from an ID."""
|
||||
|
||||
fingerprint: str
|
||||
evidence_ids: tuple[str, ...]
|
||||
|
||||
|
||||
def hy_node_fingerprint(node: MemoryNode) -> str:
|
||||
"""Hash the complete native serialized node, including vectors and metadata.
|
||||
|
||||
Timestamp precision follows HY's to_dict contract (integer Unix seconds).
|
||||
Attest the node read from the backend, not an unpersisted input object.
|
||||
"""
|
||||
if type(node) is not MemoryNode:
|
||||
raise permission_error("hy_store_node_shape")
|
||||
return sha256(
|
||||
json.dumps(node.to_dict(), sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
_CHAT_LAYERS = frozenset(
|
||||
{
|
||||
MemoryLayer.L0_BASIC_INFO,
|
||||
MemoryLayer.L1_RAW,
|
||||
MemoryLayer.L2_FACT,
|
||||
MemoryLayer.L4_IDENTITY,
|
||||
MemoryLayer.L7_INTENTION,
|
||||
}
|
||||
)
|
||||
_UPDATE_FIELDS = frozenset(
|
||||
{
|
||||
"content",
|
||||
"embedding",
|
||||
"tags",
|
||||
"tag_ners",
|
||||
"layer",
|
||||
"memory_at",
|
||||
"gmt_modified",
|
||||
"is_latest",
|
||||
"status",
|
||||
"supersedes",
|
||||
"superseded_by",
|
||||
"source_raw_memory_id",
|
||||
# SDK 2.0.2 sends both of these on an in-place reconcile UPDATE.
|
||||
# They are metadata only and must never carry identity, permission or
|
||||
# tenant state: `owner` is a provenance label whose only values are
|
||||
# "user" / "agent" (who said this line), and `custom` is the SDK's own
|
||||
# free-form notes field. Neither may hold a user_id, scope, grant or
|
||||
# tenancy marker, so allowing the field names does not widen the
|
||||
# cross-user boundary that the store still enforces per node.
|
||||
"owner",
|
||||
"custom",
|
||||
}
|
||||
)
|
||||
_DEDUP_AUDIT_FIELDS = frozenset({"shadow_reason", "dedup_kept_id"})
|
||||
_DIGEST_COUNTER_FIELD = "s2_evidence_count"
|
||||
|
||||
|
||||
def _write_view(node: MemoryNode) -> dict[str, Any]:
|
||||
"""Only native Chroma's empty-string and float32 storage normalization."""
|
||||
payload: dict[str, Any] = node.to_dict()
|
||||
payload["source_session_id"] = payload["source_session_id"] or ""
|
||||
# The pinned public SDK serializes tag_ners as JSON, but its Chroma read path omits this
|
||||
# field from the JSON decode list. Decode only the exact list-of-strings
|
||||
# shape that the write path produced; malformed values still fail the
|
||||
# complete-node comparison below.
|
||||
tag_ners = payload.get("tag_ners")
|
||||
if (
|
||||
isinstance(tag_ners, list)
|
||||
and tag_ners
|
||||
and all(isinstance(value, str) and len(value) == 1 for value in tag_ners)
|
||||
and tag_ners[0] == "["
|
||||
and tag_ners[-1] == "]"
|
||||
):
|
||||
tag_ners = "".join(tag_ners)
|
||||
if isinstance(tag_ners, str):
|
||||
try:
|
||||
decoded = json.loads(tag_ners)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
decoded = None
|
||||
if isinstance(decoded, list) and all(isinstance(value, str) for value in decoded):
|
||||
payload["tag_ners"] = decoded
|
||||
if payload.get("tag_ners") in ([], ["[", "]"]):
|
||||
payload["tag_ners"] = []
|
||||
# HY Chroma encodes None as ""; these JSON fields keep "" when json.loads
|
||||
# fails (vector_store_chroma.py:741-788). Only an absent link is equivalent.
|
||||
for field in ("supersedes", "superseded_by"):
|
||||
if payload[field] == "":
|
||||
payload[field] = None
|
||||
if payload["embedding"] is not None:
|
||||
payload["embedding"] = [
|
||||
struct.unpack("f", struct.pack("f", n))[0] for n in payload["embedding"]
|
||||
]
|
||||
if not all(math.isfinite(n) for n in payload["embedding"]):
|
||||
raise ValueError("nonfinite_embedding")
|
||||
return payload
|
||||
|
||||
|
||||
def _same_written_vector(actual: list[float] | None, expected: list[float] | None) -> bool:
|
||||
if actual == expected:
|
||||
return True
|
||||
if actual is None or expected is None or len(actual) != len(expected):
|
||||
return False
|
||||
# Chroma's cosine index can renormalize float32 vectors on readback. Bound
|
||||
# only this write roundtrip; the saved full-node proof uses exact readback.
|
||||
return all(
|
||||
math.isclose(left, right, rel_tol=4 * 2**-23, abs_tol=0.0)
|
||||
for left, right in zip(actual, expected, strict=True)
|
||||
)
|
||||
|
||||
|
||||
_NATIVE_WRITES = frozenset(
|
||||
{"upsert", "upsert_batch", "update_payload", "update_embedding", "delete"}
|
||||
)
|
||||
|
||||
|
||||
class HyPermissionGraphStore:
|
||||
"""Lease-gated graph store for HY System 2's digest transaction.
|
||||
|
||||
The native graph is deliberately absent from ordinary ingest leases. Digest
|
||||
needs it to write L6 schemas, but every operation must remain inside the
|
||||
same frozen permission lease rather than exposing the raw store.
|
||||
"""
|
||||
|
||||
def __init__(self, native: Any, recheck: Any) -> None:
|
||||
self._native = native
|
||||
self._recheck = recheck
|
||||
self._closed = False
|
||||
|
||||
def recheck(self) -> None:
|
||||
if self._closed:
|
||||
raise permission_error("hy_graph_store_closed")
|
||||
self._recheck()
|
||||
|
||||
async def close(self) -> None:
|
||||
self._closed = True
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
if name.startswith("_") and name != "_run":
|
||||
raise AttributeError(name)
|
||||
attribute = getattr(self._native, name)
|
||||
if not callable(attribute):
|
||||
return attribute
|
||||
|
||||
async def guarded(*args: Any, **kwargs: Any) -> Any:
|
||||
self.recheck()
|
||||
result = await attribute(*args, **kwargs)
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
return guarded
|
||||
|
||||
|
||||
class HyPermissionVectorStore:
|
||||
"""Closed, task-exclusive adapter; initialize/close never own the native store.
|
||||
|
||||
node_permissions may attest additional native raw/profile nodes not represented
|
||||
in known_memories. Every attestation must cover all its contributors, and all
|
||||
Evidence IDs must already be in this lease. Known entries also constrain exact
|
||||
content/owner. Without a full attestation even a known entry is denied.
|
||||
|
||||
written_nodes records successful writes with the entire lease footprint, a
|
||||
conservative provenance bound, not a claim of semantic citation accuracy.
|
||||
These transient fingerprints must be persisted by the integrating factory if
|
||||
future operations are to reuse raw/profile nodes. No content is stored here.
|
||||
"""
|
||||
|
||||
__slots__ = (
|
||||
"__native",
|
||||
"__lease",
|
||||
"__agent_id",
|
||||
"__session_id",
|
||||
"__cross_evidence_scope",
|
||||
"__user_id",
|
||||
"__known",
|
||||
"__permissions",
|
||||
"__written",
|
||||
"__evidence_ids",
|
||||
"__deleted",
|
||||
"__shadowed",
|
||||
"__superseded",
|
||||
"__heads",
|
||||
"__lock",
|
||||
"__closed",
|
||||
"__failure",
|
||||
"__step_timer",
|
||||
"__refresh_permission",
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
native: Any,
|
||||
lease: ModelProcessingPermissionLease,
|
||||
*,
|
||||
agent_id: str,
|
||||
session_id: str,
|
||||
cross_evidence_scope: bool = False,
|
||||
node_permissions: Mapping[str, HyNodePermission] | None = None,
|
||||
refresh_permission: Callable[[str], Awaitable[HyNodePermission | None]] | None = None,
|
||||
step_timer: HyStepTimer | None = None,
|
||||
) -> None:
|
||||
self.__step_timer = step_timer
|
||||
self.__native = native
|
||||
self.__lease = lease
|
||||
self.__agent_id = agent_id
|
||||
self.__session_id = session_id
|
||||
self.__cross_evidence_scope = cross_evidence_scope
|
||||
self.__user_id = lease.user_id
|
||||
self.__failure: str | None = None
|
||||
self.__closed = False
|
||||
self.__lock = asyncio.Lock()
|
||||
self.__deleted: set[str] = set()
|
||||
self.__written: dict[str, HyNodePermission] = {}
|
||||
self.__shadowed: set[str] = set()
|
||||
self.__superseded: dict[str, tuple[str, ...]] = {}
|
||||
self.__heads: dict[str, tuple[str, ...]] = {}
|
||||
self.__refresh_permission = refresh_permission
|
||||
self.recheck()
|
||||
self.__evidence_ids = tuple(record.evidence_id for record in lease.records)
|
||||
current = lease.source_context.current if lease.source_context else lease.records
|
||||
conversation_mismatch = any(r.conversation_id != session_id for r in current)
|
||||
if (
|
||||
not lease.task_id
|
||||
or not agent_id
|
||||
or agent_id == "*"
|
||||
or not session_id
|
||||
or not current
|
||||
or (conversation_mismatch and not cross_evidence_scope)
|
||||
or not self.__evidence_ids
|
||||
or len(set(self.__evidence_ids)) != len(lease.records)
|
||||
or any(r.user_id != self.__user_id for r in lease.records)
|
||||
):
|
||||
self.__deny("scope")
|
||||
known = lease.source_context.known_memories if lease.source_context else ()
|
||||
self.__known = {item.hy_memory_id: item for item in known}
|
||||
if len(self.__known) != len(known):
|
||||
self.__deny("provenance")
|
||||
self.__permissions = dict(node_permissions or {})
|
||||
for key, proof in self.__permissions.items():
|
||||
if (
|
||||
not key
|
||||
or not isinstance(proof, HyNodePermission)
|
||||
or len(proof.fingerprint) != 64
|
||||
or any(c not in "0123456789abcdef" for c in proof.fingerprint)
|
||||
or not isinstance(proof.evidence_ids, tuple)
|
||||
or not proof.evidence_ids
|
||||
or len(set(proof.evidence_ids)) != len(proof.evidence_ids)
|
||||
or not set(proof.evidence_ids) <= set(self.__evidence_ids)
|
||||
):
|
||||
self.__deny("provenance")
|
||||
for key, item in self.__known.items():
|
||||
if not item.evidence_ids or not set(item.evidence_ids) <= set(self.__evidence_ids):
|
||||
self.__deny("provenance")
|
||||
known_proof = self.__permissions.get(key)
|
||||
if known_proof and not set(item.evidence_ids) <= set(known_proof.evidence_ids):
|
||||
self.__deny("provenance")
|
||||
|
||||
@property
|
||||
def written_nodes(self) -> Mapping[str, HyNodePermission]:
|
||||
"""Latest verified write proofs; each access returns a new immutable snapshot.
|
||||
|
||||
Export after SDK writes have drained and before close/governance projection.
|
||||
Merge these rows into the lineage ledger, then remove deleted_node_ids;
|
||||
this write delta does not include untouched pre-existing permissions.
|
||||
"""
|
||||
self.recheck()
|
||||
return MappingProxyType(dict(self.__written))
|
||||
|
||||
@property
|
||||
def deleted_node_ids(self) -> frozenset[str]:
|
||||
"""Verified native deletions in this operation, for lineage-ledger cleanup."""
|
||||
self.recheck()
|
||||
return frozenset(self.__deleted)
|
||||
|
||||
@property
|
||||
def retired_node_ids(self) -> frozenset[str]:
|
||||
"""Nodes HY deleted or shadowed (its logical delete) in this operation."""
|
||||
self.recheck()
|
||||
return frozenset(self.__deleted | self.__shadowed)
|
||||
|
||||
@property
|
||||
def superseded_nodes(self) -> Mapping[str, tuple[str, ...]]:
|
||||
"""Nodes HY marked superseded in this operation, with what replaced them.
|
||||
|
||||
HY writes the two directions separately and may fail either one, so a
|
||||
replacement counts whether it was recorded on the old node or only on
|
||||
the new head.
|
||||
"""
|
||||
self.recheck()
|
||||
result = {}
|
||||
for key, by in self.__superseded.items():
|
||||
heads = set(by) | {head for head, olds in self.__heads.items() if key in olds}
|
||||
heads -= self.__deleted | {key}
|
||||
if key not in self.__deleted and heads:
|
||||
result[key] = tuple(sorted(heads))
|
||||
return MappingProxyType(result)
|
||||
|
||||
@property
|
||||
def supports_fulltext(self) -> bool:
|
||||
return False
|
||||
|
||||
def recheck(self) -> None:
|
||||
"""Must also gate transport/publication after HY swallows a store failure."""
|
||||
self.__assert_open()
|
||||
try:
|
||||
self.__lease.recheck()
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
self.__failure = (
|
||||
code
|
||||
if code.startswith("model_content_") and len(code) <= 100
|
||||
else "model_content_hy_store_lease_failed"
|
||||
)
|
||||
raise permission_error(self.__failure) from None
|
||||
|
||||
def __assert_open(self) -> None:
|
||||
if self.__closed:
|
||||
self.__deny("closed")
|
||||
if self.__failure:
|
||||
raise permission_error(self.__failure)
|
||||
|
||||
async def initialize(self) -> None:
|
||||
self.recheck()
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Revoke this wrapper locally without rechecking context or closing HY.
|
||||
|
||||
Cleanup must remain possible after governance projection changes the
|
||||
lease's known-memory revisions, or after permission has been revoked.
|
||||
"""
|
||||
async with self.__lock:
|
||||
self.__closed = True
|
||||
self.__permissions.clear()
|
||||
self.__written.clear()
|
||||
|
||||
async def has_tag_embedding(self, user_id: str, tag: str) -> bool:
|
||||
self.recheck()
|
||||
# Native chat calls this best-effort helper. NotImplemented prevents it
|
||||
# from embedding tags or falling through to an unguarded entity store.
|
||||
raise NotImplementedError("hy_permission_store_tag_index_unsupported")
|
||||
|
||||
def __deny(self, suffix: str) -> NoReturn:
|
||||
self.__failure = self.__failure or f"model_content_hy_store_{suffix}"
|
||||
raise permission_error(self.__failure)
|
||||
|
||||
async def __invoke(self, method: str, *args: Any, **kwargs: Any) -> Any:
|
||||
writing = method in _NATIVE_WRITES
|
||||
self.recheck() if writing else self.__assert_open()
|
||||
step = "vector_store_write" if writing else "vector_store_read"
|
||||
first = args[0] if args else None
|
||||
try:
|
||||
with (
|
||||
self.__step_timer.step(step, items=len(first) if isinstance(first, list) else 1)
|
||||
if self.__step_timer is not None
|
||||
else nullcontext()
|
||||
):
|
||||
result = await getattr(self.__native, method)(*args, **kwargs)
|
||||
except Exception:
|
||||
self.__deny("backend_failed")
|
||||
self.recheck() if writing else self.__assert_open()
|
||||
return result
|
||||
|
||||
async def __read_native(self, key: str) -> MemoryNode | None:
|
||||
value = await self.__invoke("get_by_id", key)
|
||||
if value is not None and (type(value) is not MemoryNode or value.node_id != key):
|
||||
self.__deny("node_shape")
|
||||
return deepcopy(value)
|
||||
|
||||
def __fingerprint(self, node: MemoryNode) -> str:
|
||||
try:
|
||||
return hy_node_fingerprint(node)
|
||||
except (TypeError, ValueError, AttributeError, OverflowError):
|
||||
self.__deny("node_shape")
|
||||
|
||||
def __write_view(self, node: MemoryNode) -> dict[str, Any]:
|
||||
self.__fingerprint(node)
|
||||
try:
|
||||
return _write_view(node)
|
||||
except (TypeError, ValueError, OverflowError, struct.error):
|
||||
self.__deny("node_shape")
|
||||
|
||||
def __in_scope(self, node: MemoryNode) -> bool:
|
||||
if node.user_id != self.__user_id or node.layer not in _CHAT_LAYERS:
|
||||
return False
|
||||
if node.layer is MemoryLayer.L0_BASIC_INFO:
|
||||
# Public SDK 2.0.2 migrates verified legacy agent-level profiles into one
|
||||
# user-level chain. Existing nodes still require a full proof in
|
||||
# __load; accepting their historical agent id does not authorize
|
||||
# unproven cross-agent memories.
|
||||
return True
|
||||
if self.__cross_evidence_scope:
|
||||
return bool(node.agent_id == self.__agent_id)
|
||||
return bool(node.agent_id == self.__agent_id and node.session_id == self.__session_id)
|
||||
|
||||
def __new_node_in_scope(self, node: MemoryNode) -> bool:
|
||||
if not self.__in_scope(node):
|
||||
return False
|
||||
if node.layer is MemoryLayer.L0_BASIC_INFO:
|
||||
return node.agent_id in ("", self.__agent_id) and node.session_id in (
|
||||
"",
|
||||
self.__session_id,
|
||||
)
|
||||
return True
|
||||
|
||||
def __references(self, node: MemoryNode) -> list[str]:
|
||||
refs = []
|
||||
for field in ("supersedes", "superseded_by", "evidence_chain"):
|
||||
values = getattr(node, field)
|
||||
if field != "evidence_chain" and values == "":
|
||||
continue
|
||||
if values is not None:
|
||||
if not isinstance(values, list) or any(
|
||||
not isinstance(v, str) or not v for v in values
|
||||
):
|
||||
self.__deny("node_shape")
|
||||
refs.extend(values)
|
||||
if node.source_raw_memory_id is not None:
|
||||
if not isinstance(node.source_raw_memory_id, str) or not node.source_raw_memory_id:
|
||||
self.__deny("node_shape")
|
||||
refs.append(node.source_raw_memory_id)
|
||||
return refs
|
||||
|
||||
async def __load(
|
||||
self,
|
||||
key: str,
|
||||
first: MemoryNode | None = None,
|
||||
*,
|
||||
allow_deleted_links: bool = False,
|
||||
) -> MemoryNode | None:
|
||||
pending = [key]
|
||||
checked: dict[str, MemoryNode] = {}
|
||||
while pending:
|
||||
current = pending.pop()
|
||||
if current in checked:
|
||||
continue
|
||||
# HY dedup deletes head then ancestors. Only this operation's
|
||||
# verified tombstones may terminate a closure during deletion.
|
||||
if allow_deleted_links and current != key and current in self.__deleted:
|
||||
continue
|
||||
proof = self.__permissions.get(current)
|
||||
if proof is None or current in self.__deleted:
|
||||
return None
|
||||
value = (
|
||||
first if current == key and first is not None else await self.__read_native(current)
|
||||
)
|
||||
if value is None or not self.__in_scope(value):
|
||||
return None
|
||||
if self.__fingerprint(value) != proof.fingerprint:
|
||||
refreshed = (
|
||||
await self.__refresh_permission(current)
|
||||
if self.__refresh_permission is not None
|
||||
else None
|
||||
)
|
||||
if refreshed is None or self.__fingerprint(value) != refreshed.fingerprint:
|
||||
self.__deny("node_changed")
|
||||
self.__permissions[current] = refreshed
|
||||
proof = refreshed
|
||||
known = self.__known.get(current) if current not in self.__written else None
|
||||
if known and (value.content != known.content or value.owner != known.owner):
|
||||
self.__deny("node_changed")
|
||||
checked[current] = value
|
||||
pending.extend(self.__references(value))
|
||||
return deepcopy(checked[key])
|
||||
|
||||
async def get_by_id(self, node_id: str) -> MemoryNode | None:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
result = await self.__load(node_id)
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
async def get_by_ids(self, node_ids: list[str]) -> list[MemoryNode]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
result = [
|
||||
n for key in dict.fromkeys(node_ids) if (n := await self.__load(key)) is not None
|
||||
]
|
||||
self.recheck()
|
||||
return result
|
||||
|
||||
async def get_embeddings(self, node_ids: list[str]) -> dict[str, list[float]]:
|
||||
return {
|
||||
n.node_id: list(n.embedding)
|
||||
for n in await self.get_by_ids(node_ids)
|
||||
if n.embedding is not None
|
||||
}
|
||||
|
||||
def __scope(self, user_id: str | None, agent_id: str | None) -> None:
|
||||
if user_id not in (None, self.__user_id) or agent_id not in (None, self.__agent_id):
|
||||
self.__deny("scope")
|
||||
|
||||
async def list_by_user(
|
||||
self,
|
||||
user_id: str | None = None,
|
||||
agent_id: str | None = None,
|
||||
limit: int = 10000,
|
||||
status_filter: list[Any] | None = None,
|
||||
layers: list[Any] | None = None,
|
||||
user_ids: list[str] | None = None,
|
||||
agent_ids: list[str] | None = None,
|
||||
only_latest: bool = False,
|
||||
with_vectors: bool = True,
|
||||
allow_raw_layer: bool = False,
|
||||
) -> list[MemoryNode]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
effective_users = user_ids or ([user_id] if user_id else [])
|
||||
effective_agents = agent_ids or ([agent_id] if agent_id else [])
|
||||
if effective_users != [self.__user_id] or any(
|
||||
value != self.__agent_id for value in effective_agents
|
||||
):
|
||||
self.__deny("scope")
|
||||
rows = await self.__invoke(
|
||||
"list_by_user",
|
||||
user_id=self.__user_id,
|
||||
agent_id=agent_id,
|
||||
limit=limit,
|
||||
status_filter=status_filter,
|
||||
layers=layers,
|
||||
user_ids=[self.__user_id],
|
||||
agent_ids=effective_agents or None,
|
||||
only_latest=only_latest,
|
||||
with_vectors=with_vectors,
|
||||
allow_raw_layer=allow_raw_layer,
|
||||
)
|
||||
result = []
|
||||
for row in rows:
|
||||
if type(row) is not MemoryNode:
|
||||
self.__deny("node_shape")
|
||||
value = await self.__load(row.node_id)
|
||||
if (
|
||||
value is not None
|
||||
and (not layers or value.layer in layers or value.layer.value in layers)
|
||||
and (
|
||||
not status_filter
|
||||
or value.status in status_filter
|
||||
or value.status.value in status_filter
|
||||
)
|
||||
):
|
||||
result.append(value)
|
||||
self.recheck()
|
||||
return result[:limit]
|
||||
|
||||
async def search(
|
||||
self,
|
||||
query_embedding: list[float],
|
||||
isolation_key: str = "",
|
||||
isolation_keys: list[str] | None = None,
|
||||
user_id: str | None = None,
|
||||
user_ids: list[str] | None = None,
|
||||
agent_ids: list[str] | None = None,
|
||||
layers: list[Any] | None = None,
|
||||
limit: int = 10,
|
||||
score_threshold: float = 0.0,
|
||||
status_filter: list[Any] | None = None,
|
||||
only_latest: bool = True,
|
||||
tags_match_any: list[str] | None = None,
|
||||
created_after: float | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
async with self.__lock:
|
||||
self.recheck()
|
||||
self.__scope(user_id, None)
|
||||
key = MemoryNode.build_isolation_key(self.__user_id, self.__agent_id, self.__session_id)
|
||||
if (
|
||||
any(uid != self.__user_id for uid in user_ids or [])
|
||||
or any(aid != self.__agent_id for aid in agent_ids or [])
|
||||
or isolation_key not in ("", key)
|
||||
or any(ik != key for ik in isolation_keys or [])
|
||||
):
|
||||
self.__deny("scope")
|
||||
hits = await self.__invoke(
|
||||
"search",
|
||||
query_embedding=deepcopy(query_embedding),
|
||||
isolation_key=isolation_key,
|
||||
isolation_keys=isolation_keys,
|
||||
user_id=self.__user_id,
|
||||
user_ids=[self.__user_id],
|
||||
agent_ids=[self.__agent_id],
|
||||
layers=layers,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold,
|
||||
status_filter=status_filter,
|
||||
only_latest=only_latest,
|
||||
tags_match_any=tags_match_any,
|
||||
created_after=created_after,
|
||||
)
|
||||
result = []
|
||||
for hit in hits:
|
||||
value = await self.__load(hit["node_id"])
|
||||
if value is None:
|
||||
continue
|
||||
score = float(hit["score"])
|
||||
if not math.isfinite(score):
|
||||
self.__deny("node_shape")
|
||||
# Do not forward arbitrary backend hit metadata. Rehydrate the
|
||||
# full fingerprinted node; native search often omits embedding.
|
||||
result.append({"node_id": value.node_id, "node": value, "score": score})
|
||||
self.recheck()
|
||||
return result[:limit]
|
||||
|
||||
async def __target(
|
||||
self,
|
||||
key: str,
|
||||
first: MemoryNode | None = None,
|
||||
*,
|
||||
allow_deleted_links: bool = False,
|
||||
) -> MemoryNode:
|
||||
value = await self.__load(key, first, allow_deleted_links=allow_deleted_links)
|
||||
if value is None:
|
||||
self.__deny("target_denied")
|
||||
return value
|
||||
|
||||
async def __check_links(self, value: MemoryNode) -> None:
|
||||
for ref in self.__references(value):
|
||||
if await self.__load(ref) is None:
|
||||
self.__deny("link_denied")
|
||||
|
||||
async def __plan_upsert(self, value: MemoryNode) -> MemoryNode:
|
||||
value = deepcopy(value)
|
||||
if type(value) is not MemoryNode or not value.node_id or not self.__in_scope(value):
|
||||
self.__deny("scope")
|
||||
existing = await self.__read_native(value.node_id)
|
||||
if existing is not None:
|
||||
await self.__target(value.node_id, existing)
|
||||
elif (
|
||||
not self.__new_node_in_scope(value)
|
||||
or value.node_id in self.__permissions
|
||||
or value.node_id in self.__known
|
||||
or value.node_id in self.__deleted
|
||||
):
|
||||
self.__deny("scope")
|
||||
await self.__check_links(value)
|
||||
self.__write_view(value)
|
||||
return value
|
||||
|
||||
async def __capture(
|
||||
self,
|
||||
expected: MemoryNode,
|
||||
*,
|
||||
allow_native_modified_time: bool = False,
|
||||
) -> None:
|
||||
actual = await self.__read_native(expected.node_id)
|
||||
if actual is None:
|
||||
self.__deny("write_mismatch")
|
||||
actual_view, expected_view = self.__write_view(actual), self.__write_view(expected)
|
||||
if allow_native_modified_time:
|
||||
if actual.gmt_modified < expected.gmt_modified:
|
||||
self.__deny("write_mismatch")
|
||||
actual_view["gmt_modified"] = expected_view["gmt_modified"]
|
||||
actual_vector = actual_view.pop("embedding")
|
||||
expected_vector = expected_view.pop("embedding")
|
||||
if actual_view != expected_view or not _same_written_vector(actual_vector, expected_vector):
|
||||
self.__deny("write_mismatch")
|
||||
proof = HyNodePermission(self.__fingerprint(actual), self.__evidence_ids)
|
||||
self.__permissions[actual.node_id] = proof
|
||||
self.__written[actual.node_id] = proof
|
||||
status = getattr(actual.status, "value", actual.status)
|
||||
if status == MemoryStatus.SHADOW.value:
|
||||
self.__shadowed.add(actual.node_id)
|
||||
else:
|
||||
self.__shadowed.discard(actual.node_id)
|
||||
if status == MemoryStatus.SUPERSEDED.value:
|
||||
by = actual.superseded_by
|
||||
self.__superseded[actual.node_id] = tuple(by) if isinstance(by, list) else ()
|
||||
else:
|
||||
self.__superseded.pop(actual.node_id, None)
|
||||
olds = actual.supersedes
|
||||
if isinstance(olds, list) and olds:
|
||||
self.__heads[actual.node_id] = tuple(olds)
|
||||
else:
|
||||
self.__heads.pop(actual.node_id, None)
|
||||
|
||||
async def upsert(self, node: MemoryNode) -> str:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
expected = await self.__plan_upsert(node)
|
||||
result = await self.__invoke("upsert", deepcopy(expected))
|
||||
if result != expected.node_id:
|
||||
self.__deny("write_mismatch")
|
||||
await self.__capture(expected)
|
||||
return str(result)
|
||||
|
||||
async def upsert_batch(self, nodes: list[MemoryNode]) -> list[str]:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
if len({n.node_id for n in nodes}) != len(nodes):
|
||||
self.__deny("node_shape")
|
||||
# No forward links to uncommitted batch members. Preflight every
|
||||
# existing target before the backend can make a partial write.
|
||||
planned = [await self.__plan_upsert(n) for n in nodes]
|
||||
ids = [n.node_id for n in planned]
|
||||
result = await self.__invoke("upsert_batch", deepcopy(planned))
|
||||
if result != ids:
|
||||
self.__deny("write_mismatch")
|
||||
for expected in planned:
|
||||
await self.__capture(expected)
|
||||
return ids
|
||||
|
||||
async def update_payload(self, node_id: str, updates: dict[str, Any]) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
current = await self.__target(node_id)
|
||||
updates = deepcopy(updates)
|
||||
dedup_audit = False
|
||||
if set(updates) & _DEDUP_AUDIT_FIELDS:
|
||||
if (
|
||||
not set(updates) <= (_UPDATE_FIELDS | _DEDUP_AUDIT_FIELDS)
|
||||
or updates.get("status") != MemoryStatus.SHADOW.value
|
||||
or updates.get("is_latest") is not False
|
||||
or not isinstance(updates.get("shadow_reason"), str)
|
||||
or not updates["shadow_reason"].startswith("dedup:")
|
||||
or not isinstance(updates.get("dedup_kept_id"), str)
|
||||
or not updates["dedup_kept_id"]
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
await self.__target(updates["dedup_kept_id"])
|
||||
custom = deepcopy(current.custom)
|
||||
custom["hy_dedup"] = {
|
||||
"reason": updates.pop("shadow_reason"),
|
||||
"kept_id": updates.pop("dedup_kept_id"),
|
||||
}
|
||||
updates["custom"] = custom
|
||||
dedup_audit = True
|
||||
allowed = _UPDATE_FIELDS | ({"custom"} if dedup_audit else set())
|
||||
if self.__cross_evidence_scope and set(updates) == {"custom"}:
|
||||
custom = updates.get("custom")
|
||||
if not isinstance(custom, dict):
|
||||
self.__deny("update_fields")
|
||||
current_custom = current.custom if isinstance(current.custom, dict) else {}
|
||||
previous = current_custom.get(_DIGEST_COUNTER_FIELD, 0)
|
||||
proposed = custom.get(_DIGEST_COUNTER_FIELD)
|
||||
if (
|
||||
not isinstance(previous, int)
|
||||
or isinstance(previous, bool)
|
||||
or not isinstance(proposed, int)
|
||||
or isinstance(proposed, bool)
|
||||
or proposed != previous + 1
|
||||
or set(custom) != set(current_custom) | {_DIGEST_COUNTER_FIELD}
|
||||
or any(custom[key] != value for key, value in current_custom.items())
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
allowed |= {"custom"}
|
||||
# `custom` is an allowed field name (SDK 2.0.2 sends it on a
|
||||
# reconcile UPDATE), but the System 2 evidence counter it can hold
|
||||
# stays digest-only: only the digest scope may advance it, and only
|
||||
# by one per write. Letting the name through must not let an
|
||||
# ordinary write move that counter.
|
||||
if not self.__cross_evidence_scope and _DIGEST_COUNTER_FIELD in (
|
||||
updates.get("custom") or {}
|
||||
):
|
||||
self.__deny("update_fields")
|
||||
if not set(updates) <= allowed:
|
||||
self.__deny("update_fields")
|
||||
try:
|
||||
if "layer" in updates:
|
||||
MemoryLayer(updates["layer"])
|
||||
if "status" in updates:
|
||||
MemoryStatus(updates["status"])
|
||||
if "tag_ners" in updates and (
|
||||
not isinstance(updates["tag_ners"], list)
|
||||
or any(not isinstance(value, str) for value in updates["tag_ners"])
|
||||
):
|
||||
self.__deny("node_shape")
|
||||
expected = MemoryNode.from_dict({**current.to_dict(), **updates})
|
||||
self.__fingerprint(expected)
|
||||
except (TypeError, ValueError):
|
||||
self.__deny("node_shape")
|
||||
if not self.__in_scope(expected):
|
||||
self.__deny("scope")
|
||||
await self.__check_links(expected)
|
||||
self.__write_view(expected)
|
||||
if await self.__invoke("update_payload", node_id, updates) is not True:
|
||||
self.__deny("write_failed")
|
||||
await self.__capture(expected, allow_native_modified_time=True)
|
||||
return True
|
||||
|
||||
async def update_embedding(self, node_id: str, embedding: list[float]) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
expected = await self.__target(node_id)
|
||||
expected.embedding = deepcopy(embedding)
|
||||
self.__write_view(expected)
|
||||
if await self.__invoke("update_embedding", node_id, deepcopy(embedding)) is not True:
|
||||
self.__deny("write_failed")
|
||||
await self.__capture(expected, allow_native_modified_time=True)
|
||||
return True
|
||||
|
||||
async def delete(self, node_id: str) -> bool:
|
||||
async with _native_write_lock(), self.__lock:
|
||||
self.recheck()
|
||||
await self.__target(node_id, allow_deleted_links=True)
|
||||
if await self.__invoke("delete", node_id) is not True:
|
||||
self.__deny("write_failed")
|
||||
if await self.__read_native(node_id) is not None:
|
||||
self.__deny("write_mismatch")
|
||||
self.__deleted.add(node_id)
|
||||
self.__permissions.pop(node_id, None)
|
||||
self.__written.pop(node_id, None)
|
||||
self.__superseded.pop(node_id, None)
|
||||
self.__heads.pop(node_id, None)
|
||||
return True
|
||||
File diff suppressed because it is too large.
Load diff
@@ -144,6 +144,10 @@ class HyReconcileGuard:
|
||||
self.failures: list[str] = []
|
||||
self.bindings: HyWriteBindings | None = None
|
||||
self.imported_contents: frozenset[str] | None = None
|
||||
self.mutation_target_ids: set[str] = set()
|
||||
|
||||
def begin_write(self) -> None:
|
||||
self.mutation_target_ids.clear()
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(self.reconciler, name)
|
||||
@@ -210,6 +214,15 @@ class HyReconcileGuard:
|
||||
for operation in result.ops:
|
||||
raw_name = getattr(operation, "op", "")
|
||||
name = str(getattr(raw_name, "value", raw_name)).upper()
|
||||
if name in {"UPDATE", "SUPERSEDE", "DELETE", "SHADOW"}:
|
||||
target = getattr(operation, "memory_id", None)
|
||||
if isinstance(target, str) and target:
|
||||
self.mutation_target_ids.add(target)
|
||||
self.mutation_target_ids.update(
|
||||
key
|
||||
for key in getattr(operation, "memory_ids", ())
|
||||
if isinstance(key, str) and key
|
||||
)
|
||||
if name == "ADD":
|
||||
operation_counts["ADD"] += 1
|
||||
elif name in {"UPDATE", "MERGE", "NOOP"}:
|
||||
@@ -232,20 +245,6 @@ class HyReconcileGuard:
|
||||
"result_count": len(result.ops),
|
||||
},
|
||||
)
|
||||
timer = self.step_timer
|
||||
try:
|
||||
with (
|
||||
timer.step("write_verification", items=len(result.ops))
|
||||
if timer is not None
|
||||
else nullcontext()
|
||||
):
|
||||
await self._check_targets(result, kwargs)
|
||||
except Exception as error:
|
||||
code = str(error)
|
||||
if not code.startswith("hy_memory_write_verification_") or len(code) > 100:
|
||||
code = "hy_memory_write_verification_failed"
|
||||
self.failures.append(code)
|
||||
raise ValueError(code) from error
|
||||
return result
|
||||
|
||||
async def _check_targets(self, result: Any, kwargs: dict[str, Any]) -> None:
|
||||
|
||||
@@ -14,7 +14,7 @@ from pathlib import Path
|
||||
from typing import Literal, cast
|
||||
|
||||
HY_MEMORY_DISTRIBUTION = "hy-memory"
|
||||
HY_MEMORY_VERSION = "2.0.2"
|
||||
HY_MEMORY_VERSION = "2.0.4"
|
||||
HY_MEMORY_NAMESPACE = "governed-v2"
|
||||
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
|
||||
|
||||
@@ -31,11 +31,12 @@ _COLLECTION_LOGGERS = (
|
||||
"httpx",
|
||||
"httpcore",
|
||||
)
|
||||
_NOISY_LIBRARY_LOGGERS = ("httpx", "httpcore", "watchfiles", "jieba")
|
||||
|
||||
# One Memory deliberately stays on ultra because its existing, user-visible
|
||||
# profile/graph flow calls digest(), which the pinned public SDK exposes only
|
||||
# for ultra clients. Reader and non-reranker behavior remain pinned for
|
||||
# compatibility with public SDK 2.0.2.
|
||||
# compatibility with public SDK 2.0.4.
|
||||
HY_MEMORY_MODE = "ultra"
|
||||
# The SDK only recognises `legacy` and `hybrid` (see
|
||||
# hy_memory/pipelines/_retrieval/config.py: `ALL_READERS = (READER_LEGACY,
|
||||
@@ -139,6 +140,18 @@ def apply_diagnostic_log_level(level: DiagnosticLogLevel) -> None:
|
||||
for name in tuple(logging.Logger.manager.loggerDict):
|
||||
if any(name == root or name.startswith(f"{root}.") for root in _COLLECTION_LOGGERS):
|
||||
logging.getLogger(name).setLevel(selected)
|
||||
# Successful HTTP exchanges, file-watch notifications and tokenizer
|
||||
# initialization are useful only while actively debugging. At INFO they
|
||||
# dominated core.log and the in-app panel without describing product work.
|
||||
noisy_level = selected if selected == logging.DEBUG else max(selected, logging.WARNING)
|
||||
for root in _NOISY_LIBRARY_LOGGERS:
|
||||
logging.getLogger(root).setLevel(noisy_level)
|
||||
for name in tuple(logging.Logger.manager.loggerDict):
|
||||
if any(
|
||||
name == root or name.startswith(f"{root}.")
|
||||
for root in _NOISY_LIBRARY_LOGGERS
|
||||
):
|
||||
logging.getLogger(name).setLevel(noisy_level)
|
||||
|
||||
os.environ["MEMORY_LOG_LEVEL"] = logging.getLevelName(selected)
|
||||
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "true" if level == "debug" else "false"
|
||||
|
||||
@@ -7,7 +7,7 @@ from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord, WrittenBy
|
||||
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
|
||||
from one_memory.input_sanitization import sanitize_model_input
|
||||
|
||||
|
||||
@@ -61,17 +61,7 @@ class HyWriteBindings:
|
||||
seed.workspace,
|
||||
)
|
||||
or record.content_kind is not EvidenceContentKind.RAW_CONVERSATION
|
||||
or (record.role, record.written_by)
|
||||
not in {
|
||||
# D98: the same change as the turn assembly. A sub agent's
|
||||
# question is a `user` message marked written by the
|
||||
# agent; excluding it here would make the source context
|
||||
# disagree with what was actually sent.
|
||||
("user", WrittenBy.PERSON),
|
||||
("user", WrittenBy.AGENT),
|
||||
("assistant", WrittenBy.AGENT),
|
||||
("tool", WrittenBy.TOOL),
|
||||
}
|
||||
or record.role not in {"user", "assistant", "tool"}
|
||||
):
|
||||
raise ValueError("hy_memory_write_verification_context_scope")
|
||||
self.all_sources: list[dict[str, Any]] = [
|
||||
|
||||
@@ -152,7 +152,7 @@ class HyStepTimer:
|
||||
if value is None:
|
||||
continue
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component=STEP_COMPONENTS[name],
|
||||
event="hy.step.completed",
|
||||
outcome="succeeded",
|
||||
@@ -168,7 +168,7 @@ class HyStepTimer:
|
||||
)
|
||||
model = measured.get("model")
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="hy.bridge",
|
||||
event="hy.steps.summary",
|
||||
outcome="succeeded",
|
||||
|
||||
@@ -34,10 +34,11 @@ class ScanIssueCode(StrEnum):
|
||||
SOURCE_BUSY = "source_busy"
|
||||
UNSUPPORTED_SCHEMA = "unsupported_schema"
|
||||
BUDGET_EXCEEDED = "budget_exceeded"
|
||||
#: The file is bigger than the import's own per-file ceiling. Skipped with
|
||||
#: a reason rather than read, because one oversized history file can hold
|
||||
#: the import open long enough to take the process with it. Distinct from
|
||||
#: the access layer's `FILE_TOO_LARGE`, which is a per-root limit.
|
||||
#: The file is bigger than a per-file ceiling: either the import's own
|
||||
#: ceiling or the access gate's per-root `max_file_bytes`. Both are skipped
|
||||
#: with this reason rather than read, because one oversized history file
|
||||
#: can hold the import open long enough to take the process with it, and
|
||||
#: the user-facing cause is the same size limit.
|
||||
FILE_TOO_LARGE = "file_too_large"
|
||||
HISTORY_WINDOW_EXCLUDED = "history_window_excluded"
|
||||
|
||||
|
||||
@@ -105,7 +105,7 @@ async def iterate_in_worker[T](
|
||||
# was unwinding with, and a diagnostic is not worth an import.
|
||||
try:
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="import.async_bridge",
|
||||
event="import.source_read_finished",
|
||||
outcome="succeeded",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import sqlite3
|
||||
from contextlib import AbstractContextManager
|
||||
from datetime import UTC, datetime
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
@@ -20,6 +21,23 @@ CURSOR_MAX_SOURCE_FILE_BYTES = 64 * 1024 * 1024 * 1024
|
||||
GRANT_SCHEMA = source_grants_ddl("sqlite")
|
||||
|
||||
|
||||
class SourceAuthorizationState(StrEnum):
|
||||
"""Why a source cannot be read right now, or `ok` when it can.
|
||||
|
||||
The API used to answer `authorized=false` for all of these at once, which
|
||||
is how a revoked grant, a rebuilt history directory and a moved candidate
|
||||
all looked like the same bug and all got the same advice. The values carry
|
||||
no paths and no user content, because they are written to diagnostics.
|
||||
"""
|
||||
|
||||
OK = "ok"
|
||||
MISSING_GRANT = "missing_grant"
|
||||
REVOKED_GRANT = "revoked_grant"
|
||||
SOURCE_IDENTITY_CHANGED = "source_identity_changed"
|
||||
SOURCE_ROOT_UNAVAILABLE = "source_root_unavailable"
|
||||
CANDIDATE_CHANGED = "candidate_changed"
|
||||
|
||||
|
||||
class SourceGrantReceipt(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
@@ -146,6 +164,46 @@ class SourceGrantStore:
|
||||
raise SourceAccessError(SourceAccessErrorCode.SOURCE_IDENTITY_CHANGED)
|
||||
return authorized
|
||||
|
||||
def authorization_state(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
source_id: str,
|
||||
candidate: Path,
|
||||
) -> SourceAuthorizationState:
|
||||
"""Why reading this source would be refused.
|
||||
|
||||
Read-only and fails the same way `authorized_root` does: this names a
|
||||
state for the UI and for diagnostics, it does not widen access. The
|
||||
candidate is compared by path, so a grant that still points somewhere
|
||||
else reads as a moved candidate rather than as a rebuilt directory —
|
||||
the two need different fixes.
|
||||
"""
|
||||
with self._connect() as connection:
|
||||
row = connection.execute(
|
||||
"SELECT * FROM source_grants WHERE user_id=? AND source_id=?",
|
||||
(user_id, source_id),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return SourceAuthorizationState.MISSING_GRANT
|
||||
if row["revoked_at"] is not None or row["revision"] <= 0:
|
||||
return SourceAuthorizationState.REVOKED_GRANT
|
||||
granted = Path(row["requested_path"])
|
||||
if not granted.exists():
|
||||
return SourceAuthorizationState.SOURCE_ROOT_UNAVAILABLE
|
||||
try:
|
||||
if not root_identity_matches(granted, row["root_identity"]):
|
||||
return SourceAuthorizationState.SOURCE_IDENTITY_CHANGED
|
||||
except SourceAccessError:
|
||||
# Unreadable or gone between the two calls: there is no grant to
|
||||
# repair, only a directory to make available again.
|
||||
return SourceAuthorizationState.SOURCE_ROOT_UNAVAILABLE
|
||||
# Same directory, but the app now looks somewhere else. Re-authorizing
|
||||
# the old grant would not help; the candidate has to be granted.
|
||||
if granted != candidate:
|
||||
return SourceAuthorizationState.CANDIDATE_CHANGED
|
||||
return SourceAuthorizationState.OK
|
||||
|
||||
def revoke(self, *, user_id: str, source_id: str) -> SourceGrantReceipt:
|
||||
now = datetime.now(UTC).isoformat()
|
||||
with self._connect(writing=True) as connection:
|
||||
|
||||
@@ -106,7 +106,7 @@ class PollingLiveSyncSupervisor:
|
||||
level=(
|
||||
"warn"
|
||||
if value.state in {LiveSyncState.DEGRADED, LiveSyncState.UNAVAILABLE}
|
||||
else "info"
|
||||
else "debug"
|
||||
),
|
||||
component="import.live_sync",
|
||||
event="live_sync.state_changed",
|
||||
@@ -282,11 +282,24 @@ class PollingLiveSyncSupervisor:
|
||||
and job.status
|
||||
in {ImportStatus.PENDING, ImportStatus.RUNNING, ImportStatus.INTERRUPTED}
|
||||
]
|
||||
bounded_history_incomplete = any(
|
||||
job.source_id == source_id
|
||||
and job.operation == "history"
|
||||
and job.history_after is not None
|
||||
and job.status is not ImportStatus.COMPLETED
|
||||
for job in source_jobs
|
||||
)
|
||||
# Establish the realtime EOF independently even while history owns
|
||||
# this source. Otherwise the P1 job starts from the P2 checkpoint
|
||||
# (or from zero) and becomes a second full-history scan.
|
||||
adapter = self.import_supervisor.adapters[source_id]
|
||||
await self._live_cursors(source_id, adapter)
|
||||
live_cursors = await self._live_cursors(source_id, adapter)
|
||||
# A bounded onboarding history is the authority for everything up
|
||||
# to its EOF. An adapter without a native live baseline has no safe
|
||||
# realtime cursor until that history finishes; starting a sync from
|
||||
# zero here would bypass `history_after` and import the old corpus.
|
||||
if bounded_history_incomplete and not live_cursors:
|
||||
return None
|
||||
# Only one scan of a source may be in flight: they share the
|
||||
# cursor, and two readers means every file read twice. With a
|
||||
# catch-up still holding the source there is no free moment to ask
|
||||
|
||||
@@ -156,7 +156,7 @@ class FileWatchLiveSync:
|
||||
if source is not None
|
||||
):
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="import.live_watch",
|
||||
event="live_sync.hook_received",
|
||||
outcome="queued",
|
||||
@@ -190,7 +190,7 @@ class FileWatchLiveSync:
|
||||
try:
|
||||
await self._trigger_past(source_id, observed)
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="import.live_watch",
|
||||
event="live_sync.hook_completed",
|
||||
outcome="written",
|
||||
|
||||
@@ -79,6 +79,8 @@ class EvidenceInput(BaseModel):
|
||||
source_ordinal: str | None = Field(default=None, min_length=1, max_length=256)
|
||||
content_kind: EvidenceContentKind = EvidenceContentKind.RAW_CONVERSATION
|
||||
role: str = Field(min_length=1, max_length=128)
|
||||
# Legacy import/storage metadata only. Protocol `role` is authoritative;
|
||||
# this value must not affect identity, admission, scheduling or ownership.
|
||||
written_by: WrittenBy
|
||||
created_at: datetime | None = None
|
||||
verbatim: str = Field(min_length=1, max_length=MAX_VERBATIM_CHARACTERS)
|
||||
@@ -149,17 +151,22 @@ class EvidenceInput(BaseModel):
|
||||
)
|
||||
)
|
||||
else:
|
||||
# `role` is the protocol-level speaker identity. `written_by` was a
|
||||
# parser guess and differed between two imports of the same host
|
||||
# message (notably sub-agent/user and unknown/user), minting two
|
||||
# Evidence rows. Keep it out of every new identity.
|
||||
#
|
||||
# No workspace and no source_ordinal in the derived key: the hook
|
||||
# lane has neither (it reports no workspace and a constant ordinal),
|
||||
# so including either gave the same message a different key per lane
|
||||
# and stored it twice. Both are record metadata here, not identity.
|
||||
identity = "\0".join(
|
||||
(
|
||||
"derived-v3",
|
||||
"derived-v4",
|
||||
self.user_id,
|
||||
self.source_id,
|
||||
self.conversation_id,
|
||||
self.written_by,
|
||||
self.role,
|
||||
self.content_hash,
|
||||
)
|
||||
)
|
||||
@@ -168,6 +175,54 @@ class EvidenceInput(BaseModel):
|
||||
# two rows. The capture time is record metadata, not identity.
|
||||
return hashlib.sha256(identity.encode()).hexdigest()
|
||||
|
||||
@property
|
||||
def written_by_derived_v3_idempotency_key(self) -> str:
|
||||
"""The last identity that treated the parser's author guess as canonical.
|
||||
|
||||
New builds key by protocol role. During an upgrade this key lets a
|
||||
rescan find and reuse the row already stored by derived-v3 instead of
|
||||
inserting the same host message under a second Evidence id.
|
||||
"""
|
||||
identity = "\0".join(
|
||||
(
|
||||
"derived-v3",
|
||||
self.user_id,
|
||||
self.source_id,
|
||||
self.conversation_id,
|
||||
self.written_by,
|
||||
self.content_hash,
|
||||
)
|
||||
)
|
||||
return hashlib.sha256(identity.encode()).hexdigest()
|
||||
|
||||
@property
|
||||
def written_by_derived_v3_idempotency_keys(self) -> tuple[str, ...]:
|
||||
"""Every legacy author guess this protocol role may have carried."""
|
||||
guesses = {
|
||||
"user": (
|
||||
WrittenBy.PERSON,
|
||||
WrittenBy.AGENT,
|
||||
WrittenBy.UNKNOWN,
|
||||
),
|
||||
"assistant": (WrittenBy.AGENT, WrittenBy.UNKNOWN, WrittenBy.PERSON),
|
||||
"tool": (WrittenBy.TOOL, WrittenBy.UNKNOWN),
|
||||
}.get(self.role, tuple(WrittenBy))
|
||||
ordered = (self.written_by, *guesses)
|
||||
keys: list[str] = []
|
||||
for written_by in dict.fromkeys(ordered):
|
||||
identity = "\0".join(
|
||||
(
|
||||
"derived-v3",
|
||||
self.user_id,
|
||||
self.source_id,
|
||||
self.conversation_id,
|
||||
written_by,
|
||||
self.content_hash,
|
||||
)
|
||||
)
|
||||
keys.append(hashlib.sha256(identity.encode()).hexdigest())
|
||||
return tuple(keys)
|
||||
|
||||
@property
|
||||
def legacy_message_idempotency_key(self) -> str | None:
|
||||
"""The pre-workspace key for a message-shaped input, when one exists.
|
||||
@@ -212,7 +267,13 @@ class EvidenceInput(BaseModel):
|
||||
"""
|
||||
return self.derived_v2_idempotency_key("derived-v2", workspace_agnostic=False)
|
||||
|
||||
def derived_v2_idempotency_key(self, label: str, *, workspace_agnostic: bool) -> str:
|
||||
def derived_v2_idempotency_key(
|
||||
self,
|
||||
label: str,
|
||||
*,
|
||||
workspace_agnostic: bool,
|
||||
written_by: WrittenBy | None = None,
|
||||
) -> str:
|
||||
identity = "\0".join(
|
||||
(
|
||||
label,
|
||||
@@ -220,7 +281,7 @@ class EvidenceInput(BaseModel):
|
||||
self.source_id,
|
||||
*([] if workspace_agnostic else [self.workspace or "\uffffnull"]),
|
||||
self.conversation_id,
|
||||
self.written_by,
|
||||
written_by or self.written_by,
|
||||
self.created_at.isoformat() if self.created_at else "",
|
||||
self.source_ordinal or "",
|
||||
# v2 hashed the cleaned body; v1 predates raw_verbatim entirely,
|
||||
@@ -230,6 +291,26 @@ class EvidenceInput(BaseModel):
|
||||
)
|
||||
return hashlib.sha256(identity.encode()).hexdigest()
|
||||
|
||||
@property
|
||||
def older_written_by_idempotency_keys(self) -> tuple[str, ...]:
|
||||
keys: list[str] = []
|
||||
for written_by in WrittenBy:
|
||||
keys.append(
|
||||
self.derived_v2_idempotency_key(
|
||||
"derived-v1",
|
||||
workspace_agnostic=True,
|
||||
written_by=written_by,
|
||||
)
|
||||
)
|
||||
keys.append(
|
||||
self.derived_v2_idempotency_key(
|
||||
"derived-v2",
|
||||
workspace_agnostic=False,
|
||||
written_by=written_by,
|
||||
)
|
||||
)
|
||||
return tuple(keys)
|
||||
|
||||
@property
|
||||
def evidence_id(self) -> str:
|
||||
return str(uuid5(EVIDENCE_NAMESPACE, self.idempotency_key))
|
||||
@@ -249,6 +330,7 @@ class EvidenceRecord(BaseModel):
|
||||
source_ordinal: str | None
|
||||
content_kind: EvidenceContentKind
|
||||
role: str
|
||||
# Retained while old databases and export formats still carry the column.
|
||||
written_by: WrittenBy
|
||||
created_at: datetime | None
|
||||
verbatim: str
|
||||
|
||||
@@ -199,7 +199,7 @@ class ImportRunner:
|
||||
raise ValueError("Import job source does not match adapter source")
|
||||
job, runner_token = await in_the_background(self.job_store.claim, job_id)
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="import.parser",
|
||||
event="import.parse.started",
|
||||
outcome="started",
|
||||
@@ -310,7 +310,7 @@ class ImportRunner:
|
||||
if event.kind == "progress" and str(event.phase) != last_phase:
|
||||
phase = str(event.phase)
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="import.parser",
|
||||
event="import.phase_changed",
|
||||
outcome="changed",
|
||||
@@ -516,7 +516,7 @@ class ImportRunner:
|
||||
async def _emit_parse_result(self, job: ImportJob, started: float) -> None:
|
||||
succeeded = job.status is ImportStatus.COMPLETED
|
||||
diagnostic_event(
|
||||
level="info" if succeeded else "warn",
|
||||
level="debug" if succeeded else "warn",
|
||||
component="import.parser",
|
||||
event="import.parse.completed",
|
||||
outcome="succeeded" if succeeded else str(job.status),
|
||||
|
||||
@@ -141,11 +141,18 @@ def parse_codex_record(
|
||||
if role == "user":
|
||||
written_by = _user_author(record, payload, session)
|
||||
# Machine inputs may quote a whole transcript, including user_query tags.
|
||||
# Session provenance is not proof of who wrote any particular future turn.
|
||||
if written_by is WrittenBy.PERSON:
|
||||
# Protocol role is authoritative; host author metadata does not change
|
||||
# how a user message is cleaned.
|
||||
if not (
|
||||
content.startswith(
|
||||
"The following is the Codex agent history whose request action you are assessing."
|
||||
)
|
||||
and ">>> TRANSCRIPT START" in content
|
||||
and ">>> TRANSCRIPT END" in content
|
||||
):
|
||||
content = strip_host_wrappers(content)
|
||||
if not content:
|
||||
return ParsedSkip(ScanIssueCode.NO_TEXT)
|
||||
if not content:
|
||||
return ParsedSkip(ScanIssueCode.NO_TEXT)
|
||||
return ParsedMessage(
|
||||
conversation_id=conversation_from_relative_path(context.relative_path),
|
||||
message_id=first_string(payload.get("id"), record.get("id")),
|
||||
|
||||
@@ -44,6 +44,7 @@ from one_memory.imports.discovery import (
|
||||
discovered_off_loop,
|
||||
)
|
||||
from one_memory.imports.models import (
|
||||
MAX_VERBATIM_CHARACTERS,
|
||||
EvidenceContentKind,
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
@@ -65,6 +66,20 @@ from one_memory.imports.sources.parsing import parse_timestamp
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _clamp_verbatim(text: str) -> str:
|
||||
"""Keep one pathological record from violating EvidenceInput's limit.
|
||||
|
||||
A single pasted dump larger than MAX_VERBATIM_CHARACTERS used to fail
|
||||
validation, retry, and then wedge the whole history import on the same
|
||||
record. Keep the head and make the truncation explicit and actionable.
|
||||
"""
|
||||
if len(text) <= MAX_VERBATIM_CHARACTERS:
|
||||
return text
|
||||
marker = f"\n…[truncated: original {len(text)} chars]"
|
||||
keep = max(0, MAX_VERBATIM_CHARACTERS - len(marker))
|
||||
return text[:keep] + marker
|
||||
|
||||
#: At most one read heartbeat per second. Faster would be more writes than the
|
||||
#: counts are worth; slower leaves the job's visible state stale.
|
||||
_READ_PROGRESS_INTERVAL_SECONDS = 1.0
|
||||
@@ -355,7 +370,7 @@ class JsonlSourceAdapter:
|
||||
source_file, checkpoint, header = await asyncio.to_thread(
|
||||
self._prepare, listed, options
|
||||
)
|
||||
_log.info(
|
||||
_log.debug(
|
||||
"import_file_read source=%s file=%d/%d file_bytes=%d planned_bytes=%d",
|
||||
self.descriptor.source_id,
|
||||
index,
|
||||
@@ -376,7 +391,7 @@ class JsonlSourceAdapter:
|
||||
else DEFAULT_IMPORT_MAX_FILE_BYTES
|
||||
)
|
||||
if source_file.size > budget:
|
||||
_log.info(
|
||||
_log.debug(
|
||||
"import_file_budgeted source=%s file=%d/%d file_bytes=%d budget=%d path=%s",
|
||||
self.descriptor.source_id,
|
||||
index,
|
||||
@@ -1076,8 +1091,11 @@ def to_evidence(
|
||||
role=parsed.role,
|
||||
written_by=parsed.written_by,
|
||||
created_at=parsed.created_at,
|
||||
verbatim=parsed.verbatim,
|
||||
raw_verbatim=parsed.raw_verbatim,
|
||||
verbatim=_clamp_verbatim(parsed.verbatim),
|
||||
raw_verbatim=(
|
||||
None if parsed.raw_verbatim is None
|
||||
else _clamp_verbatim(parsed.raw_verbatim)
|
||||
),
|
||||
origin_meta={
|
||||
EvidenceOrigin.FILE: OriginMetadata(
|
||||
source_version=origin.source_version or source_version,
|
||||
@@ -1102,6 +1120,11 @@ def discovery_failure_code(
|
||||
) -> ScanIssueCode:
|
||||
if failure.reason_code is SourceAccessErrorCode.BUDGET_EXCEEDED:
|
||||
return ScanIssueCode.BUDGET_EXCEEDED
|
||||
if failure.reason_code is SourceAccessErrorCode.FILE_TOO_LARGE:
|
||||
# The file is readable and its format is fine; only its size is over
|
||||
# the access gate. Reporting that as `unsupported_schema` sends the
|
||||
# user looking for a parser problem that does not exist.
|
||||
return ScanIssueCode.FILE_TOO_LARGE
|
||||
return ScanIssueCode.UNSUPPORTED_SCHEMA
|
||||
|
||||
|
||||
@@ -1109,6 +1132,8 @@ def _scan_failure_code(code: SourceAccessErrorCode) -> ScanIssueCode:
|
||||
"""Map a per-file open failure onto a machine code, never onto free text."""
|
||||
if code is SourceAccessErrorCode.BUDGET_EXCEEDED:
|
||||
return ScanIssueCode.BUDGET_EXCEEDED
|
||||
if code is SourceAccessErrorCode.FILE_TOO_LARGE:
|
||||
return ScanIssueCode.FILE_TOO_LARGE
|
||||
if code is SourceAccessErrorCode.SYMLINK_NOT_ALLOWED:
|
||||
return ScanIssueCode.UNSUPPORTED_SCHEMA
|
||||
if code in {
|
||||
|
||||
@@ -7,7 +7,7 @@ import sqlite3
|
||||
import stat
|
||||
from collections.abc import AsyncIterator, Iterator
|
||||
from dataclasses import dataclass, replace
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from datetime import UTC, datetime, timedelta, timezone
|
||||
from functools import partial
|
||||
|
||||
from pydantic import JsonValue
|
||||
@@ -82,6 +82,11 @@ _CURSOR_HOST_CONTEXT_START = re.compile(
|
||||
)
|
||||
_CURSOR_USER_QUERY = re.compile(r"<user_query\b")
|
||||
_CURSOR_TIMESTAMP_PREFIX = re.compile(r"<timestamp>[\s\S]*?</timestamp>\s*")
|
||||
_CURSOR_TIMESTAMP_VALUE = re.compile(
|
||||
r"<timestamp>\s*"
|
||||
r"([A-Za-z]+,\s+[A-Za-z]{3}\s+\d{1,2},\s+\d{4},\s+\d{1,2}:\d{2}\s+[AP]M)"
|
||||
r"\s+\(UTC([+-]\d{1,2})(?::(\d{2}))?\)\s*</timestamp>",
|
||||
)
|
||||
_CURSOR_IMAGE_FILES_PREFIX = re.compile(r"\[Image\]\s*<image_files>[\s\S]*?</image_files>\s*")
|
||||
_CURSOR_BARE_IMAGE_PREFIX = re.compile(r"(?:\[Image\]\s*)+")
|
||||
|
||||
@@ -123,6 +128,24 @@ def _strip_cursor_timestamp(content: str | None) -> str | None:
|
||||
return _CURSOR_TIMESTAMP_PREFIX.sub("", content, count=1) if content else None
|
||||
|
||||
|
||||
def _cursor_host_timestamp(content: str | None) -> datetime | None:
|
||||
"""Parse Cursor's user-visible timestamp envelope into UTC."""
|
||||
if not content:
|
||||
return None
|
||||
match = _CURSOR_TIMESTAMP_VALUE.search(content)
|
||||
if match is None:
|
||||
return None
|
||||
try:
|
||||
local = datetime.strptime(match[1], "%A, %b %d, %Y, %I:%M %p")
|
||||
hours = int(match[2])
|
||||
minutes = int(match[3] or "0")
|
||||
sign = -1 if hours < 0 else 1
|
||||
offset = timedelta(hours=hours, minutes=sign * minutes)
|
||||
return local.replace(tzinfo=timezone(offset)).astimezone(UTC)
|
||||
except (ValueError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def _strip_cursor_host_prefix(content: str | None) -> str | None:
|
||||
if not content:
|
||||
return content
|
||||
@@ -398,10 +421,7 @@ class CursorUnifiedSourceAdapter:
|
||||
if (
|
||||
isinstance(event, ScannedEvidence)
|
||||
and event.evidence.conversation_id in transcript_ids
|
||||
and not (
|
||||
event.evidence.written_by is WrittenBy.AGENT
|
||||
and is_host_generated_user_turn(event.evidence.verbatim)
|
||||
)
|
||||
and not is_host_generated_user_turn(event.evidence.verbatim)
|
||||
):
|
||||
yield ScanSkip(
|
||||
cursor=event.cursor,
|
||||
@@ -426,10 +446,7 @@ def _shadowed_by_transcript(event: ScanEvent, transcript_ids: set[str]) -> ScanE
|
||||
if (
|
||||
isinstance(event, ScannedEvidence)
|
||||
and event.evidence.conversation_id in transcript_ids
|
||||
and not (
|
||||
event.evidence.written_by is WrittenBy.AGENT
|
||||
and is_host_generated_user_turn(event.evidence.verbatim)
|
||||
)
|
||||
and not is_host_generated_user_turn(event.evidence.verbatim)
|
||||
):
|
||||
return ScanSkip(
|
||||
cursor=event.cursor,
|
||||
@@ -703,6 +720,7 @@ def _parse_transcript_record(
|
||||
project_name = path_parts[path_parts.index("projects") + 1] if "projects" in path_parts else ""
|
||||
session_name = context.relative_path.rsplit("/", 1)[-1].removesuffix(".jsonl")
|
||||
is_subagent = f"{project_name}\0{session_name}" in subagent_ids
|
||||
host_timestamp = _cursor_host_timestamp(content) if role == "user" else None
|
||||
created_at = _transcript_fallback_time(context)
|
||||
native_message_id = record.get("id") or message.get("id")
|
||||
message_id = (
|
||||
@@ -728,7 +746,16 @@ def _parse_transcript_record(
|
||||
created_at=created_at,
|
||||
workspace=workspace,
|
||||
record_type="cursor_transcript",
|
||||
origin_metadata=OriginMetadata(raw_meta={"time_basis": "file_mtime"}),
|
||||
origin_metadata=OriginMetadata(
|
||||
raw_meta={
|
||||
"time_basis": "file_mtime",
|
||||
**(
|
||||
{"turn_event_at": host_timestamp.isoformat()}
|
||||
if host_timestamp is not None
|
||||
else {}
|
||||
),
|
||||
}
|
||||
),
|
||||
),
|
||||
tool_blocks_evidence(message.get("content"), block_types=TOOL_BLOCKS),
|
||||
)
|
||||
@@ -819,6 +846,13 @@ def _read_store_messages(
|
||||
reason_code=ScanIssueCode.UNKNOWN_RECORD,
|
||||
)
|
||||
continue
|
||||
host_timestamp = (
|
||||
_cursor_host_timestamp(content) if role == "user" else None
|
||||
)
|
||||
fallback_timestamp = datetime.fromtimestamp(
|
||||
source_file.modified_ns / 1_000_000_000,
|
||||
tz=UTC,
|
||||
) + timedelta(microseconds=rowid)
|
||||
parsed = ParsedMessage(
|
||||
conversation_id=conversation_id,
|
||||
message_id=str(record.get("id") or row["id"]),
|
||||
@@ -829,8 +863,19 @@ def _read_store_messages(
|
||||
else written_by_user
|
||||
),
|
||||
verbatim=cleaned,
|
||||
created_at=fallback_timestamp,
|
||||
workspace=_store_workspace(source_file),
|
||||
record_type="cursor_store_blob",
|
||||
origin_metadata=OriginMetadata(
|
||||
raw_meta={
|
||||
"time_basis": "file_mtime",
|
||||
**(
|
||||
{"turn_event_at": host_timestamp.isoformat()}
|
||||
if host_timestamp is not None
|
||||
else {}
|
||||
),
|
||||
}
|
||||
),
|
||||
)
|
||||
yield ScannedEvidence(
|
||||
cursor=cursor,
|
||||
|
||||
@@ -57,7 +57,8 @@ _UNANSWERED_PREFIXES = (
|
||||
|
||||
|
||||
def _is_visible_user_row(role: str, written_by: str, verbatim: str) -> bool:
|
||||
if role != "user" or written_by != WrittenBy.PERSON.value:
|
||||
del written_by # Legacy storage metadata; protocol role is authoritative.
|
||||
if role != "user":
|
||||
return False
|
||||
return not verbatim.lstrip().lower().startswith(_UNANSWERED_PREFIXES)
|
||||
|
||||
@@ -468,16 +469,54 @@ class EvidenceStore:
|
||||
if anchored is not None:
|
||||
self._write_origins(connection, anchored, evidence.origin_meta, now)
|
||||
workspace, project = self._resolve_conversation_attribution(connection, evidence)
|
||||
connection.execute(
|
||||
"""
|
||||
UPDATE evidence
|
||||
SET workspace = COALESCE(workspace, ?),
|
||||
project = COALESCE(project, ?),
|
||||
updated_at = ?
|
||||
WHERE evidence_id = ?
|
||||
""",
|
||||
(workspace, project, now, anchored),
|
||||
)
|
||||
existing = connection.execute(
|
||||
"SELECT role,revision FROM evidence WHERE evidence_id=?",
|
||||
(anchored,),
|
||||
).fetchone()
|
||||
if existing is None:
|
||||
raise RuntimeError("anchored evidence disappeared")
|
||||
if str(existing["role"]) != evidence.role:
|
||||
was_revised = True
|
||||
revision = int(existing["revision"]) + 1
|
||||
connection.execute(
|
||||
"""
|
||||
UPDATE evidence
|
||||
SET workspace=COALESCE(workspace,?),
|
||||
project=COALESCE(project,?),
|
||||
content_kind=?,role=?,created_at=COALESCE(?,created_at),
|
||||
verbatim=?,content_hash=?,revision=?,updated_at=?
|
||||
WHERE evidence_id=?
|
||||
""",
|
||||
(
|
||||
workspace,
|
||||
project,
|
||||
evidence.content_kind,
|
||||
evidence.role,
|
||||
evidence.created_at.isoformat() if evidence.created_at else None,
|
||||
evidence.verbatim,
|
||||
evidence.content_hash,
|
||||
revision,
|
||||
now,
|
||||
anchored,
|
||||
),
|
||||
)
|
||||
self._insert_current_version(
|
||||
connection,
|
||||
anchored,
|
||||
revision=revision,
|
||||
changed_at=now,
|
||||
)
|
||||
else:
|
||||
connection.execute(
|
||||
"""
|
||||
UPDATE evidence
|
||||
SET workspace=COALESCE(workspace,?),
|
||||
project=COALESCE(project,?),
|
||||
updated_at=?
|
||||
WHERE evidence_id=?
|
||||
""",
|
||||
(workspace, project, now, anchored),
|
||||
)
|
||||
record = self._get_with_connection(connection, anchored)
|
||||
if record is None:
|
||||
raise RuntimeError("anchored evidence disappeared")
|
||||
@@ -485,7 +524,11 @@ class EvidenceStore:
|
||||
from one_memory import ingest_journal
|
||||
|
||||
ingest_journal.record(connection, anchored, cause="import")
|
||||
return PutEvidenceResult(record=record, was_created=False, was_revised=False)
|
||||
return PutEvidenceResult(
|
||||
record=record,
|
||||
was_created=False,
|
||||
was_revised=was_revised,
|
||||
)
|
||||
insert = connection.execute(
|
||||
"""
|
||||
INSERT INTO evidence (
|
||||
@@ -803,27 +846,20 @@ class EvidenceStore:
|
||||
and same_scope[0]["idempotency_key"] != evidence.idempotency_key
|
||||
):
|
||||
return str(same_scope[0]["evidence_id"])
|
||||
elif (legacy_id := claim_legacy_key(evidence.legacy_derived_idempotency_key)) is not None:
|
||||
return legacy_id
|
||||
if not evidence.message_id and not evidence.correlation_id:
|
||||
# BUG-03 moved the derived key off the capture time and onto the
|
||||
# pre-cleaning text. A replay of a row stored under the old scheme
|
||||
# has to be recognized as the same message.
|
||||
replayed = connection.execute(
|
||||
"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE idempotency_key = ? AND user_id = ? AND source_id = ?
|
||||
AND conversation_id = ?
|
||||
""",
|
||||
elif (
|
||||
legacy_id := next(
|
||||
(
|
||||
evidence.prior_derived_idempotency_key,
|
||||
evidence.user_id,
|
||||
evidence.source_id,
|
||||
evidence.conversation_id,
|
||||
anchored
|
||||
for key in (
|
||||
*evidence.written_by_derived_v3_idempotency_keys,
|
||||
*evidence.older_written_by_idempotency_keys,
|
||||
)
|
||||
if (anchored := claim_legacy_key(key)) is not None
|
||||
),
|
||||
).fetchone()
|
||||
if replayed is not None:
|
||||
return str(replayed["evidence_id"])
|
||||
None,
|
||||
)
|
||||
) is not None:
|
||||
return legacy_id
|
||||
if (
|
||||
EvidenceOrigin.HOOK in evidence.origins
|
||||
and evidence.message_id is None
|
||||
@@ -838,7 +874,7 @@ class EvidenceStore:
|
||||
ON hook.evidence_id=e.evidence_id AND hook.origin='hook'
|
||||
WHERE e.user_id=? AND e.source_id=? AND e.conversation_id=?
|
||||
AND (e.workspace IS ? OR e.workspace IS NULL OR ? IS NULL)
|
||||
AND e.role=? AND e.written_by=? AND e.content_hash=?
|
||||
AND e.role=? AND e.content_hash=?
|
||||
AND e.message_id IS NOT NULL AND hook.evidence_id IS NULL
|
||||
AND ABS((julianday(e.created_at)-julianday(?))*86400.0) <= 5.0
|
||||
ORDER BY ABS((julianday(e.created_at)-julianday(?))*86400.0), e.evidence_id
|
||||
@@ -851,7 +887,6 @@ class EvidenceStore:
|
||||
evidence.workspace,
|
||||
evidence.workspace,
|
||||
evidence.role,
|
||||
evidence.written_by,
|
||||
evidence.content_hash,
|
||||
evidence.created_at.isoformat(),
|
||||
evidence.created_at.isoformat(),
|
||||
@@ -872,7 +907,7 @@ class EvidenceStore:
|
||||
AND (e.workspace IS ? OR e.workspace IS NULL OR ? IS NULL)
|
||||
-- Unary plus keeps the coverage index from looking better than
|
||||
-- the message index, which narrows to this exact text.
|
||||
AND e.role = ? AND +e.written_by = ? AND e.content_hash = ?
|
||||
AND e.role = ? AND e.content_hash = ?
|
||||
AND e.message_id IS NULL
|
||||
""",
|
||||
(
|
||||
@@ -882,7 +917,6 @@ class EvidenceStore:
|
||||
evidence.workspace,
|
||||
evidence.workspace,
|
||||
evidence.role,
|
||||
evidence.written_by,
|
||||
evidence.content_hash,
|
||||
),
|
||||
).fetchall()
|
||||
@@ -1247,7 +1281,7 @@ class EvidenceStore:
|
||||
rows = connection.execute(
|
||||
"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
|
||||
WHERE user_id = 'local' AND role = 'user'
|
||||
ORDER BY imported_at, evidence_id
|
||||
LIMIT ?
|
||||
""",
|
||||
@@ -1301,7 +1335,7 @@ class EvidenceStore:
|
||||
rows = connection.execute(
|
||||
f"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
|
||||
WHERE user_id = 'local' AND role = 'user'
|
||||
AND ({predicates})
|
||||
ORDER BY imported_at, evidence_id
|
||||
LIMIT ?
|
||||
@@ -1333,7 +1367,7 @@ class EvidenceStore:
|
||||
rows = connection.execute(
|
||||
f"""
|
||||
SELECT evidence_id FROM evidence
|
||||
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
|
||||
WHERE user_id = 'local' AND role = 'user'
|
||||
AND workspace IS NOT NULL
|
||||
AND ({predicates})
|
||||
ORDER BY imported_at, evidence_id
|
||||
@@ -1476,7 +1510,7 @@ class EvidenceStore:
|
||||
)
|
||||
values: list[object] = [user_id, *(source_ids or ())]
|
||||
visible_user = """
|
||||
role='user' AND written_by='person' AND NOT (
|
||||
role='user' AND NOT (
|
||||
lower(ltrim(verbatim)) LIKE '<codex_internal_context%' OR
|
||||
lower(ltrim(verbatim)) LIKE '<environment_context>%' OR
|
||||
lower(ltrim(verbatim)) LIKE '<heartbeat>%' OR
|
||||
@@ -1485,7 +1519,7 @@ class EvidenceStore:
|
||||
)
|
||||
"""
|
||||
selected_visible_user = """
|
||||
e.role='user' AND e.written_by='person' AND NOT (
|
||||
e.role='user' AND NOT (
|
||||
lower(ltrim(e.verbatim)) LIKE '<codex_internal_context%' OR
|
||||
lower(ltrim(e.verbatim)) LIKE '<environment_context>%' OR
|
||||
lower(ltrim(e.verbatim)) LIKE '<heartbeat>%' OR
|
||||
@@ -1638,7 +1672,7 @@ class EvidenceStore:
|
||||
if seed is None:
|
||||
return ConversationTurnPage((), {}, 0, offset, limit)
|
||||
visible_user = """
|
||||
role='user' AND written_by='person' AND NOT (
|
||||
role='user' AND NOT (
|
||||
lower(ltrim(verbatim)) LIKE '<codex_internal_context%' OR
|
||||
lower(ltrim(verbatim)) LIKE '<environment_context>%' OR
|
||||
lower(ltrim(verbatim)) LIKE '<heartbeat>%' OR
|
||||
@@ -1731,7 +1765,7 @@ class EvidenceStore:
|
||||
AND first.conversation_id = evidence.conversation_id
|
||||
AND first.workspace IS evidence.workspace
|
||||
AND (
|
||||
(first.role = 'user' AND first.written_by = 'person')
|
||||
first.role = 'user'
|
||||
OR first.content_kind = 'memory_content'
|
||||
)
|
||||
ORDER BY julianday(COALESCE(first.created_at, first.imported_at)),
|
||||
@@ -2125,8 +2159,6 @@ class EvidenceStore:
|
||||
(
|
||||
existing["content_kind"] != evidence.content_kind,
|
||||
existing["role"] != evidence.role,
|
||||
evidence.written_by is not WrittenBy.UNKNOWN
|
||||
and existing["written_by"] != evidence.written_by,
|
||||
existing["content_hash"] != evidence.content_hash,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -111,7 +111,7 @@ class ImportSupervisor:
|
||||
)
|
||||
)
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="import.supervisor",
|
||||
event="import.queued",
|
||||
outcome="queued",
|
||||
@@ -198,7 +198,7 @@ class ImportSupervisor:
|
||||
started = monotonic()
|
||||
failure_type: str | None = None
|
||||
diagnostic_event(
|
||||
level="info",
|
||||
level="debug",
|
||||
component="import.supervisor",
|
||||
event="import.started",
|
||||
outcome="started",
|
||||
@@ -316,6 +316,10 @@ class ImportSupervisor:
|
||||
"source_id": current.source_id,
|
||||
"mode": current.operation,
|
||||
"count": current.stored,
|
||||
"stored_count": current.stored,
|
||||
"revised_count": current.revised,
|
||||
"skipped_count": current.skipped,
|
||||
"error_count": current.errors,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -85,15 +85,14 @@ def _state(connection: sqlite3.Connection, evidence_id: str) -> dict[str, str |
|
||||
value = evidence.model_dump(mode="json")
|
||||
# The stored normalized content_hash is insufficient for exact-content leases.
|
||||
content = _digest(value["verbatim"])
|
||||
material = _digest(
|
||||
{key: value[key] for key in ("verbatim", "role", "written_by", "content_kind")}
|
||||
)
|
||||
material = _digest({key: value[key] for key in ("verbatim", "role", "content_kind")})
|
||||
origins = _digest(value["origin_meta"])
|
||||
identity = _digest(
|
||||
{
|
||||
key: item
|
||||
for key, item in value.items()
|
||||
if key not in {"verbatim", "origin_meta", "imported_at", "updated_at"}
|
||||
if key
|
||||
not in {"verbatim", "written_by", "origin_meta", "imported_at", "updated_at"}
|
||||
}
|
||||
)
|
||||
return dict(
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import json
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import TextIO, cast
|
||||
|
||||
@@ -104,7 +105,13 @@ def capture_hook(source_id: str, payload: object, database: Path) -> bool:
|
||||
# Cursor also runs ~/.claude/settings.json hooks; its own hook already files the turn.
|
||||
if source_id != "cursor" and "cursor_version" in payload:
|
||||
return False
|
||||
if not database.is_file() or not _sync_open(database, source_id):
|
||||
if not database.is_file():
|
||||
return False
|
||||
rejection = capture_rejection(database, source_id)
|
||||
if rejection is not None:
|
||||
# Silent, and the reason is counted rather than printed: a hook that
|
||||
# spoke would put this into the host's turn as text.
|
||||
record_capture_rejection(database, source_id, rejection)
|
||||
return False
|
||||
if source_id == "pi" and _is_automated_session(first_string(payload.get("cwd"))):
|
||||
return False
|
||||
@@ -139,17 +146,79 @@ def capture_hook(source_id: str, payload: object, database: Path) -> bool:
|
||||
|
||||
|
||||
def _sync_open(database: Path, source_id: str) -> bool:
|
||||
return capture_rejection(database, source_id) is None
|
||||
|
||||
|
||||
class CaptureRejection(StrEnum):
|
||||
"""Why a hook's turn was dropped, or nothing at all.
|
||||
|
||||
Capture fails open on purpose: a hook that raises or prints would be read
|
||||
back into the host's turn, so a refusal has to be silent and exit 0. It was
|
||||
also completely invisible, which is why "持续同步开着但一条都没进" could not be
|
||||
told apart from "capture 坏了". These values name the reason without any
|
||||
payload, path, session or message — the only part that may be recorded.
|
||||
"""
|
||||
|
||||
SYNC_DISABLED = "sync_disabled"
|
||||
GRANT_MISSING = "grant_missing"
|
||||
GRANT_REVOKED = "grant_revoked"
|
||||
|
||||
|
||||
def capture_rejection(database: Path, source_id: str) -> CaptureRejection | None:
|
||||
"""None when this turn may be captured, otherwise why it may not."""
|
||||
from one_memory.memory_server.config_state import ConfigStore
|
||||
|
||||
if not ConfigStore(database.parent).sync_enabled(source_id):
|
||||
return False
|
||||
return CaptureRejection.SYNC_DISABLED
|
||||
with main_database(database).read() as connection:
|
||||
row = connection.execute(
|
||||
"SELECT 1 FROM source_grants WHERE user_id='local' AND source_id=? "
|
||||
"AND revoked_at IS NULL",
|
||||
"SELECT revoked_at FROM source_grants WHERE user_id='local' AND source_id=?",
|
||||
(source_id,),
|
||||
).fetchone()
|
||||
return row is not None
|
||||
if row is None:
|
||||
return CaptureRejection.GRANT_MISSING
|
||||
# Indexed, not named: this connection has no row factory, and a hook is
|
||||
# not the place to add one.
|
||||
if row[0] is not None:
|
||||
return CaptureRejection.GRANT_REVOKED
|
||||
return None
|
||||
|
||||
|
||||
def record_capture_rejection(
|
||||
database: Path,
|
||||
source_id: str,
|
||||
reason: CaptureRejection,
|
||||
*,
|
||||
window_seconds: float = 600.0,
|
||||
) -> None:
|
||||
"""Note one dropped turn, at most once per window per source and reason.
|
||||
|
||||
Best effort by construction: a hook runs inside somebody's turn, so a sink
|
||||
that is slow, full or read-only must never surface here. The record is a
|
||||
count and nothing else — no payload, no path, no session or conversation
|
||||
id — because this file is meant to be handed to support as is.
|
||||
|
||||
One small file per source and reason, rather than a line per turn: capture
|
||||
runs on every prompt, and a log that grew with the conversation would be a
|
||||
privacy problem wearing a diagnosis as a disguise.
|
||||
"""
|
||||
try:
|
||||
directory = database.parent / "logs"
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
marker = directory / f"capture-rejection-{source_id}-{reason.value}.json"
|
||||
now = time.time()
|
||||
payload: dict[str, object] = {"source_id": source_id, "reason": reason.value}
|
||||
if marker.exists() and now - marker.stat().st_mtime < window_seconds:
|
||||
payload = json.loads(marker.read_text(encoding="utf8"))
|
||||
previous = payload.get("count", 1)
|
||||
payload["count"] = int(previous) + 1 if isinstance(previous, (str, int, float)) else 2
|
||||
else:
|
||||
payload["count"] = 1
|
||||
payload["window_started_at"] = datetime.fromtimestamp(now, UTC).isoformat()
|
||||
payload["last_at"] = payload["window_started_at"]
|
||||
marker.write_text(json.dumps(payload, sort_keys=True), encoding="utf8")
|
||||
except Exception:
|
||||
return
|
||||
|
||||
|
||||
def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed: bool) -> None:
|
||||
@@ -167,7 +236,7 @@ def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed
|
||||
):
|
||||
if (
|
||||
record in chunk
|
||||
and any(item.role == "user" and item.written_by is WrittenBy.PERSON for item in chunk)
|
||||
and any(item.role == "user" for item in chunk)
|
||||
and any(item.role == "assistant" for item in chunk)
|
||||
):
|
||||
intake.submit_records(chunk, priority="live")
|
||||
|
||||
@@ -160,7 +160,9 @@ class MatterDiscovery:
|
||||
task_by_id: dict[str, MemoryTask] = {}
|
||||
for chunk in chunks:
|
||||
for record in chunk:
|
||||
task = await asyncio.to_thread(processor.tasks.task_for_evidence, record.evidence_id)
|
||||
task = await asyncio.to_thread(
|
||||
processor.tasks.task_for_evidence, record.evidence_id
|
||||
)
|
||||
if task is not None and task.status is MemoryTaskStatus.COMPLETED:
|
||||
task_by_id[task.task_id] = task
|
||||
tasks = tuple(sorted(task_by_id.values(), key=lambda task: (task.created_at, task.task_id)))
|
||||
|
||||
@@ -270,7 +270,7 @@ async def reconcile_current(
|
||||
(
|
||||
r
|
||||
for r in records
|
||||
if r.written_by == "person"
|
||||
if r.role == "user"
|
||||
and r.created_at
|
||||
and (r.source_id, r.conversation_id) in related
|
||||
),
|
||||
|
||||
@@ -51,14 +51,19 @@ def validate_claim(claim: Claim, records: Mapping[str, EvidenceRecord | SampleRe
|
||||
text = record.text if isinstance(record, SampleRecord) else record.verbatim
|
||||
if not citation.quote.strip() or citation.quote not in text:
|
||||
raise ValueError("citation quote is not in the source")
|
||||
authors.add(str(record.written_by))
|
||||
author = {
|
||||
"user": "person",
|
||||
"assistant": "agent",
|
||||
"tool": "tool",
|
||||
}.get(record.role, "unknown")
|
||||
authors.add(author)
|
||||
created = record.created_at
|
||||
citations.append(
|
||||
citation.model_copy(
|
||||
update={
|
||||
"revision": record.revision,
|
||||
"source_id": record.source_id,
|
||||
"written_by": str(record.written_by),
|
||||
"written_by": author,
|
||||
"created_at": created.isoformat() if isinstance(created, datetime) else created,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -1,36 +1,25 @@
|
||||
"""Read-only, exact-revision support checks inside the caller's SQLite snapshot.
|
||||
"""Memory evidence freshness checks are retired.
|
||||
|
||||
No proof is legacy/unknown, not stale. One complete fresh proof suffices; all
|
||||
complete proofs stale means false. Malformed records raise. An exceeded budget
|
||||
means the memory is too large to verify, not that it is stale: it reads as
|
||||
current, the same as having no proof, and is reported once in a while as a
|
||||
content-free diagnostic. One such memory used to fail every COUNT over the
|
||||
library, so the homepage returned 503 for as long as it existed (D76).
|
||||
Set ONE_MEMORY_FRESHNESS_BUDGET_RAISES=1 to restore the old raising behavior.
|
||||
The caller owns BEGIN/deadline/progress-handler and error-to-unavailable mapping.
|
||||
Read paths used to re-validate ``memory_task_outputs.support_refs`` against
|
||||
live ``evidence`` rows via ``is_memory_evidence_current``. That design hid
|
||||
memories when evidence changed, but at high cost (library 503s, per-row SQL,
|
||||
segment-key breakage). Call sites still import this module; registration is
|
||||
now a no-op that always admits the row.
|
||||
|
||||
Proof rows may still be written by the ingest path; they are ignored on read.
|
||||
See ``新鲜度_todo.md`` for the remaining teardown steps.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Iterator, Sequence
|
||||
from contextlib import contextmanager
|
||||
|
||||
from one_memory.database_access import register_function
|
||||
from one_memory.diagnostic_log import diagnostic_event
|
||||
from one_memory.imports.models import EvidenceRecord
|
||||
from one_memory.memory_tasks import MemoryTaskOutput, matches_evidence_ref
|
||||
|
||||
_PROOFS = 64
|
||||
_REFERENCES = 128
|
||||
_BODY_BYTES = 65_536
|
||||
_EVIDENCE_BYTES = 8 * 1024 * 1024
|
||||
_SCOPE_STEPS = 64
|
||||
|
||||
#: Budget refusals: the check could not finish, the data is not malformed.
|
||||
#: Historical budget error codes. Retained so diagnostics/env docs do not break
|
||||
#: mid-teardown; nothing raises them on the read path anymore.
|
||||
BUDGET_ERRORS = frozenset(
|
||||
{
|
||||
"memory_freshness_proof_budget",
|
||||
@@ -39,80 +28,23 @@ BUDGET_ERRORS = frozenset(
|
||||
"memory_freshness_scope_budget",
|
||||
}
|
||||
)
|
||||
_REPORT_INTERVAL_SECONDS = 60.0
|
||||
_reported: dict[str, tuple[float, int]] = {}
|
||||
_reported_lock = threading.Lock()
|
||||
|
||||
|
||||
def budget_errors_raise() -> bool:
|
||||
"""Opt back into the pre-D76 behavior, where one large memory fails the query."""
|
||||
return os.environ.get("ONE_MEMORY_FRESHNESS_BUDGET_RAISES", "").strip() == "1"
|
||||
"""Former opt-in to raise on budget overrun. Always false after retirement."""
|
||||
return False
|
||||
|
||||
|
||||
def _report_budget(code: str) -> None:
|
||||
"""Count every budget skip; write at most one line per reason per interval."""
|
||||
now = time.monotonic()
|
||||
with _reported_lock:
|
||||
last, pending = _reported.get(code, (float("-inf"), 0))
|
||||
pending += 1
|
||||
if now - last < _REPORT_INTERVAL_SECONDS:
|
||||
_reported[code] = (last, pending)
|
||||
return
|
||||
_reported[code] = (now, 0)
|
||||
diagnostic_event(
|
||||
level="warn",
|
||||
component="memory.freshness",
|
||||
event="memory_freshness.budget_exceeded",
|
||||
outcome="skipped",
|
||||
details={"reason": code, "count": pending},
|
||||
def has_proof_table(db: sqlite3.Connection) -> bool:
|
||||
"""Whether ``memory_task_outputs`` exists. Unused by the retired check."""
|
||||
return (
|
||||
db.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_task_outputs'"
|
||||
).fetchone()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def _object(pairs: list[tuple[str, object]]) -> dict[str, object]:
|
||||
result: dict[str, object] = {}
|
||||
for key, value in pairs:
|
||||
if key in result:
|
||||
raise ValueError("memory_freshness_invalid_proof")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
class _Prefetched:
|
||||
"""Prefetched rows for the current batch, private to thread and connection.
|
||||
|
||||
Scoped, never global: a cache that outlived the transaction would answer
|
||||
from rows a later write had already replaced.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._local = threading.local()
|
||||
|
||||
def get(self) -> dict[str, list[sqlite3.Row]] | None:
|
||||
return getattr(self._local, "rows", None)
|
||||
|
||||
def evidence(self) -> dict[str, sqlite3.Row] | None:
|
||||
return getattr(self._local, "evidence", None)
|
||||
|
||||
@contextmanager
|
||||
def using(
|
||||
self,
|
||||
rows: dict[str, list[sqlite3.Row]],
|
||||
evidence: dict[str, sqlite3.Row] | None = None,
|
||||
) -> Iterator[None]:
|
||||
previous = getattr(self._local, "rows", None)
|
||||
previous_evidence = getattr(self._local, "evidence", None)
|
||||
self._local.rows = rows
|
||||
self._local.evidence = evidence
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
self._local.rows = previous
|
||||
self._local.evidence = previous_evidence
|
||||
|
||||
|
||||
_prefetched_proofs = _Prefetched()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def freshness_proofs_prefetched(
|
||||
db: sqlite3.Connection,
|
||||
@@ -120,328 +52,16 @@ def freshness_proofs_prefetched(
|
||||
*,
|
||||
evidence_ids: Sequence[str] = (),
|
||||
) -> Iterator[None]:
|
||||
"""Serve the freshness check's proof reads from one query.
|
||||
|
||||
The decisions are unchanged — the rows come from the same transaction —
|
||||
but the per-memory reads collapse into the one in `proof_rows_for`.
|
||||
"""
|
||||
with _prefetched_proofs.using(
|
||||
proof_rows_for(db, memory_ids),
|
||||
evidence_rows_for(db, evidence_ids) if evidence_ids else None,
|
||||
):
|
||||
yield
|
||||
|
||||
|
||||
def has_proof_table(db: sqlite3.Connection) -> bool:
|
||||
"""Whether this database keeps proofs at all.
|
||||
|
||||
Without the table the registered UDF answers 1 unconditionally, so there
|
||||
is nothing to prefetch and nothing to read.
|
||||
"""
|
||||
return (
|
||||
db.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_task_outputs'"
|
||||
).fetchone()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
_EVIDENCE_COLUMNS = (
|
||||
"evidence_id,user_id,source_id,workspace,project,conversation_id,"
|
||||
"message_id,correlation_id,source_ordinal,content_kind,role,written_by,"
|
||||
"created_at,verbatim,content_hash,revision,imported_at,updated_at"
|
||||
)
|
||||
|
||||
#: The budget measure the single read applies, kept as one string so the
|
||||
#: batched read cannot compute it differently.
|
||||
_EVIDENCE_SIZE_SQL = (
|
||||
"length(CAST(verbatim AS BLOB)) + "
|
||||
"length(CAST(json_array(evidence_id,user_id,source_id,workspace,project,"
|
||||
"conversation_id,message_id,correlation_id,source_ordinal,content_kind,"
|
||||
"role,written_by,created_at,content_hash,revision,imported_at,updated_at) "
|
||||
"AS BLOB))"
|
||||
)
|
||||
|
||||
|
||||
def evidence_rows_for(
|
||||
db: sqlite3.Connection,
|
||||
evidence_ids: Sequence[str],
|
||||
) -> dict[str, sqlite3.Row]:
|
||||
"""Read the cited evidence rows for a page in one query.
|
||||
|
||||
The check reads every citation twice — once for its size, against the
|
||||
evidence budget, and once for the row it validates against the proof's
|
||||
reference. Both are per citation, so both grow with the page.
|
||||
|
||||
The size comes from the same expression the single read uses, so the
|
||||
budget arithmetic cannot drift between the two paths.
|
||||
"""
|
||||
if not evidence_ids:
|
||||
return {}
|
||||
cursor = db.cursor()
|
||||
cursor.row_factory = sqlite3.Row
|
||||
by_id: dict[str, sqlite3.Row] = {}
|
||||
for start in range(0, len(evidence_ids), 900):
|
||||
batch = list(dict.fromkeys(evidence_ids[start : start + 900]))
|
||||
placeholders = ",".join("?" for _ in batch)
|
||||
for row in cursor.execute(
|
||||
f"SELECT {_EVIDENCE_COLUMNS},{_EVIDENCE_SIZE_SQL} AS measure "
|
||||
f"FROM evidence WHERE evidence_id IN ({placeholders})",
|
||||
batch,
|
||||
):
|
||||
by_id[str(row["evidence_id"])] = row
|
||||
return by_id
|
||||
|
||||
|
||||
def proof_rows_for(
|
||||
db: sqlite3.Connection,
|
||||
memory_ids: Sequence[str],
|
||||
) -> dict[str, list[sqlite3.Row]]:
|
||||
"""Read the proof rows for a whole page in one query.
|
||||
|
||||
The freshness check reads these per memory, and that is the read path's
|
||||
whole cost: measured on the old-user library, 160 memories cost 0.69 s in
|
||||
the UDF against 0.08 s for the citation subquery, and grew further while
|
||||
the library was also being written to — enough to cross the five second
|
||||
interrupt on `/api/v1/memories` and on the map.
|
||||
|
||||
Returns only ids that have rows; a caller falling back to a single read
|
||||
for anything absent keeps the same answer.
|
||||
"""
|
||||
if not memory_ids:
|
||||
return {}
|
||||
if not has_proof_table(db):
|
||||
return {}
|
||||
cursor = db.cursor()
|
||||
cursor.row_factory = sqlite3.Row
|
||||
by_id: dict[str, list[sqlite3.Row]] = {}
|
||||
for start in range(0, len(memory_ids), 400):
|
||||
batch = list(memory_ids[start : start + 400])
|
||||
placeholders = ",".join("?" for _ in batch)
|
||||
for row in cursor.execute(
|
||||
"SELECT task_id,memory_id,length(CAST(body AS BLOB)) AS size,"
|
||||
"substr(CAST(body AS BLOB),1,?) AS body FROM memory_task_outputs "
|
||||
f"WHERE memory_id IN ({placeholders})",
|
||||
(_BODY_BYTES + 1, *batch),
|
||||
):
|
||||
by_id.setdefault(str(row["memory_id"]), []).append(row)
|
||||
return by_id
|
||||
|
||||
|
||||
def _proof_revision(
|
||||
cursor: sqlite3.Cursor,
|
||||
memory_id: str,
|
||||
revision: int,
|
||||
user_id: str,
|
||||
available: set[int],
|
||||
) -> int:
|
||||
if revision in available or not available:
|
||||
return revision
|
||||
if (
|
||||
cursor.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='governed_memory_versions'"
|
||||
).fetchone()
|
||||
is None
|
||||
):
|
||||
return revision
|
||||
original = revision
|
||||
for step in range(_SCOPE_STEPS + 1):
|
||||
if revision in available:
|
||||
return revision
|
||||
row = cursor.execute(
|
||||
"SELECT v.action, typeof(v.content)='text' AND typeof(p.content)='text' "
|
||||
"AND CAST(v.content AS BLOB)=CAST(p.content AS BLOB) AS same_content, "
|
||||
"v.status=p.status AS same_status, "
|
||||
"CAST(v.content AS BLOB)=CAST(m.content AS BLOB) AS current_content "
|
||||
"FROM governed_memory_versions v "
|
||||
"JOIN governed_memory_versions p ON p.memory_id=v.memory_id "
|
||||
"AND p.revision=v.revision-1 "
|
||||
"JOIN governed_memories m ON m.memory_id=v.memory_id "
|
||||
"WHERE v.memory_id=? AND v.revision=? AND m.user_id=? AND m.revision=?",
|
||||
(memory_id, revision, user_id, original),
|
||||
).fetchone()
|
||||
if row is None or row[0] != "set_scopes" or not all(row[index] == 1 for index in (1, 2, 3)):
|
||||
return original
|
||||
if step == _SCOPE_STEPS:
|
||||
raise ValueError("memory_freshness_scope_budget")
|
||||
revision -= 1
|
||||
raise ValueError("memory_freshness_scope_budget")
|
||||
"""No-op. Kept so batched read callers need not change in this PR."""
|
||||
del db, memory_ids, evidence_ids
|
||||
yield
|
||||
|
||||
|
||||
def register_memory_freshness(db: sqlite3.Connection) -> str:
|
||||
"""Register UDF; return a predicate for alias m. Does not change row_factory.
|
||||
"""Register a always-true UDF; return SQL predicate ``1``.
|
||||
|
||||
Production initialization supplies memory_task_outputs(memory_id) index.
|
||||
Old no-index databases depend on the caller's SQL deadline. Per evaluation:
|
||||
65 rows / 64KiB each, 129 active IDs, 8MiB total Evidence material. No library
|
||||
bodies are loaded; Evidence objects live only for this evaluation. No cache
|
||||
persists across transactions (safe when a caller reuses its connection).
|
||||
Callers that embed the return value in ``WHERE`` keep working. Callers that
|
||||
invoke ``is_memory_evidence_current`` by name also get current.
|
||||
"""
|
||||
if not has_proof_table(db):
|
||||
register_function(db, "is_memory_evidence_current", 3, lambda *_: 1)
|
||||
return "1"
|
||||
|
||||
raising = budget_errors_raise()
|
||||
|
||||
def current(memory_id: str, revision: int, user_id: str) -> int:
|
||||
try:
|
||||
return checked(memory_id, revision, user_id)
|
||||
except ValueError as error:
|
||||
code = str(error)
|
||||
if raising or code not in BUDGET_ERRORS:
|
||||
raise
|
||||
_report_budget(code)
|
||||
return 1
|
||||
except sqlite3.Error as error:
|
||||
# A freshness proof is an enhancement, never the reason a person
|
||||
# cannot see their library. Under a concurrent history import the
|
||||
# 5s read deadline interrupts this UDF mid-statement; the old
|
||||
# behaviour let that escape, and the caller turned it into
|
||||
# `memory_library_unavailable`, so the memory list stayed empty for
|
||||
# as long as the import ran. Answer "current" instead and record
|
||||
# why, which is the same trade the D76 budget skip already makes.
|
||||
_report_budget("memory_freshness_sql_error")
|
||||
logging.getLogger(__name__).warning(
|
||||
"memory_freshness_udf_failed %s: %s", type(error).__name__, error
|
||||
)
|
||||
return 1
|
||||
|
||||
def checked(memory_id: str, revision: int, user_id: str) -> int:
|
||||
if not isinstance(memory_id, str) or type(revision) is not int or revision < 1:
|
||||
raise ValueError("memory_freshness_invalid_identity")
|
||||
if not isinstance(user_id, str):
|
||||
raise ValueError("memory_freshness_invalid_identity")
|
||||
cursor = db.cursor()
|
||||
cursor.row_factory = sqlite3.Row
|
||||
prefetched = _prefetched_proofs.get()
|
||||
if prefetched is not None:
|
||||
rows = prefetched.get(memory_id, ())
|
||||
else:
|
||||
rows = cursor.execute(
|
||||
"SELECT task_id,memory_id,length(CAST(body AS BLOB)) AS size,"
|
||||
"substr(CAST(body AS BLOB),1,?) AS body FROM memory_task_outputs "
|
||||
"WHERE memory_id=? LIMIT ?",
|
||||
(_BODY_BYTES + 1, memory_id, _PROOFS + 1),
|
||||
).fetchall()
|
||||
rows = list(rows)
|
||||
if len(rows) > _PROOFS:
|
||||
raise ValueError("memory_freshness_proof_budget")
|
||||
decoded = []
|
||||
for row in rows:
|
||||
if type(row["size"]) is not int or row["size"] > _BODY_BYTES:
|
||||
raise ValueError("memory_freshness_proof_budget")
|
||||
body = json.loads(row["body"], object_pairs_hook=_object)
|
||||
if not isinstance(body, dict) or type(body.get("memory_revision")) is not int:
|
||||
raise ValueError("memory_freshness_invalid_proof")
|
||||
decoded.append((row, body))
|
||||
revision = _proof_revision(
|
||||
cursor,
|
||||
memory_id,
|
||||
revision,
|
||||
user_id,
|
||||
{body["memory_revision"] for _, body in decoded},
|
||||
)
|
||||
applicable = []
|
||||
for row, body in decoded:
|
||||
if body["memory_revision"] != revision:
|
||||
continue
|
||||
raw_refs = body.get("support_refs", [])
|
||||
if not isinstance(raw_refs, list) or any(
|
||||
not isinstance(ref, dict)
|
||||
or set(ref) - {"evidence_id", "revision", "content_hash", "identity_hash"}
|
||||
for ref in raw_refs
|
||||
):
|
||||
raise ValueError("memory_freshness_invalid_reference")
|
||||
output = MemoryTaskOutput.model_validate_json(row["body"], strict=True)
|
||||
if output.memory_id != memory_id or output.task_id != row["task_id"]:
|
||||
raise ValueError("memory_freshness_proof_identity")
|
||||
if len(output.support_refs) > _REFERENCES:
|
||||
raise ValueError("memory_freshness_reference_budget")
|
||||
for ref in output.support_refs:
|
||||
if (
|
||||
set(ref) - {"evidence_id", "revision", "content_hash", "identity_hash"}
|
||||
or not ref["evidence_id"]
|
||||
or ref["revision"] < 1
|
||||
or any(
|
||||
len(ref[key]) != 64
|
||||
or any(char not in "0123456789abcdef" for char in ref[key])
|
||||
for key in ("content_hash", "identity_hash")
|
||||
if key in ref
|
||||
)
|
||||
):
|
||||
raise ValueError("memory_freshness_invalid_reference")
|
||||
proof_ids = [ref["evidence_id"] for ref in output.support_refs]
|
||||
if len(proof_ids) != len(set(proof_ids)):
|
||||
raise ValueError("memory_freshness_duplicate_reference")
|
||||
applicable.append(output)
|
||||
if not applicable:
|
||||
return 1
|
||||
active = cursor.execute(
|
||||
"SELECT evidence_id FROM governed_memory_evidence "
|
||||
"WHERE memory_id=? AND active=1 LIMIT ?",
|
||||
(memory_id, _REFERENCES + 1),
|
||||
).fetchall()
|
||||
if len(active) > _REFERENCES:
|
||||
raise ValueError("memory_freshness_reference_budget")
|
||||
ids = {row[0] for row in active}
|
||||
complete = [
|
||||
output
|
||||
for output in applicable
|
||||
if output.support_refs and {ref["evidence_id"] for ref in output.support_refs} == ids
|
||||
]
|
||||
if not complete:
|
||||
return 1
|
||||
records: dict[str, EvidenceRecord | None] = {}
|
||||
used = 0
|
||||
prefetched_evidence = _prefetched_proofs.evidence()
|
||||
for identifier in ids:
|
||||
evidence_row = (
|
||||
prefetched_evidence.get(identifier) if prefetched_evidence is not None else None
|
||||
)
|
||||
if evidence_row is not None:
|
||||
size = evidence_row["measure"]
|
||||
if type(size) is not int or size < 0:
|
||||
raise ValueError("memory_freshness_invalid_evidence")
|
||||
else:
|
||||
measured = cursor.execute(
|
||||
f"SELECT {_EVIDENCE_SIZE_SQL} FROM evidence WHERE evidence_id=?",
|
||||
(identifier,),
|
||||
).fetchone()
|
||||
if measured is None:
|
||||
records[identifier] = None
|
||||
continue
|
||||
if type(measured[0]) is not int or measured[0] < 0:
|
||||
raise ValueError("memory_freshness_invalid_evidence")
|
||||
size = measured[0]
|
||||
used += size
|
||||
if used > _EVIDENCE_BYTES:
|
||||
raise ValueError("memory_freshness_evidence_budget")
|
||||
if evidence_row is None:
|
||||
evidence_row = cursor.execute(
|
||||
f"SELECT {_EVIDENCE_COLUMNS} FROM evidence WHERE evidence_id=?",
|
||||
(identifier,),
|
||||
).fetchone()
|
||||
if evidence_row is None:
|
||||
records[identifier] = None
|
||||
continue
|
||||
# evidence_ref explicitly excludes origin_meta/imported_at/updated_at.
|
||||
# `sqlite3.Row` iterates values, not column names, so pair the
|
||||
# names with the values explicitly.
|
||||
fields = {
|
||||
name: value
|
||||
for name, value in zip(evidence_row.keys(), tuple(evidence_row), strict=False)
|
||||
if name != "measure"
|
||||
}
|
||||
records[identifier] = EvidenceRecord.model_validate({**fields, "origin_meta": {}})
|
||||
for output in complete:
|
||||
if all(
|
||||
(record := records[ref["evidence_id"]]) is not None
|
||||
and record.user_id == user_id
|
||||
and matches_evidence_ref(record, ref)
|
||||
for ref in output.support_refs
|
||||
):
|
||||
return 1
|
||||
return 0
|
||||
|
||||
register_function(db, "is_memory_evidence_current", 3, current)
|
||||
return "is_memory_evidence_current(m.memory_id,m.revision,m.user_id)"
|
||||
register_function(db, "is_memory_evidence_current", 3, lambda *_: 1)
|
||||
return "1"
|
||||
@@ -290,7 +290,7 @@ def read_memory_library(
|
||||
COALESCE(
|
||||
MIN(
|
||||
CASE WHEN e.role = 'user'
|
||||
AND e.written_by IN ('person', 'user')
|
||||
AND e.role = 'user'
|
||||
THEN e.created_at END
|
||||
),
|
||||
MIN(e.created_at)
|
||||
|
||||
@@ -10,7 +10,6 @@ from fastapi.params import Depends
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from one_memory.database_access import main_database
|
||||
from one_memory.hy_memory_bridge import HyMemoryBridgeError
|
||||
from one_memory.imports.models import (
|
||||
EvidenceInput,
|
||||
EvidenceOrigin,
|
||||
@@ -186,6 +185,10 @@ class MemoryNoteStore:
|
||||
]
|
||||
if task is None or task.status == MemoryTaskStatus.WAITING_FOR_MODEL:
|
||||
state = "waiting_for_model"
|
||||
elif task.status == MemoryTaskStatus.WAITING_FOR_PERMISSION:
|
||||
# Upgrade compatibility: old rows are claimable and will leave
|
||||
# this state as soon as the direct SDK runner claims them.
|
||||
state = "waiting_for_permission"
|
||||
elif task.status == MemoryTaskStatus.COMPLETED:
|
||||
state = "ready" if memory_ids else "no_memories"
|
||||
elif task.status == MemoryTaskStatus.RUNNING:
|
||||
@@ -195,22 +198,6 @@ class MemoryNoteStore:
|
||||
else:
|
||||
state = "failed"
|
||||
error = task.last_error_code if task else None
|
||||
if self.processor.model_content_config is not None and state in {
|
||||
"waiting_for_model",
|
||||
"failed",
|
||||
}:
|
||||
try:
|
||||
active = self.processor.profiles.active(self.processor.secrets)
|
||||
if active is not None:
|
||||
self.processor._guard_records((record,), active)
|
||||
except HyMemoryBridgeError as failure:
|
||||
if str(failure).startswith("model_content_"):
|
||||
state = "waiting_for_permission"
|
||||
error = str(failure)
|
||||
else:
|
||||
raise
|
||||
except SecretStoreUnavailableError:
|
||||
error = "secure_storage_unavailable"
|
||||
return MemoryNoteReceipt(
|
||||
evidence_id=record.evidence_id,
|
||||
text=sanitize_model_input(record.verbatim),
|
||||
@@ -274,11 +261,8 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
# unchanged; replaying the same request returns the original
|
||||
# `entry:<evidence_id>` job instead of a second HY call.
|
||||
if readmit:
|
||||
# `/{evidence_id}/process` is the explicit "the condition
|
||||
# that parked it may have changed" signal: the person just
|
||||
# granted consent or fixed the model. `submit_evidence`
|
||||
# alone would return an existing parked row unchanged, so
|
||||
# without this readmit the note stays parked forever.
|
||||
# `/{evidence_id}/process` is the explicit retry signal.
|
||||
# It also revives pre-upgrade permission waiters.
|
||||
await asyncio.to_thread(notes.intake.queue.readmit_parked)
|
||||
submitted = await asyncio.to_thread(
|
||||
notes.intake.submit_evidence,
|
||||
@@ -296,10 +280,6 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
|
||||
last_round=1,
|
||||
priority="live",
|
||||
)
|
||||
except HyMemoryBridgeError as error:
|
||||
if not str(error).startswith("model_content_"):
|
||||
raise
|
||||
return await asyncio.to_thread(notes.receipt, record.evidence_id)
|
||||
except SecretStoreUnavailableError:
|
||||
saved = await asyncio.to_thread(notes.receipt, record.evidence_id)
|
||||
return saved.model_copy(
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
Loaded 100 of 400 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user