Compare commits

..
Author SHA1 Message Date
alvinfeiandClaude Opus 5.5 754555e3c3 fix(memhub): restore and freeze the home when an upgrade migration fails
A failed SDK migration left a half-migrated home (schema migrations,
moved legacy work, renamed Chroma collections) that the next run
resumed on top of, the maintenance fence re-enabled the LaunchAgent so
launchd retried at once, a deferred service idled forever, and capture
hooks kept writing into the legacy database.

Restore: before the first mutation the run records a restore point
(the pre-cutover bundle, or a fresh bundle when resuming an old ledger
row). Any failure before SDK authority is committed restores the
management database (with WAL), config.json and hy-memory/governed-v2
through a verified staging copy and a journaled rename swap; the failed
state is kept aside in backups/sdk-migration-failed/. A failure after
the commit, or an unreadable authority, never restores. An interrupted
swap is finished on the next start; meanwhile status reports required
and Core refuses to serve. The migration's own code stays the result;
the restore outcome is logged and reported separately.

Freeze: a deferred migration turns the service enable marker off, makes
the runtime service exit instead of idling, and leaves a frozen capture
fence that hooks honour. Desktop launch, memhub install/upgrade and now
start/restart re-run the migration and unfreeze on success, restoring
the previous auto-start.

The failure dialog says when data was restored, and its GitHub issue now
prefills the body: from/to version, code, failing step, whether data was
restored, macOS version and the log path shown with ~. migrate.invoked
records the pre-upgrade version. sdk_data_plane_* codes now reach the
dialog too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 19:11:04 +08:00
feitianxiang b43c967baa Merge pull request #195 from Tencent-Hunyuan/feat/memory-panel-all-scenes
feat(memhub): memory panel lists and searches chat, working and coding
2026-10-27 18:04:32 +08:00
alvinfeiandClaude Opus 5.5 dac2c15b3f feat(memhub): memory panel lists and searches chat, working and coding
Pin hy-memory 2.0.6 and use its scene-aware APIs in the memory panel.

- Empty query: list_memories(scene="all") mixes the three scenes newest
  first, 20 per page, with per-scene totals and the source filter.
- Search: one SDK search(scene="all", rerank=False, limit=200) per
  query; Core keeps the keyword matches in a server-side result window
  (window_id, 300s idle TTL, 32 windows, ids only) and every page is cut
  from it, so pages never repeat or skip. An expired window re-searches
  and returns page 1 with window_reset. Pages are read fresh, so edits
  and deletes show.
- Items carry scene, kind, variant, task/solution and times; the panel
  shows 对话 / 工作 / 编码 badges. Detail, edit and delete pass
  kind/variant (experience edits the solution, optionally the task);
  an ambiguous id is 409, a task on a non-experience 422.
- memhub memories list/show print scene/kind/variant.
- SDK errors are re-exported through hy_sdk_runtime to keep the SDK
  import boundary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 16:54:08 +08:00
feitianxiang 0b775d7d61 Merge pull request #194 from Tencent-Hunyuan/release/hy-memory-2.0.6
release: hy-memory 2.0.6
2026-10-27 15:59:35 +08:00
alvinfeiandClaude Opus 5.5 4e217f845e release: hy-memory 2.0.6
Published hy-memory==2.0.6 to PyPI: search rerank switch, size-gated
MMR and near-duplicate demotion, scene="all" search and list, per-scene
list/memory_stats, productivity update/delete, reconcile time order,
HTTP 409/400 for ambiguous or bad namespaces, and the stats UTC fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 15:55:00 +08:00
feitianxiang 2102a298d4 Merge pull request #193 from Tencent-Hunyuan/fix/sdk-near-dup-gate
fix(sdk): gate near-duplicate demotion with the MMR top-k threshold
2026-10-27 15:42:38 +08:00
alvinfeiandClaude Opus 5.5 e540386db1 fix(sdk): gate near-duplicate demotion with the MMR top-k threshold
near_dup_demote compares each candidate with every earlier
non-duplicate by Jaccard, quadratic when candidates differ. Its pool
grows with limit (about max(3*limit, 30) plus keyword hits), so a
limit-200 page could cost ~180k comparisons. It now runs only when the
caller's limit is below MMR_MAX_TOP_K, like MMR; large pages want every
relevant item, similar ones included. The read reports
near_dup_applied next to mmr_applied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 15:34:41 +08:00
feitianxiang 1374ac26b7 Merge pull request #192 from Tencent-Hunyuan/feat/sdk-search-scenes
feat(sdk): rerank switch, size-gated MMR, scene="all" search and list, productivity edit/delete
2026-10-27 15:22:41 +08:00
alvinfeiandClaude Opus 5.5 47dafbb102 feat(sdk): rerank switch, size-gated MMR, scene="all" search and list, productivity edit/delete
search:
- rerank=True (default) keeps today's behavior; rerank=False skips the
  local cross-encoder, graph rerank and the coding facts' BM25 rerank for
  that call (carried on ReadRequest, so concurrent calls are independent).
- MMR now runs only when the caller's limit is below MMR_MAX_TOP_K (20).
  It is O(k^2 * pool) Jaccard and made large result pages slow.
- scene="all" searches chat, working and coding concurrently with one
  shared embedding and fuses them by Reciprocal Rank Fusion (k=60) over
  each lane's own rank, so no lane is buried by its score scale. Items
  carry scene, kind, memory_id, content, fused score, lane score/rank,
  source and times. A failed or disabled lane is reported in "errors".

list_memories:
- scene="coding"|"working" honour include_sources/exclude_sources with
  the filtered total.
- scene="all" lists the three scenes newest first with limit/offset over
  the merged order, per-scene totals and the same unified item fields.

update/delete:
- kind/variant disambiguate like get(); an experience update replaces
  its solution (task= also replaces the task and its search vectors), a
  fact update replaces the statement; delete removes experiences and
  facts with their vectors. Chat calls are unchanged.
- The HTTP server maps an ambiguous id to 409 and a bad namespace to 400
  instead of 500, passes kind/variant through, and strips the query
  string from PUT/DELETE paths, which leaked it into the memory id.

memory_stats now reads productivity times (written as naive UTC) as UTC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 15:13:12 +08:00
feitianxiang 49ad1b3086 Merge pull request #185 from Tencent-Hunyuan/fix/migration-chroma-settings
fix: list and count every memory scene, and migrate without a Chroma clash
2026-10-27 14:35:45 +08:00
alvinfeiandClaude Opus 5.5 3064515b43 Merge main into fix/migration-chroma-settings
Keep both CHANGELOG entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 14:30:30 +08:00
feitianxiang e5c8cc3819 Merge pull request #190 from Tencent-Hunyuan/feat/mcp-three-tools
feat(memhub): cut the MCP server to recall, write and diagnostic trace
2026-10-27 14:13:53 +08:00
alvinfeiandClaude Opus 5.5 27f64f1b7a chore(memhub): bump build to 31 and teach new-harness skills the three tools
The custom-Harness connection Skill covered pairing and the MCP config
but never told the model what the tools are, and not every Harness
reads the MCP initialize instructions. It now has a "Using MemHub after
connecting" section for MemHub_recall (including raw-mode results),
MemHub_write and MemHub_diagnostic_trace. The command-only Skill (Pi)
says recall can return original-conversation excerpts and that the
host has no write path.

Build number 30 -> 31.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 14:07:09 +08:00
kristinhe1111 3c5451db24 Merge pull request #183 from Tencent-Hunyuan/fix/home-recent-update-metadata
fix(memhub): stabilize recent updates and memory details
2026-10-27 13:22:45 +08:00
alvinfeiandClaude Opus 5.5 a08f69a747 feat(memhub): cut the MCP server to recall, write and diagnostic trace
MCP hosts now see three tools: MemHub_recall, MemHub_write and
MemHub_diagnostic_trace. MemHub_ask, MemHub_conversations,
MemHub_context, MemHub_memory_write and MemHub_memory_write_status are
removed, with their broker routes (/ask, /conversations, /memory-writes)
and client functions. Broker /context stays: the WorkBuddy injection
hook, recall-cli --context and the Hermes provider still use it.

MemHub_write saves one hand-added memory exactly like the app's
"添加记忆": broker POST /memory-notes authenticates the installation and
calls Core /api/v1/memory-notes with trusted headers, so the note takes
the direct-write lane ahead of history and is reconciled through
add_extracted. It takes {text, request_id?} and returns the note receipt
with the request_id to retry with.

MemHub_recall falls back to the SDK raw layer when no generation model
is configured, or when the memory search is empty and the library holds
no memory yet. The receipt says so with mode "raw" and fallback_reason;
raw hits are conversation excerpts (content_format "raw", source,
session_id). Raw reads keep the /context raw-lane authority: a host
needs om_conversations and its granted, unrevoked sources, so recall's
own grant never widens to original conversations.

Stored grants holding om_ask or om_conversations still load; new
pairings get om_recall, om_conversations and om_diagnostic_trace.
Skill text (SKILL_VERSION 19), connection prompt, docs and tests follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 13:16:55 +08:00
kristinheandCursor 099f5ae7c0 test(memhub): update memory search affordance
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 13:13:55 +08:00
kristinhe1111 e7625032d8 Merge pull request #187 from Tencent-Hunyuan/fix/memory-filter-clear-query
fix(memhub): 清空搜索框时移除已提交的记忆关键词
2026-10-27 13:11:07 +08:00
kristinhe1111 3aa3b5957d Merge pull request #186 from Tencent-Hunyuan/fix/hy-debug-pipeline-logs
fix(memhub): surface HY pipeline logs in Debug
2026-10-27 12:24:15 +08:00
kristinheandCursor 409ab9a9b4 fix(memhub): clear submitted memory query with input
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 12:22:55 +08:00
alvinfeiandClaude Opus 5.5 c2f50dce06 fix: claim hand-added memories first, and never let older memories win
MemHub: "添加记忆" (add a memory) went into the same durable queue as
history, which is claimed through a ten-minute event-time window, oldest
first. A note is stamped "now", so with a backlog it ran only after the
whole history. Jobs from one-memory-manual are now claimed ahead of the
window, still one at a time per source and in session order; history
keeps its strict timeline.

SDK reconcile: memories are not written in the order they happened, so a
new memory can be older than an existing one. Both reconcile prompts now
say only memory_at decides which is newer, that an older memory never
supersedes or overwrites a newer one (it is added as a past state), and
that an unknown time is not newer. The Chinese prompt now lists memory_at
among the new-memory fields, which it had left out. enforce_time_order
makes this hold in code: a SUPERSEDE or UPDATE aimed at a memory with a
later memory_at becomes an ADD.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 12:21:01 +08:00
kristinheandCursor 2735d8693a fix(memhub): prioritize memory list space
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 12:19:49 +08:00
kristinheandCursor 47bbfb4b27 fix(memhub): simplify memory note copy
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 12:11:54 +08:00
kristinheandCursor 3f169c9951 style(memhub): format language selector
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 12:01:58 +08:00
kristinheandCursor 24f699a726 fix(memhub): surface HY pipeline logs in Debug
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:43:35 +08:00
kristinheandCursor cb97aa5822 fix(memhub): show attributed app calls on home
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:43:25 +08:00
alvinfeiandClaude Opus 5.5 38a4da6849 fix: list and count every memory scene, and migrate without a Chroma clash
SDK: list_memories(scene="chat"|"coding"|"working") lists one kind of
memory. coding and working list the scene's whole store, newest update
first, with workspace_id still narrowing it, and total is the real count
instead of the page size. memory_stats() returns each scene's count and
latest created/updated time plus their total, reading no memory text.
Without a workspace, coding and working memories could not be listed or
counted, so a 45-memory library listed 22.

MemHub migration: reattaching legacy vectors opened Chroma with its own
settings and never closed it, so the readback that followed in the same
process could not open the store ("An instance of Chroma already exists
... with different settings"). Every 1.0.4 home with legacy vectors
failed its first upgrade launch. Offline code now opens Chroma as the SDK
does and closes it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 11:41:10 +08:00
kristinheandCursor 48a5d70026 fix(memhub): stabilize language setting layout
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:20:53 +08:00
kristinheandCursor b21c929728 fix(memhub): stabilize memory detail layout
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:12:47 +08:00
kristinheandCursor b9b3c587ff fix(memhub): hide recent update internals
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:01:14 +08:00
feitianxiang 5a5650fc94 Merge pull request #182 from Tencent-Hunyuan/chore/bump-build-30
chore(release): bump MemHub build number to 30
2026-10-27 09:21:00 +08:00
alvinfeiandClaude Opus 5.5 0e149b6c9c chore(release): bump MemHub build number to 30
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 09:17:07 +08:00
feitianxiang 0dd2bcd6b1 Merge pull request #181 from Tencent-Hunyuan/fix/core-ready-before-recovery
fix(memhub): make Core ready before it resumes unfinished work
2026-10-27 09:13:25 +08:00
alvinfeiandClaude Opus 5.5 567a8b349d test(desktop): let the attach-race winner exist before the loop gives up
The test wrote the winning Core's receipt and token from inside the
spawn, without awaiting it, while the readiness loop ran with zero
delay. On a slow runner the loop finished first, the re-read found a
receipt without its token, and start() reported a false conflict. The
probe now waits for the winner's files, which is the order the test is
about.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 09:00:50 +08:00
alvinfeiandClaude Opus 5.5 d89afb3887 fix(memhub): make Core ready before it resumes unfinished work
A home with pending imports and write jobs kept /health closed while
the lifespan resumed them, so the Runtime hit its 60s readiness budget,
killed a healthy Core with core_start_timeout, and the next Core began
the same recovery again. The write drain also started before readiness
and held the single writer the rest of startup needed.

Readiness is now only "the stores are open and Core answers". After
core.ready a recovery task runs, step by step: the post-cutover
metadata repair, the durable write drain, raw retention (or the legacy
pending-dispatch replay), interrupted imports, live sync, source
reconcile and the legacy catch-up. A failed step is logged and named in
/health without stopping readiness or the steps after it. /health
reports recovery as pending, running, done or failed. Shutdown cancels
recovery first and still releases the content-backfill gate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 23:59:38 +08:00
feitianxiang 0abf87d4b6 Merge pull request #180 from Tencent-Hunyuan/fix/memhub-sdk-only-migration-guard
fix(memhub): serve SDK only and stop on an unfinished upgrade migration
2026-10-26 22:34:53 +08:00
alvinfeiandClaude Opus 5.5 4e5a44b8dc fix(memhub): pass the lint gate
Format cli.test.ts, and mark the map's search form role as intentional:
biome's useSemanticElements also fails main (#179 merge), and a <search>
wrapper would add a box around the flex row .map-search styles.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 22:33:12 +08:00
alvinfeiandClaude Opus 5.5 bf36ba765b fix(memhub): serve SDK only and stop on an unfinished upgrade migration
Every 1.0.4 home must pass the offline SDK cutover, but the runtime
service could start Core on a home whose migration failed, so it served
legacy. Callers also asked Core for the memory authority before many
calls and kept v1 data-plane branches that only ever 409 on SDK.

Migration and startup:
- Write every migration step to logs/sdk-migration.log (content-free
  JSON lines) from Desktop, CLI, the runtime service and Core.
- Migrate before the runtime service spawns Core; Core refuses a legacy
  home with sdk_migration_required.
- Desktop shows a dialog that reveals the log and opens a GitHub issue,
  then quits. CLI install/upgrade no longer roll back to a version
  without the gate; start/restart/status/doctor name the log.
- Cycle the service when the migration status is unreadable so no
  pre-upgrade Runtime keeps serving.

SDK only:
- Web, host-bridge and CLI call v2 only; drop the authority query.
- Delete Core's v1 data plane, the Reader and /api/v2/memory-authority.
- Fix MCP memory write on v2, MemHub_conversations, memhub recall and
  status, v2 call events and WorkBuddy delivery reports.
- Remove dead features: matters, boundary peek/ack, memory plan/apply,
  the legacy library and its v1 operations.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 22:14:46 +08:00
feitianxiang e9cce69650 Merge pull request #179 from Tencent-Hunyuan/fix/memory-search-button
fix(memhub): search on submit and set SDK recall limits apart
2026-10-26 16:08:51 +08:00
alvinfeiandClaude Opus 5.5 2fed7adf6b fix(memhub): set the SDK context recall lanes apart
POST /api/v2/context/search passed the caller's `limit` as the normal,
schema and proactive intention top-k alike, so limit=5 ran three full
recall lanes including the intention lane the SDK keeps off by default.
The v2 request now takes `limit` (default 10), `schema_limit` (default
3) and `intention_limit` (default 1) and passes each through. The host
broker's default item cap follows v2's default of 10.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 15:48:48 +08:00
alvinfeiandClaude Opus 5.5 7308ed9df2 fix(memhub): search the memory list on submit, over 20 results
- The map's memory search no longer queries on every keystroke: typing
  edits the words and the search runs on the 搜索 button or Enter.
  Submitting the words already applied re-reads the page.
- Core asks SDK search for a 20-memory relevance window instead of 500;
  the larger window made a single search slow enough to freeze the page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 15:27:20 +08:00
feitianxiang 1c0cbf6d05 Merge pull request #176 from Tencent-Hunyuan/fix/memory-page-sdk-consistency
fix(memhub): make the SDK memory page one consistent page
2026-10-26 09:10:22 +08:00
alvinfeiandClaude Opus 5.5 71854a3510 fix(memhub): make the SDK memory page one consistent page
Root cause: the page read its rows from an unfiltered limit=300 window
while its count, source labels and filter came from the paged library,
and the cutover backfill (import_exact) had written every legacy memory
with source "default" and session "default_session".

- Web: on SDK authority the list, count, source/time labels and graph
  all come from one library page and the graph for the same source and
  window, read together and polled together. "default" is a named,
  filterable 未识别来源 bucket; impossible conversation times are not
  shown as when a conversation happened; detail timestamps carry their
  year; empty states tell "no match" from "empty source"; a finished
  memory note re-reads the page at once.
- Core: library search keeps the window's rows that contain the query
  words (the Chroma store has no keyword channel), reports when the
  window was full, and the graph endpoint takes the list's source and
  offset and no longer chains "default_session" memories together.
- Repair v2 links SDK memories to legacy evidence through
  governed_memory_evidence and write-job ids across every pre-cutover
  bundle, and settles remaining future memory_at values to gmt_created.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-25 23:23:33 +08:00
feitianxiang fcb1bcd866 Merge pull request #174 from Tencent-Hunyuan/chore/bump-build-22
chore(release): bump MemHub build number to 22
2026-10-25 15:32:06 +08:00
alvinfeiandClaude Opus 5.5 6305e4cea8 chore(release): bump MemHub build number to 22
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-25 15:31:24 +08:00
feitianxiang a4b0969248 Merge pull request #173 from Tencent-Hunyuan/fix/memory-library-build23-acceptance
fix(memhub): repair Build 23 memory library issues
2026-10-25 15:28:12 +08:00
alvinfeiandClaude Opus 5.5 e0920311a1 fix(memhub): close gaps in the Build 23 memory library repair
- Cursor: resume WAL-backed stores from their rowid. The cursor identity
  now carries the WAL change token, but resume_checkpoint/touched_cursor
  still compared the bare dev:ino, so every change re-read the store from
  rowid 0 and reported a rewind.
- Library search: leave the proactive intention lane off, list only
  active hits, bound the per-hit client.get fan-out, and count skipped
  rows from the single page parse instead of re-validating the page.
- Metadata repair: pool evidence per SDK memory so disagreeing sources
  stay unknown instead of last-output-wins; close the legacy, coding and
  Chroma handles; register both openings in the main-database gate,
  which the previous commit left failing.
- Runtime: the same-build opt-in fails closed when the selected tree's
  manifest is unreadable, so it cannot bypass downgrade refusal.
- Detail: keep the heading within 72 characters and break English text
  at a word boundary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-25 15:21:31 +08:00
alvinfeiandCursor e6426f6fd0 fix(memhub): repair SDK memory library acceptance issues
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-25 14:53:38 +08:00
feitianxiang 99e3c8d9d7 Merge pull request #172 from Tencent-Hunyuan/fix/sdk-memory-native-detail
fix(memhub): reuse the memory map for SDK-native details
2026-10-25 09:54:03 +08:00
alvinfeiandCursor 896a9c6c83 fix(memhub): render SDK memory details natively
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-25 09:46:34 +08:00
kristinhe 193fabeeb4 fix(memhub): open an SDK memory where the map put it
On SDK authority, clicking a memory switched the whole page to the SDK
library — a second screen made of status tabs and a ledger, where the memory
page's design has one list, one graph and one inspector. Those three already
exist: they are the components the mock-data screen was built from. This wires
the real SDK reads into them instead of standing a second screen beside them.

- opening a memory reads it through v2 and draws it in the map's inspector;
  edits and deletes go to updateSdkMemory / deleteSdkMemory, and a write
  re-reads the page instead of waiting for the poll
- the left list gets a real library page (50 a page, with the total and each
  memory's source and conversation time), so the count, the paging footer and
  the provenance line read the way the design reads
- a memory with no cited excerpt no longer renders an evidence block and a
  source line that can only say "not available"

The SDK library panel is left as it was; it is no longer on this path.
2026-10-25 09:36:02 +08:00
feitianxiang 0aeb3e8760 Merge pull request #167 from Tencent-Hunyuan/fix/simplify-sdk-migration
fix(memhub): migrate to SDK authority without copying memory
2026-10-24 23:18:32 +08:00
feitianxiang 70aaa7be85 Merge pull request #170 from Tencent-Hunyuan/fix/upgrade-runtime-and-sdk-memory-map
fix(memhub): install the bundled Runtime on upgrade and show SDK memories
2026-10-24 23:12:03 +08:00
alvinfeiandClaude Opus 5.5 18046152ab feat(memhub): draw the memory map from SDK memories on SDK authority
The default memory view is the map, and on SDK authority it had nothing
to draw: its list and graph read only legacy authority. It now reads SDK
memories (`/api/v2/memory-library`) and `/api/v2/memories/graph` and
draws them with the same map model, so the map stays the default view.
Opening a memory from the map goes to the SDK library, which reads,
edits and deletes it through v2; governed detail is refused there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 22:22:55 +08:00
alvinfeiandClaude Opus 5.5 7b946a28ef fix(memhub): install the bundled Runtime on upgrade and show SDK memories
Build 21 acceptance found an upgraded App on the previous Runtime and a
memory page that never left "reading":

- Desktop skipped `installPackagedRuntime` whenever the service owner was
  "cli", which it also assumes itself when a plist exists without an owner
  receipt. The App now always installs its bundled Runtime (a newer build
  from `memhub upgrade` is kept) and refuses to start on a `current` older
  than its own build (`packaged_runtime_not_current`). The startup failure
  event now records the cause, with the home path shortened.
- On SDK authority the default memory map loaded nothing: its list and
  graph read only legacy authority. The memory page now opens the v2 SDK
  library there, and `/api/v2/memories/graph` serves the map from SDK
  memories plus the Kuzu overlay. A failed authority probe is retried
  instead of leaving the page unknown forever.
- Every completed SDK write logged RetiredAuthorityAccess: the retired
  Evidence facade raised from `getattr(..., None)`. It now has a no-op
  `release_sdk_staging`.
- A repair of an already migrated home is reported as `complete`, not a
  second `cutover`, and the CLI treats it as non-rollbackable too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 22:08:14 +08:00
alvinfeiandClaude Opus 5.5 e5ce0cd964 refactor(memhub): remove the private retrieval vector index
All memory and raw reads and writes go through the Hy Memory SDK. The
one_memory.retrieval index kept a second, MemHub-owned vector copy of
conversation text in a private Chroma store under <data_dir>/retrieval-vectors/,
which that rule does not allow. It was experimental and off by default.

Remove the vector store, runtime, provisioning, supervisor, worker, index
store, chunking and mapped content, the offset tokenizer only chunking used,
the semantic raw lanes that read the index, the retrieval_index_enabled and
retrieval_minimum_free_bytes settings, app.state.retrieval_runtime and their
tests. The context route now calls the lexical raw reader directly, which is
exactly what it did with the index disabled; the v2 coordinator keeps its
lexical lane with the same statuses it produced when no index was configured.

Historical migrations and the legacy retrieval tables in the drop list are
unchanged. An existing retrieval-vectors directory is left on disk.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 18:13:47 +08:00
alvinfeiandClaude Opus 5.5 d8837b4fb2 fix(memhub): reattach stranded vectors and read memories back before cutover
Build 20 cut over showing 37 of 3317 memories, with "memory unavailable"
on every page:

- `/api/v2/memory-authority` read the cutover receipt, which cleanup drops
  and a fresh home never has, so the UI chose the legacy views the API
  refuses. It now reads the authority singleton the gates use.
- #145 renamed the SDK vector collections (`one_memory_*` to `memhub_*`)
  and dropped the step that carried old ones over, so every later Runtime
  read an empty collection. `sdk-migrate` now renames them through Chroma's
  public client while the Runtime is stopped, merging any newer writes,
  and also repairs homes that already cut over.
- The cutover only checked the backup. It now opens the public SDK client
  as the Runtime will and reads the legacy memory ids back; every chat
  memory must read back or the run stops before anything irreversible.
  A migrated home is checked against its pre-cutover bundle.

Cleanup also VACUUMs the main database to return the dropped tables' space.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 18:02:53 +08:00
feitianxiang 0c36a6c960 Merge pull request #168 from Tencent-Hunyuan/feat/single-sdk-migrate-command
feat(memhub): migrate every install with one sdk-migrate command
2026-10-24 11:47:25 +08:00
alvinfeiandClaude Opus 5.5 9bdce08847 feat(memhub): migrate every install with one sdk-migrate command
Only Desktop ran the SDK-authority migration, so homes upgraded with
`memhub upgrade` (Linux, and macOS without the App) stayed on legacy
authority for good. The migration was also four Core commands that
Desktop orchestrated, which a second caller would have had to repeat.

Core now has one resumable `sdk-migrate --runtime-build <build>`: it
takes the host capture fence itself, backs up (reusing an existing
bundle), prepares the queue, commits authority and drops the legacy
tables, printing one progress line per step. `migration-bundle`,
`sdk-cutover` and `sdk-cleanup` are folded into it.

`runSdkMigration` in runtime-control keeps the Runtime stopped around
that command and is shared by Desktop startup and CLI install/upgrade.
The CLI runs it with the new Runtime's Core before starting the
service; a failure rolls back, and after cutover a slow start keeps the
new version because an older Runtime refuses a migrated home.

A rehearsal on a real home also showed freezing queue input failed on
completed jobs whose payload the queue had already released; those are
now skipped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 11:46:01 +08:00
alvinfeiandClaude Opus 5.5 7cbc2fd632 fix(memhub): migrate to SDK authority without copying memory
Every released build wrote memory and raw turns through the SDK first;
the Governance/Evidence tables only mirrored them. The old migration
still compared the mirror with the SDK item by item, and its layer
labels did not say which Coding store held a row. Build 19 therefore
quarantined experiences that were on disk, quarantine had no exit, and
startup stayed blocked on every retry.

The cutover now copies and compares no memory. It verifies the bundle,
makes the write queue independent of the legacy tables, discards the
staging copy earlier builds left, commits authority and drops the
tables, including the evidence_search index that was left behind.
Invoked legacy tasks and pending deletions are reported, not blocking.
A home blocked by build 19 resumes its run and reuses its bundle.

Also fixes legacy-authority regressions from #145 found by the full
core suite: startup backfill, visibility recompute and deletion retry
were no longer started, deletion rows stayed pending, and correcting a
memory without a model returned 500.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 11:01:33 +08:00
kristinhe1111 6b0bd86679 Merge pull request #156 from Tencent-Hunyuan/fix/build-17-test-index
docs(memhub): regenerate the core test index for build 17
2026-10-23 20:22:05 +08:00
kristinhe1111 91250d7c87 fix(desktop): keep an unknown startup failure code in the dialog
Build 17 routed every startup error through the friendly-text map, so three
desktop contracts that assert the dialog body (page_load_failed,
runtime_broker_unavailable, host_permission_store_unavailable) started failing.

Only codes we wrote copy for become sentences now; anything else keeps the code
it was thrown with, which is what support reads.
2026-10-23 20:02:05 +08:00
kristinhe1111 3c89a535fb docs(memhub): regenerate the core test index for build 17
The build-17 test file was added without regenerating TEST_INDEX.md, so the
public structure gate failed on main. Generated artifact only.
2026-10-23 19:49:44 +08:00
kristinhe1111 1151878265 Merge pull request #155 from Tencent-Hunyuan/release/memhub-build-17
fix(memhub): build 17 — migration fixes carry through to first launch
2026-10-23 19:34:32 +08:00
kristinhe e82c4c939e fix(memhub): name a blocked cutover without its counts
The cutover gate reports sdk_data_plane_cutover_work_pending plus a JSON body of
table counts. That string went straight into the document the startup dialog
reads, so the person saw braces and table names.

Collapse it to sdk_migration_work_pending and let the dialog say the data is
already upgraded but earlier memory work is still finishing. Any other code that
is not a plain identifier becomes sdk_migration_cutover_failed, so no free text
reaches the dialog.
2026-10-23 19:33:31 +08:00
kristinhe ca4a66349b fix(memhub): explain the wait and the failure on first launch
Three things a person meets before the app can open:

- The waiting window had no drag region element, so a frameless window could not
  be moved at all; the CSS alone never applied to anything.
- The page said only that the service was starting. It now says a first launch
  or a memory upgrade can take a few minutes and the app opens by itself, in all
  three interface languages.
- A migration failure surfaced as a raw code in one path and a generic sentence
  in the other. Every migration code now maps to a sentence that says what
  happened, what happens to their memories, and what to do next; the dispatch
  path reuses the same mapping instead of printing error.message.

Build number 17 carries the diagnostics fix and the merged SDK migration fix.
2026-10-23 19:19:06 +08:00
kristinhe 82044d6172 fix(memhub): name the migration failure instead of one opaque code
A full disk, an unconfigured model profile and a Coding classification gap all
reached the person as sdk_migration_backfill_failed, so the startup dialog could
only ever say the upgrade failed.

Classify the exception chain before the code is written: ENOSPC/EDQUOT becomes
sdk_migration_disk_space_insufficient, SDK productivity errors become
sdk_migration_coding_inventory_failed, the legacy working_fact import failure
keeps its own name, and existing sdk_data_plane_/sdk_migration_ codes pass
through. Anything unrecognized keeps its message out of the document.
2026-10-23 19:18:56 +08:00
kristinhe 7f23b74c36 fix(memhub): separate local diagnostics from safe exports 2026-10-23 19:03:50 +08:00
kristinhe 617e2d0ba6 fix(memhub): keep diagnostics from blocking startup 2026-10-23 19:03:50 +08:00
feitianxiang 083b837f32 Merge pull request #154 from Tencent-Hunyuan/fix/sdk-migration-turns-and-coding
fix(memhub): harden SDK migration inventory and fencing
2026-10-23 17:37:15 +08:00
alvinfeiandCursor 1dc7e957c0 fix(memhub): harden SDK migration inventory
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-23 17:34:59 +08:00
feitianxiang 3b44d8381c Merge pull request #153 from Tencent-Hunyuan/fix/sdk-migration-turns-and-coding
fix(memhub): repair legacy SDK authority migration
2026-10-23 17:27:39 +08:00
alvinfeiandCursor 790d6d3d5e fix(memhub): repair SDK authority migration
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-23 17:26:40 +08:00
feitianxiang 4c311b2e93 Merge pull request #149 from Tencent-Hunyuan/fix/memhub-1.0.5-release-doc-test
test(memhub): update 1.0.5 release doc expectations
2026-10-22 21:31:04 +08:00
alvinfeiandCursor 801a2af36b test(memhub): update release documentation expectations
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 21:30:31 +08:00
feitianxiang 0a8e43e570 Merge pull request #148 from Tencent-Hunyuan/release/memhub-1.0.5
release(memhub): bump to 1.0.5 and pin hy-memory 2.0.5
2026-10-22 21:22:40 +08:00
alvinfeiandCursor f5a38c38b3 release(memhub): pin hy-memory 2.0.5
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 21:20:40 +08:00
feitianxiang d08e070816 Merge pull request #147 from Tencent-Hunyuan/release/hy-memory-2.0.5
release: hy-memory 2.0.5
2026-10-22 20:58:49 +08:00
alvinfeiandCursor 29bd9146a4 release: hy-memory 2.0.5
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 20:52:21 +08:00
26c33ee40f refactor(sdk): internalize legacy data migration (#146)
* refactor(sdk): keep legacy migration out of client APIs

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(migration): stage SDK data before cutover

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 20:49:23 +08:00
d926844173 refactor(memhub): remove private Hy Memory bridge (#145)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: kristinhe <kristinhe@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 20:02:27 +08:00
de4292ed4e feat(hermes): add managed native prefetch provider (#143)
* feat: add native Hermes memory provider

* feat(hermes): support scoped private CA trust

* feat(hermes): bound write timeout separately from reads

* fix(hermes): support truststore scoped CA snapshots

* feat(hermes): add managed prefetch provider

Co-authored-by: kindong <727777477@qq.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: kindong <727777477@qq.com>
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 16:51:15 +08:00
456 changed files with 26581 additions and 69590 deletions

No files matched your search

+167 -53
View File
@@ -2,6 +2,156 @@
Hy Memory and MemHub are versioned and released independently.
## Unreleased
- The SDK migration no longer fails its first run on a home with legacy
vectors: reattaching them left a Chroma instance open with its own settings,
and the readback that followed could not open the same store
("An instance of Chroma already exists ... with different settings").
- MemHub "添加记忆" (add a memory) no longer waits behind unprocessed history.
Notes are claimed ahead of the event-time window that keeps history strictly
old to new; they still run one at a time and in order within their session.
- The SDK memory page now reads its list, count, source and time labels and
graph from one page response for one source filter, so rows, totals and the
map no longer disagree. Memories with no recorded source are shown and
filterable as "unidentified source"; library search keeps only memories
that contain the words within the SDK's relevance window.
- Metadata repair v2 follows `governed_memory_evidence` across every
pre-cutover bundle, which restores the source of memories the cutover
backfilled without one, and moves impossible future conversation times back
to the write time.
- Reused the existing three-column memory map for SDK-authority list/detail
views and removed the retired "original evidence" surface. SDK-native status,
source, conversation time and creation time now render without fabricated
revisions; source and text filters are applied by the paged Core endpoint.
- SDK memory search now calls the SDK search contract instead of listing the
entire library in Core. Corrupt empty records are skipped and diagnosed
without mutating data or failing a page, and post-cutover metadata repair
restores source and original times from the verified migration bundle.
- Kept Cursor's WAL change token separate from the file mtime so a hash can no
longer appear as a conversation date centuries in the future.
- A trusted packaged Runtime now repairs a dirty same-build installation by
staging a clean replacement, and `runtime_service_not_ready` is rendered as
user-facing startup guidance instead of an internal code.
- Fixed legacy-to-SDK startup migration to adopt native Coding Experience and
Working Fact IDs instead of importing them as Chat layers. Raw migration now
rebuilds bounded user/assistant turns from the latest three days, excludes
tools, and enforces the same rolling retention after cutover.
- Paused host capture writes with a live Desktop migration fence; Cursor
History catch-up resumes from its existing checkpoint after cutover.
- Removed MemHub's private `hy_memory_bridge` implementation. MemHub now calls
public `HyMemoryClient` methods at runtime; prompt/reconcile/response
monkey-patches and private VDB/Coding runtime access were deleted. Kuzu Graph
projection/admin and stopped-writer App migration are the documented product
exceptions.
- Added a MemHub-managed native Hermes MemoryProvider with automatic prefetch,
persistent per-session memory-ID deduplication, authorized Broker reads, and
recall-context stripping to prevent memory recursion.
- Added an extensible MemHub interface-localization layer with Simplified
Chinese, English, and Spanish catalogs shared by Web and Electron-native
surfaces. Interface language remains independent from memory content and
model processing language.
- Fixed governed-memory revisions and repeated evidence links failing when the
visibility dirty-marker trigger encounters the same memory twice. Startup
now replaces the affected legacy trigger without changing schema versions
or governed and visibility data.
- Kept post-write authority verification failures available for read-only
reconciliation while preventing automatic or generic confirmed retries from
invoking the model again. Automatic reconciliation skips those sticky rows
before applying its batch limit, and linked durable jobs exclusively own
retry scheduling for their memory tasks.
- Removed MemHub's task-scoped HyPermission store wrappers and global
model-content consent switch. The trusted embedded Hy Memory SDK now uses
its native local stores directly.
- Existing installations ignore the retired config flag, readmit previously
parked pre-invocation work, and drop only the obsolete
`hy_node_permissions` proof table. Evidence, governed memories, model
profiles, SDK stores, and source/host grants are preserved.
## Hy Memory 2.0.6 — 2026-10-27
- Published
[`hy-memory==2.0.6`](https://pypi.org/project/hy-memory/2.0.6/).
- `search(..., rerank=False)` skips every reranking stage for that one
call (the local cross-encoder, the memory-graph rerank, and the BM25 rerank of
working/coding facts), so results come back in retrieval-score order. The
flag travels on the read request, so concurrent searches do not affect each
other. The default (`True`) keeps current behavior; `/api/v1/search` accepts
it too.
- MMR diversity selection now runs only when the requested `limit` is
below 20 (`MMR_MAX_TOP_K`) and is skipped for larger limits, where its
quadratic token-overlap pass is slow and gains little. Previously it ran for
every limit. `MEMORY_HYBRID_V2_MMR_LAMBDA` still sets the weight.
Near-duplicate demotion follows the same gate: it compares each candidate
with every earlier non-duplicate, which is quadratic in a pool that grows
with `limit`, and large result pages should show similar items anyway.
- `search(scene="all")` searches chat, working and coding at the same time
and merges them with Reciprocal Rank Fusion (k=60) over each scene's own
ranking, so no scene is crowded out by the other scenes' score scales. Each
result says which scene and kind it came from. A scene that fails or is turned
off is reported in `errors` and does not fail the search. Source filters apply
to all three.
- `list_memories(scene="all")` lists chat, working and coding memories
together, newest update first, with `limit`/`offset` paging over the merged
order and `total` summing all three. `scene="coding"` and `"working"` now
honor `include_sources` / `exclude_sources`, and `total` counts only the
filtered memories.
- `update()` and `delete()` now work on working/coding experiences and
facts, not just chat memories. They take the same optional `kind` / `variant`
as `get()`, and an ID found in more than one place raises
`AmbiguousMemoryIdError` instead of guessing. Updating an experience replaces
its solution and keeps its task, unless a new `task=` is also passed; updating
a fact replaces its statement. Deleting removes the stored row and its vectors.
- `list_memories(scene="chat" | "coding" | "working")` lists one kind of
memory. `coding` and `working` list that scene's whole store, newest update
first (`workspace_id` still narrows it), and `total` is now the real count
rather than the page size. The new `memory_stats()` returns the count and the
latest created/updated time for each scene and their total, without reading
any memory text. Previously coding and working memories could not be listed
or counted without a workspace, so a library of 45 memories listed 22.
- Reconcile keeps time order. Both reconcile prompts now state that only
`memory_at` decides which memory is newer, that an older memory must never
supersede or overwrite a newer one, and that an unknown time is not newer;
the Chinese prompt now lists `memory_at` among the new-memory fields. The
reconciler also enforces this in code: a SUPERSEDE or UPDATE aimed at a memory
with a later `memory_at` becomes an ADD, so the newer memory stays current.
- The HTTP server returns 409 `ambiguous_memory_id` and 400 for a bad
`kind`/`variant` instead of 500, passes `kind`/`variant` (and `task` on PUT)
to get/update/delete, and no longer reads a PUT/DELETE query string as part
of the memory id.
- `memory_stats()` reads working/coding times, which are stored as naive UTC,
as UTC rather than local time.
## Hy Memory 2.0.5 — 2026-10-22
- Published
[`hy-memory==2.0.5`](https://pypi.org/project/hy-memory/2.0.5/).
- Extended `get()` across Chat, Coding Experience, Working Fact, and raw IDs;
the returned `layer`, `kind`, and `variant` identify the actual record.
- Added explicit raw retrieval through the existing `search()` and
`list_memories()` contracts, including user, agent, session, and source
filters.
- Added public embedding-service injection and corrected extraction language
detection for structured OpenAI-style messages.
- Added lifecycle status filters, durable idempotent write outcomes, and
quiesced local data-plane snapshot/restore.
- Kept legacy migration mechanics out of the public Client API; separate
migration-only raw/import methods from development previews were removed.
- Physical delete now reports partial VDB/Graph cleanup as failure.
- Coding search no longer injects the version-evolution diff block by default;
opt in with `MEMORY_CODING_CHAIN_BLOCK_ENABLED=true`.
- Serialized SDK SQLite cache/history connection lifecycle and tightened
Coding reconciliation to its agent/session/workspace authority.
- Clarified persistent caller context through `meta`, while `metadata` remains
legacy processing configuration.
## Hy Memory 2.0.5.dev4 — 2026-10-21
- Development preview used for SDK-authority migration validation; not the
final `2.0.5` release.
- Published as
[`hy-memory==2.0.5.dev4`](https://pypi.org/project/hy-memory/2.0.5.dev4/).
## Hy Memory 2.0.5.dev3 — 2026-10-21
- Development preview; not the final `2.0.5` release.
@@ -28,6 +178,12 @@ Hy Memory and MemHub are versioned and released independently.
APIs, offline local data-plane snapshot/restore, and fail-closed physical
delete receipts.
## Hy Memory 2.0.4 — 2026-10-17
- Published
[`hy-memory==2.0.4`](https://pypi.org/project/hy-memory/2.0.4/) as a
maintenance release.
## Hy Memory 2.0.3 — 2026-10-17
- Added an optional runtime HTTP transport factory so embedded products can
@@ -38,58 +194,6 @@ Hy Memory and MemHub are versioned and released independently.
unchanged memory extraction, reconciliation, storage, and Coding/Working
behavior.
## Unreleased
- Added a MemHub-managed native Hermes MemoryProvider with automatic prefetch,
persistent per-session memory-ID deduplication, authorized Broker reads, and
recall-context stripping to prevent memory recursion.
- Added an extensible MemHub interface-localization layer with Simplified
Chinese, English, and Spanish catalogs shared by Web and Electron-native
surfaces. Interface language remains independent from memory content and
model processing language.
- `search(..., layers=["l1_raw"])` now explicitly searches SDK raw records in
pro/ultra mode and keeps existing user/agent/session/source filters; default
search continues to hide raw.
- Added `import_memory_exact` for idempotent migration of an existing semantic
memory with its canonical ID, lifecycle and supersession links, without
extraction or reconciliation.
- Added SDK-authoritative lifecycle listing with explicit status filters,
idempotent durable write outcomes, exact raw import/get/list APIs, and
quiesced local data-plane snapshot/restore contracts for the MemHub
authority migration.
- Physical delete now reports incomplete VDB/Graph cleanup as a partial failure
instead of claiming success after only one store was changed.
- Coding search no longer injects the version-evolution diff block
(`chain_block`) by default. Enable with
`MEMORY_CODING_CHAIN_BLOCK_ENABLED=true` or
`CodingConfig.chain_block_enabled=True` (write-side observe unchanged).
- Fixed governed-memory revisions and repeated evidence links failing when the
visibility dirty-marker trigger encounters the same memory twice. Startup
now replaces the affected legacy trigger without changing schema versions
or governed and visibility data.
- Serialized each SDK SQLite store's shared connection lifecycle so concurrent
history/cache operations cannot race with `close()` and crash the process.
- Scoped Coding reconciliation targets to the current HY authority,
conversation, and workspace, preventing near-duplicate sessions from
mutating an experience or working fact owned by another evidence context.
- Kept post-write authority verification failures available for read-only
reconciliation while preventing automatic or generic confirmed retries from
invoking the model again. Automatic reconciliation skips those sticky rows
before applying its batch limit, and linked durable jobs exclusively own
retry scheduling for their memory tasks.
- Clarified that `add(metadata=...)` carries legacy SDK processing options,
while persistent caller business context belongs in `meta`.
- Added an opt-in `get(..., include_raw=True)` path for reading `L1_RAW`
memories, including memories written in lite mode. The default remains
hidden for backward compatibility.
- Removed MemHub's task-scoped HyPermission store wrappers and global
model-content consent switch. The trusted embedded Hy Memory SDK now uses
its native local stores directly.
- Existing installations ignore the retired config flag, readmit previously
parked pre-invocation work, and drop only the obsolete
`hy_node_permissions` proof table. Evidence, governed memories, model
profiles, SDK stores, and source/host grants are preserved.
## Hy Memory 2.0.2 — 2026-10-17
- Published
@@ -104,7 +208,17 @@ Hy Memory and MemHub are versioned and released independently.
Because PyPI release metadata is immutable, the next SDK patch release must
carry the updated license metadata; `2.0.1` retains its published metadata.
## MemHub 1.0.3 — Unreleased
## MemHub 1.0.5 — Unreleased
- Updated the source version to `1.0.5` (build 14).
- Pinned the public SDK to the stable `hy-memory==2.0.5` release.
## MemHub 1.0.4 — 2026-10-18
- Updated the source version to `1.0.4` (build 13).
- Published the `v1.0.4` source and release metadata.
## MemHub 1.0.3 — 2026-10-17
- Updated the source version to `1.0.3` (build 7).
- Pinned the public SDK to `hy-memory==2.0.3`.
+7 -6
View File
@@ -273,8 +273,8 @@ catalog 也可以在没有基础 L0 时形成只读的合成画像。
```python
memory = client.get("memory_id")
# lite 模式写入的是 L1_RAW;按 ID 读取时必须显式选择返回原文。
raw_memory = client.get("memory_id", include_raw=True)
# raw ID 返回 L1_RAW,fact ID 返回对应事实层。
raw_memory = client.get("raw_id")
result = client.update("memory_id", "更新后的内容")
result = client.delete("memory_id")
result = client.delete_all(user_id="user_001")
@@ -284,7 +284,7 @@ page = client.list_memories(
limit=100,
offset=0,
order="desc",
layers=None,
layers=["l1_raw"], # 显式列出 raw;不传仍沿用默认语义层列表
with_evolution=False,
)
@@ -294,9 +294,10 @@ result = client.clone_user("user_001", "user_002", agent_id=None)
对应异步方法为 `async_get()`、`async_update()`、`async_delete()`、
`async_delete_all()`、`async_list_memories()`、`async_clone_user()`。
`get()` / `async_get()` 默认继续隐藏 `L1_RAW`,因此不会改变现有调用方的原文访问
边界。只有显式传入 `include_raw=True` 才返回该层;这也是 lite 模式按 ID 读取
写入内容及其 `meta` 的方式。MemHub 的 HTTP 读取接口不开放此参数。
`get()` / `async_get()` 是统一的 ID 精确读取:返回结果的 `layer`、`kind` 和
`variant` 表明实际类型。授权由调用 SDK 前的产品边界完成,不由 SDK 根据 layer
猜测。`list_memories()` 默认仍不包含 `L1_RAW`;显式传
`layers=["l1_raw"]` 时按同一列表合同返回 raw。
普通 `list_memories()` 返回 `vdb` 桶,并在 Graph 有结果时附带 `graph`;传入
`workspace_id` 时返回生产力记忆的 `scene="productivity"` 和 `experiences`。
+13
View File
@@ -65,6 +65,12 @@ from .snapshot import (
# ====== 配置 ======
from .config import MemoryConfig
from .embedding import (
EmbeddingDimensionMismatchError,
EmbeddingService,
EmbeddingServiceProtocol,
)
from .retrieval import AmbiguousMemoryIdError, MemoryKind, MemoryVariant
# ====== Embedding 端点覆盖(per-call,不改 model/dims,只换接入端点)======
from .core.embed_service import embed_endpoint_override
@@ -125,6 +131,13 @@ __all__ = [
"snapshot_local_data_plane",
# 配置
"MemoryConfig",
# 公共扩展协议 / 精确读取类型
"EmbeddingService",
"EmbeddingServiceProtocol",
"EmbeddingDimensionMismatchError",
"MemoryKind",
"MemoryVariant",
"AmbiguousMemoryIdError",
# Embedding 端点覆盖
"embed_endpoint_override",
# 高级 API
+14 -7
View File
@@ -743,6 +743,7 @@ RECONCILE_PROMPT_ZH = """你是一个记忆整合系统(Memory Reconciler)
* content
* owner
* memory_at(发生时间)
* tags
new memories 没有 ID。
@@ -791,6 +792,7 @@ new memories 没有 ID。
* 无损合并
* 保留所有有效信息
* 不要产生重复 memory
* 两者在某个值上不一致时,保留 memory_at 更晚的那个值;绝不用更早的 memory 覆盖更新的值
### 同一次事件的补充 vs 另一次发生
@@ -823,6 +825,7 @@ new memories 没有 ID。
* 新 memory 与旧 memory 在同一维度冲突
* 两者不能同时为真
* 旧 memory 已不再代表当前状态
* 新 memory 的 memory_at 不早于旧 memory 的 memory_at
例如:
@@ -837,6 +840,7 @@ new memories 没有 ID。
* wording 差异
* 同义表达
* 可共存偏好
* 新 memory 的 memory_at 早于旧 memory:它是更早的状态,绝不能取代更新的状态(见「时间规则」)
要求:
@@ -859,7 +863,7 @@ new memories 没有 ID。
### 把多条旧记忆折叠进同一条链
如果**两条或更多**已有记忆描述的是**同一维度**,而新记忆是该维度上的最新变化,
如果**两条或更多**已有记忆描述的是**同一维度**,而新记忆是该维度上的最新变化(它的 memory_at 晚于所有被折叠的记忆),
你可以把它们折叠进同一条演化链。此时用 `memory_ids`(有序列表,**旧→新**)
代替单个 `memory_id`:
@@ -951,13 +955,16 @@ SUPERSEDE 不需要复制旧内容,因为旧节点仍存在。
## 时间规则
`memory_at: null`
每条 memory(新的和已有的)都带 memory_at,表示它发生的时间。记忆**不一定按发生顺序写入**:
历史可能晚导入,导入还没处理完时用户也可能手动添加记忆。所以一条 new memory 可能**比已有记忆更早**。
表示时间未知。
不要编造时间顺序。
不要仅因为“看起来更新”就 supersede。
* 新旧**只看 memory_at**。"New memories" 只表示"正在写入",不表示"更近"。
* 更早的 memory 绝不能取代更新的 memory:不能 SUPERSEDE 一条更新的记忆,也不能用 UPDATE 拿更早的值覆盖更新的值。
* 如果 new memory 比与它冲突的已有记忆更早:保留已有记忆作为当前状态;把 new memory 以"过去状态"ADD
(用过去时,并带上时间,例如"2023 年用户住在上海"),没有新信息就不输出。
* `memory_at: null` 表示时间未知。未知**不算更新**:不要用 memory_at 为 null 的 memory 去 SUPERSEDE 一条有时间的记忆。
* 不要编造时间顺序。
* 不要仅因为“看起来更新”就 supersede。
---
+105 -3
View File
@@ -221,6 +221,8 @@ Rules:
* merge losslessly
* preserve all useful information
* do NOT create duplicates
* if the two disagree on a value, keep the value from the memory with the later
`memory_at`; never let an older memory overwrite a newer value
### Same occurrence vs. another occurrence
@@ -255,6 +257,7 @@ Use ONLY when:
* the new memory contradicts an existing memory
* both cannot be true at the same time
* the old memory would now give the wrong current-state answer
* the new memory's `memory_at` is the same as or LATER than the existing memory's `memory_at`
Examples:
@@ -268,6 +271,8 @@ Do NOT use for:
* compatible preferences
* wording differences
* semantic equivalence
* a new memory whose `memory_at` is EARLIER than the existing memory's — it is an
older state and must never replace a newer one (see "Time order")
Rules:
@@ -289,7 +294,8 @@ Rules:
### Folding several existing memories into one chain
If TWO OR MORE existing memories track the SAME dimension and the new memory is
the latest change on that dimension, you may fold them into a single evolution
the latest change on that dimension — its `memory_at` is later than every folded
memory's `memory_at` — you may fold them into a single evolution
chain. Use `memory_ids` (an ordered list, OLDEST → NEWEST) instead of a single
`memory_id`:
@@ -318,8 +324,7 @@ older memories' text into it; they remain queryable in the chain history.
* Separate dated occurrences stay separate ADDs; use `same_series` only for the
same subject and same concrete activity/event type.
* Keep each memory focused on one coherent topic.
* Do not invent chronology.
* `memory_at: null` means unknown time.
* Follow "Time order" below.
* **NEVER merge or supersede across different subjects.** The subject is whoever
the content is about: "The user ..." vs "John ..." vs "Mary ...". Memories about
different people are independent facts even when the sentence is otherwise
@@ -332,6 +337,27 @@ older memories' text into it; they remain queryable in the chain history.
---
# Time order
Every memory, new or existing, has `memory_at`: when it happened. Memories are
NOT always written in the order they happened — history can be imported late,
and a memory can be added while older history is still being processed. A new
memory can therefore be OLDER than an existing one.
* Decide which memory is newer ONLY by `memory_at`. "New memories" means "being
written now", not "more recent".
* An older memory must never replace a newer one: no SUPERSEDE of a newer
memory, and no UPDATE that overwrites a newer value with an older one.
* If a new memory is older than an existing memory it conflicts with, keep the
existing memory as the current state. ADD the new one as a past state, worded
in the past with its time (for example "In 2023 the user lived in Shanghai"),
or output nothing if it adds no information.
* `memory_at: null` means unknown time. Unknown is NOT newer: never SUPERSEDE a
memory that has a known `memory_at` with one whose `memory_at` is null.
* Do not invent chronology.
---
# Optional: `related`
On any op you MAY add a `related` array recording how the memory being written
@@ -441,6 +467,81 @@ def append_reconcile_structure_prompt(
return prompt + RECONCILE_STRUCTURE_METADATA_PROMPT_EN
def _as_aware(value: Any) -> Optional[datetime]:
"""memory_at as a comparable aware datetime; None when unknown or unreadable."""
if isinstance(value, str) and value:
try:
value = datetime.fromisoformat(value)
except (TypeError, ValueError):
return None
if not isinstance(value, datetime):
return None
# Stored nodes rebuild memory_at from epoch seconds as naive local time
# (MemoryNode.from_dict), so a naive value is local, not UTC.
return value if value.tzinfo is not None else value.astimezone()
def enforce_time_order(
ops: List["ReconcileOp"],
new_memory_at: Any,
existing: Dict[str, Any],
) -> List["ReconcileOp"]:
"""An older memory must never replace a newer one, whatever the model says.
Memories are not always written in the order they happened: history is
imported late, and a person can add a memory while older history is still
being processed. The prompt tells the model to compare `memory_at`; this
makes it hold. A SUPERSEDE or UPDATE aimed at an existing memory whose
`memory_at` is later than the memory being written is turned into an ADD,
so the newer memory stays the current state. Unknown times are not
compared. A content-less UPDATE (the legacy DELETE mapping) is dropped.
"""
written_at = _as_aware(new_memory_at)
if written_at is None:
return ops
kept: List[ReconcileOp] = []
for op in ops:
if op.op not in ("SUPERSEDE", "UPDATE"):
kept.append(op)
continue
targets = list(op.memory_ids) if op.memory_ids else [op.memory_id]
newer = [
target for target in targets
if target and (node := existing.get(target)) is not None
and (target_at := _as_aware(getattr(node, "memory_at", None))) is not None
and target_at > written_at
]
if not newer:
kept.append(op)
continue
logger.warning(
"[reconciler] %s blocked: memory_at %s is older than existing %s; "
"kept the newer memory",
op.op, written_at.isoformat(), ",".join(newer),
)
if not op.content:
continue
kept.append(ReconcileOp(
op="ADD",
content=op.content,
layer=op.layer,
owner=op.owner,
tags=list(op.tags),
tag_ners=list(op.tag_ners),
topic_paths=list(op.topic_paths),
event_facet=dict(op.event_facet),
topic_assignments=list(op.topic_assignments),
structure_references=list(op.structure_references),
topic_paths_provided=op.topic_paths_provided,
event_facet_provided=op.event_facet_provided,
topic_assignments_provided=op.topic_assignments_provided,
speculate=op.speculate,
related=list(op.related),
reason=f"time_order: older than {','.join(newer)}; {op.reason}"[:500],
))
return kept
def build_reconcile_prompt(
*,
new_memories: List[str],
@@ -1399,6 +1500,7 @@ class MemoryReconciler:
_tok_total += response.tokens_used
raw_response = response.content
ops = self._parse_ops(raw_response)
ops = enforce_time_order(ops, current_time, candidate_map)
ops = self._inherit_exact_input_structure(
ops, new_memories_with_meta,
)
+1288 -487
View File
File diff suppressed because it is too large. Load diff
+192 -4
View File
@@ -217,6 +217,21 @@ def _in_clause(column: str, values: Optional[List[str]]) -> Tuple[str, List[Any]
return f"{column} IN ({placeholders})", list(values)
def _source_where(
include_sources: Optional[List[str]], exclude_sources: Optional[List[str]],
) -> Tuple[str, List[Any]]:
"""Caller-source filter as SQL (``1`` when unfiltered); exclusion wins."""
clauses: List[str] = []
params: List[Any] = []
if include_sources:
clauses.append(f"source IN ({','.join('?' for _ in include_sources)})")
params.extend(include_sources)
if exclude_sources:
clauses.append(f"source NOT IN ({','.join('?' for _ in exclude_sources)})")
params.extend(exclude_sources)
return (" AND ".join(clauses) or "1"), params
def _scope_matches(
row_workspace_id: Optional[str],
row_branch: Optional[str],
@@ -416,6 +431,36 @@ class CodingMemoryStore:
self._initialized = False
@staticmethod
def database_path(
config: MemoryConfig,
db_path: Optional[str] = None,
variant: str = "working",
) -> str:
"""Return the authoritative SQLite path without creating anything."""
if db_path is None:
data_dir = (
os.path.dirname(config.history.db_path)
if config.history.db_path else "./data"
)
db_path = os.path.join(data_dir, "coding_memory.db")
normalized = (
"coding"
if str(variant or "").strip().lower() == "coding"
else "working"
)
return _variant_db_path(db_path, "_cv" if normalized == "coding" else "")
@classmethod
def database_exists(
cls,
config: MemoryConfig,
db_path: Optional[str] = None,
variant: str = "working",
) -> bool:
"""Whether a coding namespace has an existing SQLite database."""
return os.path.isfile(cls.database_path(config, db_path, variant))
async def initialize(self) -> None:
if self._initialized:
return
@@ -573,6 +618,86 @@ class CodingMemoryStore:
logger.info(f"[coding-store] deleted experience_id={experience_id}")
return True
async def edit_experience(
self,
experience_id: str,
*,
user_id: Optional[str] = None,
task: Optional[str] = None,
solution: Optional[str] = None,
search_keys: Optional[List[str]] = None,
) -> Optional[CodingMemory]:
"""调用方手动改一条经验(client.update 用)。返回改后的经验;不存在返回 None。
与 update() 的区别:update 是写入链路的覆盖式重写,带写授权校验(同
agent/session/workspace 才能改);这里是用户对已有条目的直接编辑,只动
给定的字段,其余原样保留,因此不做授权校验。
向量行只嵌 task + search_keys(solution 不进向量),所以只改 solution 时
不重嵌、不碰向量库;改了 task / search_keys 才删旧 key 行重写。
updated_at 总是刷新。
"""
prev = await self.get_by_id(experience_id, user_id=user_id)
if prev is None:
return None
updated = deepcopy(prev)
if task is not None:
updated.task = task
if solution is not None:
updated.solution = solution
if search_keys is not None:
updated.search_keys = list(search_keys)
updated.updated_at = datetime.utcnow()
await _run_in_pool(self._upsert_sql_sync, updated)
if updated.task != prev.task or updated.search_keys != prev.search_keys:
await self._delete_vec_keys_by_count(experience_id, 1 + len(prev.search_keys))
await self._upsert_vec_keys(updated)
logger.info(f"[coding-store] edited experience_id={experience_id}")
return updated
async def edit_fact(
self,
fact_id: str,
*,
statement: str,
user_id: Optional[str] = None,
) -> Optional[WorkingFact]:
"""调用方手动改一条事实的 statement。返回改后的事实;不存在返回 None。
statement 是事实层的检索文本,所以 active 事实会重嵌并覆盖它那一行向量
(node_id 就是 fact_id,upsert 即覆盖)。superseded 事实的向量行早已撤掉,
这里不把它加回去,否则会让过期结论重新占召回名额。
"""
prev = await self.get_fact(fact_id, user_id=user_id)
if prev is None:
return None
updated = deepcopy(prev)
updated.statement = statement
await _run_in_pool(self._upsert_fact_sql_sync, updated)
if updated.status == "active":
await self._upsert_fact_vec(updated)
logger.info(f"[coding-store] edited fact_id={fact_id}")
return updated
async def delete_fact(self, fact_id: str, user_id: Optional[str] = None) -> bool:
"""物理删除一条事实(SQLite 行 + 事实向量行),返回是否真有删到。
别处留下的引用(经验的 source_fact_ids、后继事实的 fact_link)不清:
召回侧本来就按存在性过滤死链(get_many / get_facts 查不到自然掉队)。
"""
prev = await self.get_fact(fact_id, user_id=user_id)
if prev is None:
logger.info(f"[coding-store] delete fact skipped (not found): {fact_id}")
return False
if self._fact_vec is not None:
try:
await self._fact_vec.delete(fact_id)
except Exception as e:
logger.debug(f"[coding-store] delete fact vec {fact_id} ignored: {e}")
await _run_in_pool(self._delete_fact_sql_sync, fact_id, prev.user_id)
logger.info(f"[coding-store] deleted fact_id={fact_id}")
return True
# ------------------------------------------------------------
# 读取
# ------------------------------------------------------------
@@ -657,29 +782,84 @@ class CodingMemoryStore:
branch: Optional[str] = None,
limit: int = 100,
offset: int = 0,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
) -> List["CodingMemory"]:
"""按 workspace_id(+可选 branch)纯过滤列出该 user 的完整 coding memory。
作用域规则见 _scope_where(与 search 路径共用同一套)。无语义排序,按
updated_at 倒序(最近修改优先)。
updated_at 倒序(最近修改优先)。include_sources / exclude_sources 按
source 列在 SQL 里过滤(排除优先),所以 LIMIT/OFFSET 在过滤之后生效。
"""
return await _run_in_pool(
self._list_by_workspace_sync, user_id, workspace_id, branch, limit, offset
self._list_by_workspace_sync, user_id, workspace_id, branch, limit, offset,
include_sources, exclude_sources,
)
def _list_by_workspace_sync(
self, user_id: str, workspace_id: str, branch: Optional[str],
limit: int, offset: int,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
) -> List["CodingMemory"]:
scope_sql, scope_params = _scope_where(workspace_id, branch)
source_sql, source_params = _source_where(include_sources, exclude_sources)
with self._lock:
assert self._conn is not None
sql = (f"SELECT * FROM coding_memory_meta WHERE user_id = ? AND {scope_sql} "
f"ORDER BY updated_at DESC LIMIT ? OFFSET ?")
params: List[Any] = [user_id] + scope_params + [limit, offset]
f"AND {source_sql} "
f"ORDER BY updated_at DESC, experience_id ASC LIMIT ? OFFSET ?")
params: List[Any] = ([user_id] + scope_params + source_params
+ [limit, offset])
cur = self._conn.execute(sql, params)
return [_row_to_memory(r) for r in cur.fetchall()]
async def stats(
self,
user_id: str,
workspace_id: Optional[str] = None,
branch: Optional[str] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
) -> Dict[str, Any]:
"""经验数、最新创建/修改时间、active 事实数。纯聚合,不读正文。
作用域与 list_by_workspace 相同:workspace_id 为空表示该 user 全部;
来源过滤同样与 list_by_workspace 一致,count 因此恒等于分页可翻到的总数。
"""
return await _run_in_pool(
self._stats_sync, user_id, workspace_id, branch,
include_sources, exclude_sources,
)
def _stats_sync(
self, user_id: str, workspace_id: Optional[str], branch: Optional[str],
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
) -> Dict[str, Any]:
scope_sql, scope_params = _scope_where(workspace_id, branch)
source_sql, source_params = _source_where(include_sources, exclude_sources)
with self._lock:
assert self._conn is not None
row = self._conn.execute(
f"SELECT COUNT(*) AS n, MAX(created_at) AS created, "
f"MAX(updated_at) AS updated FROM coding_memory_meta "
f"WHERE user_id = ? AND {scope_sql} AND {source_sql}",
[user_id] + scope_params + source_params,
).fetchone()
facts = self._conn.execute(
f"SELECT COUNT(*) AS n FROM working_fact "
f"WHERE user_id = ? AND status = 'active' AND {scope_sql} "
f"AND {source_sql}",
[user_id] + scope_params + source_params,
).fetchone()
return {
"count": int(row["n"] or 0),
"latest_created_at": _parse_dt(row["created"]) if row["created"] else None,
"latest_updated_at": _parse_dt(row["updated"]) if row["updated"] else None,
"facts": int(facts["n"] or 0),
}
# ------------------------------------------------------------
# 召回
# ------------------------------------------------------------
@@ -1058,6 +1238,14 @@ class CodingMemoryStore:
(experience_id, user_id),
)
def _delete_fact_sql_sync(self, fact_id: str, user_id: str) -> None:
with self._lock:
assert self._conn is not None
self._conn.execute(
"DELETE FROM working_fact WHERE fact_id = ? AND user_id = ?",
(fact_id, user_id),
)
def _get_by_id_sync(
self, experience_id: str, user_id: Optional[str]
) -> Optional[CodingMemory]:
+9 -2
View File
@@ -50,6 +50,7 @@ class CodingWriter:
embed_service: EmbedService,
fact_reconciler: Optional[WorkingFactReconciler] = None,
cache: Any = None,
variant: str = "working",
):
self.store = store
self.extractor = extractor
@@ -59,6 +60,11 @@ class CodingWriter:
self.fact_reconciler = fact_reconciler or WorkingFactReconciler(
store, embed_service, llm_provider
)
self.variant = (
"coding"
if str(variant or "").strip().lower() == "coding"
else "working"
)
# cache 仅用于把 request_id 串给各阶段(各阶段自己打点);writer 不集中打点。
self.cache = cache
@@ -217,6 +223,7 @@ class CodingWriter:
return {
"success": True,
"scene": "productivity",
"variant": self.variant,
"request_id": request_id,
"elapsed_ms": round((time.perf_counter() - t0) * 1000, 2),
"ops": [op.to_dict() for op in ops],
@@ -278,11 +285,11 @@ class CodingWriter:
f"[coding-write] crosslink fact {f.fact_id} failed: {e}"
)
@staticmethod
def _empty_response(request_id: str, t0: float) -> Dict[str, Any]:
def _empty_response(self, request_id: str, t0: float) -> Dict[str, Any]:
return {
"success": True,
"scene": "productivity",
"variant": self.variant,
"request_id": request_id,
"elapsed_ms": round((time.perf_counter() - t0) * 1000, 2),
"ops": [],
+53
View File
@@ -0,0 +1,53 @@
"""Public structural contract for injected embedding services."""
from typing import Any, Dict, List, Protocol, Tuple, runtime_checkable
@runtime_checkable
class EmbeddingService(Protocol):
"""Embedding service accepted by :class:`HyMemoryClient`.
Implementations are borrowed by the client: their lifecycle remains owned
by the caller and ``HyMemoryClient.close()`` will not close them.
"""
async def embed(self, text: str, use_cache: bool = True) -> List[float]:
...
async def embed_with_meta(
self, text: str, use_cache: bool = True
) -> Tuple[List[float], Dict[str, Any]]:
...
async def embed_batch(
self, texts: List[str], use_cache: bool = True
) -> List[List[float]]:
...
async def embed_queued(
self, text: str, use_cache: bool = True
) -> List[float]:
...
async def embed_queued_with_meta(
self, text: str, use_cache: bool = True
) -> Tuple[List[float], Dict[str, Any]]:
...
def get_embedding_dims(self) -> int:
...
# Explicit alias for callers that prefer the conventional Protocol suffix.
EmbeddingServiceProtocol = EmbeddingService
class EmbeddingDimensionMismatchError(ValueError):
"""Raised before initialization when service and store dimensions differ."""
__all__ = [
"EmbeddingService",
"EmbeddingServiceProtocol",
"EmbeddingDimensionMismatchError",
]
@@ -160,6 +160,28 @@ def near_dup_demote(
return out
# MMR 只在调用方请求的 top-k(search 的 limit)小于这个值时自动启用。
# mmr_select 的贪心是 O(k^2 * pool) 次 Jaccard,k 大时代价陡增,而大 k 本身已
# 足够容纳多样性,收益很小。权重仍由 MEMORY_HYBRID_V2_MMR_LAMBDA 控制。
MMR_MAX_TOP_K = 20
def mmr_applies(top_k: int, lambda_: float) -> bool:
"""本次检索是否走 MMR:λ 在有效区间内,且 top_k < MMR_MAX_TOP_K。"""
return 0.0 < float(lambda_) < 0.999 and int(top_k) < MMR_MAX_TOP_K
def near_dup_applies(top_k: int) -> bool:
"""本次检索是否走近重复降权:与 MMR 同一个 top-k 门槛。
near_dup_demote 让每个候选和此前所有"非重复"候选两两算 Jaccard,候选
互不相似时是 O(pool^2);池子随 limit 放大(约 max(3*limit, 30) + 关键词
命中),limit 达到 MMR_MAX_TOP_K 时代价同样陡增。大 k 的调用方(如记忆
面板翻页)要的是全部相关项,相似项也应照常出现,降权没有收益。
"""
return int(top_k) < MMR_MAX_TOP_K
def mmr_select(
hits: Sequence[Dict[str, Any]],
limit: int,
+5
View File
@@ -401,6 +401,11 @@ class ReadRequest:
intention_limit: int = 0 # Intention 路的 topk,默认 0 关闭
# 重排开关(按次生效)。False 时跳过全部重排阶段:本地 cross-encoder
# (recall.local_rerank_enabled) 与记忆关系图重排 (_apply_graph_rerank),
# 结果按召回分排序。走请求对象而非环境变量,并发请求互不影响。
rerank: bool = True
# 时间过滤
created_after: Optional[float] = None # Unix timestamp (float),只返回 gmt_created >= 此值的记忆
+18 -5
View File
@@ -57,7 +57,9 @@ from ._retrieval.ner import (
memory_matches_time_hints,
)
from ._retrieval.diversity import (
near_dup_applies,
near_dup_demote,
mmr_applies,
mmr_select,
social_boilerplate_demote,
pool_df_demote,
@@ -1279,7 +1281,10 @@ class BaseReadPipeline(ReadPipeline):
SAME_SERIES_TIMELINE_ON = bool(
getattr(self.config.recall, "same_series_timeline_enabled", False)
)
LOCAL_RERANK_ON = bool(
# 按次的重排开关(search(rerank=False)):关掉本地 cross-encoder 与图重排,
# 结果按召回分排序。只读请求对象,不碰共享状态,并发请求互不影响。
RERANK_ON = bool(getattr(request, "rerank", True))
LOCAL_RERANK_ON = RERANK_ON and bool(
getattr(self.config.recall, "local_rerank_enabled", False)
)
LOCAL_RERANK_POOL_FACTOR = max(1, int(
@@ -1431,6 +1436,11 @@ class BaseReadPipeline(ReadPipeline):
MIN_SCORE = _resolve_normal_min_score(request.min_score)
final_limit = request.limit if request.limit > 0 else 10
# MMR 只在调用方的 top-k(final_limit,即 search 的 limit)< MMR_MAX_TOP_K
# 时启用;池子(LOCAL_RERANK 下是 final_limit 的若干倍)不参与判定。
MMR_ON = mmr_applies(final_limit, MMR_LAMBDA)
# 近重复降权同样是候选两两比较,用同一个 top-k 门槛。
NEAR_DUP_ON = near_dup_applies(final_limit)
vdb_sem_limit = max(final_limit * 3, 30)
vdb_kw_limit = max(final_limit * 5, 80)
graph_limit = max(
@@ -2462,7 +2472,7 @@ class BaseReadPipeline(ReadPipeline):
# 只把已选出的 top-N 换个顺序收益很小。
relation_paths: List[Dict[str, Any]] = []
_graph_stats: Dict[str, Any] = {}
if self._expansion_on():
if RERANK_ON and self._expansion_on():
_t_graph = datetime.now()
_pool_before = len(normal_results)
normal_results, relation_paths, _graph_stats = await self._apply_graph_rerank(
@@ -2525,7 +2535,7 @@ class BaseReadPipeline(ReadPipeline):
df_thr=POOL_DF_THR,
penalty=POOL_DF_PENALTY,
)
if NEAR_DUP_THR > 0 and NEAR_DUP_PENALTY > 0:
if NEAR_DUP_ON and NEAR_DUP_THR > 0 and NEAR_DUP_PENALTY > 0:
normal_results = near_dup_demote(
normal_results,
jaccard_thr=NEAR_DUP_THR,
@@ -2538,7 +2548,7 @@ class BaseReadPipeline(ReadPipeline):
))
if LOCAL_RERANK_ON else final_limit
)
if 0.0 < MMR_LAMBDA < 0.999 and len(normal_results) > _pre_rerank_limit:
if MMR_ON and len(normal_results) > _pre_rerank_limit:
normal_results = mmr_select(
normal_results,
_pre_rerank_limit,
@@ -2715,7 +2725,7 @@ class BaseReadPipeline(ReadPipeline):
reverse=True,
)
if (
0.0 < MMR_LAMBDA < 0.999
MMR_ON
and len(multi_seed_pool) > _pre_rerank_limit
):
multi_seed_pool = mmr_select(
@@ -3544,6 +3554,9 @@ class BaseReadPipeline(ReadPipeline):
"episode_capsule_render_mode": EPISODE_CAPSULE_RENDER_MODE,
"same_series_timeline_enabled": SAME_SERIES_TIMELINE_ON,
"local_rerank_enabled": LOCAL_RERANK_ON,
"rerank": RERANK_ON,
"mmr_applied": MMR_ON,
"near_dup_applied": NEAR_DUP_ON,
"local_rerank_model": getattr(
self.config.recall,
"local_rerank_model",
+9 -3
View File
@@ -19,6 +19,7 @@ import logging
from .base import WritePipeline, ReadPipeline
from ..config import MemoryConfig
from ..core.embed_service import EmbedService
from ..embedding import EmbeddingService
from ..data.vector_store import create_vector_store
from ..data.vector_store_base import VectorStoreBase
@@ -50,7 +51,12 @@ class ComponentFactory:
s2 = await factory.get_system2_writer() # System2Writer (ultra mode)
"""
def __init__(self, config: Optional[MemoryConfig] = None):
def __init__(
self,
config: Optional[MemoryConfig] = None,
*,
embedding_service: Optional[EmbeddingService] = None,
):
self.config = config or MemoryConfig.from_env()
# 实例缓存(单例)
@@ -62,7 +68,7 @@ class ComponentFactory:
self._readers: Dict[str, ReadPipeline] = {}
# 共享资源
self._shared_embed_service: Optional[EmbedService] = None
self._shared_embed_service: Optional[EmbeddingService] = embedding_service
self._shared_vector_store: Optional[VectorStoreBase] = None
self._shared_vector_store_initialized: bool = False
self._shared_graph_store = None
@@ -72,7 +78,7 @@ class ComponentFactory:
# 共享资源
# ================================================================
def _get_shared_embed_service(self) -> EmbedService:
def _get_shared_embed_service(self) -> EmbeddingService:
if self._shared_embed_service is None:
self._shared_embed_service = EmbedService(self.config)
return self._shared_embed_service
+23
View File
@@ -0,0 +1,23 @@
"""Public types and errors for exact memory-ID lookup."""
from typing import Literal, Sequence
MemoryKind = Literal["chat", "experience", "fact"]
MemoryVariant = Literal["working", "coding"]
class AmbiguousMemoryIdError(LookupError):
"""An ID exists in more than one public memory namespace."""
def __init__(self, memory_id: str, namespaces: Sequence[str]):
self.memory_id = memory_id
self.namespaces = tuple(namespaces)
joined = ", ".join(self.namespaces)
super().__init__(
f"memory ID {memory_id!r} is ambiguous across namespaces: {joined}; "
"pass kind and/or variant to disambiguate"
)
__all__ = ["MemoryKind", "MemoryVariant", "AmbiguousMemoryIdError"]
+45 -7
View File
@@ -33,9 +33,28 @@ import threading
from datetime import datetime
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Any, Dict, Optional
from urllib.parse import parse_qs, urlparse
from .retrieval import AmbiguousMemoryIdError
logger = logging.getLogger("hy_memory.server")
def _namespace_args(values: Dict[str, Any]) -> Dict[str, Any]:
"""kind / variant from a query string (lists) or a JSON body (scalars).
They disambiguate an id that exists in more than one namespace (chat,
experience, fact; working or coding). Absent means "resolve it".
"""
args: Dict[str, Any] = {}
for key in ("kind", "variant"):
value = values.get(key)
if isinstance(value, list):
value = value[0] if value else None
if value:
args[key] = str(value)
return args
# Lazy-init global client
_client = None
_client_lock = None
@@ -148,7 +167,6 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
if path == "/api/v1/metrics":
minutes = 5
if "?" in self.path:
from urllib.parse import parse_qs, urlparse
qs = parse_qs(urlparse(self.path).query)
minutes = int(qs.get("minutes", ["5"])[0])
client = _get_client()
@@ -161,7 +179,8 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
if m:
memory_id = m.group(1)
client = _get_client()
result = client.get(memory_id)
query = parse_qs(urlparse(self.path).query)
result = client.get(memory_id, **_namespace_args(query))
if result is None:
_json_response(self, 404, {"error": "not_found", "memory_id": memory_id})
else:
@@ -170,6 +189,10 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
_json_response(self, 404, {"error": "not_found", "path": path})
except AmbiguousMemoryIdError as e:
_json_response(self, 409, {"error": "ambiguous_memory_id", "detail": str(e)})
except ValueError as e:
_json_response(self, 400, {"error": str(e)})
except Exception as e:
logger.error(f"[server] GET {path} error: {e}", exc_info=True)
_json_response(self, 500, {"error": str(e)})
@@ -213,7 +236,7 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
_json_response(self, 500, {"error": str(e)})
def do_PUT(self):
path = self.path.rstrip("/")
path = self.path.split("?")[0].rstrip("/")
body = _read_json_body(self)
if body is None:
_json_response(self, 400, {"error": "invalid JSON body"})
@@ -228,29 +251,43 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
_json_response(self, 400, {"error": "content is required"})
return
client = _get_client()
result = client.update(memory_id, content)
result = client.update(
memory_id,
content,
**_namespace_args(body),
**({"task": body["task"]} if body.get("task") else {}),
)
_json_response(self, 200, result)
return
_json_response(self, 404, {"error": "not_found", "path": path})
except AmbiguousMemoryIdError as e:
_json_response(self, 409, {"error": "ambiguous_memory_id", "detail": str(e)})
except ValueError as e:
_json_response(self, 400, {"error": str(e)})
except Exception as e:
logger.error(f"[server] PUT {path} error: {e}", exc_info=True)
_json_response(self, 500, {"error": str(e)})
def do_DELETE(self):
path = self.path.rstrip("/")
path = self.path.split("?")[0].rstrip("/")
try:
m = re.match(r"^/api/v1/memories/([^/]+)$", path)
if m:
memory_id = m.group(1)
client = _get_client()
result = client.delete(memory_id)
query = parse_qs(urlparse(self.path).query)
result = client.delete(memory_id, **_namespace_args(query))
_json_response(self, 200, result)
return
_json_response(self, 404, {"error": "not_found", "path": path})
except AmbiguousMemoryIdError as e:
_json_response(self, 409, {"error": "ambiguous_memory_id", "detail": str(e)})
except ValueError as e:
_json_response(self, 400, {"error": str(e)})
except Exception as e:
logger.error(f"[server] DELETE {path} error: {e}", exc_info=True)
_json_response(self, 500, {"error": str(e)})
@@ -455,7 +492,8 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
if key in body:
kwargs[key] = body[key]
for key in ("limit", "min_score", "schema_min_score", "schema_limit", "reader"):
for key in ("limit", "min_score", "schema_min_score", "schema_limit", "reader",
"rerank"):
if key in body:
kwargs[key] = body[key]
+68 -6
View File
@@ -8,6 +8,7 @@ Uses a two-tier strategy:
This avoids langdetect's known weakness with short Chinese text.
"""
import json
import logging
import re
from typing import List, Optional
@@ -135,6 +136,30 @@ def detect_language(text: str) -> str:
return "zh" if ratio > 0.1 else "en"
def _spoken_text(value: str) -> str:
"""Ignore text-block metadata when a message content is structured JSON."""
text = value.strip()
if text.startswith("["):
try:
blocks = json.loads(text)
except (ValueError, TypeError):
pass
else:
if (
isinstance(blocks, list)
and blocks
and all(
isinstance(block, dict)
and isinstance(block.get("type"), str)
and block["type"] in {"text", "input_text", "output_text"}
and isinstance(block.get("text"), str)
for block in blocks
)
):
return "\n".join(block["text"] for block in blocks)
return text
def extract_content_for_detection(
content: str = "",
messages: Optional[List] = None,
@@ -153,20 +178,57 @@ def extract_content_for_detection(
Concatenated content text suitable for language detection.
"""
if content and content.strip():
return content.strip()
value = content.strip()
# Writer serializes message batches to JSON before calling Extractor.
# Detect the spoken text, not the leading ASCII keys (role/content),
# without changing the raw evidence sent to the model.
if value.startswith("["):
try:
decoded = json.loads(value)
except (ValueError, TypeError):
pass
else:
if (
isinstance(decoded, list)
and decoded
and all(
isinstance(item, dict)
and isinstance(item.get("role"), str)
and isinstance(item.get("content"), str)
for item in decoded
)
):
# In a mixed batch, the latest user turn is the clearest
# instruction for the language; assistant/tool metadata
# must not decide it first.
user_text = [
_spoken_text(item["content"])
for item in decoded
if item["role"] == "user" and item["content"].strip()
]
if user_text:
return user_text[-1]
spoken = "\n".join(_spoken_text(item["content"]) for item in decoded)
if spoken.strip():
return spoken
return _spoken_text(value)
if messages:
parts = []
user_parts = []
for msg in messages:
if isinstance(msg, dict):
c = msg.get("content", "")
c, role = msg.get("content", ""), msg.get("role")
elif hasattr(msg, "content"):
c = msg.content
c, role = msg.content, getattr(msg, "role", None)
else:
continue
if c and c.strip():
parts.append(c.strip())
return "\n".join(parts)
if isinstance(c, str) and c.strip():
text = _spoken_text(c)
parts.append(text)
if role == "user":
user_parts.append(text)
return user_parts[-1] if user_parts else "\n".join(parts)
return ""
+17 -41
View File
@@ -9,10 +9,17 @@
## Architecture Principles
1. Memory 读写与存储 backend **只经 SDK**(config / env)。
2. MemHub 对 memory 的原子操作 **只经 SDK**;缺接口先补 SDK。
2. MemHub 运行时对 memory 的原子操作 **只经 SDK 公共 Client**;缺正常业务
接口先补 SDK。Runtime writers 停止后的 App 启动迁移可直接操作底层存储,
但不得把迁移专用方法加入公共 Client。
3. MemHub **不存 memory 实体**;**可以存 `memory_id`** 做组织/回执。原文进入 **SDK raw**。
4. SDK raw 只保留滚动三天;一条 raw 对应一个完整 turn,只含真人 user 与
assistant,tool 不持久化。提取后的高层 memory 不受该窗口影响。
5. `source_raw_ids` 是 SDK 内部溯源关系,不是产品 Evidence 合同;记忆详情不提供
“原始依据”或按 raw ID 展开原文的入口。
生产入口仍建议经 `HyMemoryBridge`(共享 Runtime、模型配置、密钥、harness/model 门禁),但 bridge **不得**再把 SDK 结果镜像成第二套 memory 正文权威。
生产入口通过共享 Runtime 租约取得公开 `HyMemoryClient`;MemHub 只在调用前执行
模型配置、密钥和 harness/model 门禁,不维护第二套 memory 正文权威。
来源授权、模型授权和宿主召回授权仍是 MemHub 管理面门禁;memory
数据平面的权威在 SDK。
@@ -30,46 +37,15 @@ reader: hybrid
## MemHub 运行时边界
MemHub 对 SDK 使用公开、可复现的本地策略(经 bridge / `hy_runtime_config` 注入):
MemHub 通过公开 `HyMemoryClient` 配置使用可复现的本地策略:
- vector / graph / cache / history / coding 等 **一律 SDK config/env**;
- 模型与 embedding 来自版本化 Model Profile;密钥进安全存储;
- MemHub 管理库可存 harness/model/队列状态与 **`memory_id` 引用**。
发布包默认 SDK 行为环境变量仍集中在
`apps/core/src/one_memory/hy_runtime_config.py`,并由下表锁定:
| 变量 | 值 |
|---|---|
| `MEMORY_MODE` | `ultra` |
| `MEMORY_ENABLE_GRAPH` | `true` |
| `MEMORY_GRAPH_RERANK_ENABLED` | `false` |
| `MEMORY_GRAPH_TEMPORAL_SPINE` | `true` |
| `MEMORY_TOPIC_SUMMARY_ENABLED` | `false` |
| `MEMORY_EPISODE_SUMMARY_ENABLED` | `false` |
| `MEMORY_EPISODE_RECALL_ENABLED` | `true` |
| `MEMORY_DYNAMIC_EPISODE_ENABLED` | `true` |
| `MEMORY_SESSION_EPISODE_ENABLED` | `false` |
| `MEMORY_SESSION_EPISODE_CONTEXT_ENABLED` | `false` |
| `MEMORY_EPISODE_CAPSULE_CONTEXT_ENABLED` | `true` |
| `MEMORY_EPISODE_CAPSULE_QUERY_MIN` | `0.65` |
| `MEMORY_EPISODE_CAPSULE_ASSIGNMENT_MIN` | `0.60` |
| `MEMORY_EPISODE_CAPSULE_FIRST_RANK_MAX` | `3` |
| `MEMORY_EPISODE_CAPSULE_MEMBER_LIMIT` | `3` |
| `MEMORY_ENABLE_INTENTION` | `false` |
| `MEMORY_ENABLE_QUERY_REWRITE` | `false` |
| `MEMORY_FEW_SHOT_ENABLED` | `true` |
| `MEMORY_HYBRID_V2_QUERY_EXPAND` | `1` |
| `MEMORY_HYBRID_V2_GEO_INTENT_PRIOR` | `0.6` |
| `MEMORY_HYBRID_V2_DIVERSITY` | `1` |
| `MEMORY_HYBRID_V2_TEMPORAL_ADMIT_PRIOR` | `0.55` |
| `MEMORY_HYBRID_V2_TEMPORAL_BOOST` | `0.18` |
| `MEMORY_CROSS_DOMAIN_CON_THRESHOLD` | `0.5` |
| `MEMORY_READER_ENABLE_SUMMARY` | `false` |
| `MEMORY_SUMMARY_ENABLED_IN_SYS2` | `false` |
| `MEMORY_LOCAL_RERANK_ENABLED` | `false` |
| `RECONCILE_ENABLED` | `true` |
| `HY_MEMORY_READER` | `hybrid` |
MemHub 不再通过 import-time 环境变量覆盖 SDK 的提取、reconcile、episode、
intention、few-shot、summary 或去重策略。产品只配置模型/密钥/endpoint、本地
backend 路径、公开 embedding service,以及每次公开 Client 调用的参数。
独立脚本直连 SDK 时须自配存储目录与模型端点。
@@ -77,7 +53,7 @@ MemHub 对 SDK 使用公开、可复现的本地策略(经 bridge / `hy_runtim
下面是独立调试示例。
产品代码经 `HyMemoryBridge` 获取已配置的 client(管理面门禁),memory I/O
产品代码经 `hy_sdk_runtime.SdkClientHandle` 获取已配置的公开 client;memory I/O
由 SDK 完成。
```python
@@ -228,7 +204,8 @@ result = client.search(
- 区分空结果、权限拒绝、不可用、超时、部分结果;
- 未发布的 Matter 占位功能不保留表或 API;未来组织功能另立合同。
生产入口:`hy_memory_bridge.py`、`recall.py`、`packages/host-bridge`。
生产入口:`hy_sdk_runtime.py`、`sdk_write_processor.py`、`recall.py` 和
`packages/host-bridge`。
## 关闭客户端
@@ -245,8 +222,7 @@ Runtime 长期复用 client;仅在停止、配置切换或受控恢复时重
```text
harness / user action (+ authz, model, queue)
-> HyMemoryBridge
-> Hy Memory SDK
-> public HyMemoryClient
-> SDK stores (config/env)
-> optional MemHub persist memory_id / org metadata
-> Recall Broker / UI (SDK read + MemHub authz)
+5 -5
View File
@@ -18,13 +18,13 @@ If you run into a problem, please open an Issue with the version, platform,repro
## 发行与平台合同
源码当前固定使用公开 PyPI 开发预览 `hy-memory==2.0.5.dev3`。MemHub 源码版本为 `1.0.4`
(build 13),GitHub 仓库为
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.6`。MemHub 源码版本为 `1.0.5`
(build 14),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
`v1.0.4` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
`v1.0.5` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
@@ -45,7 +45,7 @@ Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命
## 仓库结构
- `apps/core`:Python / FastAPI Kernel,负责授权、模型、队列、SDK bridge 与 API。
- `apps/core`:Python / FastAPI Kernel,负责授权、模型、队列、SDK Runtime 边界与 API。
- `apps/cli`:安装、控制和诊断共享 Runtime 的 `one-memory` CLI。
- `apps/web`:React / TypeScript 管理界面。
- `apps/desktop`:Electron 桌面壳与图形安装入口。
@@ -117,7 +117,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
## Hy Memory
SDK 依赖固定为公开 PyPI 开发预览 `hy-memory==2.0.5.dev3`。SDK 配置、权限边界及 search
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.6`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+3 -3
View File
@@ -3,9 +3,9 @@
"dataLicense": "CC0-1.0",
"SPDXID": "SPDXRef-DOCUMENT",
"name": "one-memory",
"documentNamespace": "https://one-memory.local/spdx/2026-10-22T08:09:31Z",
"documentNamespace": "https://one-memory.local/spdx/2026-10-22T13:18:16Z",
"creationInfo": {
"created": "2026-10-22T08:09:31Z",
"created": "2026-10-22T13:18:16Z",
"creators": [
"Tool: one-memory-generate-sbom"
]
@@ -2352,7 +2352,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.5.dev3",
"versionInfo": "2.0.6",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+2 -2
View File
@@ -2,14 +2,14 @@
MemHub itself is licensed under Apache-2.0 (see `LICENSE`).
It bundles the runtime dependencies listed below.
Generated from the installed dependency tree on 2026-10-22T08:09:31Z
Generated from the installed dependency tree on 2026-10-22T13:18:16Z
by `python3 scripts/generate_sbom.py`. Do not edit by hand.
Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.5.dev3
## hy-memory 2.0.6
License: Apache-2.0
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/cli",
"version": "1.0.4",
"version": "1.0.5",
"description": "MemHub bootstrap CLI for installing and managing the local-first shared memory runtime",
"license": "Apache-2.0",
"repository": {
+23 -48
View File
@@ -21,18 +21,21 @@ const TOKEN = "probe-token-not-a-real-credential";
const MEMORY = {
memory_id: "probe-1",
user_id: "local",
kind: "chat",
status: "confirmed",
content: "部署脚本要先跑 verify_merge_plan。",
scopes: ["all-apps"],
evidence_ids: ["probe-e1"],
origin: "extracted",
layer: "l2_fact",
status: "active",
is_latest: true,
supersedes: [],
superseded_by: [],
tags: [],
revision: 2,
created_at: "2026-09-19T00:00:00Z",
updated_at: "2026-09-19T00:00:00Z",
deleted_at: null,
owner: null,
source_raw_ids: ["probe-raw-1"],
user_id: "local",
agent_id: "codex",
session_id: "probe-session",
source: "codex",
memory_at: 1758240000,
gmt_created: 1758240000,
};
const ROUTES = new Map([
@@ -124,26 +127,6 @@ const ROUTES = new Map([
},
],
],
[
"/api/v1/memory-tasks",
[
{
task_id: "probe-task",
request_id: "probe-request",
user_id: "local",
kind: "extract",
input_ref: "probe",
status: "waiting_for_model",
profile_id: "probe-profile",
profile_revision: 1,
profile_snapshot: {},
attempts: 0,
last_error_code: "model_missing",
created_at: "2026-09-19T00:00:00Z",
updated_at: "2026-09-19T00:00:01Z",
},
],
],
[
"/api/v1/model-profiles",
[
@@ -159,28 +142,20 @@ const ROUTES = new Map([
},
],
],
["/api/v1/home-summary", { memory_count: 1, latest: null, observed_at: "T" }],
[
"/api/v1/memory-library",
{ total: 1, offset: 0, limit: 100, items: [MEMORY] },
],
[
"/api/v1/memories/probe-1",
"/api/v2/home-summary",
{
memory: MEMORY,
evidence: [
{
evidence_id: "probe-e1",
available: true,
source_id: "codex",
written_by: "user",
created_at: "2026-09-19T00:00:00Z",
verbatim: "部署前先跑 verify_merge_plan,别靠读 diff 猜冲突。",
},
],
history: [{ memory_id: "probe-1", revision: 2, action: "confirm" }],
schema_version: 2,
memory_count: 1,
latest: MEMORY,
observed_at: "2026-09-19T00:00:01Z",
},
],
[
"/api/v2/memory-library",
{ memories: [MEMORY], total: 1, offset: 0, limit: 100 },
],
["/api/v2/memories/probe-1", MEMORY],
]);
const seen = [];
+42 -1
View File
@@ -1,6 +1,7 @@
import { mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { inflateRawSync } from "node:zlib";
import { DiagnosticLogger } from "@one-memory/runtime-control";
import { afterEach, expect, it, vi } from "vitest";
@@ -29,6 +30,27 @@ async function fixture() {
return dataDir;
}
function readZipEntry(archive: Buffer, expected: string): string {
let offset = 0;
while (archive.readUInt32LE(offset) === 0x04034b50) {
const method = archive.readUInt16LE(offset + 8);
const compressedSize = archive.readUInt32LE(offset + 18);
const nameLength = archive.readUInt16LE(offset + 26);
const extraLength = archive.readUInt16LE(offset + 28);
const nameStart = offset + 30;
const dataStart = nameStart + nameLength + extraLength;
const name = archive.subarray(nameStart, nameStart + nameLength).toString();
const compressed = archive.subarray(dataStart, dataStart + compressedSize);
if (name === expected) {
if (method === 0) return compressed.toString();
if (method === 8) return inflateRawSync(compressed).toString();
throw new Error("unsupported zip compression");
}
offset = dataStart + compressedSize;
}
throw new Error(`missing ${expected}`);
}
it("prints matching diagnostic events as NDJSON without starting Runtime", async () => {
const dataDir = await fixture();
let stdout = "";
@@ -74,6 +96,25 @@ it("exports one support zip to the requested downloads directory", async () => {
});
expect(receipt.path).toMatch(/MemHub-日志-3800-\d{8}-\d{4}\.zip$/);
expect(await readdir(downloads)).toEqual([path.basename(receipt.path)]);
const archive = await readFile(receipt.path);
expect(JSON.parse(readZipEntry(archive, "summary.json")).commit).toBeNull();
});
it("fails closed for an explicitly supplied invalid commit", async () => {
const dataDir = await fixture();
const downloads = await mkdtemp(path.join(os.tmpdir(), "om-cli-downloads-"));
roots.push(downloads);
vi.spyOn(process.stdout, "write").mockImplementation(() => true);
await expect(
run(["logs", "export", "--json"], {
ONE_MEMORY_DATA_DIR: dataDir,
ONE_MEMORY_EXPORT_DIR: downloads,
ONE_MEMORY_BUILD_NUMBER: "3800",
ONE_MEMORY_COMMIT: "unknown",
}),
).rejects.toThrow("diagnostic_commit_invalid");
expect(await readdir(downloads)).toEqual([]);
});
it("never emits a secret from a malformed manually written event", async () => {
+530 -35
View File
@@ -3,7 +3,9 @@ import { tmpdir } from "node:os";
import path from "node:path";
import {
claimRuntimeServiceForApp,
clearSdkMigrationBlock,
readRuntimeServiceOwner,
recordSdkMigrationBlock,
resolveRuntimeCliConfig,
} from "@one-memory/runtime-control";
import { describe, expect, it, vi } from "vitest";
@@ -57,6 +59,11 @@ const runtimeMocks = vi.hoisted(() => ({
})),
uninstallUserService: vi.fn(async (..._args: unknown[]) => "removed"),
restartUserService: vi.fn(async () => undefined),
runSdkMigration: vi.fn(
async (): Promise<{ state: string; code?: string }> => ({
state: "not_required",
}),
),
rollbackLocalInstall: vi.fn(async () => undefined),
planCliLink: vi.fn(async ({ home, installRoot }) => ({
schema_version: 1,
@@ -126,6 +133,7 @@ vi.mock("@one-memory/runtime-control", async (importOriginal) => {
installUserService: runtimeMocks.installUserService,
uninstallUserService: runtimeMocks.uninstallUserService,
restartUserService: runtimeMocks.restartUserService,
runSdkMigration: runtimeMocks.runSdkMigration,
rollbackLocalInstall: runtimeMocks.rollbackLocalInstall,
planCliLink: runtimeMocks.planCliLink,
installCliLink: runtimeMocks.installCliLink,
@@ -784,7 +792,7 @@ describe("public One Memory CLI", () => {
}
});
it("upgrade failure restarts the restored artifact with the same data identity", async () => {
it("upgrade restart failure keeps the new version and never restarts the old one", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce({
schema_version: 1,
@@ -826,25 +834,27 @@ describe("public One Memory CLI", () => {
command: "upgrade",
code: "restart failed",
});
expect(runtimeMocks.rollbackLocalInstall).toHaveBeenCalledOnce();
// The CLI manages the objective aliases the package does not own
// (`memhub` + `memhub-cli`); both are rolled back.
expect(aliasMocks.rollbackManagedCliLink).toHaveBeenCalledTimes(2);
expect(runtimeMocks.restartUserService).toHaveBeenCalledTimes(2);
for (const call of runtimeMocks.restartUserService.mock
.calls as unknown[][]) {
expect(call[0]).toMatchObject({
dataDir: "/tmp/custom data",
installRoot: "/tmp/install/current",
});
}
expect(JSON.parse(streams.stderr())).toMatchObject({
rolled_back: false,
});
// Product rule: the previous Runtime may predate the SDK migration gate,
// so it is never restored after the migration was handed the home.
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
expect(aliasMocks.rollbackManagedCliLink).not.toHaveBeenCalled();
expect(runtimeMocks.restartUserService).toHaveBeenCalledOnce();
expect(
(runtimeMocks.restartUserService.mock.calls as unknown[][])[0]?.[0],
).toMatchObject({
dataDir: "/tmp/custom data",
installRoot: "/tmp/install/current",
});
});
// O-22 (2026-09-29 C8 H17): `upgrade --yes` on a manifest-valid but
// crash-looping version reported exit 0 and left `current` on the bad
// version. A restart that merely returns is not proof the new Runtime came
// up, so the CLI now verifies health and rolls back if it never does.
it("rolls back and fails when the upgraded Runtime never becomes healthy", async () => {
it("fails without rolling back when the upgraded Runtime never becomes healthy", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce({
schema_version: 1,
@@ -884,9 +894,219 @@ describe("public One Memory CLI", () => {
command: "upgrade",
code: "upgrade_health_check_failed",
});
// Rolled back to the previous version and restarted it.
// Kept on the new version; the old Runtime is not restarted.
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
expect(runtimeMocks.restartUserService).toHaveBeenCalledOnce();
});
const upgradeArguments = [
"upgrade",
"--source",
"/tmp/dist",
"--home",
"/tmp/home",
"--install-root",
"/tmp/install",
"--data-dir",
"/tmp/custom data",
"--yes",
];
const presentCliLink = {
schema_version: 1 as const,
platform: "linux" as const,
home: "/tmp/home",
install_root: "/tmp/install",
bin_directory: "/tmp/bin",
status: "present" as const,
blocked_reason: null,
link_path: "/tmp/bin/one-memory",
target_path: "/tmp/install/current/bin/one-memory",
};
it("says the data was restored when a failed migration put it back", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce(presentCliLink);
runtimeMocks.runSdkMigration.mockResolvedValueOnce({
state: "deferred",
code: "sdk_migration_memory_unreadable",
dataRestored: true,
} as never);
const streams = captureStreams();
let code: number;
try {
code = await run(
upgradeArguments,
{},
{ waitForRuntimeReady: async () => true },
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
const failure = JSON.parse(streams.stderr());
expect(failure).toMatchObject({
code: "sdk_migration_memory_unreadable",
data_restored: true,
log_file: "/tmp/custom data/logs/sdk-migration.log",
});
expect(failure.message).toContain("记忆数据已恢复到升级前的状态。");
expect(failure.message).toContain(
"/tmp/custom data/logs/sdk-migration.log",
);
});
it("migrates the home with the new Core before restarting the Runtime", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce(presentCliLink);
const order: string[] = [];
runtimeMocks.runSdkMigration.mockImplementationOnce(async () => {
order.push("migrate");
return { state: "cutover" };
});
runtimeMocks.restartUserService.mockImplementationOnce(async () => {
order.push("restart");
});
const output = captureStdout();
let code: number;
try {
code = await run(
upgradeArguments,
{},
{ waitForRuntimeReady: async () => true },
);
} finally {
output.restore();
}
expect(code!).toBe(0);
expect(order).toEqual(["migrate", "restart"]);
expect(runtimeMocks.runSdkMigration).toHaveBeenCalledWith(
expect.objectContaining({ dataDir: "/tmp/custom data" }),
);
});
it("stays on the new version, not serving, when the migration does not finish", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce(presentCliLink);
runtimeMocks.runSdkMigration.mockResolvedValueOnce({
state: "deferred",
code: "sdk_migration_disk_space_insufficient",
});
const streams = captureStreams();
let code: number;
try {
code = await run(
upgradeArguments,
{},
{ waitForRuntimeReady: async () => true },
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
const failure = JSON.parse(streams.stderr());
expect(failure).toMatchObject({
command: "upgrade",
code: "sdk_migration_disk_space_insufficient",
// The one file support needs, and where to send it.
log_file: "/tmp/custom data/logs/sdk-migration.log",
issues_url: "https://github.com/Tencent-Hunyuan/Hy-Memory/issues/new",
});
expect(failure.message).toContain(
"/tmp/custom data/logs/sdk-migration.log",
);
expect(failure.message).toContain("GitHub");
expect(failure.message).not.toContain("回滚到上一版本");
expect(runtimeMocks.runSdkMigration).toHaveBeenCalledWith(
expect.objectContaining({ caller: "cli-upgrade" }),
);
// No rollback: the old Runtime has no migration gate. The new service is
// (re)started so its gate, not the old Runtime, holds the data dir.
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
expect(aliasMocks.rollbackManagedCliLink).not.toHaveBeenCalled();
expect(runtimeMocks.restartUserService).toHaveBeenCalledOnce();
expect(
(runtimeMocks.restartUserService.mock.calls as unknown[][])[0]?.[0],
).toMatchObject({
installRoot: "/tmp/install/current",
});
});
it("stays on the new version when the migration itself throws", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce(presentCliLink);
runtimeMocks.runSdkMigration.mockRejectedValueOnce(
new Error("sdk_migration_fence_failed"),
);
const streams = captureStreams();
let code: number;
try {
code = await run(
upgradeArguments,
{},
{ waitForRuntimeReady: async () => true },
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
expect(JSON.parse(streams.stderr())).toMatchObject({
command: "upgrade",
code: "sdk_migration_fence_failed",
log_file: "/tmp/custom data/logs/sdk-migration.log",
issues_url: "https://github.com/Tencent-Hunyuan/Hy-Memory/issues/new",
});
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
});
it("still rolls back a failure before the migration touched the home", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce(presentCliLink);
aliasMocks.installManagedCliLink.mockRejectedValueOnce(
new Error("cli_link_failed"),
);
const streams = captureStreams();
let code: number;
try {
code = await run(
upgradeArguments,
{},
{ waitForRuntimeReady: async () => true },
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
expect(JSON.parse(streams.stderr())).toMatchObject({
command: "upgrade",
code: "cli_link_failed",
});
expect(runtimeMocks.runSdkMigration).not.toHaveBeenCalled();
expect(runtimeMocks.rollbackLocalInstall).toHaveBeenCalledOnce();
expect(runtimeMocks.restartUserService).toHaveBeenCalledTimes(2);
// The previous Runtime was never stopped; nothing is (re)started.
expect(runtimeMocks.restartUserService).not.toHaveBeenCalled();
});
it("keeps the new version when a migrated home is slow to come up", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce(presentCliLink);
runtimeMocks.runSdkMigration.mockResolvedValueOnce({ state: "cutover" });
const streams = captureStreams();
let code: number;
try {
code = await run(
upgradeArguments,
{},
{ waitForRuntimeReady: async () => false },
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
expect(JSON.parse(streams.stderr())).toMatchObject({
command: "upgrade",
code: "runtime_not_ready_after_sdk_migration",
});
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
});
it("reports success once the upgraded Runtime is healthy", async () => {
@@ -1384,7 +1604,7 @@ describe("public One Memory CLI", () => {
expect(JSON.parse(output.value()).apply).toBe(true);
});
it("rolls back the distribution when service installation fails", async () => {
it("keeps the distribution when service installation fails after the migration ran", async () => {
runtimeMocks.applyLocalInstall.mockResolvedValueOnce({
schema_version: 1,
source: "/tmp/distribution",
@@ -1425,13 +1645,14 @@ describe("public One Memory CLI", () => {
command: "install",
status: "error",
code: "service failed",
rolled_back: false,
});
expect(runtimeMocks.rollbackLocalInstall).toHaveBeenCalledOnce();
expect(runtimeMocks.rollbackCliLink).toHaveBeenCalledOnce();
expect(aliasMocks.rollbackManagedCliLink).toHaveBeenCalledTimes(2);
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
expect(runtimeMocks.rollbackCliLink).not.toHaveBeenCalled();
expect(aliasMocks.rollbackManagedCliLink).not.toHaveBeenCalled();
});
it("upgrades only an existing managed CLI and rolls back a failed restart", async () => {
it("upgrades only an existing managed CLI and keeps it after a failed restart", async () => {
runtimeMocks.planCliLink.mockResolvedValueOnce({
schema_version: 1,
platform: "linux",
@@ -1471,7 +1692,7 @@ describe("public One Memory CLI", () => {
status: "error",
code: "restart failed",
});
expect(runtimeMocks.rollbackLocalInstall).toHaveBeenCalledOnce();
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
expect(runtimeMocks.installCliLink).not.toHaveBeenCalled();
});
@@ -1895,6 +2116,189 @@ describe("lifecycle commands accept --json", () => {
});
});
// A Runtime service that refuses to serve an unmigrated home must be reported
// as that, with the migration log and where to send it, not as a generic
// "not ready within budget".
describe("lifecycle commands report a migration block", () => {
async function blockedHome() {
const dataDir = await mkdtemp(path.join(tmpdir(), "memhub-blocked-"));
await recordSdkMigrationBlock(
dataDir,
"sdk_migration_disk_space_insufficient",
"runtime-service",
);
return dataDir;
}
// darwin: kickstart "restarts" but never becomes ready. Elsewhere: no
// service, so the direct spawn fails. Both must surface the block.
const blockedControls = () => ({
restartManagedRuntime: vi.fn(async () => "restarted" as const),
waitForRuntimeReady: vi.fn(async () => false),
stopRuntime: vi.fn(async () => "stopped" as const),
ensureRuntime: vi.fn(async () => {
throw new Error("Timed out waiting for MemHub Runtime");
}),
});
it("restart --json prints the block code, log file and issues URL", async () => {
const dataDir = await blockedHome();
const output = captureStdout();
let code: number;
try {
code = await run(
["restart", "--json"],
{ ONE_MEMORY_DATA_DIR: dataDir },
blockedControls(),
);
} finally {
output.restore();
await rm(dataDir, { recursive: true, force: true });
}
expect(code!).toBe(1);
const lines = output.value().trim().split("\n");
const document = JSON.parse(lines[lines.length - 1]!);
expect(document).toMatchObject({
command: "restart",
state: "blocked",
code: "sdk_migration_disk_space_insufficient",
log_file: path.join(dataDir, "logs", "sdk-migration.log"),
issues_url: "https://github.com/Tencent-Hunyuan/Hy-Memory/issues/new",
});
expect(output.value()).not.toContain("runtime_not_ready_within_budget");
});
it("restart on a frozen home retries the migration in the foreground first", async () => {
const dataDir = await mkdtemp(path.join(tmpdir(), "memhub-frozen-"));
await recordSdkMigrationBlock(
dataDir,
"sdk_migration_memory_unreadable",
"desktop",
{ frozen: { service_was_enabled: true }, data_restored: true },
);
const controls = {
...blockedControls(),
migrateSdkData: vi.fn(async () => ({
state: "deferred" as const,
code: "sdk_migration_memory_unreadable",
dataRestored: true,
})),
};
const output = captureStdout();
let code: number;
try {
code = await run(
["restart", "--json"],
{ ONE_MEMORY_DATA_DIR: dataDir },
controls,
);
} finally {
output.restore();
await rm(dataDir, { recursive: true, force: true });
}
expect(code!).toBe(1);
expect(controls.migrateSdkData).toHaveBeenCalledWith(
expect.objectContaining({ dataDir, caller: "cli-restart" }),
);
// Still frozen: the service is not started to retry on its own.
expect(controls.restartManagedRuntime).not.toHaveBeenCalled();
expect(controls.ensureRuntime).not.toHaveBeenCalled();
expect(JSON.parse(output.value())).toMatchObject({
command: "restart",
state: "blocked",
code: "sdk_migration_memory_unreadable",
data_restored: true,
});
});
it("restart on a frozen home restarts the Runtime once the retry succeeds", async () => {
const dataDir = await mkdtemp(path.join(tmpdir(), "memhub-frozen-"));
await recordSdkMigrationBlock(
dataDir,
"sdk_migration_memory_unreadable",
"desktop",
{ frozen: { service_was_enabled: false } },
);
const order: string[] = [];
const migrateSdkData = vi.fn(async () => {
order.push("migrate");
await clearSdkMigrationBlock(dataDir);
return { state: "cutover" as const };
});
const output = captureStreams();
let code: number;
try {
code = await run(
["restart", "--json"],
{ ONE_MEMORY_DATA_DIR: dataDir },
{
migrateSdkData,
waitForRuntimeReady: vi.fn(async () => true),
restartManagedRuntime: vi.fn(async () => {
order.push("restart");
return "restarted" as const;
}),
stopRuntime: vi.fn(async () => "stopped" as const),
ensureRuntime: vi.fn(async () => {
order.push("restart");
return { pid: 1 } as never;
}),
},
);
} finally {
output.restore();
await rm(dataDir, { recursive: true, force: true });
}
expect(code!).toBe(0);
expect(migrateSdkData).toHaveBeenCalledWith(
expect.objectContaining({ caller: "cli-restart" }),
);
expect(order).toEqual(["migrate", "restart"]);
});
it("restart in text mode names the log file and GitHub issues", async () => {
const dataDir = await blockedHome();
const output = captureStdout();
try {
expect(
await run(
["restart"],
{ ONE_MEMORY_DATA_DIR: dataDir },
blockedControls(),
),
).toBe(1);
} finally {
output.restore();
await rm(dataDir, { recursive: true, force: true });
}
expect(output.value()).toContain("sdk_migration_disk_space_insufficient");
expect(output.value()).toContain(
path.join(dataDir, "logs", "sdk-migration.log"),
);
expect(output.value()).toContain(
"https://github.com/Tencent-Hunyuan/Hy-Memory/issues/new",
);
});
it("ui --json reports the block instead of runtime_not_running", async () => {
const dataDir = await blockedHome();
const output = captureStdout();
let code: number;
try {
code = await run(["ui", "--json"], { ONE_MEMORY_DATA_DIR: dataDir });
} finally {
output.restore();
await rm(dataDir, { recursive: true, force: true });
}
expect(code!).not.toBe(0);
expect(JSON.parse(output.value())).toMatchObject({
command: "ui",
state: "blocked",
code: "sdk_migration_disk_space_insufficient",
log_file: path.join(dataDir, "logs", "sdk-migration.log"),
});
});
});
// The objective names `memhub host install <host>` (host as a positional). Both
// that and the `--host <id>` flag form must reach the same command; naming both
// at once is ambiguous and refused rather than guessed.
@@ -2004,7 +2408,7 @@ describe("install gates on Runtime readiness", () => {
});
}
it("fails and rolls back when the installed Runtime never becomes ready", async () => {
it("fails without rolling back when the installed Runtime never becomes ready", async () => {
vi.clearAllMocks();
stubInstallPlan();
const streams = captureStreams();
@@ -2033,10 +2437,10 @@ describe("install gates on Runtime readiness", () => {
}
expect(code!).toBe(1);
expect(streams.stderr()).toContain("install_runtime_not_ready");
expect(runtimeMocks.rollbackLocalInstall).toHaveBeenCalledOnce();
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
expect(
lifecycleMocks.rollbackPreparedServiceInstall,
).toHaveBeenCalledOnce();
).not.toHaveBeenCalled();
});
it("takes CLI ownership after a successful install over an App-owned Runtime", async () => {
@@ -2081,7 +2485,7 @@ describe("install gates on Runtime readiness", () => {
}
});
it("restores App ownership before relaunching the old Runtime after install failure", async () => {
it("never hands the home back to the App Runtime after the migration ran", async () => {
vi.clearAllMocks();
stubInstallPlan();
const root = await mkdtemp(path.join(tmpdir(), "memhub-cli-owner-"));
@@ -2091,14 +2495,6 @@ describe("install gates on Runtime readiness", () => {
appBundle: "/Applications/MemHub.app",
appExecutable: "/Applications/MemHub.app/Contents/MacOS/MemHub",
});
lifecycleMocks.rollbackPreparedServiceInstall.mockImplementationOnce(
async (_prepared: unknown, rollback: () => Promise<void>) => {
await rollback();
await expect(readRuntimeServiceOwner(dataDir)).resolves.toMatchObject({
owner: "app",
});
},
);
const streams = captureStreams();
try {
expect(
@@ -2121,15 +2517,114 @@ describe("install gates on Runtime readiness", () => {
{ waitForRuntimeReady: async () => false },
),
).toBe(1);
// The App Runtime may predate the SDK migration gate: CLI keeps it.
await expect(readRuntimeServiceOwner(dataDir)).resolves.toMatchObject({
owner: "app",
owner: "cli",
});
expect(
lifecycleMocks.rollbackPreparedServiceInstall,
).not.toHaveBeenCalled();
} finally {
streams.restore();
await rm(root, { recursive: true, force: true });
}
});
it("keeps the install, not serving, when its migration is deferred", async () => {
vi.clearAllMocks();
stubInstallPlan();
runtimeMocks.runSdkMigration.mockResolvedValueOnce({
state: "deferred",
code: "sdk_migration_memory_unreadable",
});
const waitForRuntimeReady = vi.fn(async () => true);
const streams = captureStreams();
let code: number;
try {
code = await run(
[
"install",
"--source",
"/tmp/dist",
"--home",
"/tmp/home",
"--install-root",
"/tmp/install",
"--bin-dir",
"/tmp/bin",
"--data-dir",
"/tmp/data",
"--yes",
],
{},
{ waitForRuntimeReady },
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
const failure = JSON.parse(streams.stderr());
expect(failure).toMatchObject({
command: "install",
code: "sdk_migration_memory_unreadable",
log_file: "/tmp/data/logs/sdk-migration.log",
issues_url: "https://github.com/Tencent-Hunyuan/Hy-Memory/issues/new",
});
expect(failure.message).toContain("/tmp/data/logs/sdk-migration.log");
expect(runtimeMocks.runSdkMigration).toHaveBeenCalledWith(
expect.objectContaining({ caller: "cli-install" }),
);
// The new (gated) service owns the data dir; the previous registration is
// never re-bootstrapped and nothing is rolled back.
expect(lifecycleMocks.applyPreparedServiceInstall).toHaveBeenCalledWith(
expect.objectContaining({ restorePreviousOnFailure: false }),
);
expect(runtimeMocks.rollbackLocalInstall).not.toHaveBeenCalled();
expect(
lifecycleMocks.rollbackPreparedServiceInstall,
).not.toHaveBeenCalled();
expect(waitForRuntimeReady).not.toHaveBeenCalled();
});
it("still rolls back an install that failed before its migration ran", async () => {
vi.clearAllMocks();
stubInstallPlan();
runtimeMocks.installCliLink.mockRejectedValueOnce(
new Error("cli_link_failed"),
);
const streams = captureStreams();
let code: number;
try {
code = await run(
[
"install",
"--source",
"/tmp/dist",
"--home",
"/tmp/home",
"--install-root",
"/tmp/install",
"--bin-dir",
"/tmp/bin",
"--data-dir",
"/tmp/data",
"--yes",
],
{},
{ waitForRuntimeReady: async () => true },
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
expect(JSON.parse(streams.stderr())).toMatchObject({
command: "install",
code: "cli_link_failed",
});
expect(runtimeMocks.runSdkMigration).not.toHaveBeenCalled();
expect(runtimeMocks.rollbackLocalInstall).toHaveBeenCalledOnce();
});
it("succeeds when the installed Runtime becomes ready", async () => {
vi.clearAllMocks();
stubInstallPlan();
+482 -252
View File
@@ -32,6 +32,8 @@ import {
processIsAlive,
type ResolvedRuntimeRelease,
RuntimeDiscoveryError,
readInstalledRuntimeSource,
readSdkMigrationBlock,
removeRuntimeServiceOwner,
resolveRuntimeCliConfig,
resolveRuntimeRelease,
@@ -39,7 +41,11 @@ import {
rollbackCliLink,
rollbackLocalInstall,
runRuntimeCli,
runSdkMigration,
runtimeInstallLayout,
type SdkMigrationResult,
type SdkMigrationSource,
sdkMigrationLogPath,
stopRuntimeProcess,
} from "@one-memory/runtime-control";
import axios from "axios";
@@ -175,21 +181,10 @@ import {
removeManagedCliLink,
rollbackManagedCliLink,
} from "./managed-cli-link.js";
import {
executeMemoryMutation,
inspectMemoryPlan,
type MemoryFlowDocument,
memoryFlowExitCode,
memoryPlanAcceptsContent,
parseMemoryContent,
planExistingMemoryMutation,
planMemoryAdd,
readMemoryMutationReceipt,
renderMemoryFlowText,
} from "./memory-flows.js";
import {
isMemoryId,
type MemoryQuery,
parseMemoryContent,
readMemoryDetail,
readMemoryPage,
renderMemoryDetailText,
@@ -354,6 +349,9 @@ export interface RunControls {
/** Wait for a restarted Runtime to become healthy after `upgrade`.
* Injectable so the upgrade tests do not spend real seconds polling. */
waitForRuntimeReady?: (dataDir: string) => Promise<boolean>;
/** Run the SDK data migration before install/upgrade starts the new
* Runtime. Injectable so the tests do not spawn a real Core. */
migrateSdkData?: typeof migrateSdkDataBeforeStart;
restartManagedRuntime?: typeof restartManagedService;
stopRuntime?: typeof stopRuntimeProcess;
ensureRuntime?: typeof ensureRuntimeProcess;
@@ -1048,15 +1046,21 @@ async function runCommand(
entry: "cli",
}));
const target = path.join(downloads, diagnosticZipFilename(build));
const environmentCommit = environment.ONE_MEMORY_COMMIT?.trim();
const embeddedCommit =
typeof __ONE_MEMORY_COMMIT__ === "string"
? __ONE_MEMORY_COMMIT__.trim()
: "";
const commit =
environmentCommit ||
(embeddedCommit && embeddedCommit !== "unknown"
? embeddedCommit
: undefined);
const receipt = await exportDiagnosticZip({
dataDir: config.dataDir,
target,
productVersion: build,
commit:
environment.ONE_MEMORY_COMMIT?.trim() ||
(typeof __ONE_MEMORY_COMMIT__ === "string"
? __ONE_MEMORY_COMMIT__
: "unknown"),
...(commit === undefined ? {} : { commit }),
coreSha256: await sha256File(config.coreCommand).catch(() => undefined),
home: os.homedir(),
crashReportsDir: path.join(
@@ -1164,6 +1168,41 @@ async function runCommand(
// declared command, and lifecycle is exactly where scripts need it.
if (arguments_.length !== (asJson ? 2 : 1)) return usage();
const config = resolveRuntimeCliConfig(environment, process.argv[1]);
// A failed upgrade migration froze this home: its service exits without
// migrating. `start`/`restart` is an explicit retry, run here in the
// foreground; success lifts the freeze and turns the service back on.
const frozen = await readSdkMigrationBlock(config.dataDir).catch(
() => null,
);
if (frozen?.frozen !== undefined) {
const migration = await (
controls.migrateSdkData ?? migrateSdkDataBeforeStart
)({
coreCommand: config.coreCommand,
dataDir: config.dataDir,
runtimeBuild:
config.buildIdentity?.buildNumber ??
config.buildIdentity?.version ??
"unknown",
asJson,
caller: command === "restart" ? "cli-restart" : "cli-start",
});
if (migration.state === "deferred") {
const failure = sdkMigrationFailure(
command,
migration.code ?? "sdk_migration_incomplete",
config.dataDir,
"blocked",
migration.dataRestored === true,
);
process.stdout.write(
asJson
? `${JSON.stringify({ ...failure, state: "blocked" })}\n`
: `${failure.message}\n`,
);
return EXIT_FAILED;
}
}
const enableSnapshot = await setManagedServiceEnabled(config.dataDir, true);
const restoreEnableState = () =>
restoreManagedServiceEnabled(config.dataDir, enableSnapshot);
@@ -1186,6 +1225,28 @@ async function runCommand(
: state === "already_running"
? "MemHub Runtime 已在运行。\n"
: "Runtime 服务已注册但未就绪;请运行 memhub logs 查看原因。\n";
// A Runtime that did not come up because its service refuses to serve an
// unmigrated home reports that cause, the migration log and where to send
// it, instead of a generic budget timeout.
const migrationBlocked = async (): Promise<string | null> => {
const blocked = await runtimeMigrationBlock(config.dataDir, command);
if (blocked === null) return null;
return asJson
? `${JSON.stringify({
schema_version: 1,
product: PRODUCT,
command,
state: "blocked",
code: blocked.code,
message: blocked.message,
log_file: blocked.log_file,
issues_url: blocked.issues_url,
})}\n`
: `${blocked.message}\n`;
};
const notReady = async () =>
(await migrationBlocked()) ??
lifecycle("not-ready", "runtime_not_ready_within_budget");
// A launchd-managed Runtime must be driven through launchctl. Spawning a
// second process while the service holds the data directory is exactly what
// produced "Runtime exited before becoming ready".
@@ -1230,16 +1291,12 @@ async function runCommand(
process.stdout.write(lifecycle("restarted", "runtime_verified"));
return EXIT_OK;
}
process.stdout.write(
lifecycle("not-ready", "runtime_not_ready_within_budget"),
);
process.stdout.write(await notReady());
await restoreEnableState();
return EXIT_FAILED;
}
if (restarted === "not-ready") {
process.stdout.write(
lifecycle("not-ready", "runtime_not_ready_within_budget"),
);
process.stdout.write(await notReady());
await restoreEnableState();
return EXIT_FAILED;
}
@@ -1295,16 +1352,12 @@ async function runCommand(
process.stdout.write(lifecycle("started", "runtime_verified"));
return EXIT_OK;
}
process.stdout.write(
lifecycle("not-ready", "runtime_not_ready_within_budget"),
);
process.stdout.write(await notReady());
await restoreEnableState();
return EXIT_FAILED;
}
if (startedManaged === "not-ready") {
process.stdout.write(
lifecycle("not-ready", "runtime_not_ready_within_budget"),
);
process.stdout.write(await notReady());
await restoreEnableState();
return EXIT_FAILED;
}
@@ -1333,6 +1386,11 @@ async function runCommand(
// A spawned Runtime that exits before ready is a failure of this command,
// not a crash: report it as a structured document (JSON) or a plain
// message, with a non-zero exit, instead of letting it escape.
const blocked = await migrationBlocked();
if (blocked !== null) {
process.stdout.write(blocked);
return EXIT_FAILED;
}
const message = error instanceof Error ? error.message : String(error);
process.stdout.write(
asJson
@@ -1529,6 +1587,24 @@ async function runCommand(
return uiExitCode(result);
};
if (attachment === null) {
const migration = await runtimeMigrationBlock(config.dataDir, "ui");
if (migration !== null) {
emit(
asJson
? `${JSON.stringify({
schema_version: 1,
product: PRODUCT,
command: "ui",
state: "blocked",
code: migration.code,
message: migration.message,
log_file: migration.log_file,
issues_url: migration.issues_url,
})}\n`
: `ui 未打开:${migration.code}\n${migration.message}\n`,
);
return uiExitCode({ state: "blocked", code: migration.code });
}
return emitUi(
{ state: "blocked", code: "runtime_not_running" },
`先运行 ${PRODUCT} start 启动 Runtime。`,
@@ -1633,7 +1709,7 @@ async function runCommand(
});
if (runtime.state !== "running") {
const document: ContextSearchDocument = {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
state: "unavailable",
code: runtime.code,
@@ -1657,7 +1733,7 @@ async function runCommand(
timeout_ms: parsed.timeoutMs,
})
: ({
schema_version: 1,
schema_version: 2,
product: PRODUCT,
state: "unavailable",
code: attached.failure ?? "runtime_attach_failed",
@@ -2423,20 +2499,20 @@ async function runCommand(
};
}
if (blocked || !reader) {
if (
parsed.kind === "update" ||
parsed.kind === "delete" ||
parsed.kind === "plan-add" ||
parsed.kind === "plan-existing" ||
parsed.kind === "apply"
) {
const document: MemoryFlowDocument = {
schema_version: 1,
if (parsed.kind === "update" || parsed.kind === "delete") {
const document = {
schema_version: 2,
product: PRODUCT,
state: "unknown",
operation: parsed.kind,
memory_id: parsed.memoryId,
code: blocked?.code ?? "runtime_attach_failed",
};
emitMemoryFlow(document, asJson);
emit(
asJson
? `${JSON.stringify(document)}\n`
: `记忆操作未完成:${document.state}(${document.code})\n`,
);
return runtime.state === "stopped" ? 3 : 1;
}
const document =
@@ -2530,73 +2606,7 @@ async function runCommand(
);
return outcome.state === "committed" ? EXIT_OK : EXIT_FAILED;
}
if (parsed.kind === "plan-add") {
const content = await readMemoryContentFile(parsed.contentFile);
if (content === null) return usage();
const document = planMemoryAdd({
content,
memoryKind: parsed.memoryKind,
scopes: parsed.scopes,
evidenceIds: parsed.evidenceIds,
});
emitMemoryFlow(document, asJson);
return memoryFlowExitCode(document);
}
if (parsed.kind === "plan-existing") {
let content: string | undefined;
if (parsed.contentFile) {
const read = await readMemoryContentFile(parsed.contentFile);
if (read === null) return usage();
content = read;
}
const document = await planExistingMemoryMutation(reader, {
operation: parsed.operation,
memoryId: parsed.memoryId,
...(content === undefined ? {} : { content }),
});
emitMemoryFlow(document, asJson);
return memoryFlowExitCode(document);
}
if (!operations) {
const document: MemoryFlowDocument = {
schema_version: 1,
product: PRODUCT,
state: "unknown",
code: "runtime_attach_failed",
};
emitMemoryFlow(document, asJson);
return EXIT_FAILED;
}
const file = await readBoundedJson(parsed.planFile);
if (file === null) return usage();
const plan = unwrapPlan(file);
const identity = inspectMemoryPlan(plan);
if (!identity) return usage();
let content: string | undefined;
if (parsed.contentFile) {
const read = await readMemoryContentFile(parsed.contentFile);
if (read === null) return usage();
content = read;
}
if (
(identity.needsContent && content === undefined) ||
(!identity.needsContent && content !== undefined) ||
!memoryPlanAcceptsContent(plan, content)
) {
return usage();
}
const attempt = await claimPlanAttempt(parsed.planFile, {
schema_version: 1,
operation_id: identity.operationId,
plan_digest: identity.digest,
});
if (attempt === "invalid") return usage();
const document =
attempt === "prior"
? await readMemoryMutationReceipt(reader, plan)
: await executeMemoryMutation(plan, content, operations, reader);
emitMemoryFlow(document, asJson);
return memoryFlowExitCode(document);
return usage();
}
if (named === "calls" || named === "import-file") {
const parsed = parseUiWriteCommand(named, arguments_.slice(1));
@@ -2806,10 +2816,15 @@ async function runCommand(
if (!ready) throw new Error("runtime_service_activation_unverified");
return plan;
})
.catch((error: unknown) => {
.catch(async (error: unknown) => {
const migration =
error instanceof Error &&
error.message === "runtime_service_activation_unverified"
? await runtimeMigrationBlock(layout.dataDir, "service-install")
: null;
process.stderr.write(
`${JSON.stringify(
serviceErrorDocument("service-install", error),
migration ?? serviceErrorDocument("service-install", error),
null,
2,
)}\n`,
@@ -3143,6 +3158,11 @@ async function runCommand(
null;
let installReceiptWritten = false;
let serviceApplied = false;
let migrated = false;
// `null` until the migration is handed this home; "failed" if it threw.
let migrationState: SdkMigrationResult["state"] | "failed" | null = null;
let deferredCode: string | null = null;
let deferredRestored = false;
let previousOwner:
| Awaited<ReturnType<typeof claimCliServiceOwnership>>
| undefined;
@@ -3153,7 +3173,39 @@ async function runCommand(
}
shellPathReceipt = await applyShellPath(shellPath);
previousOwner = await claimCliServiceOwnership(layout.dataDir);
await applyPreparedServiceInstall(preparedService);
// `install` over an existing home (reinstall, App-to-CLI takeover) must
// migrate it too, before the new Runtime starts serving it.
migrationState = "failed";
const migration = await (
controls.migrateSdkData ?? migrateSdkDataBeforeStart
)({
coreCommand: layout.coreCommand,
dataDir: layout.dataDir,
runtimeBuild: String(plan.build_number ?? plan.version),
asJson: true,
caller: "cli-install",
sourceRuntime: await readInstalledRuntimeSource(
plan.install_root,
receipt.previous_current_target,
),
});
migrationState = migration.state;
// A deferred migration keeps this install: the new Runtime service is
// registered and its migration gate keeps Core down until the home is
// migrated. Rolling back would hand the home to the previous (App or CLI)
// Runtime, which has no gate and would serve the legacy data.
if (migration.state === "deferred") {
deferredCode = migration.code ?? "sdk_migration_incomplete";
deferredRestored = migration.dataRestored === true;
}
migrated =
migration.state === "cutover" || migration.state === "complete";
// From here the previous LaunchAgent is never re-bootstrapped: it may
// be an older Runtime with no migration gate.
await applyPreparedServiceInstall({
...preparedService,
restorePreviousOnFailure: false,
});
serviceApplied = true;
// The receipt exists so a later command can find the data dir without
// flags. An isolated install named its data dir explicitly, and writing
@@ -3169,6 +3221,7 @@ async function runCommand(
});
installReceiptWritten = true;
}
if (deferredCode !== null) throw new SdkMigrationDeferred(deferredCode);
// Registering the service is not the same as a working install: on a host
// where Core cold-start stalls, launchd reports a pid while status is
// still `runtime_service_running`. Gate on real readiness (same predicate
@@ -3204,6 +3257,47 @@ async function runCommand(
);
}
};
// Once the migration has been handed this home nothing rolls back: the
// previous version (App or CLI) may be an older Runtime without the
// migration gate, and restoring it could serve a home that is not on SDK
// authority. This install stays; its service keeps Core down until the
// home is migrated. Only a failure before the migration ran (the
// previous Runtime was never stopped and the service never touched) is
// rolled back below.
if (migrationState !== null) {
if (!serviceApplied && !migrated && deferredCode === null) {
// The migration threw. Hand the data dir to this version's gated
// service anyway so that the previous Runtime does not come back.
await applyPreparedServiceInstall({
...preparedService,
restorePreviousOnFailure: false,
}).catch(() => undefined);
}
process.stderr.write(
`${JSON.stringify(
deferredCode !== null
? sdkMigrationFailure(
"install",
deferredCode,
layout.dataDir,
"upgrade",
deferredRestored,
)
: migrated
? notReadyAfterMigration("install")
: migrationState === "failed"
? sdkMigrationFailure(
"install",
sdkMigrationErrorCode(error),
layout.dataDir,
)
: notReadyAfterUpgrade("install", error),
null,
2,
)}\n`,
);
return EXIT_FAILED;
}
if (serviceApplied) {
await rollbackPreparedServiceInstall(
preparedService,
@@ -3575,9 +3669,18 @@ async function runCommand(
let previousOwner:
| Awaited<ReturnType<typeof claimCliServiceOwnership>>
| undefined;
const rollbackUpgrade = async (
dataDir: string | undefined,
): Promise<void> => {
let migrated = false;
// `null` until the migration is handed this home; "failed" if it threw.
let migrationState: SdkMigrationResult["state"] | "failed" | null = null;
/**
* Restore the previous version's files and ownership. Only used for a
* failure before the migration ran: the previous Runtime was never stopped
* and the service never restarted, so nothing new starts serving. After
* the migration has been handed the home the upgrade never rolls back:
* the previous Runtime may predate the migration gate and would serve a
* home that is not on SDK authority.
*/
const rollbackUpgrade = async (): Promise<void> => {
for (const receipt of memhubReceipts.reverse()) {
await rollbackManagedCliLink(receipt);
}
@@ -3587,51 +3690,81 @@ async function runCommand(
() => undefined,
);
}
await restartUserService(
runtimeInstallLayout({
platform: plan.platform,
home: parsed.home,
installRoot: canonicalManagedServiceRoot(
plan.platform,
plan.current_link,
),
dataDir,
...hourlyIsolationFromEnvironment(environment),
}),
serviceOptions,
).catch(() => undefined);
};
/**
* After a migration that did not finish, make sure the process holding the
* data dir is the new version's service (gated: it keeps Core down while
* the migration is deferred) and not the previous Runtime. The migration
* fence already stopped it unless the status probe itself failed.
*/
const keepUpgradedRuntimeOffLegacyHome = async (): Promise<void> => {
try {
await restartUserService(layout, serviceOptions);
} catch {
await (controls.stopRuntime ?? stopRuntimeProcess)({
dataDir: parsed.dataDir ?? layout.dataDir,
probeIdentity,
}).catch(() => undefined);
}
};
try {
for (const link of companionLinks) {
memhubReceipts.push(await installManagedCliLink(link));
}
previousOwner = await claimCliServiceOwnership(layout.dataDir);
migrationState = "failed";
const migration = await (
controls.migrateSdkData ?? migrateSdkDataBeforeStart
)({
coreCommand: layout.coreCommand,
dataDir: parsed.dataDir ?? layout.dataDir,
runtimeBuild: String(plan.build_number ?? plan.version),
asJson,
caller: "cli-upgrade",
sourceRuntime: await readInstalledRuntimeSource(
plan.install_root,
receipt.previous_current_target,
),
});
migrationState = migration.state;
if (migration.state === "deferred") {
// Stay on the new version. Its Runtime service refuses to spawn Core
// while the migration is deferred, so nothing serves the legacy home;
// rolling back would restart the old Runtime, which has no such gate.
await keepUpgradedRuntimeOffLegacyHome();
process.stderr.write(
`${JSON.stringify(sdkMigrationFailure("upgrade", migration.code ?? "sdk_migration_incomplete", parsed.dataDir ?? layout.dataDir, "upgrade", migration.dataRestored === true), null, 2)}\n`,
);
return EXIT_FAILED;
}
// From here a failure keeps the new version: an older Runtime either
// refuses a migrated home or, predating the migration gate, would serve
// one that is not on SDK authority.
migrated =
migration.state === "cutover" || migration.state === "complete";
await restartUserService(layout, serviceOptions);
// A restart that returns without error only means launchd/systemd
// accepted the command. O-22: the first `upgrade` implementation
// reported success even when the new artifact crash-looped, leaving
// `current` on a version that never came up. Verify the restarted
// Runtime actually becomes healthy; if it does not, roll back to the
// previously working version.
// Runtime actually becomes healthy; if it does not, fail loudly. It is
// not rolled back (product rule: SDK authority or not serving at all).
const dataDir = parsed.dataDir ?? layout.dataDir;
const ready = await (controls.waitForRuntimeReady ?? waitForRuntimeReady)(
dataDir,
);
if (!ready) {
await rollbackUpgrade(parsed.dataDir);
if (!ready && migrated) {
process.stderr.write(
`${JSON.stringify(
{
schema_version: 1,
product: PRODUCT,
command: "upgrade",
status: "error",
code: "upgrade_health_check_failed",
message: "新版本重启后未在限定时间内就绪,已回滚到上一版本。",
},
null,
2,
)}\n`,
`${JSON.stringify(notReadyAfterMigration("upgrade"), null, 2)}\n`,
);
return EXIT_FAILED;
}
if (!ready) {
// Not rolled back: see `rollbackUpgrade`. If the service is blocked on
// the migration (it re-runs it before spawning Core), say so.
const blocked = await runtimeMigrationBlock(dataDir, "upgrade");
process.stderr.write(
`${JSON.stringify(blocked ?? notReadyAfterUpgrade("upgrade"), null, 2)}\n`,
);
return EXIT_FAILED;
}
@@ -3644,9 +3777,32 @@ async function runCommand(
});
}
} catch (error) {
await rollbackUpgrade(parsed.dataDir);
if (migrationState === null) {
await rollbackUpgrade();
process.stderr.write(
`${JSON.stringify(serviceErrorDocument("upgrade", error), null, 2)}\n`,
);
return EXIT_FAILED;
}
if (migrated) {
// An older Runtime refuses a migrated home: keep the new version.
process.stderr.write(
`${JSON.stringify(notReadyAfterMigration("upgrade"), null, 2)}\n`,
);
return EXIT_FAILED;
}
if (migrationState === "failed") {
// The migration itself threw: its outcome is unknown, so treat the
// home as not migrated and keep the old Runtime off it.
await keepUpgradedRuntimeOffLegacyHome();
process.stderr.write(
`${JSON.stringify(sdkMigrationFailure("upgrade", sdkMigrationErrorCode(error), parsed.dataDir ?? layout.dataDir), null, 2)}\n`,
);
return EXIT_FAILED;
}
// `not_required`: the restart itself failed. Stay on the new version.
process.stderr.write(
`${JSON.stringify(serviceErrorDocument("upgrade", error), null, 2)}\n`,
`${JSON.stringify(notReadyAfterUpgrade("upgrade", error), null, 2)}\n`,
);
return EXIT_FAILED;
}
@@ -4014,11 +4170,173 @@ export function runtimeIsReady(
);
}
const SDK_MIGRATION_STEPS: Record<string, string> = {
bundle: "正在备份记忆数据,数据较多时需要几分钟……",
prepare: "正在整理待处理的写入任务……",
cutover: "正在切换记忆数据源……",
cleanup: "正在清理旧数据表……",
};
/**
* Migrate a home written by an older release before the new Runtime serves it.
*
* Runs the new Runtime's own Core with the Runtime stopped. Progress goes to
* stderr so stdout stays one result document.
*/
async function migrateSdkDataBeforeStart(options: {
coreCommand: string;
dataDir: string;
runtimeBuild: string;
asJson: boolean;
caller?: "cli-install" | "cli-upgrade" | "cli-start" | "cli-restart";
sourceRuntime?: SdkMigrationSource | null;
}): Promise<SdkMigrationResult> {
return runSdkMigration({
dataDir: options.dataDir,
core: { command: options.coreCommand },
runtimeBuild: options.runtimeBuild,
...(options.caller === undefined ? {} : { caller: options.caller }),
...(options.sourceRuntime === undefined
? {}
: { sourceRuntime: options.sourceRuntime }),
onProgress: (step) => {
process.stderr.write(
options.asJson
? `${JSON.stringify({ schema_version: 1, product: PRODUCT, event: "sdk_migration.progress", step })}\n`
: `${SDK_MIGRATION_STEPS[step] ?? step}\n`,
);
},
});
}
class SdkMigrationDeferred extends Error {
constructor(readonly code: string) {
super(code);
}
}
function notReadyAfterMigration(command: "install" | "upgrade") {
return {
schema_version: 1,
product: PRODUCT,
command,
status: "error",
code: "runtime_not_ready_after_sdk_migration",
message:
"记忆数据已迁移到新版本,但新版本 Runtime 没有在限定时间内就绪。旧版本无法打开已迁移的数据,所以没有回滚。请运行 memhub restart,或 memhub doctor 查看原因。",
};
}
/**
* The new version did not come up and was not rolled back: the previous one
* may predate the SDK migration gate. `error` carries the underlying code
* (a failed service restart), never free text from a child process.
*/
function notReadyAfterUpgrade(command: "install" | "upgrade", error?: unknown) {
return {
schema_version: 1,
product: PRODUCT,
command,
status: "error",
code:
error !== undefined
? serviceErrorDocument(command, error).code
: command === "upgrade"
? "upgrade_health_check_failed"
: "install_runtime_not_ready",
rolled_back: false,
message:
"新版本 Runtime 没有启动或没有在限定时间内就绪。为避免旧版本打开记忆数据,已保留新版本,没有回滚。请运行 memhub restart 重试,或运行 memhub doctor 查看原因;如果仍然失败,请在 GitHub 提交反馈:" +
MIGRATION_ISSUES_URL,
issues_url: MIGRATION_ISSUES_URL,
};
}
const SDK_MIGRATION_FAILURES: Record<string, string> = {
sdk_migration_disk_space_insufficient:
"磁盘空间不足,记忆数据迁移没有完成。请腾出与记忆数据同等大小的空间后运行 memhub restart 重试。",
sdk_migration_memory_unreadable:
"核对发现新版本还读不到你的全部记忆,没有切换数据源。",
sdk_migration_model_profile_required:
"迁移需要先有可用的模型配置,用来核对新版本能否读回记忆。请先用 memhub models setup 完成配置,再运行 memhub restart 重试。",
};
const MIGRATION_ISSUES_URL =
"https://github.com/Tencent-Hunyuan/Hy-Memory/issues/new";
/**
* Why MemHub is not serving: the home has not reached SDK authority.
*
* Product rule: after an upgrade the home is on SDK authority or MemHub does
* not serve it at all. Nothing here rolls back to an older Runtime (it has no
* migration gate and would serve the legacy home); the user gets the code, the
* migration log and where to report it.
*/
function sdkMigrationFailure(
command: string,
code: string,
dataDir: string,
context: "upgrade" | "blocked" = "upgrade",
dataRestored = false,
) {
const logFile = sdkMigrationLogPath(dataDir);
// Core put the home back exactly as it was before the upgrade, so the next
// (fixed) version migrates from the original data.
const restored = dataRestored ? "记忆数据已恢复到升级前的状态。" : "";
const reason =
SDK_MIGRATION_FAILURES[code] ??
"记忆数据迁移没有完成。可以运行 memhub restart 重试。";
const lead =
context === "upgrade"
? `${reason}已保留新版本;在迁移完成前 MemHub 不会启动,也不会用旧版本打开未迁移的数据。`
: `MemHub 没有启动:记忆数据还没有迁移到新版本(${code})。${reason}`;
return {
schema_version: 1,
product: PRODUCT,
command,
status: "error",
code,
message: `${lead}原有记忆数据不会丢失。${restored}迁移日志:${logFile}。请在 GitHub 提交反馈并附上这个日志文件:${MIGRATION_ISSUES_URL}`,
log_file: logFile,
issues_url: MIGRATION_ISSUES_URL,
...(dataRestored ? { data_restored: true } : {}),
};
}
/** A stable code from a thrown migration error, never its free text. */
function sdkMigrationErrorCode(error: unknown): string {
const message = error instanceof Error ? error.message : "";
return /^[a-z][a-z0-9_]{0,95}$/.test(message)
? message
: "sdk_migration_incomplete";
}
/**
* The Runtime service refuses to spawn Core while this home's migration is
* deferred and leaves a block note; report that instead of a generic
* "not ready within budget".
*/
async function runtimeMigrationBlock(
dataDir: string,
command: string,
): Promise<ReturnType<typeof sdkMigrationFailure> | null> {
const block = await readSdkMigrationBlock(dataDir).catch(() => null);
if (block === null) return null;
return sdkMigrationFailure(
command,
block.code,
dataDir,
"blocked",
block.data_restored === true,
);
}
async function waitForRuntimeReady(
dataDir: string,
timeoutMs = 45_000,
): Promise<boolean> {
const deadline = Date.now() + timeoutMs;
const started = Date.now();
const deadline = started + timeoutMs;
while (Date.now() < deadline) {
const runtime = await inspectRuntimeStatus({
dataDir,
@@ -4026,6 +4344,17 @@ async function waitForRuntimeReady(
requireServiceReceipt: true,
}).catch(() => null);
if (runtimeIsReady(runtime)) return true;
// The service deferred its migration during this wait and will not spawn
// Core: stop waiting. An older note may predate a retry still running, so
// only a note written after the wait began ends it early.
// A frozen home: its service exits without migrating, so nothing will
// come up until a foreground migration lifts the freeze.
const blocked = await readSdkMigrationBlock(dataDir).catch(() => null);
if (
blocked !== null &&
(blocked.frozen !== undefined || Date.parse(blocked.at) >= started)
)
return false;
await new Promise((resolve) => setTimeout(resolve, 500));
}
return false;
@@ -4730,10 +5059,10 @@ async function runMaintenanceRestore(options: {
const instanceId = randomUUID();
const child = spawn(options.coreCommand, ["serve", "--maintenance"], {
env: {
// Do not pass the caller's Runtime tuning variables (for example
// retrieval-index flags) into a maintenance process: a restore must
// start the product's minimal maintenance configuration, not inherit
// an unrelated experiment from the shell.
// Do not pass the caller's Runtime tuning variables into a
// maintenance process: a restore must start the product's minimal
// maintenance configuration, not inherit an unrelated experiment from
// the shell.
PATH: process.env.PATH,
HOME: process.env.HOME,
LANG: process.env.LANG,
@@ -5945,21 +6274,7 @@ type MemoryCommand =
contentFile: string;
confirmed: boolean;
}
| { kind: "delete"; memoryId: string; confirmed: boolean }
| {
kind: "plan-add";
contentFile: string;
memoryKind: string;
scopes: string[];
evidenceIds: string[];
}
| {
kind: "plan-existing";
operation: "correct" | "delete";
memoryId: string;
contentFile?: string;
}
| { kind: "apply"; planFile: string; contentFile?: string };
| { kind: "delete"; memoryId: string; confirmed: boolean };
async function parseMemoryCommand(
rest: readonly string[],
@@ -5999,31 +6314,6 @@ async function parseMemoryCommand(
? { kind: "delete", memoryId, confirmed }
: null;
}
if (action === "plan" || action === "apply") return null;
if (action === "plan") return parseMemoryPlanCommand(tokens.slice(1));
if (action === "apply") {
let planFile: string | undefined;
let contentFile: string | undefined;
let confirmed = false;
for (let index = 1; index < tokens.length; index += 1) {
const flag = tokens[index];
if (flag === "--confirm") {
if (confirmed) return null;
confirmed = true;
continue;
}
const value = tokens[index + 1];
if (value === undefined || value.startsWith("--")) return null;
if (flag === "--plan-file" && planFile === undefined) planFile = value;
else if (flag === "--content-file" && contentFile === undefined)
contentFile = value;
else return null;
index += 1;
}
return planFile && confirmed
? { kind: "apply", planFile, ...(contentFile ? { contentFile } : {}) }
: null;
}
if (action !== "list") return null;
let status: string | undefined;
let offset = 0;
@@ -6050,73 +6340,11 @@ async function parseMemoryCommand(
return { kind: "list", query: { status, offset, limit } };
}
function parseMemoryPlanCommand(
tokens: readonly string[],
): MemoryCommand | null {
const operation = tokens[0];
const add = operation === "add";
if (!add && !["correct", "delete"].includes(operation)) {
return null;
}
const memoryId = add ? undefined : tokens[1];
if (!add && (!memoryId || !isMemoryId(memoryId))) return null;
let contentFile: string | undefined;
let memoryKind = "chat";
let kindSet = false;
const scopes: string[] = [];
const evidenceIds: string[] = [];
for (let index = add ? 1 : 2; index < tokens.length; index += 1) {
const flag = tokens[index];
const value = tokens[index + 1];
if (value === undefined || value.startsWith("--")) return null;
if (flag === "--content-file" && contentFile === undefined)
contentFile = value;
else if (flag === "--kind" && add && !kindSet) {
memoryKind = value;
kindSet = true;
} else if (flag === "--scope" && add && !scopes.includes(value))
scopes.push(value);
else if (flag === "--evidence" && add && !evidenceIds.includes(value))
evidenceIds.push(value);
else return null;
index += 1;
}
if (add) {
return contentFile && scopes.length > 0 && evidenceIds.length > 0
? {
kind: "plan-add",
contentFile,
memoryKind,
scopes,
evidenceIds,
}
: null;
}
if (
(operation === "correct" && !contentFile) ||
(operation === "delete" && (contentFile !== undefined || scopes.length > 0))
) {
return null;
}
return {
kind: "plan-existing",
operation: operation as "correct" | "delete",
memoryId: memoryId as string,
...(contentFile ? { contentFile } : {}),
};
}
async function readMemoryContentFile(file: string): Promise<string | null> {
const bytes = await readBoundedFile(file, 100_000 * 4 + 2);
return bytes ? parseMemoryContent(bytes) : null;
}
function emitMemoryFlow(document: MemoryFlowDocument, asJson: boolean): void {
emit(
asJson ? `${JSON.stringify(document)}\n` : renderMemoryFlowText(document),
);
}
function emitModelWriteDocument(
document: ModelWriteDocument,
asJson: boolean,
@@ -6289,6 +6517,8 @@ function serviceErrorDocument(
"服务文件不属于当前 MemHub 安装;请保留文件并确认其来源后再处理。",
runtime_service_managed_upgrade_failed:
"新服务启动失败,旧 MemHub 服务已恢复;检查新安装日志后重试。",
runtime_service_replacement_failed:
"新服务启动失败;记忆数据尚未处于新版本可用的状态,所以没有恢复旧服务。请运行 memhub doctor 检查服务状态。",
runtime_service_managed_upgrade_rollback_failed:
"新服务启动失败且旧服务恢复失败;请运行 memhub doctor 检查服务状态。",
runtime_service_file_modified:
+2 -2
View File
@@ -387,7 +387,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
flag: "--content-file <path>",
summary:
"从有界普通文件读取新增/纠正正文;计划只保存摘要,执行时再次校验。",
"从有界普通文件读取 update 的新正文;预览只显示摘要,--confirm 时才写入。",
},
DIRECT_CONFIRM_OPTION,
JSON_OPTION,
@@ -485,7 +485,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "recall",
summary:
"从当前授权的原文与治理记忆统一检索上下文;本地只读,不调用模型、不触发补索引。",
"从当前授权的原文与已沉淀记忆统一检索上下文;本地只读,不调用模型、不触发补索引。",
usage:
"memhub recall <query> [--kind <raw|memory>] [--source <id>] [--limit <n>] [--character-budget <n>] [--timeout-ms <n>] [--json]\n memhub recall --query-file <path> [...]\n memhub recall --stdin [...]",
options: [
+70 -135
View File
@@ -43,61 +43,35 @@ const ATTACHED = {
};
const RESPONSE = {
schema_version: 1,
schema_version: 2,
request_id: "context-request-1",
state: "grounded",
paths: {
raw: {
status: "ok",
reason: null,
elapsed_ms: 2,
returned_count: 1,
truncated: false,
truncated_by: [],
},
memory: {
status: "ok",
reason: null,
elapsed_ms: 3,
returned_count: 1,
truncated: false,
truncated_by: [],
},
},
items: [
{
kind: "raw",
source_id: "codex",
conversation_id: "conversation-1",
anchor_evidence_id: "evidence-1",
messages: [
{
evidence_id: "evidence-1",
revision: 1,
role: "user",
written_by: "user",
created_at: "2026-09-19T00:00:00Z",
text: "Synthetic raw citation.",
truncated: false,
},
],
raw_id: "raw-1",
content: "Synthetic raw citation.",
source: "codex",
session_id: "session-1",
memory_at: "2026-09-19T00:00:00Z",
score: 0.8,
truncated: false,
},
{
kind: "memory",
memory_id: "memory-1",
revision: 2,
content: "Synthetic governed memory.",
memory_kind: "working",
content: "Synthetic SDK memory.",
layer: "l2_fact",
status: "active",
source: "codex",
updated_at: "2026-09-19T00:00:01Z",
evidence_ids: ["evidence-1"],
score: 0.7,
truncated: false,
},
],
used_characters: 49,
used_characters: 44,
truncated: false,
total_ms: 5,
adoption: "unknown",
};
beforeEach(() => {
@@ -115,42 +89,17 @@ beforeEach(() => {
});
describe("unified context search", () => {
it("accepts an empty tool message in a valid partial context response", async () => {
it("accepts an empty raw item in a valid partial context response", async () => {
transport.post.mockResolvedValue({
status: 200,
data: {
...RESPONSE,
state: "partial",
paths: {
raw: {
...RESPONSE.paths.raw,
status: "degraded",
reason: "lexical_only",
returned_count: 1,
},
memory: {
...RESPONSE.paths.memory,
status: "empty",
returned_count: 0,
},
},
items: [
{
kind: "raw",
source_id: "workbuddy",
conversation_id: "conversation-1",
anchor_evidence_id: "evidence-1",
messages: [
{
evidence_id: "evidence-1",
revision: 1,
role: "tool",
written_by: "tool",
created_at: "2026-10-05T20:48:12Z",
text: "",
truncated: true,
},
],
...RESPONSE.items[0],
source: "workbuddy",
content: "",
truncated: true,
},
],
@@ -165,8 +114,9 @@ describe("unified context search", () => {
streams.restore();
}
expect(JSON.parse(streams.stdout())).toMatchObject({
schema_version: 2,
state: "partial",
items: [{ kind: "raw", messages: [{ text: "" }] }],
items: [{ kind: "raw", content: "" }],
});
});
@@ -178,12 +128,12 @@ describe("unified context search", () => {
streams.restore();
}
const output = streams.stdout();
expect(output).toContain("Synthetic governed memory.");
expect(output).toContain("Synthetic SDK memory.");
expect(output).toContain("Synthetic raw citation.");
expect(output).not.toMatch(/^- raw |memory-1|evidence-1|采用状态/m);
expect(output).not.toMatch(/^- raw |memory-1|raw-1|采用状态/m);
});
it("queries the current read-only context endpoint and preserves citations", async () => {
it("queries the SDK context endpoint directly and preserves item ids", async () => {
const streams = captureStreams();
try {
expect(
@@ -209,7 +159,7 @@ describe("unified context search", () => {
streams.restore();
}
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/context/search",
"/api/v2/context/search",
{
query: "synthetic question",
kinds: ["raw", "memory"],
@@ -222,12 +172,18 @@ describe("unified context search", () => {
);
const output = JSON.parse(streams.stdout());
expect(output).toMatchObject({
schema_version: 2,
state: "grounded",
request_id: "context-request-1",
adoption: "unknown",
});
expect(output.items[0].messages[0].evidence_id).toBe("evidence-1");
expect(output.items[1].evidence_ids).toEqual(["evidence-1"]);
expect(output).not.toHaveProperty("paths");
expect(output.items[0]).toMatchObject({ kind: "raw", raw_id: "raw-1" });
expect(output.items[1]).toMatchObject({
kind: "memory",
memory_id: "memory-1",
layer: "l2_fact",
});
// No authority pre-query: released Core always serves the SDK data plane.
expect(transport.get).not.toHaveBeenCalled();
expect(transport.put).not.toHaveBeenCalled();
expect(transport.patch).not.toHaveBeenCalled();
@@ -242,19 +198,8 @@ describe("unified context search", () => {
status: 200,
data: {
...RESPONSE,
paths: {
...RESPONSE.paths,
raw: {
status: "not_requested",
reason: "not_requested",
elapsed_ms: 0,
returned_count: 0,
truncated: false,
truncated_by: [],
},
},
items: [RESPONSE.items[1]],
used_characters: 26,
used_characters: 21,
},
});
const streams = captureStreams();
@@ -394,29 +339,7 @@ describe("unified context search", () => {
status: 200,
data: {
...RESPONSE,
paths: {
...RESPONSE.paths,
memory: {
...RESPONSE.paths.memory,
status: "empty",
returned_count: 0,
},
},
items: [
{
...(RESPONSE.items[0] as Record<string, unknown>),
messages: [
{
...(
RESPONSE.items[0] as {
messages: Array<Record<string, unknown>>;
}
).messages[0],
text,
},
],
},
],
items: [{ ...RESPONSE.items[0], content: text }],
used_characters: 10,
},
});
@@ -429,7 +352,7 @@ describe("unified context search", () => {
expect(JSON.parse(streams.stdout())).toMatchObject({
state: "grounded",
used_characters: 10,
items: [{ kind: "raw", messages: [{ text }] }],
items: [{ kind: "raw", content: text }],
});
});
@@ -441,29 +364,7 @@ describe("unified context search", () => {
status: 200,
data: {
...RESPONSE,
paths: {
...RESPONSE.paths,
memory: {
...RESPONSE.paths.memory,
status: "empty",
returned_count: 0,
},
},
items: [
{
...(RESPONSE.items[0] as Record<string, unknown>),
messages: [
{
...(
RESPONSE.items[0] as {
messages: Array<Record<string, unknown>>;
}
).messages[0],
text,
},
],
},
],
items: [{ ...RESPONSE.items[0], content: text }],
used_characters: 12,
},
});
@@ -479,6 +380,40 @@ describe("unified context search", () => {
});
});
it.each([
[
"a raw item from a source that was not asked for",
["--source", "codex"],
{ ...RESPONSE.items[0], source: "cursor" },
],
[
"a raw item when only memory was asked for",
["--kind", "memory"],
RESPONSE.items[0],
],
["a score outside 0..1", [], { ...RESPONSE.items[1], score: 1.5 }],
])("rejects %s as an invalid response", async (_label, flags, item) => {
transport.post.mockResolvedValue({
status: 200,
data: {
...RESPONSE,
items: [item],
used_characters: Array.from(String(item.content)).length,
},
});
const streams = captureStreams();
try {
expect(await run(["recall", "query", ...flags, "--json"])).toBe(1);
} finally {
streams.restore();
}
expect(JSON.parse(streams.stdout())).toMatchObject({
schema_version: 2,
state: "invalid",
code: "response_invalid",
});
});
it("reports endpoint denial without exposing the response body", async () => {
transport.post.mockResolvedValue({
status: 403,
+43 -162
View File
@@ -3,31 +3,6 @@ import { PRODUCT } from "./version.js";
export const CONTEXT_QUERY_MAX_CHARACTERS = 2_000;
const CONTEXT_KINDS = ["raw", "memory"] as const;
const PATH_STATES = [
"ok",
"empty",
"not_requested",
"not_authorized",
"unavailable",
"degraded",
] as const;
const PATH_REASONS = [
"not_requested",
"permission_required",
"source_revoked",
"read_unavailable",
"timeout_budget",
"evidence_changed",
"memory_changed",
"lexical_only",
] as const;
const LIMIT_REASONS = [
"count_budget",
"character_budget",
"timeout_budget",
"metadata_budget",
] as const;
export interface ContextSearchInput {
query: string;
kinds: Array<(typeof CONTEXT_KINDS)[number]>;
@@ -37,8 +12,9 @@ export interface ContextSearchInput {
timeout_ms: number;
}
/** The CLI's JSON document mirrors Core's SDK-native context schema v2. */
export interface ContextSearchDocument {
schema_version: 1;
schema_version: 2;
product: string;
state:
| "grounded"
@@ -50,12 +26,10 @@ export interface ContextSearchDocument {
| "invalid";
code?: string;
request_id?: string;
paths?: Record<string, unknown>;
items: Array<Record<string, unknown>>;
used_characters?: number;
truncated?: boolean;
total_ms?: number;
adoption?: "unknown";
}
export async function searchContext(
@@ -67,7 +41,7 @@ export async function searchContext(
const validated = validateContextResponse(outcome.data, input);
return (
validated ?? {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
state: "invalid",
code: "response_invalid",
@@ -131,13 +105,7 @@ export function renderContextText(document: ContextSearchDocument): string {
: "查到以下相关内容:",
];
for (const item of document.items) {
if (item.kind === "memory") {
lines.push(`- ${String(item.content)}`);
} else {
for (const message of item.messages as Array<Record<string, unknown>>) {
lines.push(`- ${String(message.text)}`);
}
}
lines.push(`- ${String(item.content)}`);
}
if (document.truncated) lines.push("结果已按请求预算截断。");
return `${lines.join("\n")}\n`;
@@ -149,84 +117,37 @@ function validateContextResponse(
): ContextSearchDocument | null {
const response = record(value);
if (
response?.schema_version !== 1 ||
response?.schema_version !== 2 ||
!oneOf(response.state, ["grounded", "empty", "partial", "unavailable"]) ||
!boundedText(response.request_id, 64) ||
!record(response.paths) ||
!Array.isArray(response.items) ||
response.items.length > request.limit ||
!integer(response.used_characters, 0, request.character_budget) ||
typeof response.truncated !== "boolean" ||
!number(response.total_ms, 0) ||
response.adoption !== "unknown"
!number(response.total_ms, 0)
) {
return null;
}
const paths = response.paths as Record<string, unknown>;
const rawPath = validatePath(paths.raw);
const memoryPath = validatePath(paths.memory);
if (!rawPath || !memoryPath) return null;
const requested = new Set(request.kinds);
if (
(!requested.has("raw") && rawPath.status !== "not_requested") ||
(!requested.has("memory") && memoryPath.status !== "not_requested")
) {
return null;
}
const items: Array<Record<string, unknown>> = [];
let characters = 0;
for (const itemValue of response.items) {
const item = validateItem(itemValue, request);
if (!item || !requested.has(item.kind as "raw" | "memory")) return null;
items.push(item);
characters +=
item.kind === "memory"
? characterCount(item.content)
: (item.messages as Array<Record<string, unknown>>).reduce(
(sum, message) => sum + characterCount(message.text),
0,
);
characters += characterCount(item.content);
}
if (characters !== response.used_characters) return null;
if (response.state === "empty" && items.length !== 0) return null;
return {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
state: response.state,
request_id: response.request_id as string,
paths: { raw: rawPath, memory: memoryPath },
items,
used_characters: response.used_characters as number,
truncated: response.truncated,
total_ms: response.total_ms as number,
adoption: "unknown",
};
}
function validatePath(value: unknown): Record<string, unknown> | null {
const path = record(value);
if (
!path ||
!oneOf(path.status, PATH_STATES) ||
!number(path.elapsed_ms, 0) ||
!integer(path.returned_count, 0, 20) ||
typeof path.truncated !== "boolean" ||
!enumList(path.truncated_by, LIMIT_REASONS) ||
!(
path.reason === null ||
path.reason === undefined ||
oneOf(path.reason, PATH_REASONS)
)
) {
return null;
}
return {
status: path.status,
reason: path.reason ?? null,
elapsed_ms: path.elapsed_ms,
returned_count: path.returned_count,
truncated: path.truncated,
truncated_by: path.truncated_by,
};
}
@@ -235,88 +156,65 @@ function validateItem(
request: ContextSearchInput,
): Record<string, unknown> | null {
const item = record(value);
if (!item || !oneOf(item.kind, CONTEXT_KINDS)) return null;
if (
!item ||
!oneOf(item.kind, CONTEXT_KINDS) ||
boundedText(item.content, 20_000, true) === null ||
!number(item.score, 0) ||
(item.score as number) > 1 ||
typeof item.truncated !== "boolean"
) {
return null;
}
if (item.kind === "memory") {
if (
!boundedText(item.memory_id, 512) ||
!integer(item.revision, 1) ||
boundedText(item.content, 20_000, true) === null ||
boundedText(item.memory_kind, 64, true) === null ||
!boundedText(item.updated_at, 64) ||
!textList(item.evidence_ids, 128, 512) ||
typeof item.truncated !== "boolean"
boundedText(item.layer, 64, true) === null ||
boundedText(item.status, 32, true) === null ||
!optionalText(item.source, 128) ||
!optionalText(item.updated_at, 64)
) {
return null;
}
return {
kind: "memory",
memory_id: item.memory_id,
revision: item.revision,
content: item.content,
memory_kind: item.memory_kind,
updated_at: item.updated_at,
evidence_ids: item.evidence_ids,
layer: item.layer,
status: item.status,
source: item.source ?? null,
updated_at: item.updated_at ?? null,
score: item.score,
truncated: item.truncated,
};
}
if (
!boundedText(item.source_id, 128) ||
!boundedText(item.raw_id, 512) ||
!boundedText(item.source, 128) ||
(request.raw_source_ids !== undefined &&
!request.raw_source_ids.includes(item.source_id as string)) ||
!boundedText(item.conversation_id, 512) ||
!boundedText(item.anchor_evidence_id, 512) ||
!Array.isArray(item.messages) ||
item.messages.length < 1 ||
item.messages.length > 3 ||
typeof item.truncated !== "boolean"
!request.raw_source_ids.includes(item.source as string)) ||
!boundedText(item.session_id, 512) ||
!optionalText(item.memory_at, 64)
) {
return null;
}
const messages = item.messages.map(validateMessage);
if (messages.some((message) => message === null)) return null;
return {
kind: "raw",
source_id: item.source_id,
conversation_id: item.conversation_id,
anchor_evidence_id: item.anchor_evidence_id,
messages,
raw_id: item.raw_id,
content: item.content,
source: item.source,
session_id: item.session_id,
memory_at: item.memory_at ?? null,
score: item.score,
truncated: item.truncated,
};
}
function validateMessage(value: unknown): Record<string, unknown> | null {
const message = record(value);
if (
!message ||
!boundedText(message.evidence_id, 512) ||
!integer(message.revision, 1) ||
boundedText(message.role, 32, true) === null ||
boundedText(message.written_by, 32, true) === null ||
!(
message.created_at === null ||
boundedText(message.created_at, 64) !== null
) ||
boundedText(message.text, 20_000, true) === null ||
typeof message.truncated !== "boolean"
) {
return null;
}
return {
evidence_id: message.evidence_id,
revision: message.revision,
role: message.role,
written_by: message.written_by,
created_at: message.created_at,
text: message.text,
truncated: message.truncated,
};
}
function failed(
outcome: Exclude<ReadOutcome<unknown>, { state: "ok" }>,
): ContextSearchDocument {
return {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
state: outcome.state,
code: outcome.code,
@@ -374,27 +272,10 @@ function oneOf<const T extends readonly string[]>(
return typeof value === "string" && values.includes(value as T[number]);
}
function enumList(
value: unknown,
values: readonly string[],
): value is string[] {
function optionalText(value: unknown, maximum: number): boolean {
return (
Array.isArray(value) &&
value.length <= 4 &&
new Set(value).size === value.length &&
value.every((item) => oneOf(item, values))
);
}
function textList(
value: unknown,
maximumItems: number,
maximumText: number,
): value is string[] {
return (
Array.isArray(value) &&
value.length <= maximumItems &&
new Set(value).size === value.length &&
value.every((item) => boundedText(item, maximumText) !== null)
value === null ||
value === undefined ||
boundedText(value, maximum) !== null
);
}
+4 -11
View File
@@ -8,14 +8,12 @@ import axios from "axios";
*/
const READ_ONLY_PATHS = [
"/api/v1/call-events",
"/api/v1/home-summary",
"/api/v1/import-jobs",
"/api/v1/live-sync",
"/api/v1/memory-library",
"/api/v1/memory-processing",
"/api/v1/memory-tasks",
"/api/v1/model-profiles",
"/api/v1/sources",
"/api/v2/home-summary",
"/api/v2/memory-library",
"/api/v2/user-profile",
] as const;
@@ -67,13 +65,10 @@ export function createUiSessionRequest(attachment: {
};
}
const MEMORY_DETAIL = /^\/api\/v1\/memories\/[A-Za-z0-9][\w.-]{0,127}$/;
const IMPORT_JOB =
/^\/api\/v1\/import-jobs\/[A-Za-z0-9][\w.-]{0,127}(?:\/issues)?$/;
const OPERATION_RECEIPT =
/^\/api\/v1\/(?:source-connections|model-profile-creations|model-probe-operations)\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const MEMORY_MUTATION_RECEIPT =
/^\/api\/v1\/memory-mutations\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
/**
* Core registers these static routes before `/{memory_id}`, so a memory whose
* id happens to be one of them would reach a different handler: `graph`
@@ -173,7 +168,7 @@ export function createCoreContextQuery(attachment: {
let response: { status: number; data: unknown };
try {
response = await client.post(
"/api/v1/context/search",
"/api/v2/context/search",
body,
options?.signal ? { signal: options.signal } : undefined,
);
@@ -197,10 +192,8 @@ export function isReadOnlyPath(path: string): boolean {
if ((READ_ONLY_PATHS as readonly string[]).includes(path)) return true;
if (IMPORT_JOB.test(path)) return true;
if (OPERATION_RECEIPT.test(path)) return true;
if (MEMORY_MUTATION_RECEIPT.test(path)) return true;
if (SDK_MEMORY_DETAIL.test(path)) return true;
if (!MEMORY_DETAIL.test(path)) return false;
const segment = path.slice("/api/v1/memories/".length);
if (!SDK_MEMORY_DETAIL.test(path)) return false;
const segment = path.slice("/api/v2/memories/".length);
return !(RESERVED_MEMORY_SEGMENTS as readonly string[]).includes(segment);
}
+24 -212
View File
@@ -1,18 +1,20 @@
import type {
GovernedMemory,
HomeSummary,
ImportIssue,
ImportJob,
LiveSyncHealth,
MemoryDetail,
MemoryEvidenceExcerpt,
MemoryLibraryPage,
MemoryProcessingSummary,
ModelProfile,
SourceResponse,
} from "@one-memory/contracts";
/** The only memory states a response may carry, and the only ones a filter may ask for. */
export const MEMORY_STATUSES = ["confirmed", "deleted", "superseded"] as const;
/** SDK memory states `memhub memories list --status` may ask for. */
export const MEMORY_STATUSES = [
"active",
"superseded",
"shadow",
"negated",
"conflicted",
"archived",
] as const;
/**
* HTTP 200 is not a contract. Everything here re-checks the shape and the
@@ -23,7 +25,6 @@ export const MEMORY_STATUSES = ["confirmed", "deleted", "superseded"] as const;
const ID_MAX = 128;
const LABEL_MAX = 256;
const TEXT_MAX = 100_000;
const LIST_MAX = 500;
const LIVE_SYNC_STATES = [
"idle",
@@ -39,14 +40,6 @@ const PROVIDER_KINDS = [
"local_onnx",
] as const;
const CREDENTIAL_STATES = ["available", "missing", "unavailable"] as const;
const MEMORY_KINDS = [
"chat",
"working",
"coding",
"schema",
"intention",
] as const;
const MEMORY_ORIGINS = ["extracted", "answered", "agent", "manual"] as const;
const IMPORT_STATUSES = [
"pending",
"running",
@@ -57,16 +50,6 @@ const IMPORT_STATUSES = [
"cancelled",
] as const;
const SCAN_PHASES = ["discover", "read", "emit", "done"] as const;
const MEMORY_TASK_KINDS = ["extract", "system2", "ultra", "sweeper"] as const;
const MEMORY_TASK_STATUSES = [
"pending",
"running",
"waiting_for_model",
"completed",
"failed",
"cancelled",
"deferred_outside_window",
] as const;
const CAPABILITY_STATES = ["not_implemented", "implemented"] as const;
const VERIFICATION_STATES = ["fixture_only", "real_replayed"] as const;
const SYNC_COMPLETENESS = [
@@ -87,16 +70,6 @@ const LOCATION_STATES = [
] as const;
const LIVE_SYNC_MODES = ["polling", "watch_and_polling"] as const;
export interface ValidatedMemoryTask {
task_id: string;
kind: (typeof MEMORY_TASK_KINDS)[number];
status: (typeof MEMORY_TASK_STATUSES)[number];
attempts: number;
last_error_code: string | null;
created_at: string;
updated_at: string;
}
export interface SourceConnectionCandidate {
source_id: string;
display_name: string;
@@ -352,45 +325,6 @@ export function validateImportIssues(
});
}
export function validateMemoryTasks(
value: unknown,
): ValidatedMemoryTask[] | null {
return list(value, (item) => {
const task = record(item);
if (!task) return null;
const taskId = text(task.task_id, ID_MAX);
if (
task.last_error_code !== null &&
text(task.last_error_code, LABEL_MAX) === null
) {
return null;
}
const error = task.last_error_code as string | null;
const createdAt = text(task.created_at, LABEL_MAX);
const updatedAt = text(task.updated_at, LABEL_MAX);
const attempts = count(task.attempts);
if (
taskId === null ||
createdAt === null ||
updatedAt === null ||
attempts === null ||
!oneOf(task.kind, MEMORY_TASK_KINDS) ||
!oneOf(task.status, MEMORY_TASK_STATUSES)
) {
return null;
}
return {
task_id: taskId,
kind: task.kind,
status: task.status,
attempts,
last_error_code: error,
created_at: createdAt,
updated_at: updatedAt,
};
});
}
export function validateProcessing(
value: unknown,
): MemoryProcessingSummary | null {
@@ -444,150 +378,32 @@ export function validateModelProfiles(value: unknown): ModelProfile[] | null {
});
}
export function validateHomeSummary(value: unknown): HomeSummary | null {
/** `GET /api/v2/home-summary`: the SDK-authority home card. */
export interface SdkHomeSummary {
schema_version: 2;
memory_count: number;
latest: { memory_id: string } | null;
observed_at: string;
}
export function validateHomeSummary(value: unknown): SdkHomeSummary | null {
const summary = record(value);
if (
!summary ||
summary.schema_version !== 2 ||
count(summary.memory_count) === null ||
text(summary.observed_at, LABEL_MAX) === null
) {
return null;
}
return summary as unknown as HomeSummary;
}
/** The page must be the page that was asked for, not merely a page. */
export function validateMemoryPage(
value: unknown,
request: { offset: number; limit: number },
): MemoryLibraryPage | null {
const page = record(value);
if (!page) return null;
const total = count(page.total);
if (
total === null ||
page.offset !== request.offset ||
page.limit !== request.limit
) {
return null;
}
const items = list(page.items, validateGovernedMemory);
if (items === null || items.length > request.limit) return null;
if (items.length > 0 && total < request.offset + items.length) return null;
return { items, total, offset: page.offset, limit: page.limit };
}
export function validateMemoryDetail(
value: unknown,
memoryId: string,
): MemoryDetail | null {
const detail = record(value);
if (!detail) return null;
const memory = validateGovernedMemory(detail.memory);
if (memory === null || memory.memory_id !== memoryId) return null;
const cited = new Set(memory.evidence_ids);
const returned = new Set<string>();
const evidence = list(detail.evidence ?? [], (item) => {
const excerpt = record(item);
if (!excerpt) return null;
const evidenceId = text(excerpt.evidence_id, ID_MAX);
// Core builds this list from the memory's own citations; anything else is
// another record's excerpt and must not be shown under this memory.
if (
evidenceId === null ||
!cited.has(evidenceId) ||
returned.has(evidenceId) ||
typeof excerpt.available !== "boolean"
) {
const latest = summary.latest;
if (latest !== null) {
const memory = record(latest);
if (!memory || typeof memory.memory_id !== "string" || !memory.memory_id) {
return null;
}
returned.add(evidenceId);
const available = excerpt.available === true;
const verbatim = available ? text(excerpt.verbatim ?? "", TEXT_MAX) : "";
if (verbatim === null) return null;
return {
evidence_id: evidenceId,
available,
source_id: optional(excerpt.source_id, ID_MAX),
written_by: optional(excerpt.written_by, LABEL_MAX),
created_at: optional(excerpt.created_at, LABEL_MAX),
// An excerpt the user cannot read has no text, whatever the server sent.
verbatim,
} as unknown as MemoryEvidenceExcerpt;
});
if (evidence === null || evidence.length !== memory.evidence_ids.length) {
return null;
}
const history = list(detail.history ?? [], (item) => {
const version = record(item);
if (
!version ||
typeof version.revision !== "number" ||
!Number.isSafeInteger(version.revision) ||
version.revision < 1 ||
text(version.action, LABEL_MAX) === null ||
!oneOf(version.status, MEMORY_STATUSES) ||
text(version.content, TEXT_MAX) === null ||
identifiers(version.scopes) === null ||
text(version.created_at, LABEL_MAX) === null
) {
return null;
}
return {
revision: version.revision,
action: version.action,
status: version.status,
content: version.content,
scopes: identifiers(version.scopes) as string[],
created_at: version.created_at,
};
});
if (history === null) return null;
return {
memory,
evidence,
history,
index_sync: null,
} as unknown as MemoryDetail;
}
export function validateGovernedMemory(value: unknown): GovernedMemory | null {
const memory = record(value);
if (
!memory ||
!text(memory.memory_id, ID_MAX) ||
!text(memory.user_id, LABEL_MAX) ||
!text(memory.content, TEXT_MAX) ||
!oneOf(memory.kind, MEMORY_KINDS) ||
!oneOf(memory.origin, MEMORY_ORIGINS) ||
typeof memory.revision !== "number" ||
!Number.isSafeInteger(memory.revision) ||
memory.revision < 1 ||
!oneOf(memory.status, MEMORY_STATUSES) ||
!text(memory.created_at, LABEL_MAX) ||
!text(memory.updated_at, LABEL_MAX)
) {
return null;
}
const scopes = identifiers(memory.scopes);
const evidenceIds = identifiers(memory.evidence_ids);
const tags = identifiers(memory.tags ?? []);
if (
scopes === null ||
scopes.length < 1 ||
evidenceIds === null ||
evidenceIds.length < 1 ||
tags === null ||
tags.length > 32
) {
return null;
}
return {
...memory,
scopes,
evidence_ids: evidenceIds,
tags,
} as unknown as GovernedMemory;
return summary as unknown as SdkHomeSummary;
}
function identifiers(value: unknown): string[] | null {
@@ -618,10 +434,6 @@ function text(value: unknown, max: number): string | null {
return typeof value === "string" && value.length <= max ? value : null;
}
function optional(value: unknown, max: number): string | undefined {
return typeof value === "string" && value.length <= max ? value : undefined;
}
function count(value: unknown): number | null {
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0
? value
@@ -134,11 +134,6 @@ function progress(): ProgressDocument {
generation: {
summary_state: { state: "ok" },
summary: { waiting_for_model: 1, failed: 0 },
tasks_state: { state: "ok" },
tasks: [{ task_id: "t1", status: "waiting_for_model" }],
returned: 1,
validated: 1,
output_truncated: false,
},
};
}
+21 -12
View File
@@ -65,6 +65,9 @@ export function renderDoctorText(document: DoctorDocument): string {
model: "模型",
pending_operations: "待处理操作",
};
const migration = document.checks.find(
(check) => check.id === "core" && check.details?.log_file,
);
return [
`MemHub 检查:${doctorState(document.state)}`,
...document.checks.map(
@@ -73,7 +76,13 @@ export function renderDoctorText(document: DoctorDocument): string {
),
...(document.state === "healthy"
? []
: ["下一步:打开 MemHub 的“帮助与诊断”查看并处理异常。"]),
: migration
? [
`升级迁移没有完成(错误码:${migration.code}),MemHub 不会启动。`,
`下一步:把迁移日志附在 GitHub 反馈里发给我们:${migration.details?.log_file}`,
`反馈地址:${migration.details?.issues_url}`,
]
: ["下一步:打开 MemHub 的“帮助与诊断”查看并处理异常。"]),
"",
].join("\n");
}
@@ -107,6 +116,14 @@ function coreCheck(status: StatusDocument): DoctorCheck {
id: "core",
state: status.runtime.state === "stopped" ? "warn" : "fail",
code: status.runtime.code,
...(status.runtime.log_file
? {
details: {
log_file: status.runtime.log_file,
issues_url: status.runtime.issues_url ?? "",
},
}
: {}),
};
}
@@ -200,8 +217,7 @@ function modelCheck(status: StatusDocument): DoctorCheck {
function pendingCheck(progress: ProgressDocument | null): DoctorCheck {
if (
progress?.imports.state !== "ok" ||
progress.generation.summary_state.state !== "ok" ||
progress.generation.tasks_state.state !== "ok"
progress.generation.summary_state.state !== "ok"
) {
return {
id: "pending_operations",
@@ -212,27 +228,20 @@ function pendingCheck(progress: ProgressDocument | null): DoctorCheck {
const jobAttention = progress.imports.jobs.filter((item) =>
["paused", "interrupted", "failed"].includes(text(item.status) ?? ""),
).length;
const taskAttention = progress.generation.tasks.filter((item) =>
["waiting_for_model", "failed"].includes(text(item.status) ?? ""),
).length;
const summary = progress.generation.summary;
const waiting = summary ? number(summary.waiting_for_model) : 0;
const failed = summary ? number(summary.failed) : 0;
const attention = jobAttention + taskAttention + waiting + failed;
const attention = jobAttention + waiting + failed;
return {
id: "pending_operations",
state: attention > 0 ? "warn" : "pass",
code: attention > 0 ? "operations_need_attention" : "pending_operations_ok",
details: {
import_jobs_validated: progress.imports.validated,
generation_tasks_validated: progress.generation.validated,
jobs_need_attention: jobAttention,
tasks_need_attention: taskAttention,
waiting_for_model: waiting,
failed,
output_truncated:
progress.imports.output_truncated ||
progress.generation.output_truncated,
output_truncated: progress.imports.output_truncated,
},
};
}
+31 -1
View File
@@ -1,4 +1,4 @@
import { mkdtemp, rm } from "node:fs/promises";
import { mkdtemp, rm, symlink } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { DiagnosticLogger } from "@one-memory/runtime-control";
@@ -41,11 +41,15 @@ it("redacts unsafe details and absolute stack frames from the read surface", ()
session_id: "11111111-1111-4111-8111-111111111111",
details: {
count: 1,
reason: "/Users/me/private-query.txt",
status: "sk-live-secret-token",
source_id: "credential.private",
prompt: "secret-token-123",
authorization: "Bearer secret-token-123",
} as unknown as Record<string, string | number | boolean | null>,
safe_stack: [
"packages/runtime-control/src/example.ts:42:7",
"packages/runtime/sk-live-secret-token.ts:1:1",
"/Users/me/private.py:1",
"C:\\Users\\me\\private.py:1",
],
@@ -56,6 +60,8 @@ it("redacts unsafe details and absolute stack frames from the read surface", ()
"packages/runtime-control/src/example.ts:42:7",
]);
expect(JSON.stringify(redacted)).not.toContain("secret-token-123");
expect(JSON.stringify(redacted)).not.toContain("sk-live-secret-token");
expect(JSON.stringify(redacted)).not.toContain("credential.private");
expect(JSON.stringify(redacted)).not.toContain("/Users/me");
expect(JSON.stringify(redacted)).not.toContain("C:\\\\Users");
});
@@ -132,3 +138,27 @@ it("follows a newly appended event without stopping Runtime", async () => {
});
expect(seen).toEqual(["service.ready", "core.failed"]);
});
it("fails follow mode when the diagnostic directory is unsafe", async () => {
const dataDir = await mkdtemp(
path.join(os.tmpdir(), "om-cli-follow-unsafe-"),
);
const external = await mkdtemp(
path.join(os.tmpdir(), "om-cli-follow-external-"),
);
roots.push(dataDir, external);
await symlink(external, path.join(dataDir, "logs"));
const query = parseLogQuery(["--follow"]);
expect(query).not.toBeNull();
if (!query) throw new Error("query_missing");
await expect(
followLogs({
dataDir,
query,
signal: new AbortController().signal,
maxEvents: 1,
emit: () => undefined,
wait: async () => undefined,
}),
).rejects.toThrow("log_directory_unreadable");
});
+6 -102
View File
@@ -3,6 +3,7 @@ import path from "node:path";
import type { DiagnosticEventV1 } from "@one-memory/runtime-control";
import {
diagnosticLogFiles,
projectDiagnosticEventForSafeOutput,
readDiagnosticEvents,
} from "@one-memory/runtime-control";
@@ -24,54 +25,6 @@ export interface LogQuery {
const ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/;
const NAME = /^[a-z][a-z0-9]*(?:[._-][a-z0-9]+){0,15}$/;
const SAFE_DETAIL_KEYS = new Set([
"action",
"attempt",
"count",
"method",
"mode",
"phase",
"platform",
"reason",
"result_count",
"source_id",
"spawned",
"status",
"version",
"parent_session_id",
"invalid_lines",
"truncated",
"runtime_state",
"core_state",
"broker_state",
"instance_id",
"open_file_count",
"memory_bytes",
"thread_count",
"database_error_count",
"file_count",
"sqlite_file_count",
"byte_file_count",
"total_bytes",
"copied_bytes",
"bytes_per_second",
"stored_count",
"revised_count",
"skipped_count",
"error_count",
"pending_count",
"prepared_count",
"queue_depth",
"in_flight_count",
"call_count",
"created_count",
"updated_count",
"superseded_count",
"deleted_count",
"operation_kind",
"file_kind",
]);
export function parseLogQuery(arguments_: readonly string[]): LogQuery | null {
if (arguments_.filter((item) => item === "--json").length > 1) return null;
const values: Partial<LogQuery> = { errorsOnly: false, follow: false };
@@ -237,63 +190,13 @@ export function renderLogText(
return `${lines.join("\n")}\n`;
}
/** Defence in depth for the CLI read surface. `DiagnosticLogger` already
* allow-lists keys, but a future writer, a manually edited log, or a malformed
* line must not turn `memhub logs` into an exfiltration path. */
/** Defence in depth for the CLI read surface. Local writers accept arbitrary
* bounded scalar keys, so every outward read surface must reuse the same
* field-and-value projection as the support-bundle exporter. */
export function redactDiagnosticEvent(
event: DiagnosticEventV1,
): DiagnosticEventV1 {
const redacted: DiagnosticEventV1 = {
schema_version: 1,
timestamp: event.timestamp,
level: event.level,
service: event.service,
component: event.component,
event: event.event,
outcome: event.outcome,
session_id: event.session_id,
...(event.trace_id === undefined ? {} : { trace_id: event.trace_id }),
...(event.incident_id === undefined
? {}
: { incident_id: event.incident_id }),
...(event.instance_id === undefined
? {}
: { instance_id: event.instance_id }),
...(event.operation_id === undefined
? {}
: { operation_id: event.operation_id }),
...(event.request_id === undefined ? {} : { request_id: event.request_id }),
...(event.job_id === undefined ? {} : { job_id: event.job_id }),
...(event.task_id === undefined ? {} : { task_id: event.task_id }),
...(event.duration_ms === undefined
? {}
: { duration_ms: event.duration_ms }),
...(event.state_before === undefined
? {}
: { state_before: event.state_before }),
...(event.state_after === undefined
? {}
: { state_after: event.state_after }),
...(event.error_code === undefined ? {} : { error_code: event.error_code }),
...(event.error_type === undefined ? {} : { error_type: event.error_type }),
...(event.safe_stack === undefined
? {}
: {
safe_stack: event.safe_stack.filter(
(frame) =>
!frame.startsWith("/") &&
!/^[A-Za-z]:\\/.test(frame) &&
!frame.includes(".."),
),
}),
};
const details = Object.fromEntries(
Object.entries(event.details ?? {}).filter(([key]) =>
SAFE_DETAIL_KEYS.has(key),
),
);
if (Object.keys(details).length > 0) redacted.details = details;
return redacted;
return projectDiagnosticEventForSafeOutput(event);
}
export async function followLogs(options: {
@@ -315,6 +218,7 @@ export async function followLogs(options: {
...options.query,
tail: 5_000,
});
if (report.failure) throw new Error(report.failure);
for (const event of report.events) {
const identity = JSON.stringify(event);
if (seen.has(identity)) continue;
-700
View File
@@ -1,700 +0,0 @@
import { createHash, randomUUID, timingSafeEqual } from "node:crypto";
import type { GovernedMemory } from "@one-memory/contracts";
import {
buildMemoryActionRequest,
buildMemoryCorrectRequest,
buildMemoryCreateRequest,
parseMemoryReceipt,
} from "@one-memory/operations";
import type { CoreReader } from "./core-client.js";
import type { CoreOperations } from "./core-operations.js";
import { validateMemoryDetail } from "./core-responses.js";
import { isMemoryId } from "./memory-report.js";
import { PRODUCT } from "./version.js";
const MEMORY_KINDS = [
"chat",
"working",
"coding",
"schema",
"intention",
] as const;
const OPERATIONS = ["add", "correct", "delete"] as const;
const MEMORY_ORIGINS = ["extracted", "answered", "agent", "manual"] as const;
const MUTATION_RECEIPT_KEYS = new Set([
"schema_version",
"request_id",
"state",
"operation",
"memory_id",
"result_revision",
"result_status",
"memory_state",
"migrated_from",
]);
const CONTENT_MAX_CHARACTERS = 100_000;
const ID = /^[A-Za-z0-9][\w.-]{0,127}$/;
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
interface PlanBody {
schema_version: 1;
kind: "memory_mutation";
operation: (typeof OPERATIONS)[number];
request_id: string;
memory_id?: string;
expected_revision?: number;
expected_status?: string;
expected_origin?: string;
memory_kind?: (typeof MEMORY_KINDS)[number];
scopes?: string[];
evidence_ids?: string[];
content_sha256?: string;
content_characters?: number;
current?: {
status: string;
origin: string;
revision: number;
scopes: string[];
evidence: Array<{
evidence_id: string;
available: boolean;
source_id?: string;
written_by?: string;
}>;
};
}
export type MemoryMutationPlan = PlanBody & { digest: string };
export type MemoryFlowDocument =
| {
schema_version: 1;
product: string;
state: "preview";
plan: MemoryMutationPlan;
}
| {
schema_version: 1;
product: string;
state: "committed" | "recovered";
operation_id: string;
receipt: Record<string, unknown>;
}
| {
schema_version: 1;
product: string;
state: "rejected" | "unknown" | "invalid";
operation_id?: string;
code: string;
};
export function parseMemoryContent(bytes: Buffer): string | null {
if (bytes.byteLength > CONTENT_MAX_CHARACTERS * 4 + 2) return null;
let decoded: string;
try {
decoded = new TextDecoder("utf-8", { fatal: true }).decode(bytes);
} catch {
return null;
}
const content = decoded.replace(/\r?\n$/, "");
return content.trim().length > 0 &&
[...content].length <= CONTENT_MAX_CHARACTERS &&
![...content].some((character) => {
const code = character.charCodeAt(0);
return code <= 8 || (code >= 11 && code <= 31) || code === 127;
})
? content
: null;
}
export function planMemoryAdd(input: {
content: string;
memoryKind: string;
scopes: string[];
evidenceIds: string[];
}): MemoryFlowDocument {
if (
!validContent(input.content) ||
!oneOf(input.memoryKind, MEMORY_KINDS) ||
!validScopes(input.scopes) ||
!validIds(input.evidenceIds, 500)
) {
return invalid("memory_add_input_invalid");
}
return preview(
withDigest({
schema_version: 1,
kind: "memory_mutation",
operation: "add",
request_id: randomUUID(),
memory_kind: input.memoryKind,
scopes: input.scopes,
evidence_ids: input.evidenceIds,
...contentIdentity(input.content),
}),
);
}
export async function planExistingMemoryMutation(
reader: CoreReader,
input: {
operation: Exclude<(typeof OPERATIONS)[number], "add">;
memoryId: string;
content?: string;
},
): Promise<MemoryFlowDocument> {
if (
!isMemoryId(input.memoryId) ||
(input.operation === "correct" && input.content === undefined) ||
(input.content !== undefined && !validContent(input.content))
) {
return invalid("memory_plan_input_invalid");
}
const outcome = await reader<unknown>(`/api/v1/memories/${input.memoryId}`);
if (outcome.state !== "ok") {
return invalid(`memory_${outcome.state}:${outcome.code}`);
}
const detail = validateMemoryDetail(outcome.data, input.memoryId);
if (!detail) return invalid("memory_response_invalid");
const memory = detail.memory;
if (memory.status !== "confirmed") {
return invalid("memory_operation_not_allowed");
}
const body: PlanBody = {
schema_version: 1,
kind: "memory_mutation",
operation: input.operation,
request_id: randomUUID(),
memory_id: memory.memory_id,
expected_revision: memory.revision,
expected_status: memory.status,
expected_origin: memory.origin,
...(input.content ? contentIdentity(input.content) : {}),
current: {
status: memory.status,
origin: memory.origin,
revision: memory.revision,
scopes: [...memory.scopes],
evidence: detail.evidence.map((item) => ({
evidence_id: item.evidence_id,
available: item.available,
...(item.source_id ? { source_id: item.source_id } : {}),
...(item.written_by ? { written_by: item.written_by } : {}),
})),
},
};
return preview(withDigest(body));
}
export function inspectMemoryPlan(
value: unknown,
): { operationId: string; digest: string; needsContent: boolean } | null {
const plan = validatePlan(value);
return plan
? {
operationId: plan.request_id,
digest: plan.digest,
needsContent: plan.operation === "add" || plan.operation === "correct",
}
: null;
}
export function memoryPlanAcceptsContent(
value: unknown,
content: string | undefined,
): boolean {
const plan = validatePlan(value);
if (!plan) return false;
if (plan.operation !== "add" && plan.operation !== "correct") {
return content === undefined;
}
if (content === undefined) return false;
const identity = contentIdentity(content);
return (
identity.content_sha256 === plan.content_sha256 &&
identity.content_characters === plan.content_characters
);
}
export async function executeMemoryMutation(
value: unknown,
content: string | undefined,
operations: CoreOperations,
reader?: CoreReader,
): Promise<MemoryFlowDocument> {
const plan = validatePlan(value);
if (!plan) return invalid("memory_plan_invalid");
if (!memoryPlanAcceptsContent(plan, content)) {
return invalid("memory_content_mismatch");
}
const { path, body } = operationRequest(plan, content);
const outcome = await operations.post<unknown>(path, body);
if (outcome.state === "rejected") {
return {
schema_version: 1,
product: PRODUCT,
state: "rejected",
operation_id: plan.request_id,
code: outcome.code,
};
}
if (outcome.state === "unknown") {
return reader
? readMemoryMutationReceipt(reader, plan)
: pending(plan.request_id, outcome.code);
}
const memory = validateWriteResponse(outcome.data, plan);
if (!memory || !matchesReceipt(memory, plan, content)) {
return reader
? readMemoryMutationReceipt(reader, plan)
: pending(plan.request_id, "memory_receipt_invalid");
}
return {
schema_version: 1,
product: PRODUCT,
state: "committed",
operation_id: plan.request_id,
receipt: {
memory_id: memory.memory_id,
status: memory.status,
origin: memory.origin,
revision: memory.revision,
scopes: memory.scopes,
evidence_ids: memory.evidence_ids,
content_sha256: contentIdentity(memory.content).content_sha256,
content_characters: [...memory.content].length,
created_at: memory.created_at,
updated_at: memory.updated_at,
},
};
}
/**
* Validate the write response through the shared `parseMemoryReceipt`, which
* owns the GovernedMemory shape and the revision/status binding Core must
* satisfy. The CLI keeps only its own plan binding (`matchesReceipt`, which
* also checks `origin`, `kind` and the echoed content), so the two surfaces
* judge a write response identically and no shape rule is duplicated here.
*
* `add` creates the first revision and expects `confirmed`; the shared parser
* derives that from `expectedRevision: 0`. Every other operation advances the
* revision the caller acted against and expects `confirmed` (correct/scope)
* or `deleted` (delete).
*/
function validateWriteResponse(
value: unknown,
plan: MemoryMutationPlan,
): GovernedMemory | null {
const expectedStatus = plan.operation === "delete" ? "deleted" : "confirmed";
// For `add` there is no id or prior revision to bind yet, so the parser's
// own receipt is the only source for both; the plan binding that follows is
// what proves the response belongs to this request.
const memoryId =
plan.operation === "add"
? String((value as { memory_id?: unknown } | null)?.memory_id ?? "")
: (plan.memory_id as string);
if (memoryId === "") return null;
return parseMemoryReceipt(value, {
memoryId,
expectedRevision:
plan.operation === "add" ? 0 : (plan.expected_revision as number),
expectedStatus,
});
}
export async function readMemoryMutationReceipt(
reader: CoreReader,
value: unknown,
): Promise<MemoryFlowDocument> {
const plan = validatePlan(value);
if (!plan) return invalid("memory_plan_invalid");
const outcome = await reader<unknown>(
`/api/v1/memory-mutations/${encodeURIComponent(plan.request_id)}`,
);
if (outcome.state !== "ok") {
return pending(plan.request_id, `memory_mutation_receipt_${outcome.code}`);
}
const receipt = validateMutationReceipt(outcome.data, plan);
if (receipt === "not_found") {
return pending(
plan.request_id,
"memory_mutation_receipt_not_found_after_attempt",
);
}
if (!receipt) {
return pending(plan.request_id, "memory_mutation_receipt_invalid");
}
return {
schema_version: 1,
product: PRODUCT,
state: "recovered",
operation_id: plan.request_id,
receipt,
};
}
export function memoryFlowExitCode(document: MemoryFlowDocument): number {
return ["preview", "committed", "recovered"].includes(document.state) ? 0 : 1;
}
export function renderMemoryFlowText(document: MemoryFlowDocument): string {
if (document.state === "preview") {
const plan = document.plan;
const lines = [
`记忆操作预览:${plan.operation}`,
`request_id:${plan.request_id}`,
];
if (plan.memory_id) lines.push(`memory_id:${plan.memory_id}`);
if (plan.expected_revision) {
lines.push(
`当前版本:${plan.expected_revision} 状态:${plan.expected_status} 来源方式:${plan.expected_origin}`,
);
}
if (plan.scopes) lines.push(`使用范围:${plan.scopes.join(",")}`);
if (plan.current) {
for (const evidence of plan.current.evidence) {
lines.push(
`证据 ${evidence.evidence_id}:source=${evidence.source_id ?? "unknown"} author=${evidence.written_by ?? "unknown"} available=${evidence.available}`,
);
}
}
if (plan.content_characters !== undefined) {
lines.push(
`正文:${plan.content_characters} 字符,sha256=${plan.content_sha256}`,
);
}
lines.push("尚未写入;保存 JSON 计划后用 memories apply --confirm 执行。");
return `${lines.join("\n")}\n`;
}
if (document.state === "committed" || document.state === "recovered") {
return `记忆操作${document.state === "recovered" ? "已从治理回执恢复" : "已提交"}:${document.operation_id}\n${JSON.stringify(document.receipt)}\n`;
}
return `记忆操作未完成:${document.state}(${"code" in document ? document.code : "memory_operation_state_invalid"})${document.operation_id ? ` request_id=${document.operation_id}` : ""}\n`;
}
function operationRequest(
plan: MemoryMutationPlan,
content: string | undefined,
): { path: string; body: Record<string, unknown> } {
if (plan.operation === "add") {
// The shared create builder is the Web console's manual-add body, which
// always creates `chat` (the Web has no kind picker). The CLI exposes
// `--kind`, so it builds through the same builder and then sets the kind,
// keeping every other field (user_id/scope/evidence/origin) shared.
const request = buildMemoryCreateRequest(
{
content: content as string,
scopes: plan.scopes ?? [],
evidence_ids: plan.evidence_ids ?? [],
origin: "manual",
},
plan.request_id,
);
const draft = request.body.draft as Record<string, unknown>;
draft.kind = plan.memory_kind;
return { path: request.path, body: request.body };
}
if (plan.operation === "correct") {
return buildMemoryCorrectRequest(
plan.memory_id as string,
plan.expected_revision as number,
content as string,
undefined,
plan.request_id,
);
}
return buildMemoryActionRequest(
plan.memory_id as string,
plan.operation,
plan.expected_revision as number,
plan.request_id,
);
}
function matchesReceipt(
memory: GovernedMemory,
plan: MemoryMutationPlan,
content: string | undefined,
): boolean {
if (plan.operation === "add") {
return (
memory.status === "confirmed" &&
memory.origin === "manual" &&
memory.revision === 1 &&
memory.kind === plan.memory_kind &&
same(memory.scopes, plan.scopes ?? []) &&
same(memory.evidence_ids, plan.evidence_ids ?? []) &&
memory.content === content
);
}
if (
memory.memory_id !== plan.memory_id ||
memory.revision !== (plan.expected_revision as number) + 1 ||
memory.origin !== plan.expected_origin
) {
return false;
}
if (plan.operation === "delete" && memory.status !== "deleted") return false;
if (plan.operation === "correct" && memory.status !== plan.expected_status) {
return false;
}
if (plan.operation === "correct" && memory.content !== content) return false;
const expectedScopes =
plan.scopes ?? (plan.current?.scopes as string[] | undefined);
return expectedScopes ? same(memory.scopes, expectedScopes) : true;
}
function validateMutationReceipt(
value: unknown,
plan: MemoryMutationPlan,
): Record<string, unknown> | "not_found" | null {
const receipt = record(value);
if (
!receipt ||
Object.keys(receipt).length !== MUTATION_RECEIPT_KEYS.size ||
Object.keys(receipt).some((key) => !MUTATION_RECEIPT_KEYS.has(key)) ||
receipt.schema_version !== 2 ||
receipt.request_id !== plan.request_id
) {
return null;
}
if (receipt.state === "not_found") {
return [
receipt.operation,
receipt.memory_id,
receipt.result_revision,
receipt.result_status,
receipt.memory_state,
receipt.migrated_from,
].every((item) => item === null)
? "not_found"
: null;
}
const expectedOperation =
plan.operation === "add" ? "create" : plan.operation;
const expectedRevision =
plan.operation === "add" ? 1 : (plan.expected_revision as number) + 1;
const expectedStatus = plan.operation === "delete" ? "deleted" : "confirmed";
// A migrated receipt records an operation from before v2; no plan made
// today can have produced it.
if (
receipt.state !== "committed" ||
receipt.migrated_from !== null ||
receipt.operation !== expectedOperation ||
typeof receipt.memory_id !== "string" ||
!isMemoryId(receipt.memory_id) ||
(plan.memory_id !== undefined && receipt.memory_id !== plan.memory_id) ||
receipt.result_revision !== expectedRevision ||
receipt.result_status !== expectedStatus ||
!["present", "deleted"].includes(String(receipt.memory_state))
) {
return null;
}
return {
schema_version: 2,
request_id: plan.request_id,
state: "committed",
operation: expectedOperation,
memory_id: receipt.memory_id,
result_revision: expectedRevision,
result_status: expectedStatus,
memory_state: receipt.memory_state,
proof_scope: "governance_transaction_only",
downstream_sync_verified: false,
payload_fingerprint_verified: false,
};
}
function validatePlan(value: unknown): MemoryMutationPlan | null {
const plan = record(value);
if (
plan?.schema_version !== 1 ||
plan.kind !== "memory_mutation" ||
!oneOf(plan.operation, OPERATIONS) ||
typeof plan.request_id !== "string" ||
!UUID.test(plan.request_id) ||
typeof plan.digest !== "string"
) {
return null;
}
const body = { ...plan };
delete body.digest;
const expected = withDigest(body as unknown as PlanBody);
if (!safeEqual(expected.digest, plan.digest)) return null;
if (plan.operation === "add") {
if (
!oneOf(plan.memory_kind, MEMORY_KINDS) ||
!validScopes(plan.scopes) ||
!validIds(plan.evidence_ids, 500) ||
!contentFields(plan)
) {
return null;
}
} else if (
typeof plan.memory_id !== "string" ||
!isMemoryId(plan.memory_id) ||
!Number.isSafeInteger(plan.expected_revision) ||
(plan.expected_revision as number) < 1 ||
plan.expected_status !== "confirmed" ||
!oneOf(plan.expected_origin, MEMORY_ORIGINS) ||
!validCurrent(plan) ||
(plan.operation === "correct" && !contentFields(plan))
) {
return null;
}
return plan as unknown as MemoryMutationPlan;
}
function contentFields(plan: Record<string, unknown>): boolean {
return (
typeof plan.content_sha256 === "string" &&
/^[0-9a-f]{64}$/.test(plan.content_sha256) &&
Number.isSafeInteger(plan.content_characters) &&
(plan.content_characters as number) >= 1 &&
(plan.content_characters as number) <= CONTENT_MAX_CHARACTERS
);
}
function validCurrent(plan: Record<string, unknown>): boolean {
const current = record(plan.current);
if (
!current ||
current.status !== plan.expected_status ||
current.origin !== plan.expected_origin ||
current.revision !== plan.expected_revision ||
!validScopes(current.scopes) ||
!Array.isArray(current.evidence) ||
current.evidence.length < 1 ||
current.evidence.length > 500
) {
return false;
}
return current.evidence.every((value) => {
const evidence = record(value);
return (
evidence !== null &&
typeof evidence.evidence_id === "string" &&
ID.test(evidence.evidence_id) &&
typeof evidence.available === "boolean" &&
(evidence.source_id === undefined ||
(typeof evidence.source_id === "string" &&
evidence.source_id.length <= 128)) &&
(evidence.written_by === undefined ||
(typeof evidence.written_by === "string" &&
evidence.written_by.length <= 256))
);
});
}
function contentIdentity(content: string): {
content_sha256: string;
content_characters: number;
} {
return {
content_sha256: createHash("sha256").update(content, "utf8").digest("hex"),
content_characters: [...content].length,
};
}
function validContent(content: string): boolean {
return (
content.trim().length > 0 &&
[...content].length <= CONTENT_MAX_CHARACTERS &&
![...content].some((character) => {
const code = character.charCodeAt(0);
return code <= 8 || (code >= 11 && code <= 31) || code === 127;
})
);
}
function validScopes(value: unknown): value is string[] {
if (
!Array.isArray(value) ||
value.length < 1 ||
value.length > 128 ||
new Set(value).size !== value.length ||
value.some(
(scope) =>
typeof scope !== "string" ||
scope.trim() !== scope ||
scope.length < 1 ||
scope.length > 512 ||
scope === "*",
)
) {
return false;
}
return !value.includes("private") || value.length === 1;
}
function validIds(value: unknown, maximum: number): value is string[] {
return (
Array.isArray(value) &&
value.length >= 1 &&
value.length <= maximum &&
new Set(value).size === value.length &&
value.every((item) => typeof item === "string" && ID.test(item))
);
}
function withDigest(body: PlanBody): MemoryMutationPlan {
return {
...body,
digest: createHash("sha256")
.update(JSON.stringify(body), "utf8")
.digest("hex"),
};
}
function preview(plan: MemoryMutationPlan): MemoryFlowDocument {
return {
schema_version: 1,
product: PRODUCT,
state: "preview",
plan,
};
}
function invalid(code: string): MemoryFlowDocument {
return {
schema_version: 1,
product: PRODUCT,
state: "invalid",
code,
};
}
function pending(operationId: string, code: string): MemoryFlowDocument {
return {
schema_version: 1,
product: PRODUCT,
state: "unknown",
operation_id: operationId,
code,
};
}
function same(left: readonly string[], right: readonly string[]): boolean {
return (
left.length === right.length &&
left.every((value, index) => value === right[index])
);
}
function safeEqual(left: string, right: string): boolean {
if (left.length !== right.length) return false;
return timingSafeEqual(Buffer.from(left), Buffer.from(right));
}
function oneOf<const T extends readonly string[]>(
value: unknown,
values: T,
): value is T[number] {
return typeof value === "string" && values.includes(value as T[number]);
}
function record(value: unknown): Record<string, unknown> | null {
return value !== null && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: null;
}
@@ -1,211 +0,0 @@
import { readFileSync } from "node:fs";
import { describe, expect, it } from "vitest";
import type { CoreReader } from "./core-client.js";
import {
planExistingMemoryMutation,
planMemoryAdd,
readMemoryMutationReceipt,
} from "./memory-flows.js";
// Written by a real Core and checked against it on the Core side, so these are
// the receipts the CLI meets in use rather than ones shaped to fit the parser.
const FIXTURE = JSON.parse(
readFileSync(
new URL(
"../../../packages/contracts/fixtures/memory-mutation-receipts.v2.json",
import.meta.url,
),
"utf8",
),
) as { receipts: Record<"add" | "correct" | "delete", Receipt> };
interface Receipt {
schema_version: number;
request_id: string;
memory_id: string;
result_revision: number;
result_status: string;
operation: string;
}
function reader(data: unknown): CoreReader {
return async <T>() => ({ state: "ok", data: data as T });
}
function detail(sample: Receipt) {
const revision = sample.result_revision - 1;
return {
memory: {
memory_id: sample.memory_id,
user_id: "local",
kind: "chat",
status: "confirmed",
content: "Synthetic memory.",
scopes: ["all-apps"],
evidence_ids: ["evidence-1"],
origin: "manual",
tags: [],
revision,
created_at: "2026-09-19T00:00:00Z",
updated_at: "2026-09-19T00:00:01Z",
deleted_at: null,
},
evidence: [
{
evidence_id: "evidence-1",
available: true,
source_id: "fixture",
written_by: "person",
created_at: "2026-09-19T00:00:00Z",
verbatim: "Synthetic source text.",
},
],
history: [
{
revision,
action: "create",
status: "confirmed",
content: "Synthetic memory.",
scopes: ["all-apps"],
created_at: "2026-09-19T00:00:01Z",
},
],
index_sync: null,
};
}
describe("memory mutation receipt v2 from Core", () => {
it("recovers an add from the receipt Core returns for a create", async () => {
const sample = FIXTURE.receipts.add;
expect(sample.schema_version).toBe(2);
const preview = planMemoryAdd({
content: "Synthetic memory.",
memoryKind: "chat",
scopes: ["all-apps"],
evidenceIds: ["evidence-1"],
});
if (preview.state !== "preview") throw new Error("preview expected");
// The request id is the one binding a receipt to its plan; the plan picks it.
const result = await readMemoryMutationReceipt(
reader({ ...sample, request_id: preview.plan.request_id }),
preview.plan,
);
expect(result).toMatchObject({
state: "recovered",
receipt: {
schema_version: 2,
operation: "create",
memory_id: sample.memory_id,
result_revision: 1,
result_status: "confirmed",
memory_state: "present",
},
});
});
it.each<["correct" | "delete", { content?: string }]>([
["correct", { content: "Corrected synthetic memory." }],
["delete", {}],
])(
"recovers %s from the receipt Core returns for it",
async (operation, extra) => {
const sample = FIXTURE.receipts[operation];
const preview = await planExistingMemoryMutation(reader(detail(sample)), {
operation,
memoryId: sample.memory_id,
...extra,
});
if (preview.state !== "preview") {
throw new Error(`preview expected: ${JSON.stringify(preview)}`);
}
const result = await readMemoryMutationReceipt(
reader({ ...sample, request_id: preview.plan.request_id }),
preview.plan,
);
expect(result).toMatchObject({
state: "recovered",
receipt: {
schema_version: 2,
operation: sample.operation,
memory_id: sample.memory_id,
result_revision: sample.result_revision,
result_status: sample.result_status,
},
});
},
);
it("does not take a v1 receipt for a v2 one", async () => {
const sample = FIXTURE.receipts.delete;
const preview = await planExistingMemoryMutation(reader(detail(sample)), {
operation: "delete",
memoryId: sample.memory_id,
});
if (preview.state !== "preview") throw new Error("preview expected");
const { migrated_from: _dropped, ...v1 } = sample as Receipt & {
migrated_from: null;
};
// As v1 was, and with v2's fields but v1's number: either alone is refused.
for (const receipt of [
{ ...v1, schema_version: 1 },
{ ...sample, schema_version: 1 },
]) {
const result = await readMemoryMutationReceipt(
reader({ ...receipt, request_id: preview.plan.request_id }),
preview.plan,
);
expect(result).toMatchObject({
state: "unknown",
code: "memory_mutation_receipt_invalid",
});
}
});
it("does not take a not-found receipt that names a migrated operation", async () => {
const sample = FIXTURE.receipts.delete;
const preview = await planExistingMemoryMutation(reader(detail(sample)), {
operation: "delete",
memoryId: sample.memory_id,
});
if (preview.state !== "preview") throw new Error("preview expected");
const notFound = {
schema_version: 2,
request_id: preview.plan.request_id,
state: "not_found",
operation: null,
memory_id: null,
result_revision: null,
result_status: null,
memory_state: null,
};
expect(
await readMemoryMutationReceipt(
reader({ ...notFound, migrated_from: null }),
preview.plan,
),
).toMatchObject({
code: "memory_mutation_receipt_not_found_after_attempt",
});
expect(
await readMemoryMutationReceipt(
reader({ ...notFound, migrated_from: "reject" }),
preview.plan,
),
).toMatchObject({ code: "memory_mutation_receipt_invalid" });
});
it("plans nothing on a memory that is not in effect", async () => {
const sample = FIXTURE.receipts.delete;
const superseded = detail(sample);
superseded.memory.status = "superseded";
superseded.history[0].status = "superseded";
const result = await planExistingMemoryMutation(reader(superseded), {
operation: "delete",
memoryId: sample.memory_id,
});
expect(result).toMatchObject({
state: "invalid",
code: "memory_operation_not_allowed",
});
});
});
+32 -3
View File
@@ -15,6 +15,29 @@ export function isMemoryId(value: string): boolean {
);
}
const CONTENT_MAX_CHARACTERS = 100_000;
/** A memory body read from `--content-file`: UTF-8, non-blank, bounded, and
* free of control characters other than tab and newline. */
export function parseMemoryContent(bytes: Buffer): string | null {
if (bytes.byteLength > CONTENT_MAX_CHARACTERS * 4 + 2) return null;
let decoded: string;
try {
decoded = new TextDecoder("utf-8", { fatal: true }).decode(bytes);
} catch {
return null;
}
const content = decoded.replace(/\r?\n$/, "");
return content.trim().length > 0 &&
[...content].length <= CONTENT_MAX_CHARACTERS &&
![...content].some((character) => {
const code = character.charCodeAt(0);
return code <= 8 || (code >= 11 && code <= 31) || code === 127;
})
? content
: null;
}
export interface MemoryQuery {
query?: string;
status?: string;
@@ -129,7 +152,7 @@ export function renderMemoryPageText(document: MemoryPageDocument): string {
];
for (const memory of document.memories) {
lines.push(
`- ${memory.memory_id} ${memory.status}/${memory.layer} source=${memory.source ?? "未知"}`,
`- ${memory.memory_id} [${memory.scene}] ${memory.status}/${memory.layer} source=${memory.source ?? "未知"}`,
` ${memory.content}${memory.content_truncated ? ` …(共 ${memory.content_characters} 字)` : ""}`,
);
}
@@ -148,6 +171,7 @@ export function renderMemoryDetailText(document: MemoryDetailDocument): string {
}
const lines = [
`记忆 ${memory.memory_id}`,
`场景:${memory.scene} 类型:${memory.kind}${memory.variant ? `(${memory.variant})` : ""}`,
`状态:${memory.status} 层:${memory.layer} 来源:${memory.source ?? "未知"}`,
`owner:${memory.owner ?? "未知"} latest:${memory.is_latest}`,
`raw IDs:${(memory.source_raw_ids as string[]).join(",") || "无"}`,
@@ -165,6 +189,10 @@ function describeMemory(memory: SdkMemoryRecord): Record<string, unknown> {
const budgeted = budget(memory.content ?? "");
return {
memory_id: memory.memory_id,
// Pre-2.0.6 Core listed chat memories only and named no scene.
scene: memory.scene ?? "chat",
kind: memory.kind ?? "chat",
variant: memory.variant ?? null,
status: memory.status,
layer: memory.layer,
is_latest: memory.is_latest,
@@ -208,8 +236,9 @@ function sdkMemory(value: unknown): SdkMemoryRecord | null {
!strings(value.superseded_by) ||
!strings(value.tags) ||
!strings(value.source_raw_ids) ||
!text(value.user_id) ||
!text(value.agent_id)
// Working/coding memories may carry an empty user or agent id.
typeof value.user_id !== "string" ||
typeof value.agent_id !== "string"
)
return null;
return value as unknown as SdkMemoryRecord;
+99 -16
View File
@@ -184,8 +184,16 @@ const CORE = new Map<string, { status: number; data: unknown }>([
},
],
[
"/api/v1/home-summary",
{ status: 200, data: { memory_count: 42, latest: null, observed_at: "T" } },
"/api/v2/home-summary",
{
status: 200,
data: {
schema_version: 2,
memory_count: 42,
latest: null,
observed_at: "T",
},
},
],
]);
@@ -194,9 +202,7 @@ const READ_ONLY_PATHS = [
"/api/v1/live-sync",
"/api/v1/memory-processing",
"/api/v1/model-profiles",
"/api/v1/home-summary",
"/api/v1/memory-library",
"/api/v1/memories/",
"/api/v2/home-summary",
"/api/v2/memory-library",
"/api/v2/memories/",
"/api/v2/user-profile",
@@ -691,6 +697,66 @@ describe("CLI-21 memory reads", () => {
expect(report.memories[1].content.length).toBeLessThan(400);
});
it("lists working and coding memories with their scene and kind", async () => {
transport.get.mockResolvedValue({
status: 200,
data: {
total: 2,
offset: 0,
limit: 20,
totals: { chat: 1, working: 1, coding: 0 },
window_id: null,
window_reset: false,
memories: [
memory("m1", "对话记忆"),
{
...memory("e1", "修复测试\n固定随机种子"),
layer: "experience",
agent_id: "",
owner: null,
source_raw_ids: [],
scene: "working",
kind: "experience",
variant: "working",
task: "修复测试",
solution: "固定随机种子",
created_at: 1_790_000_000,
updated_at: 1_790_000_100,
},
],
},
});
const streams = captureStreams();
try {
expect(await run(["memories", "list", "--json"], {})).toBe(0);
} finally {
streams.restore();
}
const report = JSON.parse(streams.stdout());
expect(report.page).toMatchObject({ state: "ok", total: 2, returned: 2 });
expect(report.memories[0]).toMatchObject({
memory_id: "m1",
scene: "chat",
kind: "chat",
variant: null,
});
expect(report.memories[1]).toMatchObject({
memory_id: "e1",
scene: "working",
kind: "experience",
variant: "working",
layer: "experience",
});
const text = captureStreams();
try {
expect(await run(["memories", "list"], {})).toBe(0);
} finally {
text.restore();
}
expect(text.stdout()).toContain("- e1 [working] active/experience");
});
it("passes the requested filters through as query parameters", async () => {
await run(
[
@@ -816,7 +882,7 @@ describe("CLI-21 memory reads", () => {
expect(report.history).toEqual([]);
});
// Core registers /api/v1/memories/graph before /{memory_id}, and that static
// Core registers /api/v2/memories/graph before /{memory_id}, and that static
// handler initializes the HY bridge. A memory id that collides with a
// reserved segment must never become a request.
it("refuses a memory id that collides with a static Core route", async () => {
@@ -830,18 +896,15 @@ describe("CLI-21 memory reads", () => {
expect(streams.stdout()).toBe("");
}
expect(transport.get).not.toHaveBeenCalled();
expect(isReadOnlyPath("/api/v1/memories/graph")).toBe(false);
expect(isReadOnlyPath("/api/v1/memories/ask")).toBe(false);
expect(isReadOnlyPath("/api/v1/memories/m1")).toBe(true);
expect(isReadOnlyPath("/api/v2/memory-library")).toBe(true);
expect(isReadOnlyPath("/api/v2/memories/graph")).toBe(false);
expect(isReadOnlyPath("/api/v2/memories/ask")).toBe(false);
expect(isReadOnlyPath("/api/v2/memories/m1")).toBe(true);
expect(isReadOnlyPath("/api/v2/memory-library")).toBe(true);
expect(isReadOnlyPath("/api/v2/home-summary")).toBe(true);
// The retired v1 data plane is no longer a CLI read.
expect(isReadOnlyPath("/api/v1/memories/m1")).toBe(false);
expect(isReadOnlyPath("/api/v1/home-summary")).toBe(false);
expect(isReadOnlyPath("/api/v1/model-content-permission")).toBe(false);
expect(
isReadOnlyPath(
"/api/v1/memory-mutations/123e4567-e89b-12d3-a456-426614174000",
),
).toBe(true);
expect(isReadOnlyPath("/api/v1/memory-mutations/../sources")).toBe(false);
});
it("refuses a memory id that would escape the memories path", async () => {
@@ -861,6 +924,26 @@ describe("CLI-21 memory reads", () => {
// was cast to the expected type, so a wrong record, an unavailable excerpt, an
// oversized page and a non-string model field all reached stdout as success.
describe("malformed success responses", () => {
it("treats a v1-shaped home summary as invalid, not as a memory count", async () => {
transport.get.mockImplementation(async (url: string) =>
url === "/api/v2/home-summary"
? {
status: 200,
data: { memory_count: 7, latest: null, observed_at: "T" },
}
: (CORE.get(url) ?? { status: 404, data: { detail: "not_found" } }),
);
const streams = captureStreams();
try {
await run(["status", "--json"], {});
} finally {
streams.restore();
}
expect(JSON.parse(streams.stdout()).sections.memories).toMatchObject({
state: "invalid",
});
});
it("refuses a detail that is not the memory that was asked for", async () => {
transport.get.mockResolvedValue({
status: 200,
@@ -315,6 +315,40 @@ describe("macOS service lifecycle", () => {
expect(deps.verifyRegistrationSnapshot).toHaveBeenCalledTimes(2);
});
it("does not re-bootstrap a previous registration that may serve a legacy home", async () => {
const f = fixture();
const oldPath = "/Users/example/Library/LaunchAgents/old-runtime.plist";
const runCommand = vi
.fn()
.mockResolvedValueOnce(undefined)
.mockRejectedValueOnce(new Error("bootstrap failed"));
const deps = dependencies({
inspectRegistration: vi.fn(async () => ({
kind: "managed" as const,
plistPath: oldPath,
content: "old managed plist",
})),
runCommand,
});
const prepared = await prepareServiceInstall(f.layout, f.options, deps);
await expect(
applyPreparedServiceInstall(
{ ...prepared, restorePreviousOnFailure: false },
deps,
),
).rejects.toThrow("runtime_service_replacement_failed");
// bootout + the failed bootstrap only; the old plist is not bootstrapped.
expect(runCommand).toHaveBeenCalledTimes(2);
expect(runCommand).not.toHaveBeenCalledWith([
"launchctl",
"bootstrap",
"gui/501",
oldPath,
]);
});
it("removes the replacement plist and restores the previous registration", async () => {
const f = fixture();
const oldPath = "/Users/example/Library/LaunchAgents/old-runtime.plist";
+16 -1
View File
@@ -72,6 +72,13 @@ export type PreparedServiceInstall = {
mode: "direct" | "bootstrap-existing" | "restart" | "replace";
previous?: ManagedSnapshot;
previouslyEnabled: boolean;
/**
* Re-bootstrap the previous (possibly older) LaunchAgent when the new one
* fails to bootstrap. `install` turns this off once the home has been handed
* to this version's SDK migration: an older Runtime has no migration gate and
* would serve a home that is not on SDK authority.
*/
restorePreviousOnFailure?: boolean;
};
const defaultDependencies: ServiceLifecycleDependencies = {
@@ -191,7 +198,8 @@ export async function applyPreparedServiceInstall(
if (!previous) throw new Error("runtime_service_upgrade_state_invalid");
await verifyPreparedRegistration(prepared, dependencies);
const domain = launchdDomain(options.uid);
let replacementFailure: "upgrade" | "rollback" | null = null;
let replacementFailure: "upgrade" | "rollback" | "not-restored" | null =
null;
const replaceCommand: ServiceCommandRunner = async (command) => {
if (command[0] !== "launchctl" || command[1] !== "bootstrap") {
await dependencies.runCommand(command);
@@ -208,6 +216,10 @@ export async function applyPreparedServiceInstall(
await dependencies.runCommand(command);
} catch {
replacementFailure = "upgrade";
if (prepared.restorePreviousOnFailure === false) {
replacementFailure = "not-restored";
throw new Error("runtime_service_replacement_failed");
}
try {
await dependencies.verifyRegistrationSnapshot(previous);
await dependencies.runCommand([
@@ -234,6 +246,9 @@ export async function applyPreparedServiceInstall(
if (replacementFailure === "upgrade") {
throw new Error("runtime_service_managed_upgrade_failed");
}
if (replacementFailure === "not-restored") {
throw new Error("runtime_service_replacement_failed");
}
throw error;
}
} catch (error) {
+4 -35
View File
@@ -7,12 +7,10 @@ import type {
} from "@one-memory/contracts";
import type { CoreReader, ReadOutcome } from "./core-client.js";
import {
type ValidatedMemoryTask,
validateImportIssues,
validateImportJob,
validateImportJobs,
validateLiveSync,
validateMemoryTasks,
validateProcessing,
validateSources,
} from "./core-responses.js";
@@ -70,11 +68,6 @@ export interface ProgressDocument {
generation: {
summary_state: ReadState;
summary: Record<string, unknown> | null;
tasks_state: ReadState;
tasks: Array<Record<string, unknown>>;
returned: number;
validated: number;
output_truncated: boolean;
};
}
@@ -134,12 +127,13 @@ export async function readProgress(
const importsPath = options.jobId
? `/api/v1/import-jobs/${options.jobId}`
: "/api/v1/import-jobs";
const [importsRead, sourcesRead, processingRead, tasksRead, issuesRead] =
// Generation progress comes from `memory-processing` alone: under SDK
// authority the per-task list (`/api/v1/memory-tasks`) is retired and empty.
const [importsRead, sourcesRead, processingRead, issuesRead] =
await Promise.all([
reader<unknown>(importsPath),
reader<unknown>("/api/v1/sources"),
reader<unknown>("/api/v1/memory-processing"),
reader<unknown>("/api/v1/memory-tasks"),
options.jobId && options.issues
? reader<unknown>(`/api/v1/import-jobs/${options.jobId}/issues`)
: Promise.resolve(null),
@@ -153,7 +147,6 @@ export async function readProgress(
: validated(importsRead, validateImportJobs);
const processing = validated(processingRead, validateProcessing);
const sources = validated(sourcesRead, validateSources);
const tasks = validated(tasksRead, validateMemoryTasks);
const issues =
options.jobId && options.issues && issuesRead
? validated(issuesRead, (value) =>
@@ -162,25 +155,19 @@ export async function readProgress(
: null;
const jobs = imports.state === "ok" ? imports.data : [];
const taskItems = tasks.state === "ok" ? tasks.data : [];
const issueItems = issues?.state === "ok" ? issues.data : [];
const importState = stateOnly(imports);
const summaryState = stateOnly(processing);
const tasksState = stateOnly(tasks);
const hasReadFailure =
importState.state !== "ok" ||
sources.state !== "ok" ||
summaryState.state !== "ok" ||
tasksState.state !== "ok" ||
(issues !== null && issues.state !== "ok");
const hasProblem =
jobs.some((job) =>
["paused", "interrupted", "failed"].includes(job.status),
) ||
taskItems.some((task) =>
["waiting_for_model", "failed"].includes(task.status),
) ||
(processing.state === "ok" &&
(processing.data.failed > 0 ||
processing.data.waiting_for_model > 0 ||
@@ -188,7 +175,6 @@ export async function readProgress(
issueItems.length > 0;
const hasActive =
jobs.some((job) => ["pending", "running"].includes(job.status)) ||
taskItems.some((task) => ["pending", "running"].includes(task.status)) ||
(processing.state === "ok" &&
(processing.data.pending > 0 || processing.data.running > 0));
@@ -229,11 +215,6 @@ export async function readProgress(
summary_state: summaryState,
summary:
processing.state === "ok" ? projectProcessing(processing.data) : null,
tasks_state: tasksState,
tasks: taskItems.slice(0, options.limit).map(projectTask),
returned: Math.min(taskItems.length, options.limit),
validated: taskItems.length,
output_truncated: taskItems.length > options.limit,
},
};
}
@@ -289,10 +270,7 @@ export function renderProgressText(document: ProgressDocument): string {
? `记忆整理:待处理 ${document.generation.summary.pending} · 进行中 ${document.generation.summary.running} · 等待模型 ${document.generation.summary.waiting_for_model} · 失败 ${document.generation.summary.failed}`
: "记忆整理:暂时无法读取",
);
if (
document.imports.output_truncated ||
document.generation.output_truncated
) {
if (document.imports.output_truncated) {
lines.push("这里只显示汇总;完整记录请使用 --json。");
}
lines.push("");
@@ -353,11 +331,6 @@ export function unreadProgress(
generation: {
summary_state: failed,
summary: null,
tasks_state: failed,
tasks: [],
returned: 0,
validated: 0,
output_truncated: false,
},
};
}
@@ -465,7 +438,3 @@ function projectProcessing(
})),
};
}
function projectTask(task: ValidatedMemoryTask): Record<string, unknown> {
return { ...task };
}
+6 -48
View File
@@ -129,18 +129,6 @@ const RESPONSES = new Map<string, { status: number; data: unknown }>([
},
},
],
[
"/api/v1/memory-tasks",
{
status: 200,
data: [
memoryTask("task-pending", "pending"),
memoryTask("task-running", "running"),
memoryTask("task-model", "waiting_for_model", "model_missing"),
memoryTask("task-failed", "failed", "provider_unavailable"),
],
},
],
[
"/api/v1/import-jobs/job-running",
{ status: 200, data: importJob("job-running", "running") },
@@ -282,9 +270,9 @@ describe("CLI-19 progress and issue reads", () => {
imported_messages: 20,
totals_state: { state: "ok" },
});
expect(
report.generation.tasks.map((task: { status: string }) => task.status),
).toEqual(["pending", "running", "waiting_for_model", "failed"]);
// The retired per-task list is no longer read or reported.
expect(report.generation).not.toHaveProperty("tasks");
expect(requestedPaths()).not.toContain("/api/v1/memory-tasks");
expect(streams.stdout()).not.toMatch(/percent|percentage|%/i);
});
@@ -374,7 +362,7 @@ describe("CLI-19 progress and issue reads", () => {
expect(streams.stdout()).not.toContain("SYNTHETIC_FOREIGN_ISSUE");
});
it("bounds each task list and emits an explicit truncation receipt", async () => {
it("bounds the import job list and emits an explicit truncation receipt", async () => {
const streams = captureStreams();
try {
expect(await run(["progress", "--limit", "1", "--json"], {})).toBe(1);
@@ -388,12 +376,6 @@ describe("CLI-19 progress and issue reads", () => {
validated: 3,
output_truncated: true,
});
expect(report.generation.tasks).toHaveLength(1);
expect(report.generation).toMatchObject({
returned: 1,
validated: 4,
output_truncated: true,
});
});
it("streams bounded progress snapshots as serial NDJSON", async () => {
@@ -428,7 +410,6 @@ describe("CLI-19 progress and issue reads", () => {
"/api/v1/import-jobs",
"/api/v1/sources",
"/api/v1/memory-processing",
"/api/v1/memory-tasks",
]) {
expect(requestedPaths().filter((item) => item === path)).toHaveLength(2);
}
@@ -500,7 +481,7 @@ describe("CLI-19 progress and issue reads", () => {
it("preserves denied, unavailable and malformed sections in one partial report", async () => {
transport.get.mockImplementation(async (url: string) => {
if (url === "/api/v1/import-jobs") return { status: 503, data: {} };
if (url === "/api/v1/memory-tasks") return { status: 403, data: {} };
if (url === "/api/v1/sources") return { status: 403, data: {} };
if (url === "/api/v1/memory-processing") {
return { status: 200, data: { pending: { secret: "nested" } } };
}
@@ -521,7 +502,7 @@ describe("CLI-19 progress and issue reads", () => {
state: "invalid",
code: "response_invalid",
});
expect(report.generation.tasks_state).toEqual({
expect(report.imports.totals_state).toEqual({
state: "denied",
code: "http_403",
});
@@ -574,7 +555,6 @@ describe("CLI-19 progress and issue reads", () => {
"/api/v1/live-sync",
"/api/v1/import-jobs",
"/api/v1/memory-processing",
"/api/v1/memory-tasks",
].some((allowed) => path === allowed || path.startsWith(`${allowed}/`)),
),
).toBe(true);
@@ -634,28 +614,6 @@ function importJob(
};
}
function memoryTask(
taskId: string,
status: string,
error: string | null = null,
) {
return {
task_id: taskId,
request_id: `request-${taskId}`,
user_id: "local",
kind: "extract",
input_ref: "synthetic",
status,
profile_id: "profile-1",
profile_revision: 1,
profile_snapshot: {},
attempts: status === "failed" ? 2 : 0,
last_error_code: error,
created_at: "T1",
updated_at: "T2",
};
}
function requestedPaths(): string[] {
return (transport.get.mock.calls as [string][]).map(([path]) => path);
}
+43 -1
View File
@@ -1,6 +1,11 @@
import { describe, expect, it } from "vitest";
import { renderStatusText, type StatusDocument } from "./status-report.js";
import { composeDoctorReport, renderDoctorText } from "./doctor-report.js";
import {
composeStatusReport,
renderStatusText,
type StatusDocument,
} from "./status-report.js";
describe("status default output", () => {
it("formats blocked task objects recursively instead of printing object coercions", () => {
@@ -35,3 +40,40 @@ describe("status default output", () => {
);
});
});
describe("status and doctor on an unfinished upgrade migration", () => {
it("names the migration log and the issues page", async () => {
const report = await composeStatusReport({
dataDir: "/tmp/isolated/.one-memory",
runtime: { state: "stopped", code: "sdk_migration_required" },
background: null,
reader: null,
});
expect(report.runtime.log_file).toBe(
"/tmp/isolated/.one-memory/logs/sdk-migration.log",
);
expect(report.runtime.issues_url).toBe(
"https://github.com/Tencent-Hunyuan/Hy-Memory/issues/new",
);
const text = renderStatusText(report, "MemHub");
expect(text).toContain("sdk_migration_required");
expect(text).toContain("/tmp/isolated/.one-memory/logs/sdk-migration.log");
const doctor = renderDoctorText(composeDoctorReport(report, null));
expect(doctor).toContain(
"/tmp/isolated/.one-memory/logs/sdk-migration.log",
);
expect(doctor).not.toContain("帮助与诊断");
});
it("adds no log path for an ordinary stopped runtime", async () => {
const report = await composeStatusReport({
dataDir: "/tmp/isolated/.one-memory",
runtime: { state: "stopped", code: "runtime_not_running" },
background: null,
reader: null,
});
expect(report.runtime.log_file).toBeUndefined();
});
});
+23 -3
View File
@@ -1,5 +1,4 @@
import type {
HomeSummary,
LiveSyncHealth,
MemoryProcessingSummary,
ModelProfile,
@@ -9,8 +8,10 @@ import type {
BackgroundHealth,
RuntimeStatus,
} from "@one-memory/runtime-control";
import { sdkMigrationLogPath } from "@one-memory/runtime-control";
import type { CoreReader, ReadOutcome } from "./core-client.js";
import {
type SdkHomeSummary,
validateHomeSummary,
validateLiveSync,
validateModelProfiles,
@@ -19,6 +20,9 @@ import {
} from "./core-responses.js";
import { PRODUCT } from "./version.js";
const MIGRATION_ISSUES_URL =
"https://github.com/Tencent-Hunyuan/Hy-Memory/issues/new";
export type SectionState =
| "ok"
| "denied"
@@ -42,6 +46,9 @@ export interface StatusDocument {
host?: string;
port?: number;
version?: string;
/** Set only when an unfinished upgrade migration is why nothing serves. */
log_file?: string;
issues_url?: string;
};
bridge: {
state: BackgroundHealth["state"] | "not_checked";
@@ -102,7 +109,7 @@ export async function composeStatusReport(options: {
reader<unknown>("/api/v1/live-sync"),
reader<unknown>("/api/v1/memory-processing"),
reader<unknown>("/api/v1/model-profiles"),
reader<unknown>("/api/v1/home-summary"),
reader<unknown>("/api/v2/home-summary"),
]);
sections.sources = section(sources, validateSources, describeSources);
sections.sync = section(sync, validateLiveSync, describeSync);
@@ -153,6 +160,12 @@ export async function composeStatusReport(options: {
host: options.runtime.host,
port: options.runtime.port,
version: options.runtime.version,
...(options.runtime.code.startsWith("sdk_migration_")
? {
log_file: sdkMigrationLogPath(options.dataDir),
issues_url: MIGRATION_ISSUES_URL,
}
: {}),
},
bridge,
sections,
@@ -177,6 +190,13 @@ export function renderStatusText(
): string {
const lines = [
`${heading}:${runtimeLine(report)}`,
...(report.runtime.log_file
? [
`升级迁移没有完成(错误码:${report.runtime.code}),MemHub 不会启动。`,
`请把迁移日志附在 GitHub 反馈里发给我们:${report.runtime.log_file}`,
`反馈地址:${report.runtime.issues_url}`,
]
: []),
`资料空间:${report.data_dir}`,
`应用连接:${BRIDGE_LABELS[report.bridge.state]}`,
];
@@ -347,7 +367,7 @@ function describeModel(profiles: ModelProfile[]): Record<string, unknown> {
};
}
function describeMemories(summary: HomeSummary): Record<string, unknown> {
function describeMemories(summary: SdkHomeSummary): Record<string, unknown> {
return {
memory_count: summary.memory_count,
};
+4 -3
View File
@@ -4,7 +4,7 @@ build-backend = "uv_build"
[project]
name = "one-memory"
version = "1.0.4"
version = "1.0.5"
description = "MemHub local-first memory core"
readme = "README.md"
requires-python = ">=3.12"
@@ -12,8 +12,9 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.5.dev3",
# Retrieval's derived vector index directly consumes these locked versions.
"hy-memory==2.0.6",
# chromadb arrives through the HY SDK; locked here so an unrelated upgrade
# cannot move the SDK's store. Built-in embedding consumes numpy directly.
"chromadb==1.5.9",
"numpy==2.5.2",
"httpx>=0.28.1",
@@ -42,12 +42,10 @@ TARGETED = (
"tests/test_database_contention.py",
"tests/test_main_database_fairness.py",
"tests/test_soak_probe_effectiveness.py",
"tests/test_evidence_reconcile_primitive.py",
"tests/test_sqlite_semantics_gate.py",
"tests/test_store_binding.py",
"tests/test_store_binding_ratchet.py",
"tests/test_main_database_entry.py",
"tests/test_memory_processing.py",
"tests/test_extraction_window.py",
"tests/test_extraction_defer.py",
"tests/test_active_memory_window.py",
@@ -2,11 +2,5 @@
import os
from one_memory.hy_runtime_config import configure_hy_behavior_environment
# HY has a few reader switches evaluated at module import time. Fix the
# product-owned, non-secret behavior before any one_memory module imports HY.
configure_hy_behavior_environment()
_version = os.environ.get("ONE_MEMORY_VERSION", "").strip()
__version__ = _version if _version else "development"
File diff suppressed because it is too large. Load diff
@@ -15,7 +15,7 @@ from one_memory.call_event_store import (
CallTool,
HostId,
)
from one_memory.context_models import ContextSearchResponse, MemoryContextItem, RawContextItem
from one_memory.context_models import SdkContextSearchResponse
from one_memory.external_memory_access import ExternalMemoryAccess
from one_memory.recall import RecallReceipt
@@ -41,75 +41,27 @@ def _host_id(access: ExternalMemoryAccess | None) -> HostId | None:
return cast(HostId, host) if host in HOSTS else None
def context_call_event(
result: ContextSearchResponse,
def sdk_context_call_event(
result: SdkContextSearchResponse,
access: ExternalMemoryAccess | None,
*,
memory_searched: bool,
timed_out: bool,
) -> CallEvent:
references: list[CallReference] = []
seen: set[tuple[str, str, int]] = set()
for item in result.items:
if isinstance(item, RawContextItem):
candidates = [
CallReference(
kind="raw",
id=message.evidence_id,
revision=message.revision,
source_id=item.source_id,
)
for message in item.messages
]
elif isinstance(item, MemoryContextItem):
candidates = [
CallReference(
kind="memory", id=item.memory_id, revision=item.revision, source_id=None
)
]
else:
continue
for reference in candidates:
identity = (reference.kind, reference.id, reference.revision)
if identity not in seen:
seen.add(identity)
references.append(reference)
timed_out = any(
path.reason == "timeout_budget" for path in (result.paths.raw, result.paths.memory)
# SDK items carry no Evidence/Governance revision, and the reference
# resolver reads the legacy store, so the event records counts only.
raw_count = sum(item.kind == "raw" for item in result.items)
memory_count = sum(item.kind == "memory" for item in result.items)
state: CallState = "timed_out" if timed_out else result.state
return _metadata_event(
tool="om_recall" if memory_searched else "om_conversations",
state=state,
elapsed_ms=result.total_ms,
access=access,
raw_count=raw_count,
memory_count=memory_count,
event_id=UUID(result.request_id),
)
state: CallState = (
"partial" if timed_out and references else "timed_out" if timed_out else result.state
)
return CallEvent.model_validate(
{
"event_id": UUID(result.request_id),
"occurred_at": datetime.now(UTC),
"host_id": _host_id(access),
"installation_id": UUID(access.installation_id) if access else None,
"tool": _context_tool(result),
"state": state,
"stage": "core_result",
"delivery": "unknown",
"adoption": "unknown",
"host_session_id": None,
"injection_elapsed_ms": None,
"elapsed_ms": result.total_ms,
"raw_count": sum(isinstance(item, RawContextItem) for item in result.items),
"memory_count": sum(isinstance(item, MemoryContextItem) for item in result.items),
"references": tuple(references),
}
)
def record_context_result(
store: CallEventStore,
result: ContextSearchResponse,
access: ExternalMemoryAccess | None,
) -> None:
store.record(context_call_event(result, access))
def _context_tool(result: ContextSearchResponse) -> CallTool:
if result.paths.memory.status not in {"not_requested", "not_authorized"}:
return "om_recall"
return "om_conversations"
def _metadata_event(
@@ -142,26 +94,6 @@ def _metadata_event(
)
def conversation_call_event(
result: object,
access: ExternalMemoryAccess | None,
elapsed_ms: float,
) -> CallEvent:
hits = tuple(getattr(result, "hits", ()) or ())
conversations = {
conversation_id
for hit in hits
if (conversation_id := getattr(hit, "conversation_id", None))
}
return _metadata_event(
tool="om_conversations",
state="grounded" if hits else "empty",
elapsed_ms=elapsed_ms,
access=access,
raw_count=min(len(conversations), 20),
)
def ask_call_event(
result: object,
access: ExternalMemoryAccess | None,
@@ -255,8 +187,9 @@ def record_legacy_result(
host_session_id=None,
injection_elapsed_ms=None,
elapsed_ms=elapsed_ms,
raw_count=0,
memory_count=len(receipt.hits),
# A raw-fallback recall returned conversation excerpts, not memories.
raw_count=len(receipt.hits) if receipt.mode == "raw" else 0,
memory_count=0 if receipt.mode == "raw" else len(receipt.hits),
references=(),
)
)
File diff suppressed because it is too large. Load diff
@@ -23,10 +23,6 @@ class Settings(BaseSettings):
live_sync_enabled: bool = True
automatic_memory_processing: bool = True
imports_enabled: bool = True
# Experimental and unreleased: ownership/error shutdown still requires acceptance.
# Resume only explicitly initialized generations; never migrate at startup.
retrieval_index_enabled: bool = False
retrieval_minimum_free_bytes: int = Field(default=256 * 1024 * 1024, ge=0)
# Bounded metadata only: query, original text and answers are never stored.
call_event_retention_days: int | None = Field(default=30, ge=1, le=365)
call_event_max_events: int | None = Field(default=10_000, ge=1, le=100_000)
@@ -47,6 +47,19 @@ class ContextSearchRequest(BaseModel):
return self
class SdkContextSearchRequest(ContextSearchRequest):
"""`POST /api/v2/context/search`: the three SDK recall lanes are set apart.
`limit` is the normal lane's top-k and the cap on returned items;
`schema_limit` and `intention_limit` are the SDK's schema (L6) and
proactive intention (L7) lanes. A caller's values are passed through.
"""
limit: int = Field(default=10, ge=1, le=20, strict=True)
schema_limit: int = Field(default=3, ge=0, le=20, strict=True)
intention_limit: int = Field(default=1, ge=0, le=20, strict=True)
class ContextMessage(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
+72 -418
View File
@@ -1,7 +1,6 @@
"""Authenticated composite read route, without generation or reindex side effects."""
"""Authenticated SDK context search route, without generation or reindex side effects."""
import asyncio
import sqlite3
import time
from collections.abc import Sequence
from datetime import UTC, datetime
@@ -11,421 +10,26 @@ from uuid import uuid4
from fastapi import APIRouter, HTTPException, Request, Response
from fastapi.params import Depends
from one_memory.background_work import (
in_interactive_read,
in_read_pool,
memory_read_pool,
raw_read_pool,
)
from one_memory.call_event_recorder import context_call_event
from one_memory.call_event_recorder import sdk_context_call_event
from one_memory.context_access import resolve_context_authority
from one_memory.context_memory import (
merge_memory_candidates,
read_memory_context,
revalidate_memory_context,
)
from one_memory.context_models import (
ContextItem,
ContextKind,
ContextLaneResult,
ContextSearchRequest,
ContextSearchResponse,
ContextSemanticStatus,
MemoryContextItem,
RawContextItem,
SdkContextItem,
SdkContextSearchRequest,
SdkContextSearchResponse,
SdkMemoryContextItem,
SdkRawContextItem,
)
from one_memory.context_raw import revalidate_raw_context
from one_memory.context_search import search_context
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception
from one_memory.external_memory_access import external_access
from one_memory.hy_runtime_config import HY_MEMORY_READER
from one_memory.hy_sdk_runtime import sdk_relevance_floor
from one_memory.hy_sdk_types import (
HySdkMemoryRecord,
parse_sdk_raw_search,
parse_sdk_search,
)
from one_memory.imports.storage import EvidenceStore
from one_memory.retrieval.raw_lane import read_raw_indexed_context
from one_memory.semantic_deadline import hy_wait_budget, search_within
CALL_EVENT_RESPONSE_BUDGET_MS = 250
RESPONSE_SERIALIZATION_RESERVE_MS = 100
INTERACTIVE_TOTAL_BUDGET_MS = 900
RAW_LANE_BUDGET_SECONDS = 0.3
MEMORY_LANE_BUDGET_SECONDS = 0.75
MEMORY_FINALIZE_RESERVE_SECONDS = 0.1
def context_router(dependencies: Sequence[Depends]) -> APIRouter:
router = APIRouter(prefix="/api/v1/context", dependencies=dependencies)
@router.post("/search", response_model=ContextSearchResponse)
async def search(
value: ContextSearchRequest, request: Request, response: Response
) -> ContextSearchResponse:
response.headers["Cache-Control"] = "no-store"
started = time.monotonic()
search_budget_ms = min(
INTERACTIVE_TOTAL_BUDGET_MS,
max(50, value.timeout_ms - RESPONSE_SERIALIZATION_RESERVE_MS),
)
deadline = started + search_budget_ms / 1000
access = external_access(request.headers)
try:
database = cast(EvidenceStore, request.app.state.evidence_store).database_path
authority = await asyncio.wait_for(
in_interactive_read(
resolve_context_authority, value, access, database, deadline=deadline
),
timeout=max(0, deadline - time.monotonic()),
)
except HTTPException:
raise
except Exception:
# Authority could not be established; do not try an unbounded local lane.
raise HTTPException(503, "context_authority_unavailable") from None
lane_observations: dict[ContextKind, ContextLaneResult] = {}
lane_failures: dict[ContextKind, tuple[str, str]] = {}
stage_timings: dict[str, float] = {}
async def raw(end: float) -> ContextLaneResult:
lane_end = min(end, time.monotonic() + RAW_LANE_BUDGET_SECONDS)
read = asyncio.create_task(
in_read_pool(
raw_read_pool(),
read_raw_indexed_context,
database,
value,
source_ids=authority.raw_sources,
deadline=lane_end,
index=getattr(request.app.state, "retrieval_runtime", None),
)
)
done, _ = await asyncio.wait({read}, timeout=max(0, lane_end - time.monotonic()))
if read not in done:
read.cancel()
read.add_done_callback(
lambda task: task.exception() if not task.cancelled() else None
)
result = ContextLaneResult(
status="degraded",
reason="timeout_budget",
searched_channels=("lexical", "semantic"),
timed_out=True,
truncated_by=("timeout_budget",),
)
else:
result = read.result()
lane_observations["raw"] = result
return result
async def memory(end: float) -> ContextLaneResult:
lane_end = min(end, time.monotonic() + MEMORY_LANE_BUDGET_SECONDS)
target = getattr(request.app.state, "recall_target", None)
lexical = asyncio.create_task(
in_read_pool(
memory_read_pool(),
read_memory_context,
database,
value,
scopes=authority.memory_scopes,
deadline=lane_end,
semantic_ids=(),
semantic_outcome="absent",
)
)
semantic_hits, outcome = await search_within(
getattr(target, "semantic_search", None),
value.query,
hy_wait_budget(lane_end - time.monotonic()),
)
done, _ = await asyncio.wait({lexical}, timeout=max(0, lane_end - time.monotonic()))
if lexical not in done:
lexical.cancel()
lexical.add_done_callback(
lambda task: task.exception() if not task.cancelled() else None
)
result = ContextLaneResult(
status="unavailable",
reason="timeout_budget",
searched_channels=("lexical", "semantic"),
semantic_status=cast(
ContextSemanticStatus,
{
"absent": "not_searched",
"answered": "completed",
"failed": "failed",
"timed_out": "timed_out",
}[outcome],
),
timed_out=True,
truncated_by=("timeout_budget",),
)
lane_observations["memory"] = result
return result
try:
lexical_result = lexical.result()
except (sqlite3.Error, OSError, ValueError) as error:
diagnostic_event(
level="warn",
component="context.search",
event="context.lexical_memory_failed",
outcome="degraded",
error_code="lexical_memory_read_failed",
**diagnostic_exception(error),
)
lexical_result = ContextLaneResult(
status="unavailable",
reason="read_unavailable",
searched_channels=("lexical",),
failure_site="memory_lexical",
error_class=type(error).__name__,
)
if outcome == "absent":
result = lexical_result
elif outcome in {"failed", "timed_out"}:
truncated_by = list(lexical_result.truncated_by)
has_lexical_memory = bool(lexical_result.items)
if (
outcome == "timed_out"
and not has_lexical_memory
and "timeout_budget" not in truncated_by
):
truncated_by.append("timeout_budget")
result = lexical_result.model_copy(
update={
"status": "degraded",
"reason": "lexical_only" if has_lexical_memory else "timeout_budget",
"truncated_by": tuple(truncated_by),
"searched_channels": ("lexical", "semantic"),
"semantic_status": "timed_out" if outcome == "timed_out" else "failed",
"timed_out": outcome == "timed_out" and not has_lexical_memory,
}
)
else:
semantic_request = value.model_copy(update={"query": ""})
enriched = asyncio.create_task(
in_interactive_read(
read_memory_context,
database,
semantic_request,
scopes=authority.memory_scopes,
deadline=lane_end,
semantic_hits=semantic_hits,
semantic_outcome=outcome,
)
)
done, _ = await asyncio.wait(
{enriched},
timeout=max(
0,
lane_end - MEMORY_FINALIZE_RESERVE_SECONDS - time.monotonic(),
),
)
if enriched not in done:
enriched.cancel()
enriched.add_done_callback(
lambda task: task.exception() if not task.cancelled() else None
)
result = lexical_result.model_copy(
update={
"status": "degraded",
"reason": "lexical_only",
"searched_channels": ("lexical", "semantic"),
"semantic_status": "completed",
"timed_out": False,
}
)
else:
try:
semantic_result = enriched.result()
except (sqlite3.Error, OSError, ValueError) as error:
diagnostic_event(
level="warn",
component="context.search",
event="context.semantic_enrichment_failed",
outcome="degraded",
error_code="semantic_enrichment_read_failed",
**diagnostic_exception(error),
)
result = lexical_result.model_copy(
update={
"status": "degraded",
"reason": "lexical_only",
"searched_channels": ("lexical", "semantic"),
"semantic_status": "completed",
"timed_out": False,
}
)
lane_observations["memory"] = result
return result
if semantic_result.status == "unavailable" and lexical_result.items:
result = lexical_result.model_copy(
update={
"status": "degraded",
"reason": "lexical_only",
"searched_channels": ("lexical", "semantic"),
"semantic_status": "completed",
"timed_out": False,
}
)
lane_observations["memory"] = result
return result
semantic_items = tuple(
item
for item in semantic_result.items
if isinstance(item, MemoryContextItem)
)
lexical_items = tuple(
item for item in lexical_result.items if isinstance(item, MemoryContextItem)
)
combined_count = len(
{item.memory_id for item in (*semantic_items, *lexical_items)}
)
merged = merge_memory_candidates(
lexical_items,
semantic_items,
limit=value.limit + 1,
)
truncated_by = list(semantic_result.truncated_by)
if combined_count > len(merged) and "count_budget" not in truncated_by:
truncated_by.append("count_budget")
result = semantic_result.model_copy(
update={
"items": tuple(merged),
# An answered semantic search may legitimately
# return no IDs. Lexical can still have a grounded
# hit; retaining semantic's `empty` status while
# attaching that hit violates the lane contract
# and makes the aggregator reject the whole lane.
"status": "ok" if merged else semantic_result.status,
"truncated_by": tuple(truncated_by),
}
)
lane_observations["memory"] = result
return result
async def finalizer(
kind: ContextKind, items: tuple[ContextItem, ...], end: float
) -> tuple[ContextItem, ...]:
if kind == "raw":
return await in_interactive_read(
revalidate_raw_context,
database,
tuple(item for item in items if isinstance(item, RawContextItem)),
source_ids=authority.raw_sources,
deadline=end,
)
return await in_interactive_read(
revalidate_memory_context,
database,
tuple(item for item in items if isinstance(item, MemoryContextItem)),
scopes=authority.memory_scopes,
deadline=end,
)
result = await search_context(
value,
raw_reader=raw if authority.raw_allowed else None,
memory_reader=memory if authority.memory_allowed else None,
finalizer=finalizer,
deadline=deadline,
timing_observer=stage_timings.update,
failure_observer=lambda kind, site, error_class: lane_failures.__setitem__(
kind, (site, error_class)
),
)
final = result.model_copy(update={"total_ms": (time.monotonic() - started) * 1000})
serialization_started = time.monotonic()
final.model_dump_json()
stage_timings["serialization_ms"] = (time.monotonic() - serialization_started) * 1000
for phase, duration in stage_timings.items():
diagnostic_event(
level="info",
component="context.search",
event="context_stage.completed",
outcome="succeeded",
request_id=final.request_id,
duration_ms=max(0, round(duration)),
details={"phase": phase},
)
for kind in ("raw", "memory"):
path = getattr(final.paths, kind)
observed = lane_observations.get(kind)
failure = lane_failures.get(kind)
lexical_hits = (
sum(
1
for item in observed.items
if isinstance(item, MemoryContextItem) and "lexical" in item.matched_by
)
if observed is not None and kind == "memory"
else 0
)
semantic_hits = (
sum(
1
for item in observed.items
if isinstance(item, MemoryContextItem)
and "governed_semantic" in item.matched_by
)
if observed is not None and kind == "memory"
else 0
)
timed_out = path.reason == "timeout_budget" or (
observed.timed_out if observed is not None else False
)
diagnostic_failure_site = failure[0] if failure is not None else None
diagnostic_error_class = failure[1] if failure is not None else None
if observed is not None:
diagnostic_failure_site = diagnostic_failure_site or observed.failure_site
diagnostic_error_class = diagnostic_error_class or observed.error_class
diagnostic_event(
level="warn" if path.status in {"degraded", "unavailable"} else "info",
component="context.search",
event="context_lane.completed",
outcome=path.status,
request_id=final.request_id,
duration_ms=round(path.elapsed_ms),
details={
"operation_kind": kind,
"mode": ",".join(observed.searched_channels) if observed else "none",
"status": path.status,
"index_complete": observed.index_complete if observed else None,
**(
{"lexical_hits": lexical_hits, "semantic_hits": semantic_hits}
if kind == "memory"
else {}
),
"reason": path.reason,
"truncated": path.truncated,
"timed_out": timed_out,
**(
{
"failure_site": diagnostic_failure_site,
"error_class": diagnostic_error_class,
}
if diagnostic_failure_site is not None
and diagnostic_error_class is not None
else {}
),
},
)
store = getattr(request.app.state, "call_event_store", None)
if store is not None and value.query != "__om_probe__":
event = context_call_event(final, access)
try:
await asyncio.to_thread(
store.record,
event,
busy_timeout_ms=CALL_EVENT_RESPONSE_BUDGET_MS,
)
response.headers["X-One-Memory-Call-Recorded"] = "true"
except Exception:
response.headers["X-One-Memory-Call-Recorded"] = "false"
request.app.state.call_event_recording_error = True
return final
return router
def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
@@ -434,7 +38,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
@router.post("/search", response_model=SdkContextSearchResponse)
async def search(
value: ContextSearchRequest,
value: SdkContextSearchRequest,
request: Request,
response: Response,
) -> SdkContextSearchResponse:
@@ -455,19 +59,33 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
except Exception:
raise HTTPException(503, "context_authority_unavailable") from None
bridge = getattr(request.app.state, "sdk_library_bridge", None)
if bridge is None:
client_handle = getattr(request.app.state, "sdk_client_handle", None)
if client_handle is None:
raise HTTPException(503, "sdk_context_unavailable")
try:
client = await client_handle.client()
except Exception:
raise HTTPException(503, "sdk_context_unavailable") from None
async def raw_lane() -> tuple[SdkRawContextItem, ...]:
if "raw" not in value.kinds or not authority.raw_allowed:
return ()
hits = await bridge.search_sdk_raw(
raw = await asyncio.to_thread(
client.search,
value.query,
user_id="local",
source_ids=tuple(authority.raw_sources or ()),
user_ids=["local"],
agent_ids=[],
include_sources=list(authority.raw_sources or ()),
limit=value.limit,
min_score=0.0,
schema_min_score=0.0,
schema_limit=0,
intention_limit=0,
scene="chat",
reader=HY_MEMORY_READER,
layers=["l1_raw"],
)
hits = parse_sdk_raw_search(raw)
return tuple(
SdkRawContextItem(
raw_id=hit.raw_id,
@@ -487,15 +105,24 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
async def memory_lane() -> tuple[SdkMemoryContextItem, ...]:
if "memory" not in value.kinds or not authority.memory_allowed:
return ()
hits = await bridge.search_memories(
raw = await asyncio.to_thread(
client.search,
value.query,
user_id="local",
agent_id="*",
user_ids=["local"],
agent_ids=[],
limit=value.limit,
min_score=sdk_relevance_floor(),
schema_min_score=sdk_relevance_floor(),
schema_limit=value.schema_limit,
intention_limit=value.intention_limit,
scene="normal",
reader=HY_MEMORY_READER,
)
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
items: list[SdkMemoryContextItem] = []
for hit in hits:
memory = await bridge.get_sdk_memory(hit.memory_id)
record = await asyncio.to_thread(client.get, hit.memory_id)
memory = None if record is None else HySdkMemoryRecord.from_sdk(record)
if memory is None or memory.status != "active":
continue
items.append(
@@ -521,6 +148,29 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
)
return tuple(items)
async def record(final: SdkContextSearchResponse, *, timed_out: bool) -> None:
# Same boundary as v2 ask: only a host's call is history. The
# renderer's own searches are the person reading their library.
store = getattr(request.app.state, "call_event_store", None)
if store is None or access is None or value.query == "__om_probe__":
return
event = sdk_context_call_event(
final,
access,
memory_searched=authority.memory_allowed,
timed_out=timed_out,
)
try:
await asyncio.to_thread(
store.record,
event,
busy_timeout_ms=CALL_EVENT_RESPONSE_BUDGET_MS,
)
response.headers["X-One-Memory-Call-Recorded"] = "true"
except Exception:
response.headers["X-One-Memory-Call-Recorded"] = "false"
request.app.state.call_event_recording_error = True
lanes = (raw_lane(), memory_lane())
try:
raw_items, memory_items = await asyncio.wait_for(
@@ -528,7 +178,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
timeout=value.timeout_ms / 1000,
)
except TimeoutError:
return SdkContextSearchResponse(
final = SdkContextSearchResponse(
request_id=str(uuid4()),
state="unavailable",
items=(),
@@ -536,6 +186,8 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
truncated=True,
total_ms=(time.monotonic() - started) * 1000,
)
await record(final, timed_out=True)
return final
partial = isinstance(raw_items, Exception) or isinstance(memory_items, Exception)
raw_candidates = (
() if isinstance(raw_items, Exception) else cast(tuple[SdkContextItem, ...], raw_items)
@@ -567,7 +219,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
truncated = True
selected.append(item)
used += len(item.content)
return SdkContextSearchResponse(
final = SdkContextSearchResponse(
request_id=str(uuid4()),
state=("partial" if partial else "grounded" if selected else "empty"),
items=tuple(selected),
@@ -575,5 +227,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
truncated=truncated,
total_ms=(time.monotonic() - started) * 1000,
)
await record(final, timed_out=False)
return final
return router
@@ -1,464 +0,0 @@
"""Bounded two-lane orchestration; readers and finalizers own trusted read access."""
import asyncio
import time
from collections.abc import Awaitable, Callable, Mapping
from dataclasses import dataclass, field
from uuid import uuid4
from pydantic import TypeAdapter
from one_memory.context_models import (
ContextItem,
ContextKind,
ContextLaneResult,
ContextLimit,
ContextPath,
ContextPaths,
ContextSearchRequest,
ContextSearchResponse,
MemoryContextItem,
RawContextItem,
)
Reader = Callable[[float], Awaitable[ContextLaneResult]]
Finalizer = Callable[
[ContextKind, tuple[ContextItem, ...], float], Awaitable[tuple[ContextItem, ...]]
]
TimingObserver = Callable[[Mapping[str, float]], None]
FailureObserver = Callable[[ContextKind, str, str], None]
_KINDS: tuple[ContextKind, ...] = ("raw", "memory")
_ITEMS = TypeAdapter(tuple[ContextItem, ...])
@dataclass
class _Lane:
result: ContextLaneResult
elapsed_ms: float = 0
limits: list[ContextLimit] = field(default_factory=list)
def _identity(item: ContextItem) -> tuple[str, ...]:
if isinstance(item, RawContextItem):
return (item.kind, item.source_id, item.conversation_id, item.anchor_evidence_id)
return (item.kind, item.memory_id)
def _validate_items(kind: ContextKind, items: tuple[ContextItem, ...]) -> None:
if len(items) > 21 or any(item.kind != kind for item in items):
raise ValueError("invalid lane items")
for item in items:
if isinstance(item, RawContextItem) and item.anchor_evidence_id not in {
message.evidence_id for message in item.messages
}:
raise ValueError("missing raw anchor")
async def _read(reader: Reader, kind: ContextKind, deadline: float) -> ContextLaneResult:
value = await reader(deadline)
# Revalidate even BaseModel instances: model_construct/model_copy bypass validation.
result = ContextLaneResult.model_validate(value.model_dump())
_validate_items(kind, result.items)
if result.status not in ("ok", "empty", "degraded", "unavailable"):
raise ValueError("reader cannot decide requested authority")
if result.status in ("empty", "unavailable") and result.items:
raise ValueError("non-result lane has items")
return result
async def _finalize(
finalizer: Finalizer, kind: ContextKind, items: tuple[ContextItem, ...], deadline: float
) -> tuple[ContextItem, ...]:
value = await finalizer(kind, items, deadline)
if not isinstance(value, tuple):
raise ValueError("invalid finalization result")
checked = _ITEMS.validate_python([item.model_dump() for item in value])
_validate_items(kind, checked)
if not {_identity(item) for item in checked} <= {_identity(item) for item in items}:
raise ValueError("finalizer introduced an unselected identity")
return checked
async def _collect(
jobs: dict[ContextKind, Awaitable[object]], deadline: float
) -> tuple[
dict[ContextKind, object],
dict[ContextKind, BaseException],
set[ContextKind],
dict[ContextKind, float],
]:
"""Cancel and await every owned task on deadline, failure or caller cancellation."""
durations: dict[ContextKind, float] = {}
async def measured(kind: ContextKind, job: Awaitable[object]) -> object:
started = time.monotonic()
try:
value = await job
if time.monotonic() >= deadline:
raise TimeoutError
return value
finally:
durations[kind] = (time.monotonic() - started) * 1000
tasks = {kind: asyncio.create_task(measured(kind, job)) for kind, job in jobs.items()}
values: dict[ContextKind, object] = {}
failed: dict[ContextKind, BaseException] = {}
timed_out: set[ContextKind] = set()
try:
if not tasks:
return values, failed, timed_out, durations
done, pending = await asyncio.wait(
tasks.values(), timeout=max(0, deadline - time.monotonic())
)
for kind, task in tasks.items():
if task in pending:
timed_out.add(kind)
elif task in done:
if not task.cancelled() and isinstance(task.exception(), TimeoutError):
timed_out.add(kind)
elif task.cancelled() or task.exception() is not None:
failed[kind] = task.exception() or asyncio.CancelledError()
else:
values[kind] = task.result()
return values, failed, timed_out, durations
finally:
for task in tasks.values():
if not task.done():
task.cancel()
if tasks:
# Cooperative readers finish immediately and still get their
# normal cleanup. A bridge that suppresses cancellation must not
# hold a host response past its hard deadline while an underlying
# SDK thread winds down.
cleanup: asyncio.Future[list[object]] = asyncio.gather(
*tasks.values(), return_exceptions=True
)
current = asyncio.current_task()
if current is not None and current.cancelling():
await cleanup
else:
try:
await asyncio.wait_for(asyncio.shield(cleanup), timeout=0.05)
except TimeoutError:
cleanup.add_done_callback(_consume_cleanup)
def _consume_cleanup(task: asyncio.Future[list[object]]) -> None:
if not task.cancelled():
task.exception()
def _deduplicate(
items: tuple[ContextItem, ...], limits: list[ContextLimit]
) -> tuple[ContextItem, ...]:
unique: dict[tuple[str, ...], ContextItem] = {}
for item in items:
key = _identity(item)
previous = unique.get(key)
if isinstance(previous, MemoryContextItem) and isinstance(item, MemoryContextItem):
if item.revision != previous.revision:
# Never graft older evidence onto a newer memory version.
unique[key] = item if item.revision > previous.revision else previous
continue
evidence = tuple(dict.fromkeys((*previous.evidence_ids, *item.evidence_ids)))
if len(evidence) > 128:
limits.append("metadata_budget")
unique[key] = previous.model_copy(
update={
"evidence_ids": evidence[:128],
"truncated": previous.truncated or item.truncated or len(evidence) > 128,
}
)
elif isinstance(previous, RawContextItem) and isinstance(item, RawContextItem):
messages = {message.evidence_id: message for message in previous.messages}
for message in item.messages:
if (
message.evidence_id not in messages
or message.revision > messages[message.evidence_id].revision
):
messages[message.evidence_id] = message
if len(messages) > 3:
limits.append("metadata_budget")
# The first window already contains the anchor; dict order preserves it.
unique[key] = previous.model_copy(
update={
"messages": tuple(messages.values())[:3],
"truncated": previous.truncated or item.truncated or len(messages) > 3,
}
)
else:
unique[key] = item
# Reserve each raw anchor for its own window before deduplicating context messages.
anchors = {
(item.source_id, item.conversation_id, item.anchor_evidence_id)
for item in unique.values()
if isinstance(item, RawContextItem)
}
seen: set[tuple[str, ...]] = set()
result: list[ContextItem] = []
for item in unique.values():
if isinstance(item, MemoryContextItem):
result.append(
item.model_copy(update={"evidence_ids": tuple(dict.fromkeys(item.evidence_ids))})
)
continue
retained_messages = []
for message in item.messages:
key = (item.source_id, item.conversation_id, message.evidence_id)
if key in seen or (key in anchors and message.evidence_id != item.anchor_evidence_id):
continue
seen.add(key)
retained_messages.append(message)
result.append(item.model_copy(update={"messages": tuple(retained_messages)}))
return tuple(result)
def _characters(item: ContextItem) -> int:
if isinstance(item, MemoryContextItem):
return len(item.content)
return sum(len(message.text) for message in item.messages)
def _shares(lengths: list[int], budget: int) -> list[int]:
"""Water-fill allocations so a large first item cannot consume every character."""
allocations = [0] * len(lengths)
pending = list(range(len(lengths)))
while pending and budget:
share, remainder = divmod(budget, len(pending))
next_pending = []
for position, index in enumerate(pending):
added = min(lengths[index] - allocations[index], share + (position < remainder))
allocations[index] += added
budget -= added
if allocations[index] < lengths[index]:
next_pending.append(index)
pending = next_pending
return allocations
def _clip(item: ContextItem, budget: int) -> ContextItem:
if _characters(item) <= budget:
return item
if isinstance(item, MemoryContextItem):
return item.model_copy(update={"content": item.content[:budget], "truncated": True})
# The first raw message is the lexical anchor. Give it the first share of
# the budget, then water-fill the adjacent context; otherwise a long
# neighbor can consume the whole window and hide the answer.
anchor = item.messages[0]
anchor_budget = min(len(anchor.text), budget)
remaining = budget - anchor_budget
adjacent = list(item.messages[1:])
shares = [anchor_budget, *_shares([len(message.text) for message in adjacent], remaining)]
return item.model_copy(
update={
"truncated": True,
"messages": tuple(
message.model_copy(
update={
"text": message.text[:share],
"truncated": message.truncated or len(message.text) > share,
}
)
for message, share in zip(item.messages, shares, strict=True)
),
}
)
async def search_context(
request: ContextSearchRequest,
*,
raw_reader: Reader | None,
memory_reader: Reader | None,
finalizer: Finalizer,
deadline: float | None = None,
timing_observer: TimingObserver | None = None,
failure_observer: FailureObserver | None = None,
) -> ContextSearchResponse:
request = ContextSearchRequest.model_validate(request.model_dump())
started = time.monotonic()
deadline = min(
deadline if deadline is not None else float("inf"), started + request.timeout_ms / 1000
)
lanes: dict[ContextKind, _Lane] = {}
jobs: dict[ContextKind, Awaitable[object]] = {}
readers: dict[ContextKind, Reader | None] = {"raw": raw_reader, "memory": memory_reader}
for kind, reader in readers.items():
if kind not in request.kinds:
lanes[kind] = _Lane(ContextLaneResult(status="not_requested", reason="not_requested"))
elif reader is None:
lanes[kind] = _Lane(
ContextLaneResult(status="not_authorized", reason="permission_required")
)
elif time.monotonic() >= deadline:
lanes[kind] = _Lane(
ContextLaneResult(status="degraded", reason="timeout_budget", timed_out=True),
limits=["timeout_budget"],
)
else:
jobs[kind] = _read(reader, kind, deadline)
values, failed, timed_out, read_ms = await _collect(jobs, deadline)
for kind in jobs:
result = values.get(kind)
if kind in timed_out:
result = ContextLaneResult(
status="degraded",
reason="timeout_budget",
truncated_by=("timeout_budget",),
timed_out=True,
)
elif kind in failed or not isinstance(result, ContextLaneResult):
error = failed.get(kind)
if failure_observer is not None:
failure_observer(
kind,
"collect",
type(error).__name__ if error is not None else "InvalidLaneResult",
)
result = ContextLaneResult(
status="unavailable",
reason="read_unavailable",
failure_site="collect",
error_class=type(error).__name__ if error is not None else "InvalidLaneResult",
)
lanes[kind] = _Lane(result, read_ms.get(kind, 0), list(dict.fromkeys(result.truncated_by)))
final_jobs: dict[ContextKind, Awaitable[object]] = {}
for kind in jobs:
lane = lanes[kind]
if lane.result.status not in ("ok", "empty", "degraded"):
continue
if time.monotonic() >= deadline:
lane.result = ContextLaneResult(
status="degraded", reason="timeout_budget", timed_out=True
)
lane.limits.append("timeout_budget")
else:
final_jobs[kind] = _finalize(finalizer, kind, lane.result.items, deadline)
final_values, final_failed, final_timed_out, revalidation_ms = await _collect(
final_jobs, deadline
)
for kind in final_jobs:
lane = lanes[kind]
lane.elapsed_ms += revalidation_ms.get(kind, 0)
if kind in final_timed_out:
lane.result = ContextLaneResult(
status="unavailable", reason="timeout_budget", timed_out=True
)
lane.limits.append("timeout_budget")
elif kind in final_failed:
error = final_failed[kind]
if failure_observer is not None:
failure_observer(kind, "finalize", type(error).__name__)
lane.result = ContextLaneResult(
status="unavailable",
reason="read_unavailable",
failure_site="finalize",
error_class=type(error).__name__,
)
else:
final_items = _ITEMS.validate_python(final_values[kind])
changed = {_identity(item): item for item in lane.result.items} != {
_identity(item): item for item in final_items
}
lane.result = lane.result.model_copy(
update={
"items": final_items,
"status": "degraded" if changed else lane.result.status,
"reason": ("evidence_changed" if kind == "raw" else "memory_changed")
if changed
else lane.result.reason,
}
)
# Nothing previously verified may escape once the common release deadline expired.
if time.monotonic() >= deadline:
for kind in jobs:
if lanes[kind].result.status in ("ok", "empty", "degraded"):
lanes[kind].result = ContextLaneResult(
status="degraded", reason="timeout_budget", timed_out=True
)
lanes[kind].limits.append("timeout_budget")
fusion_started = time.monotonic()
queues = {kind: _deduplicate(lanes[kind].result.items, lanes[kind].limits) for kind in _KINDS}
interleaved = [
queues[kind][index]
for index in range(max(map(len, queues.values()), default=0))
for kind in _KINDS
if index < len(queues[kind])
]
selected = interleaved[: request.limit]
for item in interleaved[request.limit :]:
lanes[item.kind].limits.append("count_budget")
shares = _shares([_characters(item) for item in selected], request.character_budget)
items = tuple(_clip(item, share) for item, share in zip(selected, shares, strict=True))
for before, after in zip(selected, items, strict=True):
if _characters(before) != _characters(after):
lanes[after.kind].limits.append("character_budget")
if time.monotonic() >= deadline:
items = ()
for kind in jobs:
if lanes[kind].result.status in ("ok", "empty", "degraded"):
lanes[kind].result = ContextLaneResult(
status="degraded", reason="timeout_budget", timed_out=True
)
lanes[kind].limits.append("timeout_budget")
lanes[kind].elapsed_ms = (time.monotonic() - started) * 1000
paths: dict[ContextKind, ContextPath] = {}
for kind in _KINDS:
lane = lanes[kind]
emitted = [item for item in items if item.kind == kind]
limits = tuple(dict.fromkeys(lane.limits))
truncated = bool(limits) or any(item.truncated for item in emitted)
status = lane.result.status
benign_limits = set(limits) <= {"count_budget", "character_budget"}
benign_degradation = (
not lane.result.timed_out
and benign_limits
and (
lane.result.reason is None
or (kind == "raw" and lane.result.reason == "lexical_only")
)
)
if status in ("ok", "empty") or (status == "degraded" and benign_degradation):
status = "ok" if emitted or lane.result.items else "empty"
paths[kind] = ContextPath(
status=status,
reason=lane.result.reason,
elapsed_ms=lane.elapsed_ms,
returned_count=len(emitted),
truncated=truncated,
truncated_by=limits,
)
requested_paths = [paths[kind] for kind in request.kinds]
completed = any(path.status in ("ok", "empty", "degraded") for path in requested_paths)
incomplete = any(path.status not in ("ok", "empty") for path in requested_paths)
response = ContextSearchResponse(
request_id=str(uuid4()),
state="unavailable"
if not completed
else "partial"
if incomplete
else "grounded"
if items
else "empty",
paths=ContextPaths(raw=paths["raw"], memory=paths["memory"]),
items=items,
used_characters=sum(map(_characters, items)),
truncated=any(path.truncated for path in paths.values()),
total_ms=(time.monotonic() - started) * 1000,
)
checked = ContextSearchResponse.model_validate(response.model_dump())
if timing_observer is not None:
timing_observer(
{
"raw_ms": read_ms.get("raw", 0),
"memory_ms": read_ms.get("memory", 0),
"raw_revalidation_ms": revalidation_ms.get("raw", 0),
"memory_revalidation_ms": revalidation_ms.get("memory", 0),
"fusion_ms": (time.monotonic() - fusion_started) * 1000,
"total_ms": checked.total_ms,
}
)
return checked
@@ -22,7 +22,7 @@ from dataclasses import dataclass
from datetime import UTC, datetime
from functools import partial
from pathlib import Path
from typing import Any, Literal, Protocol, cast
from typing import Any, Literal, cast
from one_memory.background_work import interactive_read_pool
from one_memory.context_expand import ExpansionAuthority, ExpansionRefSigner
@@ -36,23 +36,8 @@ from one_memory.retrieval.models import (
RawEvidence,
RetrievalPath,
RetrievalResponse,
TimeRange,
)
from one_memory.retrieval.query_plan import plan_local_query
from one_memory.retrieval.raw_candidates import RawCandidateBatch
from one_memory.retrieval.raw_evidence import materialize_raw_candidates
class RawIndexLookup(Protocol):
def lookup_raw(
self,
query: str,
*,
source_ids: frozenset[str],
deadline: float,
time_range: TimeRange | None = None,
) -> tuple[RawCandidateBatch, bool]: ...
ExpandRefs = tuple[ExpansionRefSigner, ExpansionAuthority]
@@ -106,11 +91,6 @@ class LaneResult:
coverage_limited: bool = False
def _semantic_only_raw(item: RawEvidence) -> bool:
"""Semantic overlap alone is a candidate, not a confirmed recall item."""
return set(item.matched_by) == {"semantic"}
@dataclass(frozen=True)
class StageTimings:
raw_ms: float = 0.0
@@ -191,12 +171,6 @@ def _managed(
return selected, truncated
def _group_missing_requirements(item: RawEvidence | GovernedEvidence) -> list[str]:
if isinstance(item, RawEvidence) and _semantic_only_raw(item):
return ["relevance_confirmation"]
return []
def _path(
status: str,
reason: str | None,
@@ -296,8 +270,7 @@ def read_lexical_raw_lane(
("lexical", "object") if plan.object_keys else ("lexical",),
truncated,
# Starvation means: this lane matched, then dropped the item whole. An
# ordinary no-match must stay empty, and an unconfigured semantic lane
# is not starvation.
# ordinary no-match must stay empty.
starved and not items,
probe_saturated,
)
@@ -564,97 +537,6 @@ def warm_raw_read_path(
return False
def read_semantic_raw_lane(
index: RawIndexLookup | None,
database_path: Path,
query: CoordinatorQuery,
*,
source_ids: frozenset[str],
deadline: float,
) -> LaneResult:
if not source_ids:
return LaneResult((), "unavailable", "permission_required", (), False)
# No derived index is the no-model baseline, not a failure: the lexical lane
# must still answer. Semantic is genuinely unsearched in that state, so it
# must not claim a channel or a coverage result it never produced.
if index is None:
return LaneResult((), "not_searched", None, (), False)
try:
batch, unpublished = index.lookup_raw(
query.query,
source_ids=source_ids,
deadline=deadline,
time_range=plan_local_query(query.query).time_range,
)
except Exception:
return LaneResult((), "unavailable", "index_unavailable", ("semantic",), False)
if not isinstance(batch, RawCandidateBatch):
return LaneResult((), "unavailable", "index_unavailable", ("semantic",), False)
if batch.reason == "not_authorized":
return LaneResult((), "unavailable", "permission_required", ("semantic",), False)
if batch.reason == "deadline":
return LaneResult((), "unavailable", "timeout_budget", ("semantic",), True)
if batch.reason in {"vector_unavailable", "index_unavailable"}:
return LaneResult((), "unavailable", batch.reason, ("semantic",), False)
store, generation = _index_state(index, database_path)
from one_memory.retrieval.index_store import RetrievalIndexStore
store = cast("RetrievalIndexStore | None", store)
if store is None or generation is None:
return LaneResult((), "unavailable", "index_unavailable", ("semantic",), False)
try:
materialized = materialize_raw_candidates(
store,
batch,
generation_id=generation,
source_ids=source_ids,
deadline=deadline,
character_budget=query.character_budget,
)
except Exception:
return LaneResult((), "unavailable", "index_unavailable", ("semantic",), False)
if materialized.reason == "deadline":
return LaneResult((), "unavailable", "timeout_budget", ("semantic",), True)
if materialized.reason in {"index_unavailable", "not_authorized"}:
return LaneResult((), "unavailable", materialized.reason, ("semantic",), False)
complete = materialized.complete and not unpublished
if materialized.reason == "evidence_changed":
status: Literal["ok", "empty", "unavailable", "not_searched"] = "unavailable"
reason = "evidence_changed"
elif not materialized.items and not complete:
status, reason = "unavailable", "index_incomplete"
elif materialized.reason == "character_budget":
status, reason = "ok", None
else:
status, reason = ("ok" if materialized.items else "empty"), None
return LaneResult(
tuple(materialized.items),
status,
reason,
("semantic",),
materialized.reason == "character_budget",
)
def _index_state(index: RawIndexLookup, database_path: Path) -> tuple[object | None, str | None]:
store = getattr(index, "_store", None)
generation = None
snapshot = getattr(index, "snapshot", None)
if snapshot is not None:
try:
generation = snapshot().generation_id
except Exception:
generation = None
if store is None:
try:
from one_memory.retrieval.index_store import RetrievalIndexStore
store = RetrievalIndexStore(database_path)
except Exception:
store = None
return store, generation
def read_memory_lane(
database_path: Path,
query: CoordinatorQuery,
@@ -857,7 +739,6 @@ def coordinate_context_v2(
query: CoordinatorQuery,
*,
database_path: Path,
index: RawIndexLookup | None = None,
deadline: float,
expand_refs: ExpandRefs | None = None,
) -> RetrievalResponse:
@@ -871,62 +752,29 @@ def coordinate_context_v2(
lexical = read_lexical_raw_lane(
database_path, query, source_ids=authority.raw_sources, deadline=deadline
)
semantic = read_semantic_raw_lane(
index, database_path, query, source_ids=authority.raw_sources, deadline=deadline
)
timings = StageTimings(raw_ms=(time.monotonic() - raw_started) * 1000)
combined: dict[tuple[str, str], RawEvidence] = {}
for item in (*lexical.items, *semantic.items):
if not isinstance(item, RawEvidence):
continue
identity = _identity(item)
previous = combined.get(identity)
if previous is None or _semantic_only_raw(previous):
combined[identity] = item
# Semantic similarity is a candidate signal, not a verified recall
# result. Keep semantic-only candidates visible in coverage, but do not
# release their bodies for host injection unless the item is
# independently confirmed by the lexical/object lane. A global distance
# threshold is deliberately not used: the measured positive and
# negative distributions overlap.
semantic_only = {_identity(item) for item in combined.values() if _semantic_only_raw(item)}
confirmed = {_identity(item) for item in lexical.items if isinstance(item, RawEvidence)}
deferred_raw = semantic_only - confirmed
if deferred_raw:
combined = {
identity: item
for identity, item in combined.items()
if identity not in deferred_raw
}
channels = tuple(dict.fromkeys((*lexical.channels, *semantic.channels)))
for item in lexical.items:
if isinstance(item, RawEvidence):
combined.setdefault(_identity(item), item)
if lexical.status == "unavailable":
status: Literal["ok", "empty", "unavailable", "not_searched", "degraded"] = (
"unavailable"
)
reason = lexical.reason
elif semantic.status == "unavailable":
status = "unavailable"
reason = semantic.reason
elif combined:
status, reason = "ok", semantic.reason if semantic.status != "ok" else None
elif deferred_raw:
status, reason = "degraded", "relevance_confirmation"
elif lexical.status == "empty":
status, reason = ("empty" if semantic.status != "not_searched" else "ok"), None
elif combined or lexical.status == "empty":
status, reason = "ok", None
else:
status, reason = "empty", None
truncated = lexical.truncated or semantic.truncated
raw = LaneResult(
tuple(combined.values()),
status,
reason,
channels,
truncated,
lexical.channels,
lexical.truncated,
lexical.starved,
lexical.coverage_limited,
)
if deferred_raw:
object.__setattr__(raw, "coverage_limited", True)
elif "raw" in query.kinds and not authority.raw_allowed:
raw = LaneResult((), "unavailable", "permission_required", (), False)
if "memory" in query.kinds:
@@ -995,9 +843,7 @@ def coordinate_context_v2(
return _path("unavailable", raw.reason, raw.channels, "not_searched", False, 0)
if dropped and not final_raw:
return _path("unavailable", "permission_required", raw.channels, "unavailable", True, 0)
# Lexical-only is complete when the lexical lane actually answered. An
# unconfigured semantic lane is a capability absence, not proof that
# less than the authorized range was searched.
# The raw lane is complete when the lexical lane actually answered.
complete = (
raw.status in {"ok", "empty"}
and not dropped
@@ -1022,9 +868,7 @@ def coordinate_context_v2(
"count_budget"
if truncated
else "coverage_budget"
if raw.coverage_limited and not deferred_raw
else "relevance_confirmation"
if deferred_raw
if raw.coverage_limited
else "character_budget"
)
return _path(
@@ -1112,7 +956,7 @@ def coordinate_context_v2(
"group_id": f"group-{index}",
"answer_target": query.intent,
"decision_status": "missing_confirmation",
"missing_requirements": _group_missing_requirements(item),
"missing_requirements": [],
"object_refs": _object_refs(item, plan.object_keys),
"items": [group_item.model_dump(mode="json") for group_item in (item,)],
}
@@ -1185,7 +1029,6 @@ async def search_context_v2(
query: CoordinatorQuery,
*,
database_path: Path,
index: RawIndexLookup | None = None,
deadline: float,
expand_refs: ExpandRefs | None = None,
) -> RetrievalResponse:
@@ -1198,7 +1041,6 @@ async def search_context_v2(
authority,
query,
database_path=database_path,
index=index,
deadline=deadline,
expand_refs=expand_refs,
),
@@ -0,0 +1,277 @@
"""Pure conversation-window helpers shared by capture, retrieval, and writes."""
from __future__ import annotations
import os
import re
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord
from one_memory.memory_tasks import segment_evidence_record
MEMORY_TASK_TIMEOUT_MARGIN_SECONDS = 60.0
MEMORY_TASK_TIMEOUT_MIN_SECONDS = 180.0
MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES = 512
_HARNESS_PREFIXES = (
"<codex_internal_context",
"# agents.md instructions",
"<environment_context>",
"<permissions instructions>",
"<system-reminder>",
"<system_notification>",
"<instructions>",
)
_HOST_ENVELOPE = re.compile(
r"(?:"
r"^<timestamp>.*?<system_(?:notification|reminder)>|"
r"^<system_(?:notification|reminder)>|"
r'^\{\s*"request_id"\s*:.*?"steps"\s*:'
r")",
re.IGNORECASE | re.DOTALL,
)
_WHOLE_TURN_CHARACTERS = 60_000
_WINDOW_BYTE_CEILING = 2_000_000
_HISTORY_MERGE_TURNS = 8
_HISTORY_MERGE_CHARACTERS = 8_000
def memory_task_timeout_seconds(model_timeout_seconds: float) -> float:
"""Bound a whole write without treating one model request as its budget."""
return max(
MEMORY_TASK_TIMEOUT_MIN_SECONDS,
float(model_timeout_seconds) + MEMORY_TASK_TIMEOUT_MARGIN_SECONDS,
)
def _coding_tool_result_max_bytes() -> int:
raw = os.environ.get("MEMORY_CODING_TOOL_RESULT_MAX_BYTES")
if raw is None:
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
try:
value = int(raw)
except ValueError:
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
return value if value > 0 else MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
_HY_TOOL_CHARACTERS = _coding_tool_result_max_bytes()
def _conversation_chunks(
records: list[EvidenceRecord],
*,
max_characters: int | None = None,
max_turns: int | None = None,
completed_only: bool = False,
) -> list[tuple[EvidenceRecord, ...]]:
"""Build bounded, complete user/assistant/tool windows."""
turns: list[list[EvidenceRecord]] = []
for record in records:
if not _is_hy_turn_record(record):
continue
pending_proposal = bool(turns) and not _opens_turn(turns[-1][0])
if not turns or (_opens_turn(record) and not pending_proposal):
turns.append([record])
else:
turns[-1].append(record)
if completed_only and turns and _trailing_user_turn_is_open(turns[-1]):
turns.pop()
whole = _whole_turns() and max_characters is None
if whole:
size = _hy_read_length
max_characters = _WHOLE_TURN_CHARACTERS
max_turns = max_turns or 1_000_000
else:
size = lambda record: len(record.verbatim) # noqa: E731
max_characters = max_characters or 12_000
max_turns = max_turns or 24
chunks: list[tuple[EvidenceRecord, ...]] = []
for turn in turns:
original_question = turn[0] if _opens_turn(turn[0]) else None
expanded: list[EvidenceRecord] = []
for record in turn:
reserve = (
size(original_question)
if original_question is not None
and record is not original_question
and size(original_question) < max_characters
else 0
)
segment_limit = max(1, max_characters - reserve)
if size(record) <= segment_limit:
expanded.append(record)
continue
spans = _semantic_segment_spans(record.verbatim, segment_limit)
expanded.extend(
segment_evidence_record(record, index=index, start=start, end=end)
if len(spans) > 1
else record
for index, (start, end) in enumerate(spans)
)
question = expanded[0] if expanded and _opens_turn(expanded[0]) else None
current: list[EvidenceRecord] = []
for record in expanded:
if current and (
len(current) >= max_turns
or sum(size(item) for item in current) + size(record) > max_characters
or sum(_window_bytes(item) for item in current) + _window_bytes(record)
> _WINDOW_BYTE_CEILING
):
chunks.append(tuple(current))
current = (
[question]
if question is not None
and max_turns > 1
and size(question) + size(record) <= max_characters
else []
)
current.append(record)
chunks.append(tuple(current))
return _merge_tool_only_chunks(chunks)
def _semantic_segment_spans(text: str, limit: int) -> list[tuple[int, int]]:
if len(text) <= limit:
return [(0, len(text))]
spans: list[tuple[int, int]] = []
start = 0
separators = ("\n\n", "\n", "。", "!", "?", ". ", "! ", "? ")
while start < len(text):
hard_end = min(len(text), start + limit)
if hard_end == len(text):
end = hard_end
else:
floor = start + max(1, limit // 2)
candidates = [text.rfind(separator, floor, hard_end) for separator in separators]
boundary = max(candidates)
if boundary < floor:
end = hard_end
else:
matched = next(
separator for separator in separators if text.startswith(separator, boundary)
)
end = boundary + len(matched)
spans.append((start, end))
start = end
return spans
def _merge_tool_only_chunks(
chunks: list[tuple[EvidenceRecord, ...]],
) -> list[tuple[EvidenceRecord, ...]]:
if not any(chunk and all(record.role == "tool" for record in chunk) for chunk in chunks):
return chunks
merged: list[tuple[EvidenceRecord, ...]] = []
leading_tools: list[EvidenceRecord] = []
for chunk in chunks:
if not chunk:
continue
if all(record.role == "tool" for record in chunk):
if merged:
merged[-1] = (*merged[-1], *chunk)
else:
leading_tools.extend(chunk)
continue
if leading_tools:
chunk = (*leading_tools, *chunk)
leading_tools = []
merged.append(chunk)
return [] if leading_tools else merged
def _merge_historical_chunks(
chunks: list[tuple[EvidenceRecord, ...]],
*,
max_turns: int = _HISTORY_MERGE_TURNS,
max_characters: int = _HISTORY_MERGE_CHARACTERS,
) -> list[tuple[EvidenceRecord, ...]]:
if not chunks:
return []
merged: list[tuple[EvidenceRecord, ...]] = []
current: list[EvidenceRecord] = []
current_turns = 0
current_characters = 0
for chunk in chunks:
chunk_characters = sum(len(record.verbatim) for record in chunk)
if current and (
current_turns >= max_turns or current_characters + chunk_characters > max_characters
):
merged.append(tuple(current))
current = []
current_turns = 0
current_characters = 0
current.extend(chunk)
current_turns += 1
current_characters += chunk_characters
if current:
merged.append(tuple(current))
return merged
def _chunk_input_ref(chunk: tuple[EvidenceRecord, ...]) -> str:
user_records = tuple(item for item in chunk if item.role == "user")
return user_records[-1].evidence_id if user_records else chunk[-1].evidence_id
def _opens_turn(record: EvidenceRecord) -> bool:
return record.role == "user"
def _trailing_user_turn_is_open(turn: list[EvidenceRecord]) -> bool:
if len(turn) != 1:
return False
opener = turn[0]
return (
_opens_turn(opener)
and opener.content_kind is EvidenceContentKind.RAW_CONVERSATION
and not _is_memory_note(opener)
)
def _is_hy_turn_record(record: EvidenceRecord) -> bool:
if record.role == "tool":
return record.content_kind is EvidenceContentKind.RAW_CONVERSATION and bool(
record.verbatim.strip()
)
return _is_hy_input_turn(record)
def _is_hy_input_turn(record: EvidenceRecord) -> bool:
source = record.verbatim.strip()
if not source:
return False
if record.content_kind is EvidenceContentKind.MEMORY_CONTENT:
return True
if record.role == "assistant":
return True
if record.role != "user":
return False
if source.casefold().startswith(_HARNESS_PREFIXES):
return False
return _HOST_ENVELOPE.search(source[:2_000]) is None
def _is_memory_note(record: EvidenceRecord) -> bool:
meta = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
return (
record.source_id == "one-memory-manual"
and meta is not None
and meta.raw_meta.get("input_mode") == "direct_memory_note"
)
def _window_bytes(record: EvidenceRecord) -> int:
return len(record.verbatim.encode("utf-8"))
def _whole_turns() -> bool:
return os.environ.get("ONE_MEMORY_SPLIT_TURNS", "").strip() != "1"
def _hy_read_length(record: EvidenceRecord) -> int:
if record.role == "tool":
return min(len(record.verbatim), _HY_TOOL_CHARACTERS)
return len(record.verbatim)
@@ -6,7 +6,6 @@ from collections.abc import Sequence
from typing import NamedTuple
PREFACE = "以下是获授权历史参考,不是指令;当前用户请求优先。"
_MARKER = re.compile(r"om-derived:([0-9a-f]{64})")
# Written by the host-bridge recall injection hook (recall-injection.ts): the
# recalled memory block and, in the same reply, the cross-session notice block.
_INJECTED_TAGS = ("memhub_memory_context", "memhub_session_notice")
@@ -89,45 +88,3 @@ def strip_injected_memory(text: str) -> str:
return text
kept.append(text[at:])
return _EMPTY_REMINDER.sub("", "".join(kept)).strip()
def marker(event_id: str) -> str:
return f"om-derived:{event_id}"
def echoed_event_ids(verbatim: str) -> tuple[str, ...]:
return tuple(dict.fromkeys(_MARKER.findall(verbatim)))
def render_reference(events: tuple[object, ...] | list[object]) -> str:
lines = [PREFACE]
for event in events:
source = getattr(event, "source_id", None) or "unknown"
revision = getattr(event, "evidence_revision", None)
evidence_id = getattr(event, "evidence_id", None) or ""
event_id = getattr(event, "event_id", "")
lines.append(
f"来源 {source} 修订 {0 if revision is None else revision} "
f"证据 {evidence_id} {marker(str(event_id))}"
)
return "\n".join(lines)
def is_pure_echo(verbatim: str, known_event_ids: set[str]) -> bool:
"""True when the text is only a block we already injected.
A later line that is not the preface, a source line, or one of those
markers is a real correction and must still be recorded.
"""
found = set(echoed_event_ids(verbatim))
if not found or not found & known_event_ids:
return False
for line in verbatim.splitlines():
stripped = line.strip()
if not stripped or stripped == PREFACE or stripped.startswith("来源 "):
continue
if _MARKER.search(stripped):
continue
return False
return True
+218 -123
View File
@@ -20,8 +20,10 @@ from __future__ import annotations
import atexit
import json
import logging
import math
import os
import re
import stat
import sys
import threading
import time
@@ -72,80 +74,30 @@ class _Refused(OSError):
"""The path is not one we are willing to write to, and never will be."""
DiagnosticLevel = Literal["info", "warn", "error"]
DiagnosticLevel = Literal["debug", "info", "warn", "error"]
_IDENTIFIER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$")
_EVENT = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+){0,15}$")
_DETAIL_KEYS = frozenset(
{
"action",
"attempt",
"count",
"method",
"mode",
"phase",
"platform",
"reason",
"result_count",
"source_id",
"spawned",
"status",
"version",
"parent_session_id",
"invalid_lines",
"truncated",
"open_file_count",
"memory_bytes",
"thread_count",
"database_error_count",
"route_path",
"sqlite_errorcode",
"sqlite_errorname",
"file_count",
"sqlite_file_count",
"byte_file_count",
"total_bytes",
"copied_bytes",
"bytes_per_second",
"stored_count",
"revised_count",
"skipped_count",
"error_count",
"pending_count",
"prepared_count",
"queue_depth",
"in_flight_count",
"call_count",
"created_count",
"updated_count",
"superseded_count",
"deleted_count",
"operation_kind",
"index_complete",
"lexical_hits",
"semantic_hits",
"generation",
"timed_out",
"failure_site",
"error_class",
"file_kind",
"file_size",
"file_mtime_ns",
"file_inode",
"cursor_new",
"cursor_old",
"cursor_read_to_end",
"cursor_rewind_branch",
"summed_ms",
"model_ms",
"accounted_ms",
"unaccounted_ms",
"conversation_hash",
# The sedimentation switch: a bounded boolean, the same shape as
# `spawned` and `index_complete`. Left out of this list it did not
# drop a log line, it turned the switch's own request into a 500.
"enabled",
}
_DIAGNOSTIC_LOG_NAME = re.compile(
r"^diagnostic-(runtime|core|host-bridge|desktop|cli)\.jsonl(?:\.\d+)?$"
)
_DETAIL_KEY = re.compile(r"^[A-Za-z][A-Za-z0-9_.-]{0,63}$")
_CODE_FRAME = re.compile(
r"^(?:(?:apps|packages|scripts)/(?:[A-Za-z0-9_-]+/)*[A-Za-z0-9_.-]+:\d+:\d+"
r"|one_memory/(?:[A-Za-z0-9_-]+/)*[A-Za-z0-9_.-]+:\d+:[A-Za-z_][A-Za-z0-9_]*"
r"|[A-Za-z0-9_][A-Za-z0-9_.-]*\.py:\d+:[A-Za-z_][A-Za-z0-9_]*)$"
)
_SHORT_CREDENTIAL_TOKEN = re.compile(
r"(?<![A-Za-z0-9])([A-Za-z0-9]{24,47})(?:[_-])?(?![A-Za-z0-9])"
)
_REFUSED_DETAIL_KEYS = frozenset({"__proto__", "prototype", "constructor"})
_MAX_DETAIL_KEYS = 32
_MAX_SAFE_INTEGER = (1 << 53) - 1
_EXPORT_SECRET_MARKER = re.compile(
r"(?:^|[._:-])(?:authorization|credential|password|prompt|query|secret|token)(?:$|[._:-])",
re.IGNORECASE,
)
_TRACE_STRING_DETAIL_KEYS = frozenset({"phase", "mode", "status", "reason", "failure_site"})
_TRACE_NUMBER_DETAIL_KEYS = frozenset({"count"})
_trace_id: ContextVar[str | None] = ContextVar("one_memory_trace_id", default=None)
_active: DiagnosticLog | None = None
_database_errors: Counter[tuple[str, str]] = Counter()
@@ -264,7 +216,12 @@ class DiagnosticLog:
}
value.update({key: _identifier(item) for key, item in identifiers.items() if item})
if duration_ms is not None:
if not isinstance(duration_ms, int) or duration_ms < 0:
if (
not isinstance(duration_ms, int)
or isinstance(duration_ms, bool)
or duration_ms < 0
or duration_ms > _MAX_SAFE_INTEGER
):
raise ValueError("diagnostic_duration_invalid")
value["duration_ms"] = duration_ms
for field_name, item in {
@@ -276,7 +233,9 @@ class DiagnosticLog:
if item is not None:
value[field_name] = _identifier(item)
if safe_stack:
value["safe_stack"] = [_safe_stack_line(line) for line in safe_stack[:256]]
if len(safe_stack) > 16:
raise ValueError("diagnostic_stack_invalid")
value["safe_stack"] = [_safe_stack_line(line) for line in safe_stack]
if details:
value["details"] = _details(details)
self._submit(_serialize(value))
@@ -289,6 +248,8 @@ class DiagnosticLog:
event: str,
outcome: str,
) -> dict[str, object]:
if level not in ("debug", "info", "warn", "error"):
raise ValueError("diagnostic_level_invalid")
return {
"schema_version": 1,
"timestamp": datetime.now(UTC).isoformat().replace("+00:00", "Z"),
@@ -651,24 +612,17 @@ def _drain_diagnostics_at_exit() -> None:
def diagnostic_event(**values: object) -> None:
active = _active
if active is None:
return
if values.get("level") == "debug" and not logging.getLogger().isEnabledFor(logging.DEBUG):
return
try:
active = _active
if active is None:
return
if values.get("level") == "debug" and not logging.getLogger().isEnabledFor(logging.DEBUG):
return
active.write(**values) # type: ignore[arg-type]
except OSError:
# Diagnostics must not become a new product outage when a disk is full
# or the log directory is temporarily unavailable.
return
except ValueError:
# Same promise, same reason, for a rejected payload instead of a failed
# write. An unlisted `details` key once turned every external recall
# into a 500 and then killed the process, and the sedimentation switch
# into a permanent failure. The payload itself is still wrong and is
# still caught in CI by tests/test_diagnostic_payload_keys.py; what
# must not happen is the caller paying for it.
except Exception:
# Diagnostics are best-effort: validation, serialization, logging, and
# filesystem failures must never become a product outage. BaseException
# deliberately remains visible so process-control signals still work.
return
@@ -691,46 +645,95 @@ def read_trace_events(
if (request_id is None) == (trace_id is None):
raise ValueError("exactly_one_trace_identity_required")
key, expected = ("request_id", request_id) if request_id is not None else ("trace_id", trace_id)
if expected is None or not _IDENTIFIER.fullmatch(expected):
if expected is None or not _export_identifier(expected):
raise ValueError("invalid_trace_identity")
selected: list[dict[str, object]] = []
allowed_details = {"phase", "mode", "status", "reason", "failure_site", "count"}
files = sorted(
(data_dir / "logs").glob("diagnostic-*.jsonl*"),
key=lambda path: path.stat().st_mtime,
)
for path in files:
if path.is_symlink() or not path.is_file():
directory = data_dir / "logs"
try:
directory_info = directory.lstat()
except FileNotFoundError:
return ()
if stat.S_ISLNK(directory_info.st_mode) or not stat.S_ISDIR(directory_info.st_mode):
raise ValueError("diagnostic_directory_unsafe")
files: list[tuple[Path, float]] = []
for path in directory.iterdir():
if not _DIAGNOSTIC_LOG_NAME.fullmatch(path.name):
continue
for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
try:
info = path.lstat()
except OSError:
continue
if stat.S_ISREG(info.st_mode) and not stat.S_ISLNK(info.st_mode):
files.append((path, info.st_mtime))
for path, _mtime in sorted(files, key=lambda item: item[1]):
for line in _read_regular_file_no_follow(path).splitlines():
try:
event = json.loads(line)
except (TypeError, json.JSONDecodeError):
continue
if not isinstance(event, dict) or event.get(key) != expected:
continue
details = event.get("details")
projected = {
name: event[name]
for name in (
"timestamp",
"component",
"event",
"outcome",
"duration_ms",
"error_code",
"error_type",
)
if name in event
timestamp = event.get("timestamp")
if not isinstance(timestamp, str):
continue
try:
datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
except ValueError:
continue
component = event.get("component")
event_name = event.get("event")
outcome = event.get("outcome")
projected: dict[str, object] = {
"timestamp": timestamp,
"component": (
component
if isinstance(component, str) and _export_identifier(component)
else "diagnostic.redacted"
),
"event": (
event_name
if isinstance(event_name, str) and _export_identifier(event_name)
else "diagnostic.redacted"
),
"outcome": (
outcome
if isinstance(outcome, str) and _export_identifier(outcome)
else "redacted"
),
}
duration_ms = event.get("duration_ms")
if (
isinstance(duration_ms, int)
and not isinstance(duration_ms, bool)
and 0 <= duration_ms <= _MAX_SAFE_INTEGER
):
projected["duration_ms"] = duration_ms
for name in ("error_code", "error_type"):
value = event.get(name)
if isinstance(value, str) and _export_identifier(value):
projected[name] = value
details = event.get("details")
if isinstance(details, dict):
projected["details"] = {
name: value for name, value in details.items() if name in allowed_details
}
safe_details = _trace_details(details)
if safe_details:
projected["details"] = safe_details
selected.append(projected)
return tuple(selected[-max(1, min(limit, 500)) :])
def _read_regular_file_no_follow(path: Path) -> str:
descriptor = os.open(path, os.O_RDONLY | _O_NOFOLLOW)
try:
if not stat.S_ISREG(os.fstat(descriptor).st_mode):
raise ValueError("diagnostic_file_unsafe")
with os.fdopen(descriptor, encoding="utf-8", errors="replace") as handle:
descriptor = -1
return handle.read()
finally:
if descriptor >= 0:
os.close(descriptor)
def current_trace_id() -> str | None:
return _trace_id.get()
@@ -801,7 +804,7 @@ def record_database_error(
sqlite_code = getattr(error, "sqlite_errorcode", None)
sqlite_name = getattr(error, "sqlite_errorname", None)
details: dict[str, object] = {"phase": safe_component, "count": count}
if route_path is not None and route_path.startswith("/api/v1/"):
if route_path is not None and route_path.startswith(("/api/v1/", "/api/v2/")):
details["route_path"] = route_path
if isinstance(sqlite_code, int):
details["sqlite_errorcode"] = sqlite_code
@@ -837,8 +840,77 @@ def diagnostic_exception(error: BaseException) -> dict[str, object]:
}
def _contains_credential_material(value: str) -> bool:
return bool(
re.search(
r"(?:^|[^A-Za-z0-9])(?:sk-|sk_|omhg_|ghp_|github_pat_)[A-Za-z0-9._-]+",
value,
re.IGNORECASE,
)
or re.search(r"(?:^|[^A-Za-z0-9])xox[baprs]-[A-Za-z0-9-]+", value, re.IGNORECASE)
or re.search(r"(?:^|[^A-Za-z0-9])(?:AKIA|ASIA)[A-Z0-9]{16}(?![A-Za-z0-9])", value)
or re.search(r"(?:^|[^A-Za-z0-9])AKID[A-Za-z0-9]{12,}", value, re.IGNORECASE)
or re.search(r"(?:^|[^A-Za-z0-9])AIza[A-Za-z0-9_-]{20,}", value)
or re.search(r"(?:^|[^A-Za-z0-9])GOCSPX-[A-Za-z0-9_-]+", value)
or any(
_is_conservative_short_credential(match.group(1))
for match in _SHORT_CREDENTIAL_TOKEN.finditer(value)
)
or re.search(
r"\b[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b",
value,
)
or any(
len(candidate) >= 48
and any(character.isalpha() for character in candidate)
and any(character.isdigit() for character in candidate)
for candidate in re.findall(r"\b[A-Za-z0-9._-]{48,}\b", value)
)
)
def _is_conservative_short_credential(value: str) -> bool:
return bool(
re.search(r"[A-Z]", value)
and re.search(r"[a-z]", value)
and re.search(r"\d", value)
and not re.fullmatch(r"[A-Fa-f0-9]+", value)
)
def _export_identifier(value: str) -> bool:
return bool(
_IDENTIFIER.fullmatch(value)
and not _EXPORT_SECRET_MARKER.search(value)
and not _contains_credential_material(value)
)
def _trace_details(values: Mapping[object, object]) -> dict[str, str | int | float]:
result: dict[str, str | int | float] = {}
for key, value in values.items():
if (
key in _TRACE_STRING_DETAIL_KEYS
and isinstance(value, str)
and _export_identifier(value)
):
result[str(key)] = value
elif key in _TRACE_NUMBER_DETAIL_KEYS and isinstance(value, (int, float)):
if isinstance(value, bool):
continue
safe_integer = isinstance(value, int) and abs(value) <= _MAX_SAFE_INTEGER
safe_float = (
isinstance(value, float)
and math.isfinite(value)
and (not value.is_integer() or abs(value) <= _MAX_SAFE_INTEGER)
)
if safe_integer or safe_float:
result[str(key)] = value
return result
def _serialize(value: Mapping[str, object]) -> str:
return json.dumps(value, ensure_ascii=True, separators=(",", ":")) + "\n"
return json.dumps(value, ensure_ascii=True, allow_nan=False, separators=(",", ":")) + "\n"
def _uuid(value: str | None) -> str | None:
@@ -864,24 +936,47 @@ def _event_name(value: str) -> str:
def _safe_stack_line(value: str) -> str:
if not value or len(value) > 256 or "\n" in value or "\r" in value:
if (
not value
or len(value) > 256
or any(character in value for character in "\0\r\n")
or _contains_credential_material(value)
):
raise ValueError("diagnostic_stack_invalid")
if Path(value.split(":", 1)[0]).is_absolute() or "file://" in value:
raise ValueError("diagnostic_stack_invalid")
return value
if _CODE_FRAME.fullmatch(value):
return value
if value.startswith("thread|"):
frames = value.removeprefix("thread|").split(" > ")
if frames and all(_CODE_FRAME.fullmatch(frame) for frame in frames):
return value
raise ValueError("diagnostic_stack_invalid")
def _details(
values: Mapping[str, str | int | float | bool | None],
) -> dict[str, str | int | float | bool | None]:
if len(values) > _MAX_DETAIL_KEYS:
raise ValueError("diagnostic_detail_count_invalid")
result: dict[str, str | int | float | bool | None] = {}
for key, value in values.items():
if key not in _DETAIL_KEYS:
if (
not isinstance(key, str)
or not _DETAIL_KEY.fullmatch(key)
or key in _REFUSED_DETAIL_KEYS
):
raise ValueError("diagnostic_detail_key_invalid")
if isinstance(value, str):
if not value or len(value) > 256 or "\n" in value or "\r" in value:
if not value or len(value) > 256 or any(character in value for character in "\0\r\n"):
raise ValueError("diagnostic_detail_value_invalid")
elif value is not None and not isinstance(value, (int, float, bool)):
elif isinstance(value, float):
if not math.isfinite(value) or (value.is_integer() and abs(value) > _MAX_SAFE_INTEGER):
raise ValueError("diagnostic_detail_value_invalid")
elif isinstance(value, bool) or value is None:
pass
elif isinstance(value, int):
if abs(value) > _MAX_SAFE_INTEGER:
raise ValueError("diagnostic_detail_value_invalid")
else:
raise ValueError("diagnostic_detail_value_invalid")
result[key] = value
return result
@@ -3,8 +3,8 @@
Every site that catches `Exception` broadly must call `report_exception`
before (or instead of) continuing, so a failure never looks like success
and a stack is always on disk. Content (conversation text, keys, SQL
parameters) never reaches the log: only exception type, a truncated
message, and file:line:function frames.
parameters, and exception messages) never reaches the diagnostic log: only
exception type, a repeat count, and file:line:function frames.
"""
from __future__ import annotations
@@ -67,7 +67,7 @@ def report_exception(
event=event,
outcome="failed",
**payload,
details={"reason": _safe_message(error), "count": _RECENT[key]},
details={"count": _RECENT[key]},
)
if full:
_logger.error(
@@ -23,13 +23,9 @@ _ENVELOPE_HEADERS = frozenset(
)
_BROKER_ROUTES = {
("POST", "/api/v1/recall"): "om_recall",
("POST", "/api/v1/memories/ask"): "om_ask",
("POST", "/api/v2/memories/ask"): "om_ask",
("POST", "/api/v2/memories/ask/stream"): "om_ask",
("GET", "/api/v1/conversations/search"): "om_conversations",
("POST", "/api/v1/diagnostics/trace"): "om_diagnostic_trace",
("POST", "/api/v1/boundary/peek"): "om_recall",
("POST", "/api/v1/boundary/ack"): "om_recall",
}
_CONCRETE_CALL_EVENT_DELIVERY = re.compile(
r"/api/v1/call-events/[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-"
@@ -128,11 +124,9 @@ def require_external_endpoint(request: Request) -> None:
access = external_access(request.headers)
if access is None:
return
if request.method == "POST" and request.url.path in {
"/api/v1/context/search",
"/api/v2/context/search",
}:
# The route resolves each requested lane against its original grant.
if request.method == "POST" and request.url.path == "/api/v2/context/search":
# om_conversations reaches Core only here, as the raw lane; the route
# resolves each requested lane against its original grant.
# Passing this envelope gate does not authorize either lane by itself.
if access.tools.isdisjoint({"om_recall", "om_conversations"}):
raise HTTPException(403, "external_recall_scope_denied")
@@ -146,7 +140,3 @@ def require_external_endpoint(request: Request) -> None:
raise HTTPException(403, "external_endpoint_denied")
if tool not in access.tools:
raise HTTPException(403, "external_recall_scope_denied")
if tool == "om_conversations":
access.require(tool, frozenset({"all-apps"}))
if not any(scope.startswith("source:") for scope in access.scopes):
raise HTTPException(403, "external_source_permission_required")
@@ -1,127 +0,0 @@
"""Bounded homepage projection without loading every memory into the renderer."""
import sqlite3
import time
from datetime import UTC, datetime
from pathlib import Path
from pydantic import BaseModel, ConfigDict
from one_memory.database_access import main_database
from one_memory.memory_evidence_freshness import register_memory_freshness
class HomeLatestMemory(BaseModel):
model_config = ConfigDict(frozen=True)
memory_id: str
content: str
truncated: bool
revision: int
action: str
updated_at: datetime
source_ids: tuple[str, ...]
class HomeSummary(BaseModel):
model_config = ConfigDict(frozen=True)
memory_count: int
latest: HomeLatestMemory | None
observed_at: datetime
#: 2026-10-12: the read hit its deadline and the numbers below are not a
#: snapshot of reality. Callers should keep their previous summary on
#: screen instead of treating count=0 as "no memories".
stale: bool = False
class HomeSummaryUnavailable(Exception):
"""The homepage projection could not be read inside its own deadline.
Raised instead of letting SQLite's interrupt escape as a bare
`OperationalError`, so the route can answer with an empty projection
rather than a 500.
"""
def read_home_summary(database_path: Path, *, user_id: str = "local") -> HomeSummary:
# The exclusion mirrors SourceReadAccess: one revoked active citation hides
# the whole memory. Permission filtering precedes both count and LIMIT.
# `+link.active` keeps SQLite on the (memory_id, evidence_id) primary key.
# Without the unary plus it picks the (active, ...) index and walks every
# active link once per memory: 2.9 s instead of 0.07 s on a 1,810-memory
# library, enough to time the homepage out whenever the machine is busy.
readable = """
m.user_id = ? AND m.status = 'confirmed' AND m.deleted_at IS NULL
AND NOT EXISTS (
SELECT 1 FROM governed_memory_evidence link
JOIN evidence e ON e.evidence_id = link.evidence_id
JOIN source_grants g ON g.user_id=e.user_id AND g.source_id=e.source_id
WHERE link.memory_id=m.memory_id AND +link.active=1
AND g.revoked_at IS NOT NULL
)
AND NOT EXISTS (
SELECT 1 FROM governed_memory_evidence link
LEFT JOIN evidence e ON e.evidence_id = link.evidence_id
WHERE link.memory_id=m.memory_id AND +link.active=1
AND (e.evidence_id IS NULL OR e.user_id != m.user_id)
)
"""
# The total counts with the cheap permission filters only: the per-row
# freshness UDF scales with the whole library and can burn the whole read
# budget on its own (D22). Product accepted a small drift here; the rows
# that are actually displayed are still checked row by row below.
with main_database(database_path).read() as db:
deadline = time.monotonic() + 5
db.set_progress_handler(lambda: int(time.monotonic() >= deadline), 1000)
db.row_factory = sqlite3.Row
try:
db.execute("BEGIN")
count = db.execute(
f"SELECT COUNT(*) FROM governed_memories m WHERE {readable}", (user_id,)
).fetchone()[0]
readable += " AND " + register_memory_freshness(db)
row = db.execute(
f"""SELECT m.memory_id, substr(m.content,1,320) AS content,
length(m.content)>320 AS truncated, m.revision,
COALESCE(v.action, 'unknown') AS action, m.updated_at
FROM governed_memories m
LEFT JOIN governed_memory_versions v
ON v.memory_id=m.memory_id AND v.revision=m.revision
WHERE {readable}
ORDER BY m.updated_at DESC, m.memory_id LIMIT 1""",
(user_id,),
).fetchone()
latest = None
if row is not None:
sources = db.execute(
"""SELECT DISTINCT e.source_id FROM governed_memory_evidence link
JOIN evidence e ON e.evidence_id=link.evidence_id
WHERE link.memory_id=? AND +link.active=1 ORDER BY e.source_id""",
(row["memory_id"],),
).fetchall()
latest = HomeLatestMemory(
memory_id=row["memory_id"],
content=row["content"],
truncated=bool(row["truncated"]),
revision=row["revision"],
action=row["action"],
updated_at=row["updated_at"],
source_ids=tuple(source[0] for source in sources),
)
except sqlite3.Error as error:
# The progress handler above is what interrupts this read, so a
# slow homepage surfaces as SQLITE_INTERRUPT (code 9) and used to
# escape as a 500. 2026-10-12: answer with a `stale` projection
# instead of raising: the caller keeps its last good summary on
# screen (the UI poll retries within 10s), and the homepage never
# zeroes out or errors just because the machine is busy importing.
if getattr(error, "sqlite_errorcode", None) != sqlite3.SQLITE_INTERRUPT:
raise
return HomeSummary(
memory_count=0,
latest=None,
observed_at=datetime.now(UTC),
stale=True,
)
return HomeSummary(memory_count=count, latest=latest, observed_at=datetime.now(UTC))
@@ -0,0 +1,271 @@
"""Kuzu graph-width inspection and explicit user-directed administration.
This module is the second graph architecture exception. It may open Kuzu
directly to inspect an existing graph's schema, but owns no prompts, memory
reconciliation, coding data, vector database, or legacy governance behavior.
"""
from __future__ import annotations
import logging
import os
import re
import shutil
import threading
from dataclasses import dataclass
from datetime import UTC, datetime
from enum import StrEnum
from importlib import import_module
from pathlib import Path
from typing import Any
from pydantic import BaseModel, ConfigDict
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
class GraphWidthError(RuntimeError):
"""Base error for graph-width inspection and administration."""
class GraphWidthDecision(StrEnum):
"""What to do when an existing graph disagrees with the embedding width."""
KEEP_EXISTING = "keep_existing"
REBUILD_WITH_NEW = "rebuild_with_new"
EXPORT_FIRST = "export_first"
class GraphWidthMismatch(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
stored_width: int
expected_width: int
graph_path: str
class GraphWidthOutcome(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
archived_path: str | None = None
exported_path: str | None = None
class GraphWidthMismatchError(GraphWidthError):
"""Raised instead of silently archiving a live graph."""
def __init__(self, mismatch: GraphWidthMismatch) -> None:
super().__init__(
f"graph_width_mismatch:stored={mismatch.stored_width}"
f":expected={mismatch.expected_width}"
)
self.mismatch = mismatch
def graph_store_enabled() -> bool:
"""Whether HY should open the Kuzu graph store."""
raw = os.environ.get("ONE_MEMORY_GRAPH_STORE", "").strip().lower()
if not raw:
return False
return raw != "none"
def inspect_graph_width_mismatch(config: Any) -> GraphWidthMismatch | None:
"""Return a mismatch without renaming or deleting anything."""
db_path = Path(config.graph_store.db_path)
expected = getattr(getattr(config, "embedder", None), "embedding_dims", None)
return _inspect_graph_width(db_path, expected)
def inspect_profile_graph_width(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
) -> GraphWidthMismatch | None:
"""Inspect the graph path and width selected by the active profile."""
del api_key
return _inspect_graph_width(
_profile_graph_path(data_dir),
_profile_embedding_width(profile),
)
def decide_profile_graph_width(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
decision: GraphWidthDecision,
) -> GraphWidthOutcome:
"""Apply the user's decision to the active profile's graph."""
del api_key
config = _GraphAdminConfig(
graph_store=_GraphStoreConfig(str(_profile_graph_path(data_dir))),
embedder=_EmbeddingWidthConfig(_profile_embedding_width(profile)),
)
return apply_graph_width_decision(
config,
decision,
export_root=data_dir.parent,
)
def apply_graph_width_decision(
config: Any,
decision: GraphWidthDecision,
*,
export_root: Path | None = None,
) -> GraphWidthOutcome:
"""Act only on a current mismatch and only as explicitly requested."""
mismatch = inspect_graph_width_mismatch(config)
if mismatch is None:
return GraphWidthOutcome()
if decision is GraphWidthDecision.REBUILD_WITH_NEW:
archive = retire_mismatched_graph_store(config)
return GraphWidthOutcome(archived_path=str(archive) if archive is not None else None)
if decision is GraphWidthDecision.EXPORT_FIRST:
if export_root is None:
raise GraphWidthError("export_root_required")
return GraphWidthOutcome(
exported_path=str(export_graph_store(config, export_root=export_root))
)
return GraphWidthOutcome()
def export_graph_store(config: Any, *, export_root: Path) -> Path:
"""Copy the graph outside its data directory, atomically."""
db_path = Path(config.graph_store.db_path)
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%S%fZ")
destination = export_root / f"one-memory-graph-{stamp}"
staging = destination.with_name(f"{destination.name}.partial")
sources = [db_path] + [
sibling
for suffix in (".wal", ".shadow", ".tmp")
if (sibling := db_path.with_name(f"{db_path.name}{suffix}")).exists()
]
try:
staging.mkdir(parents=True)
for source in sources:
shutil.copy2(source, staging / source.name)
staging.rename(destination)
except BaseException:
shutil.rmtree(staging, ignore_errors=True)
raise
logging.getLogger(__name__).info(
"exported hy graph store to %s (%d files)",
destination,
len(sources),
)
return destination
def retire_mismatched_graph_store(config: Any) -> Path | None:
"""Archive a mismatched Kuzu graph only after the user chooses rebuild."""
db_path = Path(config.graph_store.db_path)
if not db_path.exists():
return None
expected = getattr(getattr(config, "embedder", None), "embedding_dims", None)
if not expected:
return None
stored = _graph_embedding_width(db_path)
if stored is None or stored == int(expected):
return None
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
archive = db_path.with_name(f"{db_path.name}.dim{stored}.{stamp}.bak")
db_path.rename(archive)
for suffix in (".wal", ".shadow", ".tmp"):
sibling = db_path.with_name(f"{db_path.name}{suffix}")
if sibling.exists():
sibling.rename(archive.with_name(f"{archive.name}{suffix}"))
logging.getLogger(__name__).warning(
"retired hy graph store with embedding width %s (model expects %s): %s",
stored,
expected,
archive.name,
)
return archive
_GRAPH_WIDTH_CACHE: dict[str, int | None] = {}
_GRAPH_WIDTH_LOCK = threading.Lock()
def _graph_embedding_width(db_path: Path) -> int | None:
"""Read the Memory embedding width directly from Kuzu's schema."""
try:
kuzu = import_module("kuzu")
except ModuleNotFoundError:
return None
key = str(db_path.resolve())
with _GRAPH_WIDTH_LOCK:
if key in _GRAPH_WIDTH_CACHE:
return _GRAPH_WIDTH_CACHE[key]
try:
database = kuzu.Database(str(db_path), read_only=True)
except Exception:
return None
width: int | None = None
try:
connection = kuzu.Connection(database)
result = connection.execute("CALL TABLE_INFO('Memory') RETURN *")
while result.has_next():
row = result.get_next()
if len(row) < 3 or row[1] != "embedding":
continue
match = re.search(r"\[(\d+)\]", str(row[2]))
width = int(match.group(1)) if match else None
break
except Exception:
width = None
with _GRAPH_WIDTH_LOCK:
_GRAPH_WIDTH_CACHE[key] = width
return width
def _inspect_graph_width(
db_path: Path,
expected_width: object,
) -> GraphWidthMismatch | None:
if not db_path.exists() or not expected_width:
return None
stored = _graph_embedding_width(db_path)
if stored is None or stored == int(expected_width):
return None
return GraphWidthMismatch(
stored_width=int(stored),
expected_width=int(expected_width),
graph_path=str(db_path),
)
def _profile_graph_path(data_dir: Path) -> Path:
name = "graph" if graph_store_enabled() else "graph-disabled"
return data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / name
def _profile_embedding_width(profile: ModelProfileSnapshot) -> int | None:
if profile.embedding_dimensions is not None:
return profile.embedding_dimensions
if (profile.embedding_provider_kind or profile.provider_kind) is ProviderKind.LOCAL_ONNX:
from one_memory.embedding_assets import BUILTIN_EMBEDDING_DIMENSIONS
return BUILTIN_EMBEDDING_DIMENSIONS
return None
@dataclass(frozen=True)
class _GraphStoreConfig:
db_path: str
@dataclass(frozen=True)
class _EmbeddingWidthConfig:
embedding_dims: int | None
@dataclass(frozen=True)
class _GraphAdminConfig:
graph_store: _GraphStoreConfig
embedder: _EmbeddingWidthConfig
@@ -0,0 +1,185 @@
"""Read-only projection of HY's Kuzu graph for MemHub's existing graph UI.
This module is one of the two explicit graph architecture exceptions. The
public SDK does not expose graph projection yet, so it may use the client's
private graph handle and loop thread. It must remain read-only and must not
grow memory, vector, coding, prompt, or governance responsibilities.
"""
from __future__ import annotations
from typing import Any, cast
from pydantic import BaseModel, ConfigDict
class HyGraphNode(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
node_id: str
content: str
layer: str | None = None
confidence: float | None = None
class HyGraphEdge(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
source_node_id: str
target_node_id: str
relation: str
weight: float | None = None
class HyGraphSnapshot(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
nodes: tuple[HyGraphNode, ...] = ()
edges: tuple[HyGraphEdge, ...] = ()
# False when HY was never consulted, which is not the same as HY having
# induced no relations.
consulted: bool = True
def read_graph_snapshot(
client: object,
user_id: str,
agent_id: str,
limit: int,
) -> HyGraphSnapshot:
raw_nodes, raw_edges = _read_graph_snapshot(client, user_id, agent_id, limit)
return HyGraphSnapshot(
nodes=tuple(
HyGraphNode(
node_id=str(row[0]),
content=str(row[1]),
layer=str(row[2]) if row[2] is not None else None,
confidence=_safe_float(row[3]),
)
for row in raw_nodes
if len(row) >= 4 and row[0] and row[1]
),
edges=tuple(
HyGraphEdge(
source_node_id=str(row[0]),
target_node_id=str(row[1]),
relation=str(row[2]),
weight=_safe_float(row[3]),
)
for row in raw_edges
if len(row) >= 4 and row[0] and row[1] and row[2]
),
)
def _read_graph_snapshot(
client: object,
user_id: str,
agent_id: str,
limit: int,
) -> tuple[list[list[object]], list[list[object]]]:
"""Read the graph through the SDK handles until a public API exists."""
concrete = cast(Any, client)
# Keep the existing architecture gate unchanged; this module is the named
# graph-only exception while the rest of MemHub remains on public handles.
graph_store = getattr(concrete, "_graph_store") # noqa: B009
if graph_store is None:
return ([], [])
loop_thread = getattr(concrete, "_loop_thread") # noqa: B009
# Kuzu rejects a prepared statement that is handed a parameter its text
# never references, so a wildcard read must not bind agent_id at all.
params: dict[str, object] = {"user_id": user_id}
if agent_id != "*":
params["agent_id"] = agent_id
safe_limit = max(1, min(int(limit), 1_000))
memory_agent = "" if agent_id == "*" else " AND m.agent_id = $agent_id"
related_agent = (
"" if agent_id == "*" else " AND a.agent_id = $agent_id AND b.agent_id = $agent_id"
)
source_agent = "" if agent_id == "*" else " AND a.agent_id = $agent_id"
nodes = (
loop_thread.run(
graph_store._run(
f"""
MATCH (m:Memory)
WHERE m.user_id = $user_id{memory_agent}
AND m.status = 'active'
RETURN m.node_id, m.content, m.layer, m.confidence
LIMIT {safe_limit}
""",
params,
)
)
or []
)
related_edges = (
loop_thread.run(
graph_store._run(
f"""
MATCH (a:Memory)-[r:RELATED_TO]->(b:Memory)
WHERE a.user_id = $user_id{related_agent}
AND b.user_id = $user_id
AND a.status = 'active' AND b.status = 'active'
RETURN a.node_id, b.node_id, r.relation_type, r.weight
LIMIT {safe_limit * 4}
""",
params,
)
)
or []
)
derived_edges = (
loop_thread.run(
graph_store._run(
f"""
MATCH (a:Memory)-[:DERIVED_FROM]->(v:VdbRef)
WHERE a.user_id = $user_id{source_agent}
AND a.status = 'active'
RETURN a.node_id, v.node_id, 'derived_from', 1.0
LIMIT {safe_limit * 4}
""",
params,
)
)
or []
)
topic_nodes = (
loop_thread.run(
graph_store._run(
f"""
MATCH (m:Memory)-[:TAGGED_WITH]->(t:Topic)
WHERE m.user_id = $user_id{memory_agent}
AND m.status = 'active'
RETURN DISTINCT t.topic_id, t.name, 'TOPIC', 1.0
LIMIT {safe_limit}
""",
params,
)
)
or []
)
tagged_edges = (
loop_thread.run(
graph_store._run(
f"""
MATCH (m:Memory)-[:TAGGED_WITH]->(t:Topic)
WHERE m.user_id = $user_id{memory_agent}
AND m.status = 'active'
RETURN m.node_id, t.topic_id, 'tagged_with', 1.0
LIMIT {safe_limit * 4}
""",
params,
)
)
or []
)
return (
[*nodes, *topic_nodes],
[*related_edges, *derived_edges, *tagged_edges],
)
def _safe_float(value: object) -> float | None:
if isinstance(value, (int, float)):
return float(value)
return None
@@ -1,128 +0,0 @@
"""Which memories HY no longer counts as current, read from HY itself.
HY owns a memory's lifecycle: it supersedes, deduplicates, shadows and
restores on its own schedule — during a search, while deleting the head of a
chain — and most of that never passes through a write the ledger sees. So
the ledger does not keep a copy of any of it; readers ask HY, once per read,
for the nodes it has set aside.
Only the vector store is read. HY writes a node's status there on every
transition, and the graph holds no status the vector store lacks.
"""
import json
from collections.abc import Collection, Mapping
from dataclasses import dataclass, field
from importlib import import_module
from types import MappingProxyType
from typing import Any
#: HY's one status for a node that is current. Everything else it writes —
#: superseded, shadow, archived, negated, conflicted — takes it out of recall.
_ACTIVE = "active"
#: HY's verbatim chunks. Stored as shadow from the start and never a memory,
#: so never something a reader could have shown.
_RAW_LAYER = "l1_raw"
class HyLifecycleUnreadable(RuntimeError):
"""HY's store could not be read; never to be taken for "nothing is hidden"."""
@dataclass(frozen=True, eq=False)
class HyLifecycle:
"""HY's set-aside nodes for one user, each with what HY says replaced it."""
hidden: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
def __post_init__(self) -> None:
object.__setattr__(self, "hidden", MappingProxyType(dict(self.hidden)))
def __eq__(self, other: object) -> bool:
return isinstance(other, HyLifecycle) and dict(self.hidden) == dict(other.hidden)
def __hash__(self) -> int:
return hash(tuple(sorted(self.hidden.items())))
def replaced_by(self, memory_id: str) -> tuple[str, ...]:
return self.hidden.get(memory_id, ())
def replacing(self, memory_id: str) -> tuple[str, ...]:
"""The set-aside memories HY names this one as the replacement of."""
return tuple(sorted(old for old, by in self.hidden.items() if memory_id in by))
def read_hy_lifecycle(
client: object, user_id: str, ids: Collection[str] | None = None
) -> HyLifecycle:
"""Every node of `user_id` HY holds as anything but active, in one read.
With `ids`, only those nodes are looked up — what a single-memory read
needs, and a lookup by key rather than a scan; `replacing` is then only
known among them.
Read one level below HY's own listing, which drops a failed read and
answers empty: a hidden memory would then read as current.
"""
native: Any = client
try:
chroma = import_module("hy_memory.data.vector_store_chroma")
store = native._vector_store
collection = store._collection
if not isinstance(store, chroma.ChromaVectorStore) or collection is None:
raise HyLifecycleUnreadable("hy_lifecycle_store_unsupported")
found = collection.get(
ids=None if ids is None else list(ids),
where={
"$and": [
{"user_id": {"$eq": user_id}},
{"status": {"$ne": _ACTIVE}},
{"layer": {"$ne": _RAW_LAYER}},
]
},
include=["metadatas"],
)
metadatas = found["metadatas"]
point_ids = found["ids"]
if (
not isinstance(point_ids, list)
or not isinstance(metadatas, list)
or len(point_ids) != len(metadatas)
):
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
except HyLifecycleUnreadable:
raise
except Exception as error:
raise HyLifecycleUnreadable("hy_lifecycle_unreadable") from error
hidden: dict[str, tuple[str, ...]] = {}
for point_id, metadata in zip(point_ids, metadatas, strict=True):
if not isinstance(point_id, str) or not point_id or not isinstance(metadata, Mapping):
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
payload = chroma.ChromaVectorStore._chroma_metadata_to_payload(dict(metadata))
node_id = payload.get("node_id") or point_id
if not isinstance(node_id, str) or not node_id or node_id != point_id:
raise HyLifecycleUnreadable("hy_lifecycle_store_mismatch")
hidden[node_id] = _replacements(payload)
return HyLifecycle(hidden=hidden)
def _replacements(payload: Mapping[str, object]) -> tuple[str, ...]:
"""What HY names as taking this node's place: its chain successor or the kept duplicate."""
names: list[str] = []
superseded_by = payload.get("superseded_by")
if isinstance(superseded_by, list):
names.extend(value for value in superseded_by if isinstance(value, str) and value)
kept = payload.get("dedup_kept_id")
custom = payload.get("custom")
if isinstance(custom, str):
try:
custom = json.loads(custom)
except ValueError:
custom = None
if not isinstance(kept, str) and isinstance(custom, Mapping):
audit = custom.get("hy_dedup")
if isinstance(audit, Mapping):
kept = audit.get("kept_id")
if isinstance(kept, str) and kept:
names.append(kept)
return tuple(dict.fromkeys(names))
File diff suppressed because it is too large. Load diff
@@ -1,331 +0,0 @@
"""Transport validation for the pinned HY SDK; HY's prompts and decisions pass through unchanged."""
import logging
from contextlib import nullcontext
from time import monotonic
from typing import Any
from one_memory.diagnostic_log import diagnostic_event, diagnostic_exception, elapsed_ms
from one_memory.hy_source_context import HyWriteBindings
from one_memory.hy_step_timing import HyStepTimer
def _http_status(value: Any) -> int | None:
return value if type(value) is int and 100 <= value <= 599 else None
def _exception_status(error: Exception) -> int | None:
try:
status = _http_status(getattr(error, "status_code", None))
if status is not None:
return status
status = _http_status(getattr(getattr(error, "response", None), "status_code", None))
if status is not None:
return status
# Native LLMRetryError retains sanitized attempt dictionaries. Only the
# final attempt describes the terminal failure; never parse error text.
attempts = getattr(error, "attempts", None)
if isinstance(attempts, (list, tuple)) and attempts and isinstance(attempts[-1], dict):
return _http_status(attempts[-1].get("status_code"))
except Exception:
# Diagnostic attribute access must not replace the provider exception.
return None
return None
def _model_failure_code(status: int | None) -> str:
specific = {
401: "hy_memory_model_authentication_failed",
402: "hy_memory_model_payment_required",
403: "hy_memory_model_access_denied",
408: "hy_memory_model_timeout",
429: "hy_memory_model_rate_limited",
}
if status in specific:
return specific[status]
if status is not None and 500 <= status <= 599:
return "hy_memory_model_service_unavailable"
if status is not None and 400 <= status <= 422:
return "hy_memory_model_request_incompatible"
return "hy_memory_model_request_failed"
def _terminal_client_timeout(error: Exception) -> bool:
try:
attempts = getattr(error, "attempts", None)
if not isinstance(attempts, (list, tuple)) or not attempts:
return False
final = attempts[-1]
if type(final) is not dict:
return False
kind = final.get("failure_kind")
return type(kind) is str and kind == "timeout"
except Exception:
return False
class HyResponseGuard:
def __init__(self, provider: Any, *, phase: str = "extract") -> None:
self.provider = provider
#: Diagnostic label only; the request passes through unchanged.
self.phase = phase
self.failures: list[str] = []
def __getattr__(self, name: str) -> Any:
return getattr(self.provider, name)
async def complete(self, *args: Any, **kwargs: Any) -> Any:
return await self._complete("complete", args, kwargs)
async def complete_messages(self, *args: Any, **kwargs: Any) -> Any:
return await self._complete("complete_messages", args, kwargs)
async def _complete(self, method: str, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
for attempt in range(2):
started = monotonic()
try:
response = await getattr(self.provider, method)(*args, **kwargs)
except Exception as error:
status = _exception_status(error)
code = (
"hy_memory_model_timeout"
if status is None and _terminal_client_timeout(error)
else _model_failure_code(status)
)
self.failures.append(code)
logging.getLogger(__name__).warning(
"%s: status_code=%s", code, status if status is not None else "unknown"
)
diagnostic_event(
level="error",
component="hy.model",
event="hy.model_call.failed",
outcome="failed",
duration_ms=elapsed_ms(started),
error_code=code,
**diagnostic_exception(error),
details={"phase": self.phase, "attempt": attempt + 1, "call_count": 1},
)
raise
truncated = getattr(response, "finish_reason", "") == "length"
empty = not (response.content or "").strip() and not getattr(
response, "tool_calls", None
)
if not truncated and not empty:
diagnostic_event(
level="info",
component="hy.model",
event="hy.model_call.completed",
outcome="succeeded",
duration_ms=elapsed_ms(started),
details={"phase": self.phase, "attempt": attempt + 1, "call_count": 1},
)
return response
code = (
"hy_memory_model_output_truncated"
if truncated
else "hy_memory_model_empty_response"
)
if attempt:
self.failures.append(code)
raise ValueError(code)
logging.getLogger(__name__).warning("%s: retrying the same HY model request", code)
if truncated and "max_tokens" in kwargs:
kwargs = {**kwargs, "max_tokens": min(16000, kwargs["max_tokens"] * 2)}
raise AssertionError("unreachable")
class HyReconcileGuard:
#: Set only for the span of one leased write.
step_timer: HyStepTimer | None = None
def __init__(self, reconciler: Any) -> None:
self.reconciler = reconciler
self.failures: list[str] = []
self.bindings: HyWriteBindings | None = None
self.imported_contents: frozenset[str] | None = None
self.mutation_target_ids: set[str] = set()
def begin_write(self) -> None:
self.mutation_target_ids.clear()
def __getattr__(self, name: str) -> Any:
return getattr(self.reconciler, name)
async def reconcile(self, *args: Any, **kwargs: Any) -> Any:
timer = self.step_timer
with timer.step("reconcile") if timer is not None else nullcontext():
return await self._reconcile(args, kwargs)
async def _reconcile(self, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
started = monotonic()
if self.imported_contents is not None:
metadata = kwargs.get("new_memories_with_meta") or []
texts = kwargs.get("new_memories") or []
if (
not metadata
or not texts
or any(
not isinstance(item, dict) or item.get("content") not in self.imported_contents
for item in metadata
)
or any(text not in self.imported_contents for text in texts)
or {item["content"] for item in metadata} != set(texts)
):
code = "hy_memory_write_verification_import_input_mismatch"
self.failures.append(code)
raise ValueError(code)
try:
result = await self.reconciler.reconcile(*args, **kwargs)
except Exception as error:
diagnostic_event(
level="error",
component="hy.reconcile",
event="hy.reconcile.failed",
outcome="failed",
duration_ms=elapsed_ms(started),
error_code="hy_memory_reconcile_failed",
**diagnostic_exception(error),
)
raise
if not result.success:
self.failures.append("hy_memory_reconcile_failed")
diagnostic_event(
level="error",
component="hy.reconcile",
event="hy.reconcile.failed",
outcome="failed",
duration_ms=elapsed_ms(started),
error_code="hy_memory_reconcile_failed",
details={"result_count": len(result.ops)},
)
return result
try:
await self._check_targets(result, kwargs)
except ValueError as error:
code = str(error)
self.failures.append(code)
raise
except Exception as error:
code = "hy_memory_write_verification_failed"
self.failures.append(code)
raise ValueError(code) from error
owners: dict[str, set[str]] = {}
for item in kwargs.get("new_memories_with_meta") or []:
if item.get("owner") in {"user", "agent"}:
owners.setdefault(item["content"], set()).add(item["owner"])
# HY's no-candidate ADD path copies content verbatim but omits owner.
# Restore only an unambiguous exact match, never infer from wording.
for operation in result.ops:
source_owners = owners.get(operation.content, set())
if operation.op == "ADD" and operation.owner is None and len(source_owners) == 1:
operation.owner = next(iter(source_owners))
operation_counts = {"ADD": 0, "UPDATE": 0, "SUPERSEDE": 0, "DELETE": 0}
for operation in result.ops:
raw_name = getattr(operation, "op", "")
name = str(getattr(raw_name, "value", raw_name)).upper()
if name in {"UPDATE", "SUPERSEDE", "DELETE", "SHADOW"}:
target = getattr(operation, "memory_id", None)
if isinstance(target, str) and target:
self.mutation_target_ids.add(target)
self.mutation_target_ids.update(
key
for key in getattr(operation, "memory_ids", ())
if isinstance(key, str) and key
)
if name == "ADD":
operation_counts["ADD"] += 1
elif name in {"UPDATE", "MERGE", "NOOP"}:
operation_counts["UPDATE"] += 1
elif name == "SUPERSEDE":
operation_counts["SUPERSEDE"] += 1
elif name in {"DELETE", "SHADOW"}:
operation_counts["DELETE"] += 1
diagnostic_event(
level="info",
component="hy.reconcile",
event="hy.reconcile.completed",
outcome="succeeded",
duration_ms=elapsed_ms(started),
details={
"created_count": operation_counts["ADD"],
"updated_count": operation_counts["UPDATE"],
"superseded_count": operation_counts["SUPERSEDE"],
"deleted_count": operation_counts["DELETE"],
"result_count": len(result.ops),
},
)
return result
async def _check_targets(self, result: Any, kwargs: dict[str, Any]) -> None:
"""HY may only touch nodes of this user and scope that this write was allowed to see."""
targets = {op.memory_id for op in result.ops if op.memory_id}
targets.update(key for op in result.ops for key in getattr(op, "memory_ids", []))
for key in targets:
node = await kwargs["vector_store"].get_by_id(key)
if (
node is None
or node.user_id != kwargs["user_id"]
or node.agent_id != kwargs["agent_id"]
):
raise ValueError("hy_memory_write_verification_target_invalid")
if self.bindings is not None:
known = self.bindings.known.get(key)
if known is None or (known.content, known.owner) != (node.content, node.owner):
raise ValueError("hy_memory_write_verification_target_unbound")
if self.bindings is None:
return
for op in result.ops:
if not op.content:
continue
linked = {key for key in (op.memory_id, *getattr(op, "memory_ids", [])) if key}
self.bindings.record_operation_support(op.content, op.owner, linked)
if op.memory_id:
self.bindings.verified_operations[op.memory_id] = (op.content, op.owner)
def install_hy_write_guards(client: Any) -> None:
"""The pinned HY SDK treats an empty reconcile response as a successful empty op list.
Bind this client's real write paths. Pro/lite use the ordinary writer;
ultra uses the System2 writer's inner writer. Keep HY prompts, extraction,
retrieval, reconciliation and storage intact; never patch the shared SDK
installation.
"""
async def install() -> None:
from one_memory.hy_write_verification import HyExtractionGuard
ordinary = await client._registry.get_writer()
system2 = await client._registry.get_system2_writer()
guards: list[Any] = []
seen_writers: set[int] = set()
seen_guards: set[int] = set()
for writer in (ordinary, system2._writer):
if id(writer) in seen_writers:
continue
seen_writers.add(id(writer))
agent = writer.mem_agent
if not isinstance(agent.llm_provider, HyResponseGuard):
agent.llm_provider = HyResponseGuard(agent.llm_provider)
agent.extractor.llm = agent.llm_provider
if not isinstance(agent.extractor, HyExtractionGuard):
agent.extractor = HyExtractionGuard(agent.extractor)
reconciler = writer.reconciler
provider = reconciler._get_llm()
if not isinstance(provider, HyResponseGuard):
reconciler._llm = HyResponseGuard(provider, phase="reconcile")
if not isinstance(reconciler, HyReconcileGuard):
writer._reconciler = HyReconcileGuard(reconciler)
for guard in (
agent.llm_provider,
reconciler._llm,
writer._reconciler,
agent.extractor,
):
if id(guard) not in seen_guards:
seen_guards.add(id(guard))
guards.append(guard)
client._one_memory_write_guards = tuple(guards)
client._loop_thread.run(install())
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.5.dev3"
HY_MEMORY_VERSION = "2.0.6"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -33,49 +33,8 @@ _COLLECTION_LOGGERS = (
)
_NOISY_LIBRARY_LOGGERS = ("httpx", "httpcore", "watchfiles", "jieba")
# One Memory deliberately stays on ultra because its existing, user-visible
# profile/graph flow calls digest(), which the pinned public SDK exposes only
# for ultra clients. Reader and non-reranker behavior remain pinned for
# compatibility with the SDK authority-migration development preview.
HY_MEMORY_MODE = "ultra"
# The SDK only recognises `legacy` and `hybrid` (see
# hy_memory/pipelines/_retrieval/config.py: `ALL_READERS = (READER_LEGACY,
# READER_HYBRID)`). The old spelling `hybrid_v2` was silently rejected and
# fell back to legacy, so the hybrid-v2 recall strategy never ran at all.
HY_MEMORY_READER = "hybrid"
HY_MEMORY_BEHAVIOR_ENV: dict[str, str] = {
"MEMORY_MODE": HY_MEMORY_MODE,
"MEMORY_ENABLE_GRAPH": "true",
"MEMORY_GRAPH_RERANK_ENABLED": "false",
"MEMORY_GRAPH_TEMPORAL_SPINE": "true",
"MEMORY_TOPIC_SUMMARY_ENABLED": "false",
"MEMORY_EPISODE_SUMMARY_ENABLED": "false",
"MEMORY_EPISODE_RECALL_ENABLED": "true",
"MEMORY_DYNAMIC_EPISODE_ENABLED": "true",
"MEMORY_SESSION_EPISODE_ENABLED": "false",
"MEMORY_SESSION_EPISODE_CONTEXT_ENABLED": "false",
"MEMORY_EPISODE_CAPSULE_CONTEXT_ENABLED": "true",
"MEMORY_EPISODE_CAPSULE_QUERY_MIN": "0.65",
"MEMORY_EPISODE_CAPSULE_ASSIGNMENT_MIN": "0.60",
"MEMORY_EPISODE_CAPSULE_FIRST_RANK_MAX": "3",
"MEMORY_EPISODE_CAPSULE_MEMBER_LIMIT": "3",
"MEMORY_ENABLE_INTENTION": "false",
"MEMORY_ENABLE_QUERY_REWRITE": "false",
"MEMORY_FEW_SHOT_ENABLED": "true",
"MEMORY_HYBRID_V2_QUERY_EXPAND": "1",
"MEMORY_HYBRID_V2_GEO_INTENT_PRIOR": "0.6",
"MEMORY_HYBRID_V2_DIVERSITY": "1",
"MEMORY_HYBRID_V2_TEMPORAL_ADMIT_PRIOR": "0.55",
"MEMORY_HYBRID_V2_TEMPORAL_BOOST": "0.18",
"MEMORY_CROSS_DOMAIN_CON_THRESHOLD": "0.5",
"MEMORY_READER_ENABLE_SUMMARY": "false",
"MEMORY_SUMMARY_ENABLED_IN_SYS2": "false",
"MEMORY_LOCAL_RERANK_ENABLED": "false",
"RECONCILE_ENABLED": "true",
"HY_MEMORY_READER": HY_MEMORY_READER,
}
def detailed_logging_enabled(data_dir: Path) -> bool:
return diagnostic_log_level(data_dir) == "debug"
@@ -168,7 +127,6 @@ def configure_hy_runtime_environment(data_dir: Path) -> None:
log_dir = memory_root / "logs"
memory_root.mkdir(parents=True, exist_ok=True)
log_dir.mkdir(parents=True, exist_ok=True)
os.environ.update(HY_MEMORY_BEHAVIOR_ENV)
os.environ["MEMORY_DATA_DIR"] = str(memory_root)
os.environ["MEMORY_LOG_DIR"] = str(log_dir)
level = diagnostic_log_level(data_dir)
@@ -178,9 +136,3 @@ def configure_hy_runtime_environment(data_dir: Path) -> None:
os.environ["MEMORY_MEMORY_OPERATIONS_ENABLED"] = "false"
os.environ["ANONYMIZED_TELEMETRY"] = "False"
os.environ.pop("CHROMA_OTEL_COLLECTION_ENDPOINT", None)
def configure_hy_behavior_environment() -> None:
"""Set import-time HY behavior flags without touching paths or secrets."""
os.environ.update(HY_MEMORY_BEHAVIOR_ENV)
@@ -0,0 +1,469 @@
"""Offline maintenance of the HY SDK data plane for App migrations.
Runtime product code must use public ``HyMemoryClient`` methods. The startup
migration boundary is the isolated exception: all writers are stopped, so it
may copy, restore and repair SDK-owned local stores.
"""
from __future__ import annotations
import json
import sqlite3
from collections.abc import Iterator
from contextlib import closing
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
from hy_memory import ( # type: ignore[import-untyped]
DataPlaneRestore,
DataPlaneSnapshot,
restore_local_data_plane,
snapshot_local_data_plane,
)
from one_memory.database_access import main_database
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
# v2 reads the backfill's governed_memory_evidence links; v1 missed them.
_METADATA_REPAIR_VERSION = 2
_BATCH = 500
@dataclass(frozen=True)
class RepairMetadata:
source: str | None
memory_at: int | None
created_at: int | None
@dataclass(frozen=True)
class MetadataRepairResult:
migration_id: str | None
mapped: int
vector_rows: int
coding_rows: int
def snapshot_data_plane(
*,
source_root: Path,
target_root: Path,
writers_stopped: bool,
) -> DataPlaneSnapshot:
"""Snapshot an explicitly quiesced SDK-owned local data plane."""
return snapshot_local_data_plane(
source_root=source_root,
target_root=target_root,
writers_stopped=writers_stopped,
)
def restore_data_plane(
*,
snapshot_root: Path,
target_root: Path,
writers_stopped: bool,
protective_root: Path | None = None,
) -> DataPlaneRestore:
"""Restore an explicitly quiesced SDK-owned local data plane."""
return restore_local_data_plane(
snapshot_root=snapshot_root,
target_root=target_root,
writers_stopped=writers_stopped,
protective_root=protective_root,
)
def repair_sdk_metadata_from_bundle(
*,
data_dir: Path,
database_path: Path,
now: datetime | None = None,
) -> MetadataRepairResult:
"""Restore metadata lost by historical cutovers from their verified bundles."""
migration_id = _sdk_migration_id(database_path)
if migration_id is None:
return MetadataRepairResult(None, 0, 0, 0)
receipt = data_dir / f".sdk-metadata-repair-v{_METADATA_REPAIR_VERSION}-{migration_id}.json"
if receipt.is_file():
document = json.loads(receipt.read_text(encoding="utf-8"))
return MetadataRepairResult(
migration_id,
int(document.get("mapped", 0)),
int(document.get("vector_rows", 0)),
int(document.get("coding_rows", 0)),
)
# A home can carry several cutovers (v1 and then v2). A later bundle may
# have been taken after an earlier cutover had already retired the legacy
# tables, so every bundle is read and their links are pooled.
legacy = sorted(
(data_dir / "backups").glob("*-pre-cutover.sdk-bundle/management/one-memory.sqlite3")
)
if not legacy:
# No bundle yet is not a finished repair: try again on the next start.
return MetadataRepairResult(migration_id, 0, 0, 0)
current = now or datetime.now(UTC)
pools: dict[str, _Pool] = {}
for path in legacy:
_merge_pools(pools, _legacy_pools(path, now=current))
mapping = _decide(pools)
vector_rows = _repair_vector(data_dir, mapping, now=current)
coding_rows = _repair_coding(data_dir, mapping)
result = MetadataRepairResult(
migration_id,
len(mapping),
vector_rows,
coding_rows,
)
temporary = receipt.with_suffix(".tmp")
temporary.write_text(
json.dumps(
{
"schema_version": 1,
"migration_id": migration_id,
"mapped": result.mapped,
"vector_rows": vector_rows,
"coding_rows": coding_rows,
},
sort_keys=True,
)
+ "\n",
encoding="utf-8",
)
temporary.replace(receipt)
return result
def _sdk_migration_id(database_path: Path) -> str | None:
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
row = database.execute(
"SELECT authority,migration_id FROM memory_authority_state WHERE singleton=1"
).fetchone()
if row is None or row["authority"] != "sdk":
return None
value = row["migration_id"]
return str(value) if isinstance(value, str) and value else None
@dataclass
class _Pool:
"""Everything the legacy library knew about one SDK memory."""
sources: set[str] = field(default_factory=set)
times: list[datetime] = field(default_factory=list)
created: list[datetime] = field(default_factory=list)
def _legacy_metadata(path: Path, *, now: datetime) -> dict[str, RepairMetadata]:
return _decide(_legacy_pools(path, now=now))
def _legacy_pools(path: Path, *, now: datetime) -> dict[str, _Pool]:
"""Link each SDK memory id to the legacy evidence it was made from.
Three legacy paths wrote memories the SDK now holds:
- the cutover backfill (`import_exact`) copied every governed memory into the
SDK under its legacy id, or under the id `memory_task_outputs` recorded,
without a source or a conversation time; `governed_memory_evidence` is the
link from that memory to its evidence;
- extraction tasks recorded their evidence as `support_refs`;
- the write queue recorded the SDK ids each job produced.
"""
with closing(sqlite3.connect(f"file:{path}?mode=ro", uri=True)) as database:
database.row_factory = sqlite3.Row
tables = {
str(row[0])
for row in database.execute("SELECT name FROM sqlite_master WHERE type='table'")
}
if "evidence" not in tables:
return {}
links: dict[str, set[str]] = {}
created: dict[str, list[datetime]] = {}
canonical: dict[str, str] = {}
if "memory_task_outputs" in tables:
for row in database.execute("SELECT body FROM memory_task_outputs"):
try:
body = json.loads(str(row["body"]))
except (TypeError, ValueError):
continue
legacy_id = body.get("memory_id")
canonical_id = body.get("hy_memory_id")
refs = body.get("support_refs")
if not isinstance(legacy_id, str) or not isinstance(canonical_id, str):
continue
canonical[legacy_id] = canonical_id
links.setdefault(canonical_id, set()).update(
str(ref["evidence_id"])
for ref in (refs if isinstance(refs, list) else ())
if isinstance(ref, dict) and isinstance(ref.get("evidence_id"), str)
)
if "governed_memories" in tables:
for row in database.execute(
"SELECT memory_id,created_at,updated_at FROM governed_memories"
):
legacy_id = str(row["memory_id"])
canonical_id = canonical.get(legacy_id, legacy_id)
links.setdefault(canonical_id, set())
legacy_created = _plausible_time(
row["created_at"], fallback=row["updated_at"], now=now
)
if legacy_created is not None:
created.setdefault(canonical_id, []).append(legacy_created)
if "governed_memory_evidence" in tables:
for row in database.execute(
"SELECT memory_id,evidence_id FROM governed_memory_evidence"
):
legacy_id = str(row["memory_id"])
links.setdefault(canonical.get(legacy_id, legacy_id), set()).add(
str(row["evidence_id"])
)
if "memory_write_jobs" in tables and "sdk_memory_ids_json" in _columns(
database, "memory_write_jobs"
):
job_evidence: dict[str, set[str]] = {}
if "memory_write_job_evidence" in tables:
for row in database.execute(
"SELECT job_id,evidence_id FROM memory_write_job_evidence"
):
job_evidence.setdefault(str(row["job_id"]), set()).add(str(row["evidence_id"]))
for row in database.execute(
"SELECT job_id,sdk_memory_ids_json FROM memory_write_jobs "
"WHERE sdk_memory_ids_json IS NOT NULL"
):
try:
memory_ids = json.loads(str(row["sdk_memory_ids_json"]))
except (TypeError, ValueError):
continue
for memory_id in memory_ids if isinstance(memory_ids, list) else ():
if isinstance(memory_id, str) and memory_id:
links.setdefault(memory_id, set()).update(
job_evidence.get(str(row["job_id"]), ())
)
evidence: dict[str, tuple[str, datetime | None]] = {}
wanted = sorted({identifier for refs in links.values() for identifier in refs})
for batch in _batches(wanted):
placeholders = ",".join("?" for _ in batch)
for row in database.execute(
"SELECT evidence_id,source_id,created_at,imported_at "
f"FROM evidence WHERE evidence_id IN ({placeholders})",
batch,
):
evidence[str(row["evidence_id"])] = (
str(row["source_id"]),
_plausible_time(row["created_at"], fallback=row["imported_at"], now=now),
)
pools: dict[str, _Pool] = {}
for canonical_id, refs in links.items():
pool = pools.setdefault(canonical_id, _Pool())
for identifier in refs:
if identifier not in evidence:
continue
source, timestamp = evidence[identifier]
if source and source != "default":
pool.sources.add(source)
if timestamp is not None:
pool.times.append(timestamp)
pool.created.extend(created.get(canonical_id, ()))
return pools
def _merge_pools(into: dict[str, _Pool], other: dict[str, _Pool]) -> None:
for canonical_id, pool in other.items():
merged = into.setdefault(canonical_id, _Pool())
merged.sources |= pool.sources
merged.times.extend(pool.times)
merged.created.extend(pool.created)
def _decide(pools: dict[str, _Pool]) -> dict[str, RepairMetadata]:
# Several legacy memories can fold into one SDK memory. Their evidence is
# pooled before deciding, so a disagreeing source stays unknown instead of
# whichever link happened to come last.
return {
canonical_id: RepairMetadata(
source=next(iter(pool.sources)) if len(pool.sources) == 1 else None,
memory_at=int(max(pool.times).timestamp()) if pool.times else None,
created_at=int(min(pool.created).timestamp()) if pool.created else None,
)
for canonical_id, pool in pools.items()
}
def _columns(database: sqlite3.Connection, table: str) -> set[str]:
return {str(row[1]) for row in database.execute(f"PRAGMA table_info({table})")}
def _plausible_time(
value: object,
*,
fallback: object,
now: datetime,
) -> datetime | None:
for candidate in (value, fallback):
if not isinstance(candidate, str) or not candidate:
continue
try:
parsed = datetime.fromisoformat(candidate)
except ValueError:
continue
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=UTC)
parsed = parsed.astimezone(UTC)
if parsed <= now + timedelta(days=1):
return parsed
return None
def _repair_vector(
data_dir: Path,
mapping: dict[str, RepairMetadata],
*,
now: datetime,
) -> int:
directory = data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / "vector"
if not (directory / "chroma.sqlite3").is_file():
return 0
from one_memory.sdk_vector_reattach import open_offline_chroma
client = open_offline_chroma(directory)
# The SDK client is not open yet (startup has not created its handle), so
# this is the only writer; release the store before the SDK takes it.
try:
repaired = 0
ids = tuple(mapping)
for collection in client.list_collections():
for batch in _batches(ids):
page = collection.get(ids=list(batch), include=["metadatas"])
updated_ids: list[str] = []
updated_metadata: list[dict[str, object]] = []
for memory_id, metadata in zip(
page.get("ids") or (),
page.get("metadatas") or (),
strict=True,
):
if not isinstance(metadata, dict):
continue
desired = mapping.get(str(memory_id))
if desired is None:
continue
next_metadata = dict(metadata)
changed = False
if desired.source and next_metadata.get("source") in {
None,
"",
"default",
}:
next_metadata["source"] = desired.source
changed = True
if (
desired.memory_at is not None
and next_metadata.get("memory_at") != desired.memory_at
):
next_metadata["memory_at"] = desired.memory_at
changed = True
if (
desired.created_at is not None
and next_metadata.get("gmt_created") != desired.created_at
):
next_metadata["gmt_created"] = desired.created_at
changed = True
if changed:
updated_ids.append(str(memory_id))
updated_metadata.append(next_metadata)
if updated_ids:
collection.update(ids=updated_ids, metadatas=updated_metadata)
repaired += len(updated_ids)
repaired += _settle_future_times(collection, now=now)
finally:
client.close()
return repaired
def _settle_future_times(collection: Any, *, now: datetime) -> int:
"""Move conversation times that are still in the future back to the write time.
A historical Cursor bug stored a file hash as the message time (years like
2254). The SDK lists newest first by `memory_at`, so those rows hold the
first page. Rows the bundles could date were fixed above; for the rest the
write time is the closest true moment that is known.
"""
cutoff = int((now + timedelta(days=1)).timestamp())
page = collection.get(where={"memory_at": {"$gt": cutoff}}, include=["metadatas"])
updated_ids: list[str] = []
updated_metadata: list[dict[str, object]] = []
for memory_id, metadata in zip(page.get("ids") or (), page.get("metadatas") or (), strict=True):
if not isinstance(metadata, dict):
continue
created = metadata.get("gmt_created")
settled = (
created
if isinstance(created, int) and not isinstance(created, bool) and created <= cutoff
else int(now.timestamp())
)
updated_ids.append(str(memory_id))
updated_metadata.append({**metadata, "memory_at": settled})
if updated_ids:
collection.update(ids=updated_ids, metadatas=updated_metadata)
return len(updated_ids)
def _repair_coding(
data_dir: Path,
mapping: dict[str, RepairMetadata],
) -> int:
root = data_dir / "hy-memory" / HY_MEMORY_NAMESPACE
repaired = 0
for name in ("coding.sqlite3", "coding_cv.sqlite3"):
path = root / name
if not path.is_file():
continue
with closing(sqlite3.connect(path)) as database:
tables = {
str(row[0])
for row in database.execute("SELECT name FROM sqlite_master WHERE type='table'")
}
for table, identifier in (
("coding_memory_meta", "experience_id"),
("working_fact", "fact_id"),
):
if table not in tables:
continue
rows = database.execute(
f"SELECT {identifier},source,created_at FROM {table}"
).fetchall()
for memory_id, source, created_at in rows:
desired = mapping.get(str(memory_id))
if desired is None:
continue
updates: list[str] = []
values: list[object] = []
if desired.source and source in {None, "", "default"}:
updates.append("source=?")
values.append(desired.source)
if desired.created_at is not None:
desired_created_at = (
datetime.fromtimestamp(desired.created_at, tz=UTC)
.replace(tzinfo=None)
.isoformat()
)
if created_at != desired_created_at:
updates.append("created_at=?")
values.append(desired_created_at)
if not updates:
continue
values.append(memory_id)
database.execute(
f"UPDATE {table} SET {','.join(updates)} WHERE {identifier}=?",
values,
)
repaired += 1
database.commit()
return repaired
def _batches[T](values: list[T] | tuple[T, ...]) -> Iterator[tuple[T, ...]]:
for start in range(0, len(values), _BATCH):
yield tuple(values[start : start + _BATCH])
@@ -0,0 +1,446 @@
"""HY SDK client construction and shared-runtime leasing.
This module deliberately depends only on the SDK's package-root exports. A
lease owns one client while the process shares one ``SharedRuntime`` per
physical MemHub data directory.
"""
from __future__ import annotations
import asyncio
import hashlib
import os
import threading
from collections.abc import Callable, Coroutine
from dataclasses import dataclass
from pathlib import Path
from typing import Any, ClassVar
from hy_memory import ( # type: ignore[import-untyped]
AmbiguousMemoryIdError as AmbiguousMemoryIdError, # re-export for api.py
)
from hy_memory import (
HyMemoryClient,
MemoryConfig,
SharedRuntime,
)
from one_memory.embedding_assets import (
BUILTIN_EMBEDDING_DIMENSIONS,
BUILTIN_EMBEDDING_MODEL,
)
from one_memory.hy_graph_admin import (
GraphWidthMismatchError,
graph_store_enabled,
inspect_graph_width_mismatch,
)
from one_memory.hy_sdk_types import HySdkRuntimeError
from one_memory.local_embedding import LocalEmbedder
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
_RUNTIME_LOCK = threading.RLock()
BUILTIN_RETRIEVAL_PROFILE_ID = "builtin-retrieval"
_LOCAL_ONNX_PLACEHOLDER_URL = "http://127.0.0.1/local-onnx"
DEFAULT_RELEVANCE_FLOOR = 0.35
@dataclass
class _RuntimeEntry:
runtime: SharedRuntime
references: int
_RUNTIMES: dict[Path, _RuntimeEntry] = {}
class LocalOnnxEmbeddingService:
"""Public ``EmbeddingService`` implementation backed by MemHub's ONNX model."""
_embedder: ClassVar[LocalEmbedder | None] = None
_embedder_lock: ClassVar[threading.Lock] = threading.Lock()
def __init__(self, embedder: LocalEmbedder | None = None) -> None:
if embedder is None:
with self._embedder_lock:
if self._embedder is None:
self.__class__._embedder = LocalEmbedder()
embedder = self._embedder
if embedder is None: # pragma: no cover - guarded by the assignment above
raise HySdkRuntimeError("hy_sdk_local_embedding_unavailable")
self._local_embedder = embedder
async def embed(self, text: str, use_cache: bool = True) -> list[float]:
del use_cache
vectors = await asyncio.to_thread(self._local_embedder.encode, [text])
return vectors[0]
async def embed_with_meta(
self,
text: str,
use_cache: bool = True,
) -> tuple[list[float], dict[str, Any]]:
return await self.embed(text, use_cache), {"provider": "local-onnx"}
async def embed_batch(
self,
texts: list[str],
use_cache: bool = True,
) -> list[list[float]]:
del use_cache
if not texts:
return []
return await asyncio.to_thread(self._local_embedder.encode, list(texts))
async def embed_queued(self, text: str, use_cache: bool = True) -> list[float]:
return await self.embed(text, use_cache)
async def embed_queued_with_meta(
self,
text: str,
use_cache: bool = True,
) -> tuple[list[float], dict[str, Any]]:
return await self.embed_with_meta(text, use_cache)
def get_embedding_dims(self) -> int:
return BUILTIN_EMBEDDING_DIMENSIONS
class ClientLease:
"""Own a client and release its shared runtime exactly once."""
def __init__(self, client: HyMemoryClient, data_dir: Path) -> None:
self.client = client
self.data_dir = data_dir.resolve()
self._closed = False
self._close_lock = threading.Lock()
@property
def closed(self) -> bool:
return self._closed
def close(self) -> None:
with self._close_lock:
if self._closed:
return
self._closed = True
try:
self.client.close()
finally:
_release_runtime(self.data_dir)
def __enter__(self) -> ClientLease:
return self
def __exit__(self, *_exc: object) -> None:
self.close()
class SdkClientHandle:
"""Lazy async owner for one profile-bound public SDK client lease."""
def __init__(
self,
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None = None,
) -> None:
self._profile = profile
self._api_key = api_key
self._data_dir = data_dir
self._http_transport_factory = http_transport_factory
self._lease: ClientLease | None = None
self._lock = asyncio.Lock()
async def client(self) -> HyMemoryClient:
if self._lease is not None:
return self._lease.client
async with self._lock:
if self._lease is None:
self._lease = await asyncio.to_thread(
acquire_client,
self._profile,
api_key=self._api_key,
data_dir=self._data_dir,
http_transport_factory=self._http_transport_factory,
)
return self._lease.client
async def close(self) -> None:
async with self._lock:
lease, self._lease = self._lease, None
if lease is not None:
await asyncio.to_thread(lease.close)
def builtin_retrieval_snapshot() -> ModelProfileSnapshot:
"""Return the model-free profile used by SDK-native read operations."""
return ModelProfileSnapshot(
profile_id=BUILTIN_RETRIEVAL_PROFILE_ID,
revision=0,
provider_kind=ProviderKind.LOCAL_ONNX,
base_url=_LOCAL_ONNX_PLACEHOLDER_URL,
model=BUILTIN_EMBEDDING_MODEL,
embedding_provider_kind=ProviderKind.LOCAL_ONNX,
embedding_model=BUILTIN_EMBEDDING_MODEL,
embedding_dimensions=BUILTIN_EMBEDDING_DIMENSIONS,
timeout_seconds=60,
max_concurrency=1,
context_window=None,
has_api_key=False,
)
def sdk_relevance_floor() -> float:
raw = os.environ.get("ONE_MEMORY_HY_RELEVANCE_FLOOR", "").strip()
if not raw:
return DEFAULT_RELEVANCE_FLOOR
try:
value = float(raw)
except ValueError:
return DEFAULT_RELEVANCE_FLOOR
return min(1.0, max(0.0, value))
def build_sdk_config(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
) -> MemoryConfig:
"""Map a model snapshot to SDK models, endpoints, backends, and local paths."""
generation_endpoint, generation_key = _generation_access(profile, api_key)
embedding_provider = profile.embedding_provider_kind or profile.provider_kind
embedding_service_is_local = embedding_provider is ProviderKind.LOCAL_ONNX
embedding_model = profile.embedding_model or (
BUILTIN_EMBEDDING_MODEL if embedding_service_is_local else None
)
if not embedding_model:
raise HySdkRuntimeError("hy_sdk_embedding_model_required")
dimensions: int | None
if embedding_service_is_local:
embedding_endpoint = None
embedding_key = None
dimensions = profile.embedding_dimensions or BUILTIN_EMBEDDING_DIMENSIONS
else:
embedding_endpoint = _openai_endpoint(
embedding_provider,
profile.embedding_base_url or profile.base_url,
)
embedding_key = "ollama" if embedding_provider is ProviderKind.OLLAMA else api_key
if embedding_provider is ProviderKind.OPENAI_COMPATIBLE and not embedding_key:
raise HySdkRuntimeError("hy_sdk_embedding_api_key_missing")
if embedding_provider not in {
ProviderKind.OPENAI_COMPATIBLE,
ProviderKind.OLLAMA,
}:
raise HySdkRuntimeError("hy_sdk_embedding_provider_unsupported")
dimensions = profile.embedding_dimensions
root = data_dir.resolve() / "hy-memory" / "governed-v2"
root.mkdir(parents=True, exist_ok=True)
collection = _collection_base(
embedding_provider, embedding_endpoint, embedding_model, dimensions
)
return MemoryConfig.from_dict(
{
"llm": {
"provider": "openai",
"model": profile.model,
"api_key": generation_key,
"base_url": generation_endpoint,
},
"embedder": {
"provider": "openai",
"model": (
BUILTIN_EMBEDDING_MODEL if embedding_service_is_local else embedding_model
),
"api_key": embedding_key,
"base_url": embedding_endpoint,
"embedding_dims": dimensions,
},
"vector_store": {
"provider": "chroma",
"collection_name": collection,
"persist_directory": str(root / "vector"),
"embedding_dims": dimensions,
"host": None,
"port": None,
"api_key": None,
},
"graph_store": {
"provider": "kuzu",
"db_path": str(root / "graph"),
"url": None,
"username": None,
"password": None,
"database": None,
},
"cache": {
"backend": "sqlite",
"db_path": str(root / "cache.sqlite3"),
},
"history": {"db_path": str(root / "history.sqlite3")},
"coding": {"db_path": str(root / "coding.sqlite3")},
}
)
def acquire_client(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None = None,
) -> ClientLease:
"""Construct a public SDK client and return its explicit runtime lease."""
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir)
config.http_transport_factory = http_transport_factory
mismatch = inspect_graph_width_mismatch(config)
if mismatch is not None:
raise GraphWidthMismatchError(mismatch)
key = data_dir.resolve()
runtime = _acquire_runtime(config, key)
embedding_service: LocalOnnxEmbeddingService | None = None
if (profile.embedding_provider_kind or profile.provider_kind) is ProviderKind.LOCAL_ONNX:
embedding_service = LocalOnnxEmbeddingService()
try:
client = HyMemoryClient(
config=config,
mode="ultra" if graph_store_enabled() else "pro",
runtime=runtime,
embedding_service=embedding_service,
)
except BaseException:
_release_runtime(key)
raise
return ClientLease(client, key)
def close_all_shared_runtimes() -> None:
"""Close process runtimes only after every client lease has been released."""
with _RUNTIME_LOCK:
if _RUNTIMES:
raise HySdkRuntimeError("hy_sdk_runtime_leases_active")
def _generation_access(
profile: ModelProfileSnapshot,
api_key: str | None,
) -> tuple[str, str]:
if profile.provider_kind is ProviderKind.LOCAL_ONNX:
return _LOCAL_ONNX_PLACEHOLDER_URL, "local-read-only"
if profile.provider_kind not in {
ProviderKind.OPENAI_COMPATIBLE,
ProviderKind.OLLAMA,
}:
raise HySdkRuntimeError("hy_sdk_generation_provider_unsupported")
endpoint = _openai_endpoint(profile.provider_kind, profile.base_url)
if profile.provider_kind is ProviderKind.OLLAMA:
return endpoint, "ollama"
if not api_key:
raise HySdkRuntimeError("hy_sdk_model_api_key_missing")
return endpoint, api_key
def _collection_base(
provider: ProviderKind,
endpoint: str | None,
model: str,
dimensions: int | None,
) -> str:
identity = "\0".join((provider.value, endpoint or "local", model, str(dimensions or "auto")))
return "memhub_" + hashlib.sha256(identity.encode()).hexdigest()[:16]
def vector_collection_bases(profile: ModelProfileSnapshot) -> tuple[str, tuple[str, ...]]:
"""The SDK vector collection base for this embedding space, and the older ones.
The SDK derives every vector collection (chat, Coding keys, Working facts,
each variant) by suffixing this base. Builds before #145 named the same
embedding space `one_memory_<12 hex>`, with or without a declared width;
the offline migration renames those so their vectors stay readable.
"""
provider = profile.embedding_provider_kind or profile.provider_kind
local = provider is ProviderKind.LOCAL_ONNX
model = profile.embedding_model or (BUILTIN_EMBEDDING_MODEL if local else None)
if not model:
raise HySdkRuntimeError("hy_sdk_embedding_model_required")
endpoint: str | None
dimensions: int | None
if local:
endpoint, dimensions = None, profile.embedding_dimensions or BUILTIN_EMBEDDING_DIMENSIONS
legacy_endpoint = "local-onnx"
else:
endpoint = _openai_endpoint(provider, profile.embedding_base_url or profile.base_url)
dimensions = profile.embedding_dimensions
legacy_endpoint = endpoint
legacy_model = profile.embedding_model or "bge-small-zh-v1.5"
legacy = tuple(
"one_memory_"
+ hashlib.sha256(
":".join((provider.value, legacy_endpoint, legacy_model, *declared)).encode()
).hexdigest()[:12]
for declared in ((), ("auto",), (str(profile.embedding_dimensions or "auto"),))
)
return _collection_base(provider, endpoint, model, dimensions), tuple(dict.fromkeys(legacy))
def _openai_endpoint(provider: ProviderKind, endpoint: str | None) -> str:
value = (endpoint or "").rstrip("/")
if not value:
raise HySdkRuntimeError("hy_sdk_model_endpoint_missing")
if provider is ProviderKind.OLLAMA and not value.endswith("/v1"):
return f"{value}/v1"
return value
def _acquire_runtime(config: MemoryConfig, key: Path) -> SharedRuntime:
with _RUNTIME_LOCK:
entry = _RUNTIMES.get(key)
if entry is None:
runtime = _run_sync(SharedRuntime.create(config))
_RUNTIMES[key] = _RuntimeEntry(runtime=runtime, references=1)
return runtime
entry.references += 1
return entry.runtime
def _release_runtime(key: Path) -> None:
runtime: SharedRuntime | None = None
with _RUNTIME_LOCK:
entry = _RUNTIMES.get(key)
if entry is None:
return
entry.references -= 1
if entry.references == 0:
runtime = entry.runtime
del _RUNTIMES[key]
if runtime is not None:
runtime.close()
def _run_sync[T](coroutine: Coroutine[Any, Any, T]) -> T:
"""Run SDK async construction from either sync or async caller threads."""
try:
asyncio.get_running_loop()
except RuntimeError:
return asyncio.run(coroutine)
result: list[T] = []
failure: list[BaseException] = []
def run() -> None:
try:
result.append(asyncio.run(coroutine))
except BaseException as error:
failure.append(error)
thread = threading.Thread(target=run, name="hy-sdk-runtime-create")
thread.start()
thread.join()
if failure:
raise failure[0]
return result[0]
@@ -0,0 +1,694 @@
"""Validated records and response parsers for public HY SDK operations."""
from __future__ import annotations
from collections.abc import Mapping
from datetime import UTC, datetime
from typing import Literal, NamedTuple, cast
from pydantic import BaseModel, ConfigDict, Field
from one_memory.semantic_deadline import SemanticMemoryHit
class HySdkError(RuntimeError):
"""Base error carrying a stable, product-facing code."""
def __init__(self, code: str) -> None:
super().__init__(code)
self.code = code
class HySdkRuntimeError(HySdkError):
"""The public SDK runtime could not be configured or leased."""
class HySdkMigrationError(HySdkError):
"""The privileged offline SDK data-plane migration failed."""
class HySdkReadError(HySdkError):
"""A public SDK read operation failed or returned an invalid shape."""
MemoryScene = Literal["chat", "working", "coding"]
MemoryKindName = Literal["chat", "experience", "fact"]
MemoryVariantName = Literal["working", "coding"]
class HySdkMemoryRecord(BaseModel):
"""One memory of any scene, in the unified shape the library lists.
Chat memories keep every field they always had. Working/coding memories
(experiences and facts) carry ``scene``/``kind``/``variant`` so a client
can open, edit and delete them again without guessing the namespace; an
experience also keeps ``task`` and ``solution`` apart, and ``content`` is
the readable ``task\nsolution`` text the SDK lists.
"""
model_config = ConfigDict(frozen=True, extra="forbid")
memory_id: str
content: str
layer: str
status: str
is_latest: bool
supersedes: tuple[str, ...] = ()
superseded_by: tuple[str, ...] = ()
tags: tuple[str, ...] = ()
owner: str | None = None
source_raw_ids: tuple[str, ...] = ()
user_id: str
agent_id: str
session_id: str | None = None
source: str | None = None
memory_at: int | None = None
gmt_created: int | None = None
scene: MemoryScene = "chat"
kind: MemoryKindName = "chat"
# The productivity namespace ("working" | "coding"); None for chat. Pass it
# back with `kind` on get/update/delete.
variant: MemoryVariantName | None = None
task: str | None = None
solution: str | None = None
# Epoch seconds. `updated_at` falls back to `created_at`.
created_at: int | None = None
updated_at: int | None = None
@classmethod
def from_sdk(cls, raw: object) -> HySdkMemoryRecord:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_record_shape_unknown")
if raw.get("kind") in {"experience", "fact"}:
return cls._from_productivity(raw)
if any(
not isinstance(raw.get(key), str) or not str(raw[key]).strip()
for key in ("memory_id", "content")
):
raise HySdkReadError("hy_memory_record_shape_unknown")
try:
gmt_created = (
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
)
created_at = _epoch(raw.get("created_at"))
if created_at is None:
created_at = gmt_created
updated_at = _epoch(raw.get("updated_at"))
return cls(
memory_id=str(raw["memory_id"]),
content=str(raw["content"]),
layer=str(raw.get("layer") or ""),
status=str(raw.get("status") or ""),
is_latest=bool(raw.get("is_latest", True)),
supersedes=_string_tuple(raw.get("supersedes")),
superseded_by=_string_tuple(raw.get("superseded_by")),
tags=_string_tuple(raw.get("tags")),
owner=(str(raw["owner"]) if raw.get("owner") in {"user", "agent"} else None),
source_raw_ids=_source_raw_ids(raw),
user_id=str(raw.get("user_id") or ""),
agent_id=str(raw.get("agent_id") or ""),
session_id=(
str(raw["session_id"]) if isinstance(raw.get("session_id"), str) else None
),
source=str(raw["source"]) if isinstance(raw.get("source"), str) else None,
memory_at=raw.get("memory_at") if isinstance(raw.get("memory_at"), int) else None,
gmt_created=gmt_created,
created_at=created_at,
updated_at=updated_at if updated_at is not None else created_at,
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_record_shape_unknown") from error
@classmethod
def _from_productivity(cls, raw: Mapping[str, object]) -> HySdkMemoryRecord:
"""An experience or fact from list/search(scene="all") or get().
get() names the namespace in ``variant`` (its ``scene`` is the legacy
"productivity"); list/search name it in ``scene``.
"""
kind = str(raw["kind"])
variant = next(
(
value
for value in (raw.get("variant"), raw.get("scene"))
if value in {"working", "coding"}
),
None,
)
id_key = "experience_id" if kind == "experience" else "fact_id"
memory_id = raw.get("memory_id") or raw.get(id_key)
if variant is None or not isinstance(memory_id, str) or not memory_id.strip():
raise HySdkReadError("hy_memory_record_shape_unknown")
task: str | None = None
solution: str | None = None
if kind == "experience":
task = str(raw.get("task") or "").strip()
solution = str(raw.get("solution") or "").strip()
content = "\n".join(part for part in (task, solution) if part)
else:
statement = raw.get("statement")
content = str(statement if isinstance(statement, str) else raw.get("content") or "")
if not content.strip():
raise HySdkReadError("hy_memory_record_shape_unknown")
status = str(raw.get("status") or "active")
created_at = _epoch(raw.get("created_at"))
updated_at = _epoch(raw.get("updated_at"))
try:
return cls(
memory_id=memory_id,
content=content,
layer=kind,
status=status,
is_latest=status == "active",
tags=_string_tuple(raw.get("tags")),
user_id=str(raw.get("user_id") or ""),
agent_id=str(raw.get("agent_id") or ""),
session_id=(
str(raw["session_id"]) if isinstance(raw.get("session_id"), str) else None
),
source=str(raw["source"]) if isinstance(raw.get("source"), str) else None,
gmt_created=created_at,
scene=cast(MemoryScene, variant),
kind=cast(MemoryKindName, kind),
variant=cast(MemoryVariantName, variant),
task=task,
solution=solution,
created_at=created_at,
updated_at=updated_at if updated_at is not None else created_at,
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_record_shape_unknown") from error
class HySdkMemoryPage(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
memories: tuple[HySdkMemoryRecord, ...]
total: int = Field(ge=0)
limit: int = Field(ge=1)
offset: int = Field(ge=0)
# Set on a search whose relevance window was full: matches were looked for
# only among this many most relevant memories, so `total` may undercount.
search_window: int | None = None
# Keyword search only: the server-side result window this page was cut
# from. Pass it back with the next offset to page the same results.
window_id: str | None = None
# True when the window_id the client sent had expired or did not match the
# query: the search ran again and this is page 1 of the new window.
window_reset: bool = False
# Empty-query listing only: how many memories each scene holds under the
# same filters (`total` is their sum).
totals: dict[str, int] | None = None
@classmethod
def from_sdk(cls, raw: object, *, limit: int, offset: int) -> HySdkMemoryPage:
page, _skipped = cls._parse(raw, limit=limit, offset=offset, skip_invalid=False)
return page
@classmethod
def from_sdk_skipping_invalid(
cls, raw: object, *, limit: int, offset: int
) -> tuple[HySdkMemoryPage, int]:
"""Parse a page, leaving out rows outside the public record contract.
Returns the page and how many rows were left out, so the caller can
report them. The rows themselves are never touched.
"""
return cls._parse(raw, limit=limit, offset=offset, skip_invalid=True)
@classmethod
def _parse(
cls,
raw: object,
*,
limit: int,
offset: int,
skip_invalid: bool,
) -> tuple[HySdkMemoryPage, int]:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_list_shape_unknown")
if raw.get("scene") == "all":
return cls._parse_all_scenes(raw, limit=limit, offset=offset, skip_invalid=skip_invalid)
rows: list[object] = []
total = 0
for bucket_name, rows_name in (("vdb", "memories"), ("graph", "nodes")):
bucket = raw.get(bucket_name)
if bucket is None and bucket_name == "graph":
continue
if not isinstance(bucket, Mapping) or not isinstance(bucket.get(rows_name), list):
raise HySdkReadError("hy_memory_list_shape_unknown")
bucket_total = bucket.get("total")
if (
not isinstance(bucket_total, int)
or isinstance(bucket_total, bool)
or bucket_total < 0
):
raise HySdkReadError("hy_memory_list_shape_unknown")
total += bucket_total
rows.extend(bucket[rows_name])
deduplicated: dict[str, HySdkMemoryRecord] = {}
skipped = 0
for row in rows:
try:
record = HySdkMemoryRecord.from_sdk(row)
except HySdkReadError:
if not skip_invalid:
raise
skipped += 1
continue
deduplicated.setdefault(record.memory_id, record)
page = cls(
memories=tuple(deduplicated.values()),
# `total` is the SDK's stable result-set count. Subtracting only
# invalid rows observed on this page would make it change as the
# caller paged. Until the SDK exposes a quarantined count, keep the
# authoritative total and skip malformed rows only in this page.
total=total,
limit=limit,
offset=offset,
)
return page, skipped
@classmethod
def _parse_all_scenes(
cls,
raw: Mapping[str, object],
*,
limit: int,
offset: int,
skip_invalid: bool,
) -> tuple[HySdkMemoryPage, int]:
"""``list_memories(scene="all")``: chat, working and coding, newest first."""
rows = raw.get("memories")
total = raw.get("total")
if (
not isinstance(rows, list)
or not isinstance(total, int)
or isinstance(total, bool)
or total < 0
):
raise HySdkReadError("hy_memory_list_shape_unknown")
totals_raw = raw.get("totals")
totals = (
{
str(scene): count
for scene, count in totals_raw.items()
if isinstance(count, int) and not isinstance(count, bool) and count >= 0
}
if isinstance(totals_raw, Mapping)
else None
)
memories: list[HySdkMemoryRecord] = []
seen: set[tuple[str, str, str | None]] = set()
skipped = 0
for row in rows:
try:
record = HySdkMemoryRecord.from_sdk(row)
except HySdkReadError:
if not skip_invalid:
raise
skipped += 1
continue
key = (record.memory_id, record.kind, record.variant)
if key in seen:
continue
seen.add(key)
memories.append(record)
return (
cls(
memories=tuple(memories),
total=total,
limit=limit,
offset=offset,
totals=totals,
),
skipped,
)
class HySdkRawRecord(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
raw_id: str
content: str
user_id: str
agent_id: str
session_id: str
source: str
memory_at: int | None = None
gmt_created: int | None = None
custom: dict[str, object] = Field(default_factory=dict)
@classmethod
def from_sdk(cls, raw: object) -> HySdkRawRecord:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_raw_shape_unknown")
if any(
not isinstance(raw.get(key), str) or not str(raw[key]).strip()
for key in ("raw_id", "content", "user_id", "agent_id", "session_id", "source")
):
raise HySdkReadError("hy_memory_raw_shape_unknown")
custom = raw.get("custom")
if not isinstance(custom, Mapping):
custom = {}
try:
return cls(
raw_id=str(raw["raw_id"]),
content=str(raw["content"]),
user_id=str(raw["user_id"]),
agent_id=str(raw["agent_id"]),
session_id=str(raw["session_id"]),
source=str(raw["source"]),
memory_at=raw.get("memory_at") if isinstance(raw.get("memory_at"), int) else None,
gmt_created=(
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
),
custom=dict(custom),
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_raw_shape_unknown") from error
class HySdkRawHit(HySdkRawRecord):
score: float = Field(ge=0, le=1)
@classmethod
def from_sdk_search(cls, raw: object) -> HySdkRawHit:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_raw_search_shape_unknown")
raw_id = raw.get("memory_id")
score = raw.get("score")
if (
not isinstance(raw_id, str)
or not isinstance(score, int | float)
or isinstance(score, bool)
):
raise HySdkReadError("hy_memory_raw_search_shape_unknown")
base = HySdkRawRecord.from_sdk({**raw, "raw_id": raw_id})
return cls(
**base.model_dump(),
score=max(0.0, min(1.0, float(score))),
)
class HySdkRawPage(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
raw: tuple[HySdkRawRecord, ...]
total: int = Field(ge=0)
limit: int = Field(ge=1)
offset: int = Field(ge=0)
@classmethod
def from_memory_list(
cls,
raw: object,
*,
limit: int,
offset: int,
) -> HySdkRawPage:
"""Parse ``list_memories(layers=['l1_raw'])`` into the App raw DTO."""
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
bucket = raw.get("vdb")
if not isinstance(bucket, Mapping):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
rows = bucket.get("memories")
total = bucket.get("total")
if (
not isinstance(rows, list)
or not isinstance(total, int)
or isinstance(total, bool)
or total < 0
):
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
converted = []
for row in rows:
if not isinstance(row, Mapping) or row.get("layer") != "l1_raw":
raise HySdkReadError("hy_memory_raw_list_shape_unknown")
converted.append({**row, "raw_id": row.get("memory_id")})
return cls(
raw=tuple(HySdkRawRecord.from_sdk(row) for row in converted),
total=total,
limit=limit,
offset=offset,
)
class HySdkUserProfile(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
request_id: str
profile: dict[str, object] | None
elapsed_ms: float = Field(ge=0)
@classmethod
def from_sdk(cls, raw: object) -> HySdkUserProfile:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_profile_shape_unknown")
request_id = raw.get("request_id")
profile = raw.get("profile")
elapsed_ms = raw.get("elapsed_ms")
if (
not isinstance(request_id, str)
or (profile is not None and not isinstance(profile, Mapping))
or not isinstance(elapsed_ms, int | float)
or isinstance(elapsed_ms, bool)
):
raise HySdkReadError("hy_memory_profile_shape_unknown")
return cls(
request_id=request_id,
profile=None if profile is None else dict(profile),
elapsed_ms=float(elapsed_ms),
)
def parse_sdk_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit, ...]:
rows = _reply_rows(raw, error_code)
found: list[SemanticMemoryHit] = []
seen: set[str] = set()
for item in sorted(rows, key=_score, reverse=True):
memory_id = item.get("memory_id") or item.get("id")
if not isinstance(memory_id, str) or not memory_id or memory_id in seen:
continue
seen.add(memory_id)
content = item.get("content")
member_ids, formatting_complete = _formatted_member_ids(item)
found.append(
SemanticMemoryHit(
memory_id=memory_id,
score=_score(item),
formatted_content=(
content.strip() if isinstance(content, str) and content.strip() else None
),
member_ids=member_ids,
sdk_formatted=True,
formatting_complete=formatting_complete,
)
)
return tuple(found)
class LibraryHit(NamedTuple):
"""One keyword match in a library search window, with its namespace."""
memory_id: str
kind: MemoryKindName
variant: MemoryVariantName | None
def library_keyword_hits(raw: object, query: str) -> tuple[tuple[LibraryHit, ...], int]:
"""Memories in an SDK search window whose text contains every query term.
The local SDK store has no keyword channel (Chroma's keyword search is
empty), so search is pure vector similarity: a word in the library can
score below the floor, and a word in none of it still scores above. The
library search asks the SDK for its most relevant window and keeps only the
rows that actually contain the words, most relevant first. Returns those
hits and how many distinct rows the window held.
A ``search(scene="all")`` window mixes chat memories with working/coding
experiences and facts; their text is the unified ``content`` (an
experience's task and solution, a fact's statement). The fused order is
kept as the SDK ranked it.
"""
terms = query.casefold().split()
if isinstance(raw, Mapping) and raw.get("scene") == "all":
rows = raw.get("memories")
if not isinstance(rows, list) or any(not isinstance(row, Mapping) for row in rows):
raise HySdkReadError("hy_memory_search_shape_unknown")
ordered: list[Mapping[str, object]] = list(rows)
else:
ordered = sorted(
_reply_rows(raw, "hy_memory_search_shape_unknown"), key=_score, reverse=True
)
seen: set[LibraryHit] = set()
matched: list[LibraryHit] = []
for item in ordered:
hit = _library_hit(item)
if hit is None or hit in seen:
continue
seen.add(hit)
tags = item.get("tags")
parts = [item.get(key) for key in ("content", "task", "solution", "statement")] + list(
tags if isinstance(tags, list) else ()
)
text = " ".join(part for part in parts if isinstance(part, str)).casefold()
if all(term in text for term in terms):
matched.append(hit)
return tuple(matched), len(seen)
def _library_hit(item: Mapping[str, object]) -> LibraryHit | None:
kind = item.get("kind")
if kind in {"experience", "fact"}:
variant = item.get("variant") if item.get("variant") in {"working", "coding"} else None
variant = variant or (
item.get("scene") if item.get("scene") in {"working", "coding"} else None
)
memory_id = item.get("memory_id") or item.get(
"experience_id" if kind == "experience" else "fact_id"
)
if variant is None or not isinstance(memory_id, str) or not memory_id:
return None
return LibraryHit(
memory_id,
"experience" if kind == "experience" else "fact",
"coding" if variant == "coding" else "working",
)
memory_id = item.get("memory_id") or item.get("id")
if not isinstance(memory_id, str) or not memory_id:
return None
return LibraryHit(memory_id, "chat", None)
def parse_sdk_raw_search(raw: object) -> tuple[HySdkRawHit, ...]:
return tuple(
HySdkRawHit.from_sdk_search(row)
for row in _reply_rows(raw, "hy_memory_raw_search_shape_unknown")
)
def _epoch(value: object) -> int | None:
"""Epoch seconds from an int or an ISO-8601 string (naive means UTC)."""
if isinstance(value, bool):
return None
if isinstance(value, int):
return value
if isinstance(value, float):
return int(value)
if isinstance(value, str) and value.strip():
try:
parsed = datetime.fromisoformat(value.strip())
except ValueError:
return None
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=UTC)
return int(parsed.timestamp())
return None
def _string_tuple(value: object) -> tuple[str, ...]:
if not isinstance(value, list):
return ()
return tuple(
dict.fromkeys(item.strip() for item in value if isinstance(item, str) and item.strip())
)
def _source_raw_ids(raw: Mapping[str, object]) -> tuple[str, ...]:
custom = raw.get("custom")
values: list[object] = []
if isinstance(custom, Mapping):
listed = custom.get("source_raw_memory_ids")
if isinstance(listed, list):
values.extend(listed)
singular = raw.get("source_raw_memory_id")
if isinstance(singular, str):
values.append(singular)
return tuple(
dict.fromkeys(value.strip() for value in values if isinstance(value, str) and value.strip())
)
def _reply_rows(raw: object, error_code: str) -> tuple[Mapping[str, object], ...]:
if not isinstance(raw, Mapping):
raise HySdkReadError(error_code)
keys = tuple(key for key in ("memories", "coding_memories") if key in raw)
if "scene" in raw:
scene = raw["scene"]
if not isinstance(scene, str):
raise HySdkReadError(error_code)
if scene in ("normal", "chat"):
selected = "memories"
elif scene in ("productivity", "working"):
selected = "coding_memories"
else:
raise HySdkReadError(error_code)
if selected not in keys:
raise HySdkReadError(error_code)
elif len(keys) == 1:
selected = keys[0]
else:
raise HySdkReadError(error_code)
result: list[Mapping[str, object]] = []
for key in keys:
value = raw[key]
if key == "memories" and isinstance(value, Mapping):
if set(value) != {"schema", "proactive", "normal"}:
raise HySdkReadError(error_code)
channels = tuple(value.values())
else:
channels = (value,)
for channel in channels:
if not isinstance(channel, list) or any(
not isinstance(row, Mapping) for row in channel
):
raise HySdkReadError(error_code)
if key == selected:
result.extend(channel)
return tuple(result)
def _score(memory: Mapping[str, object]) -> float:
score = memory.get("score")
return float(score) if isinstance(score, int | float) and not isinstance(score, bool) else 0.0
def _formatted_member_ids(memory: Mapping[str, object]) -> tuple[tuple[str, ...], bool]:
represented: dict[str, None] = {}
composite = False
raw_members = memory.get("member_ids")
if raw_members is not None:
composite = True
if not isinstance(raw_members, list | tuple) or any(
not isinstance(member_id, str) or not member_id for member_id in raw_members
):
return (), False
for member_id in raw_members:
represented.setdefault(member_id, None)
chain = memory.get("evolution_chain")
if chain is not None:
composite = True
if not isinstance(chain, list | tuple):
return (), False
for entry in chain:
if not isinstance(entry, Mapping):
return (), False
member_id = entry.get("node_id") or entry.get("memory_id")
if not isinstance(member_id, str) or not member_id:
return (), False
represented.setdefault(member_id, None)
if memory.get("capsule_memories") is not None or memory.get("timeline") is not None:
composite = True
if raw_members is None:
return (), False
content = memory.get("content")
if isinstance(content, str) and content.lstrip().startswith(
("Related memories:", "Matched memories:", "Memory timeline:")
):
composite = True
if not represented:
return (), False
return tuple(represented), not composite or bool(represented)
@@ -1,153 +0,0 @@
"""Per-write original Evidence and where each HY memory came from; never a memory source."""
import json
from collections.abc import Set as AbstractSet
from importlib import import_module
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
from one_memory.input_sanitization import sanitize_model_input
class HySupportedMemory(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
hy_memory_id: str
governed_memory_id: str
revision: int
content: str
owner: Literal["user", "agent"]
evidence_ids: tuple[str, ...]
class HySourceContext(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
current: tuple[EvidenceRecord, ...]
history: tuple[EvidenceRecord, ...] = ()
supporting: tuple[EvidenceRecord, ...] = ()
known_memories: tuple[HySupportedMemory, ...] = ()
class HyWriteBindings:
def __init__(self, context: HySourceContext) -> None:
if not context.current:
raise ValueError("hy_memory_write_verification_context_scope")
self.context = context
seed = context.current[0]
records = (*context.current, *context.history, *context.supporting)
workspaces = {record.workspace for record in records}
# A missing workspace is a legacy identity, not a wildcard that may
# absorb a named project. Keep NULL and named workspaces disjoint.
if None in workspaces and len(workspaces) != 1:
raise ValueError("hy_memory_write_verification_context_scope")
ids = [record.evidence_id for record in records]
if len(ids) != len(set(ids)):
raise ValueError("hy_memory_write_verification_context_scope")
for record in records:
if (
(
record.user_id,
record.source_id,
record.conversation_id,
record.workspace,
)
!= (
seed.user_id,
seed.source_id,
seed.conversation_id,
seed.workspace,
)
or record.content_kind is not EvidenceContentKind.RAW_CONVERSATION
or record.role not in {"user", "assistant", "tool"}
):
raise ValueError("hy_memory_write_verification_context_scope")
self.all_sources: list[dict[str, Any]] = [
{
"id": record.evidence_id,
"role": record.role,
"content": sanitize_model_input(record.verbatim),
"new": index < len(context.current),
"revision": record.revision,
"created_at": record.created_at.isoformat() if record.created_at else None,
}
for index, record in enumerate(records)
]
self.sources = self.all_sources[: len(context.current) + len(context.history)]
self._supports: dict[tuple[str, str], tuple[str, ...]] = {}
self.profile_order: list[str] | None = None
self.verified_operations: dict[str, tuple[str, str]] = {}
self.known = {memory.hy_memory_id: memory for memory in context.known_memories}
for memory in context.known_memories:
if not memory.evidence_ids or not set(memory.evidence_ids) <= set(ids):
raise ValueError("hy_memory_write_verification_context_scope")
self._supports[(memory.content, memory.owner)] = memory.evidence_ids
def check_content(self, content: str) -> None:
try:
parsed = json.loads(content)
except ValueError as error:
raise ValueError("hy_memory_write_verification_input_mismatch") from error
expected = [
{"role": record.role, "content": sanitize_model_input(record.verbatim)}
for record in self.context.current
]
# HY strips tool messages itself before chat extraction when it judges a turn chat.
without_tools = [item for item in expected if item["role"] != "tool"]
if parsed != expected and (without_tools == expected or parsed != without_tools):
raise ValueError("hy_memory_write_verification_input_mismatch")
def history_json(self) -> str:
return json.dumps(
[
{
"evidence_id": source["id"],
**{k: source[k] for k in ("role", "content", "revision", "created_at")},
}
for source in self.sources
if not source["new"]
],
ensure_ascii=False,
)
def turn_support(
self, owner: str | None, targets: AbstractSet[str] = frozenset()
) -> tuple[str, ...]:
"""This turn's words by the memory's author, plus what a merged memory already cited."""
roles = {"user": {"user"}, "agent": {"assistant"}}.get(owner or "")
current = [source for source in self.all_sources if source["new"]]
ids = [source["id"] for source in current if roles is None or source["role"] in roles]
if not ids:
ids = [source["id"] for source in current]
for target in sorted(targets):
if target in self.known:
ids.extend(self.known[target].evidence_ids)
return tuple(dict.fromkeys(ids))
def record_turn_supports(self, items: list[dict[str, Any]]) -> None:
for item in items:
key = (item["content"], item["owner"] or "")
self._supports.setdefault(key, self.turn_support(item["owner"]))
def record_operation_support(self, content: str, owner: str | None, targets: set[str]) -> None:
key = (content, owner or "")
merged = (*self._supports.get(key, ()), *self.turn_support(owner, targets))
self._supports[key] = tuple(dict.fromkeys(merged))
def support_for(self, content: str, owner: str | None) -> tuple[str, ...]:
return self._supports.get((content, owner or ""), ())
def profile_support(self, content: str, values: dict[str, Any]) -> tuple[str, ...]:
render = import_module("hy_memory.agent.tools.basic_profile")._render_content
if not values or render(values, field_order=self.profile_order) != content:
return ()
ids: dict[str, None] = {}
for key, value in values.items():
support = self.support_for(f"{key}: {value}", "user")
if not support:
return ()
ids.update(dict.fromkeys(support))
return tuple(ids)
@@ -1,53 +0,0 @@
"""Input checks around HY's native extractor; HY's output is published as it returns it."""
import asyncio
from typing import Any
from one_memory.hy_source_context import HyWriteBindings
def _extracted_items(info: dict[str, Any]) -> list[dict[str, Any]]:
items: list[dict[str, Any]] = []
for section in ("memory", "intentions"):
for item in info.get(section) or []:
if isinstance(item, dict) and isinstance(item.get("content"), str):
items.append({"content": item["content"], "owner": item.get("owner")})
for name, value in (info.get("basic_info") or {}).items():
items.append({"content": f"{name}: {value}", "owner": "user"})
return items
class HyExtractionGuard:
def __init__(self, extractor: Any) -> None:
self.extractor = extractor
self.failures: list[str] = []
self._lock = asyncio.Lock()
self.bindings: HyWriteBindings | None = None
def __getattr__(self, name: str) -> Any:
return getattr(self.extractor, name)
async def extract(self, *args: Any, **kwargs: Any) -> Any:
async with self._lock:
try:
return await self._extract(args, kwargs)
except ValueError as error:
if str(error).startswith("hy_memory_write_verification"):
self.failures.append(str(error))
raise
async def _extract(self, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
content = kwargs.get("content", args[0] if args else "")
bindings = self.bindings
if bindings is not None:
bindings.check_content(content)
fields = kwargs.get("basic_profile_fields")
bindings.profile_order = list(fields) if fields else None
kwargs = {**kwargs, "history_context": bindings.history_json()}
else:
# No raw-history identity was supplied; do not trust HY's text prefixes.
kwargs = {**kwargs, "history_context": ""}
result = await self.extractor.extract(*args, **kwargs)
if bindings is not None and result.success:
bindings.record_turn_supports(_extracted_items(result.info))
return result
@@ -50,6 +50,10 @@ class AuthorizedSourceFile:
size: int
modified_ns: int
changed_ns: int
# Source-specific change token (for example Cursor's WAL identity). It is
# cursor metadata only; modified_ns remains the real filesystem mtime used
# as a fallback conversation timestamp.
cursor_version: str | None = None
root_identity: str | None = None
requested_root: Path | None = None
access_check: Callable[[], None] | None = field(default=None, repr=False, compare=False)
@@ -557,12 +557,10 @@ class PollingLiveSyncSupervisor:
)
def _note_internal_error(self, source_id: str, error: Exception) -> None:
# Type alone used to be all this said. It is not enough to act on:
# `OperationalError` covers a locked database, a missing table and a
# corrupt page alike, and without the message and the frames two dozen
# of these in a log could only be guessed at. The stack is the
# redacted, `one_memory`-relative one, so it names the call site
# without printing anyone's paths or query text.
# The ordinary local log keeps the exception message for debugging.
# The structured diagnostic event deliberately keeps only the stable
# error code, source id, type, and relative stack: exception messages
# can contain paths, query text, or credentials and are exportable data.
logging.getLogger(__name__).warning(
"live_sync_internal_error %s: %s", type(error).__name__, error
)
@@ -572,10 +570,7 @@ class PollingLiveSyncSupervisor:
event="live_sync.internal_error",
outcome="failed",
error_code="live_sync_internal_error",
# `reason` is the listed key that can carry the message; see
# `diagnostic_log._DETAIL_KEYS`. Truncated because a detail value
# longer than 256 characters is rejected.
details={"source_id": source_id, "reason": str(error)[:200]},
details={"source_id": source_id},
**diagnostic_exception(error),
)
if isinstance(error, sqlite3.Error):
@@ -867,10 +867,7 @@ class ImportRunner:
outcome="failed",
job_id=job_id,
error_code="import_heartbeat_failed",
# `reason` is the listed key for this; see
# `diagnostic_log._DETAIL_KEYS`. Truncated: a detail value
# over 256 characters is rejected.
details={"reason": str(error)[:200]},
details={"phase": "heartbeat"},
**diagnostic_exception(error),
)
@@ -644,7 +644,7 @@ def is_unchanged(source_file: AuthorizedSourceFile, cursors: dict[str, SourceCur
return (
previous is not None
and previous.read_to_end
and previous.file_identity == source_file.file_identity
and previous.file_identity == _cursor_file_identity(source_file)
and previous.observed_size == source_file.size
and previous.observed_modified_ns == source_file.modified_ns
# Chmod alone leaves size, mtime and inode untouched, so a file that
@@ -691,7 +691,7 @@ def touched_cursor(
previous = options.cursors.get(source_file.safe_id)
keep = (
previous is not None
and previous.file_identity == source_file.file_identity
and _same_file(previous, source_file)
and previous.observed_size <= source_file.size
)
# A kept cursor means the next scan resumes exactly where this one ended:
@@ -871,7 +871,7 @@ def resume_checkpoint(
previous = options.cursors.get(source_file.safe_id)
if previous is None:
return None
if previous.file_identity != source_file.file_identity:
if not _same_file(previous, source_file):
return None
if source_file.size < previous.observed_size:
return None
@@ -1046,7 +1046,7 @@ def source_cursor(
) -> SourceCursor:
return SourceCursor(
file_key=source_file.safe_id,
file_identity=source_file.file_identity,
file_identity=_cursor_file_identity(source_file),
checkpoint=checkpoint,
checkpoint_anchor=checkpoint_anchor,
observed_size=source_file.size,
@@ -1054,6 +1054,27 @@ def source_cursor(
)
def _cursor_file_identity(source_file: AuthorizedSourceFile) -> str:
return (
f"{source_file.file_identity}{_VERSION_SEPARATOR}{source_file.cursor_version}"
if source_file.cursor_version
else source_file.file_identity
)
_VERSION_SEPARATOR = ":v:"
def _same_file(previous: SourceCursor, source_file: AuthorizedSourceFile) -> bool:
"""Whether a stored cursor still points into this file.
The stored identity may carry a change token (Cursor's WAL version). The
token says the file changed, which `is_unchanged` handles; it does not mean
the file was replaced, so resuming compares the bare `dev:ino` only.
"""
return previous.file_identity.partition(_VERSION_SEPARATOR)[0] == source_file.file_identity
_POSITION = re.compile(r"(?<=:)\d+(?=:|$)")
@@ -533,8 +533,8 @@ def _with_write_ahead_log(database: AuthorizedSourceFile) -> AuthorizedSourceFil
version = hashlib.blake2b(
f"{database.modified_ns}:{journal.st_mtime_ns}:{journal.st_size}:{journal.st_ino}".encode(),
digest_size=8,
).digest()
return replace(database, modified_ns=int.from_bytes(version, "big") & ((1 << 63) - 1))
).hexdigest()
return replace(database, cursor_version=version)
def _live_baseline(listing: _CursorListing) -> dict[str, SourceCursor]:
@@ -0,0 +1,35 @@
"""Passive lifecycle projection used only by retired Governance readers."""
from __future__ import annotations
from collections.abc import Mapping
from dataclasses import dataclass, field
from types import MappingProxyType
@dataclass(frozen=True, eq=False)
class HyLifecycle:
hidden: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
def __post_init__(self) -> None:
object.__setattr__(self, "hidden", MappingProxyType(dict(self.hidden)))
def __eq__(self, other: object) -> bool:
return isinstance(other, HyLifecycle) and dict(self.hidden) == dict(
other.hidden
)
def __hash__(self) -> int:
return hash(tuple(sorted(self.hidden.items())))
def replaced_by(self, memory_id: str) -> tuple[str, ...]:
return self.hidden.get(memory_id, ())
def replacing(self, memory_id: str) -> tuple[str, ...]:
return tuple(
sorted(
old
for old, replacements in self.hidden.items()
if memory_id in replacements
)
)
@@ -0,0 +1,110 @@
"""Short-lived result windows for the memory library's keyword search.
A library search asks the SDK once for a bounded relevance window and keeps
the keyword matches. Paging must cut every page from that same list: running
the search again per page can rank differently and duplicate or skip memories
between pages. The matches are kept here under an opaque ``window_id`` that
the client passes back with the next offset.
A window is bound to the filters it was searched with (query, source,
statuses). It expires after ``ttl_seconds`` without being read, and the store
keeps at most ``max_windows`` windows, dropping the least recently read first.
Only memory identities are kept, never content: each page is read fresh from
the SDK, so an edit or a delete shows on the next page read.
"""
from __future__ import annotations
import secrets
import threading
import time
from collections import OrderedDict
from collections.abc import Callable
from dataclasses import dataclass, field
from one_memory.hy_sdk_types import LibraryHit
DEFAULT_WINDOW_TTL_SECONDS = 300.0
DEFAULT_MAX_WINDOWS = 32
WindowKey = tuple[str, str | None, tuple[str, ...]]
def window_key(query: str, source: str | None, statuses: list[str]) -> WindowKey:
return (query, source, tuple(sorted(set(statuses))))
@dataclass(frozen=True)
class SearchWindow:
window_id: str
key: WindowKey
hits: tuple[LibraryHit, ...]
# Set when the SDK window was full, so matches outside it were not looked
# for and `total` may undercount.
search_window: int | None
@dataclass
class _Entry:
window: SearchWindow
last_read: float = field(default=0.0)
class LibrarySearchWindows:
def __init__(
self,
*,
ttl_seconds: float = DEFAULT_WINDOW_TTL_SECONDS,
max_windows: int = DEFAULT_MAX_WINDOWS,
clock: Callable[[], float] = time.monotonic,
) -> None:
self._ttl = ttl_seconds
self._max = max(1, max_windows)
self._clock = clock
self._entries: OrderedDict[str, _Entry] = OrderedDict()
self._lock = threading.Lock()
def get(self, window_id: str, key: WindowKey) -> SearchWindow | None:
"""The window, if it is still live and was searched with `key`."""
with self._lock:
now = self._clock()
self._expire(now)
entry = self._entries.get(window_id)
if entry is None or entry.window.key != key:
return None
entry.last_read = now
self._entries.move_to_end(window_id)
return entry.window
def put(
self,
key: WindowKey,
hits: tuple[LibraryHit, ...],
search_window: int | None,
) -> SearchWindow:
window = SearchWindow(
window_id=secrets.token_urlsafe(12),
key=key,
hits=hits,
search_window=search_window,
)
with self._lock:
now = self._clock()
self._expire(now)
self._entries[window.window_id] = _Entry(window=window, last_read=now)
while len(self._entries) > self._max:
self._entries.popitem(last=False)
return window
def __len__(self) -> int:
with self._lock:
return len(self._entries)
def _expire(self, now: float) -> None:
stale = [
window_id
for window_id, entry in self._entries.items()
if now - entry.last_read >= self._ttl
]
for window_id in stale:
del self._entries[window_id]
@@ -1,6 +1,7 @@
"""Write visible hook text into Evidence. Tool output stays out."""
import json
import os
import time
from datetime import UTC, datetime
from enum import StrEnum
@@ -74,12 +75,13 @@ _FILE_FALLBACK = {
}
_FALLBACK_EVENTS = frozenset({"Stop", "stop", "SubagentStop", "StopFailure"})
_TEXT_BLOCKS = frozenset({"text"})
SDK_MIGRATION_CAPTURE_FENCE = ".sdk-migration-capture-fence.json"
def database_path() -> Path:
from one_memory.workbuddy_boundary import database_path as boundary_database
from one_memory.config import load_settings
return boundary_database()
return load_settings().data_dir / "one-memory.sqlite3"
def capture_stdio(source_id: str, stdin: TextIO, output: TextIO) -> int:
@@ -88,17 +90,49 @@ def capture_stdio(source_id: str, stdin: TextIO, output: TextIO) -> int:
del output
if source_id not in CAPTURE_SOURCES:
return 2
database = database_path()
if sdk_migration_capture_fenced(database.parent):
return 0
try:
payload = json.loads(stdin.read() or "")
except json.JSONDecodeError:
return 0
try:
capture_hook(source_id, payload, database_path())
capture_hook(source_id, payload, database)
except Exception:
return 0
return 0
def sdk_migration_capture_fenced(data_dir: Path) -> bool:
"""Ignore stale fences; a live migration or a failed one pauses hook writes.
A failed upgrade migration leaves the fence `frozen` (no pid): the home was
put back as it was before the upgrade and takes no writes until the next
migration attempt, which replaces the fence and lifts it on success.
"""
marker = data_dir / SDK_MIGRATION_CAPTURE_FENCE
try:
document = json.loads(marker.read_text(encoding="utf-8"))
if isinstance(document, dict) and document.get("frozen") is True:
return True
pid = int(document["pid"])
if pid < 1:
raise ValueError("invalid migration fence pid")
os.kill(pid, 0)
return True
except FileNotFoundError:
return False
except ProcessLookupError:
marker.unlink(missing_ok=True)
return False
except PermissionError:
return True
except (KeyError, TypeError, ValueError, json.JSONDecodeError, OSError):
marker.unlink(missing_ok=True)
return False
def capture_hook(source_id: str, payload: object, database: Path) -> bool:
if source_id not in CAPTURE_SOURCES or not isinstance(payload, dict):
return False
@@ -224,7 +258,7 @@ def record_capture_rejection(
def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed: bool) -> None:
if not changed or not _sync_open(store.database_path, record.source_id):
return
from one_memory.memory_processing import _conversation_chunks
from one_memory.conversation_processing import _conversation_chunks
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
@@ -10,16 +10,12 @@ weights alone.
from __future__ import annotations
import threading
from dataclasses import dataclass
from hashlib import sha256
from importlib import import_module
from typing import TYPE_CHECKING, Any
from one_memory.embedding_assets import (
BUILTIN_EMBEDDING_DIMENSIONS,
BUILTIN_EMBEDDING_MAX_TOKENS,
EMBEDDING_ASSETS,
AssetVerificationError,
MissingEmbeddingAssetsError,
assets_present,
resolve_asset_dir,
@@ -34,28 +30,6 @@ _FETCH_HINT = (
)
@dataclass(frozen=True)
class TokenOffset:
"""Half-open Python Unicode code point offsets; synthetic specials use [0,0)."""
start_cp: int
end_cp: int
special: bool = False
@dataclass(frozen=True)
class TokenizedText:
"""Untruncated, unpadded encoding, including tokenizer postprocessor specials."""
token_ids: tuple[int, ...]
offsets: tuple[TokenOffset, ...]
def __post_init__(self) -> None:
# Defensive copies also keep runtime callers from retaining mutable lists.
object.__setattr__(self, "token_ids", tuple(self.token_ids))
object.__setattr__(self, "offsets", tuple(self.offsets))
class LocalEmbedder:
"""Encodes text with the built-in model, loading it on first use."""
@@ -68,43 +42,6 @@ class LocalEmbedder:
self._session: Any | None = None
self._tokenizer: Any | None = None
self._input_names: frozenset[str] = frozenset()
self._offset_lock = threading.Lock()
self._offset_tokenizer: Any | None = None
def tokenize_with_offsets(self, text: str) -> TokenizedText:
"""Tokenize original text without encode()'s truncation or batch padding.
The Python tokenizers binding returns code point offsets into the input,
even when its model normalizer changes spelling. This separate lazy
tokenizer uses the exact locked asset; it never loads an ONNX session or
mutates the tokenizer shared by encode(). Offsets are not UTF-16 indices.
"""
if not isinstance(text, str):
raise TypeError("embedding_offsets_require_text")
with self._offset_lock:
if self._offset_tokenizer is None:
directory = resolve_asset_dir()
path = directory / "tokenizer.json"
if not path.is_file():
raise MissingEmbeddingAssetsError(f"{_FETCH_HINT} Looked in {directory}.")
data = path.read_bytes()
expected = next(a.sha256 for a in EMBEDDING_ASSETS if a.filename == path.name)
if sha256(data).hexdigest() != expected:
raise AssetVerificationError("tokenizer.json: locked digest mismatch")
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_str(data.decode("utf-8"))
tokenizer.no_truncation()
tokenizer.no_padding()
self._offset_tokenizer = tokenizer
encoding = self._offset_tokenizer.encode(text, add_special_tokens=True)
offsets = tuple(
TokenOffset(start, end, bool(special))
for (start, end), special in zip(
encoding.offsets, encoding.special_tokens_mask, strict=True
)
)
return TokenizedText(tuple(encoding.ids), offsets)
def _load(self) -> tuple[Any, Any]:
# Double-checked under a lock: memory processing encodes from worker
@@ -121,7 +58,7 @@ class LocalEmbedder:
raise MissingEmbeddingAssetsError(f"{_FETCH_HINT} Looked in {directory}.")
# Imported by name for the same reason kuzu and hy_memory are in
# hy_memory_bridge: onnxruntime ships no py.typed, and a dynamic
# onnxruntime ships no py.typed, and a dynamic
# import keeps the strict checker on everything around it.
ort = import_module("onnxruntime")
from tokenizers import Tokenizer
@@ -14,6 +14,7 @@ stuck in it.
"""
import logging
import re
import sys
import threading
import time
@@ -23,6 +24,7 @@ from types import FrameType
logger = logging.getLogger("one_memory.loop_watchdog")
DEFAULT_STALL_SECONDS = 2.0
_SAFE_FALLBACK_FILE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_.-]*\.py$")
class LoopWatchdog:
@@ -120,22 +122,28 @@ def _safe_stack_for_main_thread() -> tuple[str, ...]:
def _safe_stacks_for_all_threads() -> tuple[str, ...]:
"""Return one content-free, bounded stack summary for every live thread."""
names = {thread.ident: thread.name for thread in threading.enumerate()}
"""Return bounded stack summaries without recording thread names."""
lines: list[str] = []
for ident, frame in sorted(sys._current_frames().items()):
for _ident, frame in sorted(sys._current_frames().items()):
stack: list[str] = []
# Most specific frame first so the 256-character privacy bound keeps
# the likely culprit rather than only the thread bootstrap frames.
# Most specific frame first; append only complete frames so truncation
# can never turn a valid location into free text.
for entry in reversed(traceback.extract_stack(frame)[-8:]):
parts = entry.filename.replace("\\", "/").split("/")
try:
root = parts.index("one_memory")
relative = "/".join(parts[root:])
except ValueError:
relative = parts[-1]
fallback = parts[-1]
relative = fallback if _SAFE_FALLBACK_FILE.fullmatch(fallback) else "unknown.py"
name = entry.name if entry.name.isidentifier() else "call"
stack.append(f"{relative}:{entry.lineno}:{name}")
thread_name = names.get(ident, "unknown").replace("\n", " ").replace("\r", " ")
lines.append(f"thread={thread_name[:48]}|{' > '.join(stack)}"[:256])
candidate = f"{relative}:{entry.lineno}:{name}"
separator = "" if not stack else " > "
if len("thread|") + len(" > ".join(stack)) + len(separator) + len(candidate) > 256:
break
stack.append(candidate)
if stack:
lines.append(f"thread|{' > '.join(stack)}")
if len(lines) == 16:
break
return tuple(lines)
@@ -22,7 +22,7 @@ from pydantic import BaseModel, ConfigDict, Field
from one_memory.database import enable_write_ahead_log
from one_memory.database_access import main_database
from one_memory.hy_lifecycle import HyLifecycle
from one_memory.legacy_lifecycle_types import HyLifecycle
from one_memory.library_upgrade import LibraryUpgradeError, PreparedLibrary, holds_legacy_rows
from one_memory.memory_server.migrations import governance_ddl
@@ -1,342 +0,0 @@
"""Read-only local library projection over the existing governed memory store."""
import json
import os
import sqlite3
import time
from pathlib import Path
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
from one_memory.database_access import main_database
from one_memory.hy_lifecycle import HyLifecycle
from one_memory.memory_evidence_freshness import register_memory_freshness
from one_memory.memory_governance import (
GovernedMemory,
MemoryGovernanceStore,
MemoryStatus,
lifecycle_predicate,
)
#: Read budget for one library page. 2026-10-12: raised from 5s (which starved
#: under a concurrent history import and surfaced as 503) to 15s, still well
#: under the 30s client timeout; past it the read degrades to a `stale` page.
READ_BUDGET_SECONDS = 15
class MemorySourceTime(BaseModel):
"""When the conversation a memory came from happened (D96).
`basis` is `conversation` when the cited messages carry their own time,
`import` when only the import time is known; the UI words them apart.
"""
model_config = ConfigDict(frozen=True, extra="forbid")
at: str
basis: Literal["conversation", "import"]
class MemoryLibraryPage(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
items: list[GovernedMemory]
total: int = Field(ge=0)
offset: int = Field(ge=0)
limit: int = Field(ge=1, le=500)
source_ids_by_memory: dict[str, list[str]] = Field(default_factory=dict)
source_time_by_memory: dict[str, MemorySourceTime] = Field(default_factory=dict)
#: True when the read budget ran out mid-page (typically during a history
#: import): the counts and rows above are a partial snapshot, not an
#: authoritative view. Clients should show "importing" instead of an error.
stale: bool = False
def _sort_by_conversation() -> bool:
"""D99: the list reads as a timeline of the conversations, newest first.
A memory generated today from a three-week-old dialogue belongs next to
that week, not at the top. ONE_MEMORY_LIBRARY_SORT=updated brings back the
old "newest generated first" order without a code change.
"""
return os.environ.get("ONE_MEMORY_LIBRARY_SORT", "").strip().lower() != "updated"
def _origin_join() -> str:
"""The newest cited message per memory, as one joined aggregate.
A correlated subquery would re-scan the link table for every candidate row;
this is a single pass. Memories with no readable citation keep their old
position through COALESCE.
"""
if not _sort_by_conversation():
return ""
return """LEFT JOIN (
SELECT link.memory_id AS memory_id,
MAX(COALESCE(e.created_at, e.imported_at)) AS origin_at
FROM governed_memory_evidence link
JOIN evidence e ON e.evidence_id = link.evidence_id
WHERE link.active = 1
GROUP BY link.memory_id
) origin ON origin.memory_id = m.memory_id"""
def _order_by() -> str:
if not _sort_by_conversation():
return "m.updated_at DESC, m.memory_id ASC"
# Conversations that share a timestamp keep the generated order between
# themselves; ties would otherwise fall back to the id and look arbitrary.
return "COALESCE(origin.origin_at, m.updated_at) DESC, m.updated_at DESC, m.memory_id ASC"
class _LibraryQuery(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
user_id: str = Field(min_length=1, max_length=256)
query: str | None = None
status: MemoryStatus | None = None
active_only: bool = False
source: str | None = Field(default=None, min_length=1)
offset: int = Field(default=0, ge=0, le=2**63 - 1, strict=True)
limit: int = Field(default=100, ge=1, le=500, strict=True)
def read_conversation_memories(
database_path: Path,
*,
user_id: str,
source_id: str,
conversation_id: str,
workspace: str | None,
limit: int = 100,
) -> list[GovernedMemory]:
"""Current memories backed by any active Evidence in one conversation."""
with main_database(database_path).read() as db:
db.row_factory = sqlite3.Row
freshness = register_memory_freshness(db)
rows = db.execute(
f"""
SELECT DISTINCT m.* FROM governed_memories m
JOIN governed_memory_evidence link ON link.memory_id=m.memory_id AND link.active=1
JOIN evidence e ON e.evidence_id=link.evidence_id
WHERE m.user_id=? AND m.status='confirmed' AND m.deleted_at IS NULL
AND e.user_id=? AND e.source_id=? AND e.conversation_id=?
AND e.workspace IS ? AND {freshness}
ORDER BY m.updated_at DESC, m.memory_id
LIMIT ?
""",
(user_id, user_id, source_id, conversation_id, workspace, limit),
).fetchall()
return MemoryGovernanceStore.memories_from_rows(db, rows)
def read_memory_library(
database_path: Path,
*,
user_id: str = "local",
query: str | None = None,
status: MemoryStatus | None = None,
active_only: bool = False,
source: str | None = None,
offset: int = 0,
limit: int = 100,
lifecycle: HyLifecycle | None = None,
) -> MemoryLibraryPage:
"""Read one page and its exact readable total from the same SQLite snapshot.
This is the local UI policy, not an external agent authorization entry point.
Missing grants do not revoke already stored content; explicit revocations do.
Read/parse failures propagate rather than masquerading as an empty library.
Which memories are current is HY's `lifecycle` when given, as in
`MemoryGovernanceStore.list`.
"""
request = _LibraryQuery(
user_id=user_id,
query=query,
status=status,
active_only=active_only,
source=source,
offset=offset,
limit=limit,
)
if request.active_only and request.status is not None:
raise ValueError("active_only_conflicts_with_status")
clauses = ["m.user_id = ?"]
values: list[str | int] = [request.user_id]
selected, selected_values = lifecycle_predicate(
lifecycle, MemoryStatus.CONFIRMED if request.active_only else request.status, alias="m"
)
if selected is not None:
clauses.append(selected)
values.extend(selected_values)
if request.status != MemoryStatus.DELETED:
clauses.append("m.deleted_at IS NULL")
# SourceReadAccess hides the whole memory if any active citation is revoked.
# Do this before COUNT/LIMIT; a readable citation cannot rescue a mixed memory.
clauses.append("""NOT EXISTS (
SELECT 1 FROM governed_memory_evidence link
JOIN evidence e ON e.evidence_id = link.evidence_id
JOIN source_grants g ON g.user_id = e.user_id AND g.source_id = e.source_id
WHERE link.memory_id = m.memory_id AND +link.active = 1
AND g.revoked_at IS NOT NULL
)""")
# Governance normally guarantees ownership and referential integrity. Fail
# closed for broken active links instead of exposing another user's citation.
clauses.append("""NOT EXISTS (
SELECT 1 FROM governed_memory_evidence link
LEFT JOIN evidence e ON e.evidence_id = link.evidence_id
WHERE link.memory_id = m.memory_id AND +link.active = 1
AND (e.evidence_id IS NULL OR e.user_id != m.user_id)
)""")
if request.query:
clauses.append("m.content LIKE ? ESCAPE '\\'")
escaped = request.query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
values.append(f"%{escaped}%")
if request.source is not None:
# Start from the compact active-link index, then resolve each Evidence
# identity by primary key. The old correlated EXISTS let SQLite scan
# every Evidence row for every candidate memory on large libraries.
clauses.append("""m.memory_id IN (
SELECT link.memory_id FROM governed_memory_evidence link
JOIN evidence e ON e.evidence_id = link.evidence_id
WHERE link.active = 1 AND e.user_id = ? AND e.source_id = ?
)""")
values.extend((request.user_id, request.source))
predicate = " AND ".join(clauses)
def _interrupted(error: Exception) -> bool:
return isinstance(error, sqlite3.OperationalError) and "interrupted" in str(error).lower()
with main_database(database_path).read() as db:
# 2026-10-12: 5s starved the page whenever a history import saturated
# SQLite, and the caller turned the deadline abort into a 503, so the
# memory page read as broken for the whole import. 15s stays well under
# the 30s client timeout; past it we return a `stale` page instead of
# an error — an incomplete list is honest, an error is not.
deadline = time.monotonic() + READ_BUDGET_SECONDS
db.set_progress_handler(lambda: int(time.monotonic() >= deadline), 1000)
db.row_factory = sqlite3.Row
db.execute("BEGIN")
# The total counts with the cheap permission filters only: the per-row
# freshness UDF scales with the whole library and can burn the whole
# read budget on its own (D22). Product accepted a small drift here;
# the rows that are actually displayed are still checked row by row.
try:
total = db.execute(
f"SELECT COUNT(*) FROM governed_memories m WHERE {predicate}", values
).fetchone()[0]
except sqlite3.OperationalError as error:
if not _interrupted(error):
raise
return MemoryLibraryPage(
items=[],
total=0,
offset=request.offset,
limit=request.limit,
stale=True,
)
predicate += " AND " + register_memory_freshness(db)
try:
if request.query:
current = db.execute(
f"""SELECT m.* FROM governed_memories m {_origin_join()}
WHERE {predicate}
ORDER BY {_order_by()}""",
values,
).fetchall()
rows = current[request.offset : request.offset + request.limit]
else:
rows = db.execute(
f"""SELECT m.* FROM governed_memories m {_origin_join()}
WHERE {predicate}
ORDER BY {_order_by()} LIMIT ? OFFSET ?""",
[*values, request.limit, request.offset],
).fetchall()
except sqlite3.OperationalError as error:
if not _interrupted(error):
raise
# The count already succeeded: report it, flag the page stale.
return MemoryLibraryPage(
items=[],
total=total,
offset=request.offset,
limit=request.limit,
stale=True,
)
# Static read helper only: constructing the Store would initialize schema.
items = MemoryGovernanceStore.memories_from_rows(db, rows, lifecycle=lifecycle)
# Only the authorized page participates; source filters select memories,
# not a subset of their provenance. Keep this in the count/page snapshot.
source_ids_by_memory: dict[str, list[str]] = {item.memory_id: [] for item in items}
# D96: the list shows when the cited conversation happened, not when
# the memory was generated (which is "today" for a fresh import). Read
# in the same query as the sources, so the page costs no extra round
# trip and only this page's IDs are looked at, never the whole library.
conversation_at: dict[str, str] = {}
imported_at: dict[str, str] = {}
if items:
try:
for link in db.execute(
# D104: the conversation time is when the person spoke, not
# when the turn ended. A memory's evidence covers the whole
# round (question, tool calls, reply), so MAX() lands on the
# reply and reads as a later, contradictory time next to the
# cited source. Prefer the earliest person-side evidence; only
# fall back to the earliest of any kind when the round has no
# person-authored row at all.
"""SELECT link.memory_id, e.source_id,
COALESCE(
MIN(
CASE WHEN e.role = 'user'
AND e.role = 'user'
THEN e.created_at END
),
MIN(e.created_at)
) AS conversation_at,
MAX(e.imported_at) AS imported_at
FROM governed_memory_evidence link
JOIN evidence e ON e.evidence_id = link.evidence_id
WHERE link.active = 1 AND e.user_id = ?
AND link.memory_id IN (SELECT value FROM json_each(?))
GROUP BY link.memory_id, e.source_id
ORDER BY link.memory_id, e.source_id""",
(request.user_id, json.dumps(list(source_ids_by_memory))),
):
memory_id = link["memory_id"]
source_ids_by_memory[memory_id].append(link["source_id"])
for seen, value in (
(conversation_at, link["conversation_at"]),
(imported_at, link["imported_at"]),
):
if value and str(value) > seen.get(memory_id, ""):
seen[memory_id] = str(value)
except sqlite3.OperationalError as error:
if not _interrupted(error):
raise
# Page rows are already read; ship them without source maps
# rather than failing the whole request at the last step.
return MemoryLibraryPage(
items=items,
total=total,
offset=request.offset,
limit=request.limit,
source_ids_by_memory=source_ids_by_memory,
stale=True,
)
source_time_by_memory = {
memory_id: MemorySourceTime(at=conversation_at[memory_id], basis="conversation")
if memory_id in conversation_at
else MemorySourceTime(at=imported_at[memory_id], basis="import")
for memory_id in source_ids_by_memory
if memory_id in conversation_at or memory_id in imported_at
}
return MemoryLibraryPage(
items=items,
total=total,
offset=request.offset,
limit=request.limit,
source_ids_by_memory=source_ids_by_memory,
source_time_by_memory=source_time_by_memory,
)
@@ -1,180 +0,0 @@
"""Bounded, permission-aware receipt projection; never initialize a write store."""
import sqlite3
from pathlib import Path
from typing import Literal
from pydantic import BaseModel, ConfigDict
from one_memory.database_access import main_database
#: `legacy` is a confirmation recorded before memories stopped waiting for
#: one; it changed nothing that still exists, and `migrated_from` says so.
Operation = Literal["create", "correct", "set_scopes", "undo", "delete", "hy_reconcile", "legacy"]
ResultStatus = Literal["confirmed", "deleted"]
MigratedFrom = Literal["confirm", "reject"]
OPERATIONS = {"create", "correct", "set_scopes", "undo", "delete", "hy_reconcile", "legacy"}
STATUSES = {"confirmed", "deleted"}
#: Which operation each pre-v2 one now reads as.
MIGRATED_OPERATIONS = {"confirm": "legacy", "reject": "delete"}
#: A memory may move on to these after an earlier receipt was written; the
#: receipt still reports what that request committed, not the current status.
CURRENT_STATUSES = STATUSES | {"superseded"}
class MemoryMutationReceiptForbidden(RuntimeError):
def __init__(self) -> None:
super().__init__("memory_mutation_receipt_forbidden")
class MemoryMutationReceiptUnavailable(RuntimeError):
def __init__(self) -> None:
super().__init__("memory_mutation_receipt_unavailable")
class MemoryMutationReceipt(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
schema_version: Literal[2] = 2
request_id: str
state: Literal["not_found", "committed"]
operation: Operation | None = None
memory_id: str | None = None
result_revision: int | None = None
result_status: ResultStatus | None = None
memory_state: Literal["present", "deleted"] | None = None
#: The operation as it was stored before v2, when it has no v2 name of its own.
migrated_from: MigratedFrom | None = None
def _identity(value: str, maximum: int) -> bool:
return isinstance(value, str) and 0 < len(value) <= maximum
def _read_snapshot(
database_path: Path, request_id: str, user_id: str
) -> tuple[MemoryMutationReceipt, tuple[object, ...]]:
with main_database(database_path).read(row_factory=sqlite3.Row) as db:
db.execute("PRAGMA query_only=ON")
db.execute("BEGIN")
rows = db.execute(
"SELECT user_id,request_id,memory_id,operation,result_revision "
"FROM governed_memory_mutations WHERE user_id=? AND request_id=? LIMIT 2",
(user_id, request_id),
).fetchall()
if not rows:
return MemoryMutationReceipt(request_id=request_id, state="not_found"), ()
if len(rows) != 1:
raise MemoryMutationReceiptUnavailable()
mutation = rows[0]
memory_id, operation, revision = (
mutation["memory_id"],
mutation["operation"],
mutation["result_revision"],
)
if (
not _identity(memory_id, 512)
or operation not in OPERATIONS
or type(revision) is not int
or not 1 <= revision <= 2**53 - 1
):
raise MemoryMutationReceiptUnavailable()
current = db.execute(
"SELECT user_id,revision,status,deleted_at FROM governed_memories WHERE memory_id=?",
(memory_id,),
).fetchone()
if (
current is None
or current["user_id"] != user_id
or type(current["revision"]) is not int
or current["revision"] < revision
or current["status"] not in CURRENT_STATUSES
or (current["status"] == "deleted") != (current["deleted_at"] is not None)
):
raise MemoryMutationReceiptUnavailable()
# Versions have no historical evidence snapshot. Check every retained link,
# not only active ones, before disclosing even operation/ID metadata.
links = db.execute(
"SELECT l.evidence_id,l.active,e.user_id,e.source_id,e.revision,"
"g.granted_at,g.revoked_at FROM governed_memory_evidence l "
"LEFT JOIN evidence e ON e.evidence_id=l.evidence_id "
"LEFT JOIN source_grants g ON g.user_id=e.user_id AND g.source_id=e.source_id "
"WHERE l.memory_id=? ORDER BY l.evidence_id LIMIT 10001",
(memory_id,),
).fetchall()
if not links or len(links) > 10000:
raise MemoryMutationReceiptUnavailable()
for link in links:
if (
link["user_id"] != user_id
or link["source_id"] is None
or link["active"] not in (0, 1)
):
raise MemoryMutationReceiptUnavailable()
if link["revoked_at"] is not None:
raise MemoryMutationReceiptForbidden()
versions = db.execute(
"SELECT memory_id,revision,action,status,migrated_from FROM governed_memory_versions "
"WHERE memory_id=? AND revision=? LIMIT 2",
(memory_id, revision),
).fetchall()
if len(versions) != 1:
raise MemoryMutationReceiptUnavailable()
version = versions[0]
if version["action"] != operation or version["status"] not in STATUSES:
raise MemoryMutationReceiptUnavailable()
migrated_from = version["migrated_from"]
if migrated_from is not None and MIGRATED_OPERATIONS.get(migrated_from) != operation:
raise MemoryMutationReceiptUnavailable()
if (operation == "legacy") != (migrated_from == "confirm"):
raise MemoryMutationReceiptUnavailable()
required_status = {
"create": "confirmed",
"legacy": "confirmed",
"delete": "deleted",
}
if operation in required_status and version["status"] != required_status[operation]:
raise MemoryMutationReceiptUnavailable()
if (operation == "create") != (revision == 1):
raise MemoryMutationReceiptUnavailable()
if (
operation in {"correct", "set_scopes", "hy_reconcile", "undo"}
and version["status"] != "confirmed"
):
raise MemoryMutationReceiptUnavailable()
receipt = MemoryMutationReceipt(
request_id=request_id,
state="committed",
operation=operation,
memory_id=memory_id,
result_revision=revision,
result_status=version["status"],
memory_state="deleted" if current["status"] == "deleted" else "present",
migrated_from=migrated_from,
)
return receipt, (
tuple(mutation),
tuple(current),
tuple(version),
tuple(tuple(link) for link in links),
)
def read_memory_mutation_receipt(
database_path: Path, *, request_id: str, user_id: str = "local"
) -> MemoryMutationReceipt:
"""History only, never current authorization, HY cleanup or safe-retry proof."""
if not _identity(request_id, 256):
raise ValueError("invalid_memory_mutation_request_id")
if not _identity(user_id, 256):
raise ValueError("invalid_memory_mutation_user_id")
try:
first = _read_snapshot(database_path, request_id, user_id)
current = _read_snapshot(database_path, request_id, user_id)
if first != current:
raise MemoryMutationReceiptUnavailable()
return current[0]
except MemoryMutationReceiptForbidden:
raise
except (sqlite3.Error, ValueError, TypeError, KeyError, IndexError, OSError):
raise MemoryMutationReceiptUnavailable() from None
+17 -58
View File
@@ -3,7 +3,7 @@ import json
import sqlite3
from collections.abc import Sequence
from datetime import UTC, datetime
from typing import Literal, cast
from typing import Any, Literal, cast
from uuid import UUID
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
@@ -21,11 +21,10 @@ from one_memory.imports.models import (
)
from one_memory.imports.storage import EvidenceStore
from one_memory.input_sanitization import sanitize_model_input
from one_memory.memory_governance import MemoryStatus
from one_memory.memory_processing import MemoryProcessingSupervisor
from one_memory.memory_governance import MemoryGovernanceStore, MemoryStatus
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
from one_memory.memory_tasks import MemoryTaskStatus
from one_memory.memory_tasks import MemoryTaskStatus, MemoryTaskStore
from one_memory.secrets import SecretStoreUnavailableError
SOURCE_ID = "one-memory-manual"
@@ -68,25 +67,23 @@ class MemoryNoteReceipt(BaseModel):
error: str | None = None
class SavedMemoryNoteEvidence(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
class LegacyMemoryNoteReader:
"""Read pre-cutover note receipts without retaining any write runtime."""
available: Literal[True] = True
evidence_id: str
source_id: str
revision: int
imported_at: datetime
role: str
written_by: WrittenBy
created_at: datetime | None
verbatim: str
def __init__(
self,
tasks: MemoryTaskStore,
governance: MemoryGovernanceStore,
) -> None:
self.tasks = tasks
self.governance = governance
class MemoryNoteStore:
def __init__(
self,
evidence: EvidenceStore,
processor: MemoryProcessingSupervisor,
processor: Any | None,
intake: UnifiedWriteIntake | None = None,
*,
sdk_authority: bool = False,
@@ -252,6 +249,8 @@ class MemoryNoteStore:
memory_ids=(list(durable.sdk_memory_ids) if durable is not None else []),
error=durable.detail if durable is not None else "write_missing",
)
if self.processor is None:
raise RuntimeError("legacy_memory_note_processor_unavailable")
task = self.processor.tasks.task_for_evidence(evidence_id)
memory_ids = []
if task:
@@ -356,6 +355,8 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
try:
if notes.sdk_authority:
return await asyncio.to_thread(notes.receipt, record.evidence_id)
if notes.processor is None:
raise RuntimeError("legacy_memory_note_processor_unavailable")
if notes.intake is None:
# Isolated tests construct the store without a queue.
await notes.processor.enqueue_conversation_off_thread(record)
@@ -426,48 +427,6 @@ def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
except ValueError as error:
raise HTTPException(409, str(error)) from error
@router.get("/{evidence_id}/evidence", response_model=SavedMemoryNoteEvidence)
def evidence(evidence_id: str, request: Request) -> SavedMemoryNoteEvidence:
try:
notes = store(request)
if notes.sdk_authority:
raise HTTPException(409, "memory_note_evidence_retired")
with notes.evidence._connect() as connection:
record = notes.evidence._get_with_connection(connection, evidence_id)
metadata = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT) if record else None
if (
record is None
or record.user_id != "local"
or record.source_id != SOURCE_ID
or metadata is None
or metadata.raw_meta.get("input_mode") != "direct_memory_note"
):
raise KeyError(evidence_id)
grant = connection.execute(
"SELECT revoked_at FROM source_grants WHERE user_id='local' AND source_id=?",
(SOURCE_ID,),
).fetchone()
if grant is None or grant[0] is not None:
raise ValueError("memory_note_source_revoked")
result = SavedMemoryNoteEvidence(
evidence_id=record.evidence_id,
source_id=record.source_id,
revision=record.revision,
imported_at=record.imported_at,
role=record.role,
written_by=record.written_by,
created_at=record.created_at,
verbatim=record.verbatim,
)
# The returned body and reference share one snapshot. Recheck authority
# after that read so a revoked source is not released from a stale view.
notes.require(evidence_id)
return result
except KeyError as error:
raise HTTPException(404, "memory_note_not_found") from error
except ValueError as error:
raise HTTPException(409, str(error)) from error
@router.post("/{evidence_id}/process", response_model=MemoryNoteReceipt)
async def process(evidence_id: str, request: Request) -> MemoryNoteReceipt:
notes = store(request)
File diff suppressed because it is too large. Load diff
@@ -69,8 +69,8 @@ _TYPE_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,63}$")
def safe_error_code(error: BaseException) -> str:
"""Reduce an exception to a storable code, never its message text.
A recognised code (the message *is* the code, as `HyMemoryBridgeError`
and the permission guard raise them) is kept so callers can classify it.
A recognised code (the message *is* a stable domain code) is kept so
callers can classify it.
Everything else collapses to its exception type, which is a fixed
identifier rather than content.
"""
@@ -239,8 +239,8 @@ PERMISSION_REQUIRED_CODES = frozenset(
}
)
#: Model-side codes `process_records` raises *before* it can open the bridge
#: (no active profile, missing credential, secure storage down). They prove
#: Model-side codes the SDK write coordinator raises before it can lease a
#: client (no active profile, missing credential, secure storage down). They prove
#: nothing was sent, so the write parks instead of being lost. Transport codes
#: that can arrive after a request was in flight stay `unknown`.
MODEL_NOT_READY_CODES = frozenset(
@@ -251,9 +251,9 @@ MODEL_NOT_READY_CODES = frozenset(
}
)
#: Deterministic, pre-bridge refusals: the frozen input no longer matches
#: Deterministic, pre-SDK refusals: the frozen input no longer matches
#: current Evidence, or a turn exceeds the model's input window. Both are
#: raised while validating the records, before any bridge or HY call, so
#: raised while validating the records, before any SDK call, so
#: nothing was sent and retrying the same intent changes nothing — a terminal
#: `failed`, not `unknown` (which would wrongly imply a possibly-applied
#: write) and not a retry (which would fail identically).
@@ -316,6 +316,12 @@ LEGACY_MEMORY_WRITE_JOB_COLUMNS = (
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)
EVENT_TIME_WINDOW = timedelta(minutes=10)
#: Sources whose jobs are a person adding a memory by hand ("添加记忆"), not
#: history. History is digested strictly old to new through the event-time
#: window; a note is stamped "now", so inside that window it waited behind the
#: whole unprocessed history. These jobs are claimed ahead of the window,
#: still one at a time per source and in order within their session.
DIRECT_WRITE_SOURCES = frozenset({"one-memory-manual"})
@dataclass(frozen=True)
@@ -358,12 +364,10 @@ class UnknownWriteJob:
`request_id` is the durable queue's idempotency key. It identifies the row
for queue inspection and is the value `DurableWriteQueue.get()` accepts.
It is **not** the HY request id: `MemoryProcessingSupervisor` derives the
HY handle from the task it creates, so an inquiry implementation must
obtain that task-layer handle separately rather than passing this value to
HY. This is discovery only: it does not decide whether HY has the write,
and it deliberately exposes no transition method that could turn
`unknown` back into retryable work.
It is the public SDK request id used for exact outcome inquiry. This is
discovery only: it does not decide whether HY has the write, and it
deliberately exposes no transition method that could turn `unknown` back
into retryable work.
"""
job_id: str
@@ -1759,8 +1763,8 @@ class DurableWriteQueue:
"""Retire an `unknown` job after an inquiry, never by guessing.
`unknown` means HY may already hold the write; the only safe way out is
to ask by request id (`hy_capabilities` measured that the SDK supports
this) and then record what the inquiry found. `found=True` closes it as
to call the public SDK `get_write_outcome(request_id)` and then record
what the inquiry found. `found=True` closes it as
`completed`; `found=False` gives up and marks it `dead`. The state
machine's legal transitions are the authority, so this refuses any
change that `job_state.next_state` would reject — in particular it can
@@ -2258,12 +2262,11 @@ async def execute_job(
admit_hy_invocation: Callable[[str, Callable[[], Awaitable[bool]]], Awaitable[bool]]
| None = None,
) -> None:
"""Run one durable job through the existing authorized processing path.
"""Run one durable job through the SDK-authority processor.
The executor does not call HY directly: `process_records` owns the
permission guard, the runner and governance. `hy_invoked` is recorded
before that call, so a crash mid-write is later reclaimed to `unknown`
instead of being retried blindly.
``process_sdk_job`` owns the one public SDK write. ``hy_invoked`` is
recorded immediately before that call, so a crash mid-write is later
reclaimed to ``unknown`` instead of being retried blindly.
"""
# Every store call here is synchronous SQLite work on the single writer or
# a reader; made from the loop it would block health checks and the API for
@@ -2336,38 +2339,13 @@ async def execute_job(
try:
level = ServiceLevel.REALTIME if request.priority == "live" else ServiceLevel.BACKGROUND
with service_level_context(level):
authority = getattr(supervisor, "sdk_authority", None)
if callable(authority) and authority():
sdk_result = await supervisor.process_sdk_job(
records,
request_id=request.idempotency_key,
before_hy_invocation=mark_hy_invocation,
)
raw_ids = sdk_result.raw_ids
memory_ids = tuple(memory.memory_id for memory in sdk_result.memories)
else:
completed_tasks = (
await supervisor.process_records(
records,
before_hy_invocation=mark_hy_invocation,
source_admitted=True,
)
or ()
)
receipts = []
for task in completed_tasks:
receipt = await in_the_background(supervisor.tasks.sdk_receipt, task.task_id)
if receipt is None:
raise RuntimeError("sdk_write_receipt_missing")
receipts.append(receipt)
raw_ids = tuple(
dict.fromkeys(raw_id for receipt in receipts for raw_id in receipt.raw_ids)
)
memory_ids = tuple(
dict.fromkeys(
memory_id for receipt in receipts for memory_id in receipt.memory_ids
)
)
sdk_result = await supervisor.process_sdk_job(
records,
request_id=request.idempotency_key,
before_hy_invocation=mark_hy_invocation,
)
raw_ids = sdk_result.raw_ids
memory_ids = sdk_result.memory_ids
except Exception as error: # noqa: BLE001 - classified below
# Never persist the raw message: it may be a provider response or the
# user's own text, and `last_error` is shown over HTTP as `detail`.
@@ -2382,7 +2360,7 @@ async def execute_job(
):
await in_the_background(queue.release_unstarted, job_id, owner=owner)
return
# The task layer classifies these before any bridge is opened, so HY
# The write coordinator classifies these before any SDK client is leased, so HY
# was provably not called. They must not read as `failed`:
# - authorization stop -> cancelled (the user turned it off);
# - no permission yet -> waiting_for_permission (parked, re-admitted
@@ -2445,13 +2423,10 @@ async def execute_job(
sdk_raw_ids=raw_ids,
sdk_memory_ids=memory_ids,
)
authority = getattr(supervisor, "sdk_authority", None)
release = getattr(evidence, "release_sdk_staging", None)
durable_receipt = await in_the_background(queue.sdk_receipt, job_id)
if (
callable(authority)
and authority()
and callable(release)
callable(release)
and durable_receipt is not None
and (durable_receipt.raw_ids or durable_receipt.memory_ids)
):
@@ -2632,6 +2607,39 @@ class DurableWriteDrain:
placeholders = ",".join("?" for _ in enabled_sources)
source_clause = f"AND candidate.source_key IN ({placeholders}) "
source_params = tuple(sorted(enabled_sources))
direct_sources = ",".join("?" for _ in DIRECT_WRITE_SOURCES)
# Same eligibility as the window claim below (free lease, no
# running job of its source, nothing earlier in its session), but
# not bound to the window: a note must not wait for history.
direct_rows = connection.execute(
"SELECT candidate.job_id,candidate.source_key "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ f"AND candidate.source_key IN ({direct_sources}) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.queue_seq",
(
PENDING,
now.isoformat(),
*source_params,
*sorted(DIRECT_WRITE_SOURCES),
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchall()
window = connection.execute(
"SELECT window_anchor,window_end FROM memory_write_schedule_state WHERE singleton=1"
).fetchone()
@@ -2685,6 +2693,7 @@ class DurableWriteDrain:
if active is None:
connection.execute("DELETE FROM memory_write_schedule_state WHERE singleton=1")
window = None
anchor: Any = None
if window is None:
anchor = connection.execute(
"SELECT candidate.event_at "
@@ -2713,8 +2722,10 @@ class DurableWriteDrain:
*_FIFO_BLOCKING_STATES,
),
).fetchone()
if anchor is None:
if anchor is None and not direct_rows:
return ()
has_window = window is not None or anchor is not None
if window is None and anchor is not None:
anchor_at = datetime.fromisoformat(str(anchor[0]))
if anchor_at.tzinfo is None:
anchor_at = anchor_at.replace(tzinfo=UTC)
@@ -2726,39 +2737,41 @@ class DurableWriteDrain:
"VALUES (1,?,?,?)",
(anchor_text, end_text, now.isoformat()),
)
rows = connection.execute(
"SELECT candidate.job_id,candidate.source_key "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ "AND candidate.event_at>=? AND candidate.event_at<? "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.event_at,candidate.queue_seq",
(
PENDING,
now.isoformat(),
*source_params,
anchor_text,
end_text,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchall()
rows: list[Any] = []
if has_window:
rows = connection.execute(
"SELECT candidate.job_id,candidate.source_key "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ "AND candidate.event_at>=? AND candidate.event_at<? "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.event_at,candidate.queue_seq",
(
PENDING,
now.isoformat(),
*source_params,
anchor_text,
end_text,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchall()
selected: list[str] = []
selected_sources: set[str] = set()
for row in rows:
for row in (*direct_rows, *rows):
source_key = str(row["source_key"])
if source_key in selected_sources:
continue
@@ -1,182 +0,0 @@
"""A0-03: measured answer to "can a HY write be made idempotent?".
The result decides whether the `unknown` job state can ever be retired, so it
is probed against the installed SDK rather than written down from memory.
What is verified against `hy-memory==2.0.5.dev3`:
- `HyMemoryClient.add(..., request_id=...)` accepts a caller-supplied id and
atomically claims it in the SDK cache before invoking a write. Replays return
the durable terminal result; running/unknown claims are not invoked again.
- `HyMemoryClient.get_write_outcome(request_id)` reads that durable outcome
directly; it does not depend on optional pipeline tracing.
A first pass at this probe got both answers wrong, and the failure mode is
worth keeping in mind: searching SDK source for the string "idempotent" hits a
docstring about *episode* nodes ("calls are idempotent") and a "dedup" comment
about search over-sampling. Neither is write idempotency. So the probe checks
the `add` path specifically and names the inspector class explicitly, instead
of grepping for a word that appears in unrelated prose.
Net: MemHub can query and settle an uncertain request without blind replay.
`unknown` remains a legitimate operational state while the SDK claim itself is
non-terminal, but the same request id cannot write twice.
"""
from __future__ import annotations
import importlib
import inspect
from enum import StrEnum
from typing import Any
class HyIdempotencySupport(StrEnum):
"""Whether the SDK can dedupe a replayed write."""
#: A caller key is honoured and a replay is rejected or folded together.
KEY = "key"
#: Same content is folded by the engine, without an explicit key.
CONTENT_HASH = "content_hash"
#: A replay writes a second time. Retrying after a crash duplicates.
NONE = "none"
#: Never probed. Must not survive A0-03.
UNKNOWN = "unknown"
class HyResultLookupSupport(StrEnum):
"""Whether the outcome of a past write can be queried afterwards."""
BY_REQUEST_ID = "by_request_id"
BY_MEMORY_ID_ONLY = "by_memory_id_only"
NONE = "none"
UNKNOWN = "unknown"
def _module(name: str) -> Any | None:
try:
return importlib.import_module(name)
except ImportError:
return None
def _has_callable(obj: Any, name: str) -> bool:
return obj is not None and callable(getattr(obj, name, None))
def hy_write_accepts_request_id() -> bool:
"""True when the write path carries a caller-supplied request id."""
client_mod = _module("hy_memory.client")
if client_mod is None:
return False
client = getattr(client_mod, "HyMemoryClient", None)
if client is None:
return False
add = getattr(client, "add", None)
if add is None:
return False
return "request_id" in inspect.signature(add).parameters
def _add_path_declares_idempotency() -> bool:
"""Whether the *write* path de-duplicates, judged on its own source.
Deliberately scoped to `add`/`async_add`. A whole-module search matches
unrelated prose: "calls are idempotent" describes episode-node upserts, and
a "dedup" comment describes search over-sampling. Both are about other
features and would make this probe answer `key` for the wrong reason.
"""
client_mod = _module("hy_memory.client")
if client_mod is None:
return False
client = getattr(client_mod, "HyMemoryClient", None)
if client is None:
return False
sources: list[str] = []
for name in ("add", "async_add"):
func = getattr(client, name, None)
if func is None:
continue
try:
sources.append(inspect.getsource(func).lower())
except (OSError, TypeError):
continue
if not sources:
return False
if _has_callable(client, "get_write_outcome"):
claim = getattr(client, "_claim_write_outcome", None)
if claim is not None:
return True
# Older SDKs may expose an explicit key under one of these names.
markers = ("idempotency_key", "idempotent_key", "dedupe_key", "dedup_key")
return any(marker in source for source in sources for marker in markers)
def _inspector_can_lookup_by_request_id() -> bool:
"""True when pipeline logs can be queried by the write's request id."""
inspector = _module("hy_memory.inspector")
if inspector is None:
return False
for cls_name in ("MemoryInspector", "Inspector", "PipelineInspector"):
cls = getattr(inspector, cls_name, None)
if cls is None:
continue
for method in ("get_pipeline_logs", "async_get_pipeline_logs"):
func = getattr(cls, method, None)
if func is None:
continue
# The parameter is `request_id` on `MemoryInspector`; older
# docstrings call the same thing `n`. Accept either so a rename
# does not silently downgrade this to "no lookup available".
if {"request_id", "n"} & set(inspect.signature(func).parameters):
return True
return False
def _probe_idempotency() -> HyIdempotencySupport:
if not hy_write_accepts_request_id():
# No handle at all: not even a correlation id to investigate later.
return HyIdempotencySupport.NONE
if _add_path_declares_idempotency():
return HyIdempotencySupport.KEY
# A request_id is present but nothing de-duplicates on it.
return HyIdempotencySupport.NONE
def _probe_result_lookup() -> HyResultLookupSupport:
client_mod = _module("hy_memory.client")
client = getattr(client_mod, "HyMemoryClient", None) if client_mod is not None else None
if _has_callable(client, "get_write_outcome"):
return HyResultLookupSupport.BY_REQUEST_ID
if _inspector_can_lookup_by_request_id():
return HyResultLookupSupport.BY_REQUEST_ID
if _has_callable(client, "get"):
return HyResultLookupSupport.BY_MEMORY_ID_ONLY
return HyResultLookupSupport.NONE
IDEMPOTENCY_SUPPORT: HyIdempotencySupport = _probe_idempotency()
RESULT_LOOKUP_SUPPORT: HyResultLookupSupport = _probe_result_lookup()
def probe_hy_write_idempotency() -> dict[str, Any]:
"""Full probe report, for STATUS evidence and for debugging a bumped SDK.
Re-probed on every call so a dependency upgrade that changes the answer is
visible, rather than being frozen at import time.
"""
client_mod = _module("hy_memory.client")
client = getattr(client_mod, "HyMemoryClient", None)
add = getattr(client, "add", None)
return {
"sdk_importable": client_mod is not None,
"add_signature": str(inspect.signature(add)) if add is not None else "",
"accepts_request_id": hy_write_accepts_request_id(),
"add_declares_idempotency": _add_path_declares_idempotency(),
"durable_write_outcome": (
client is not None and _has_callable(client, "get_write_outcome")
),
"inspector_lookup_by_request_id": _inspector_can_lookup_by_request_id(),
"idempotency": _probe_idempotency().value,
"result_lookup": _probe_result_lookup().value,
}
@@ -2217,34 +2217,6 @@ CREATE TABLE IF NOT EXISTS retrieval_chunks (
"""
def retrieval_documents_v2_ddl(dialect: str = "mysql") -> str:
"""The v2 documents table under its temporary `_v2` name.
`index_store.migrate_v2` creates this, copies rows from the v1 table, drops
the v1 table and renames. The rename is the only reason a `_v2` name exists
at all, and it is derived from the single definition so the copy-across
cannot drift from what fresh databases get.
"""
documents = next(
statement
for statement in _split_statements(retrieval_index_ddl(dialect))
if "CREATE TABLE IF NOT EXISTS retrieval_documents (" in statement
)
return documents.replace(
"CREATE TABLE IF NOT EXISTS retrieval_documents (",
"CREATE TABLE retrieval_documents_v2 (",
)
def retrieval_build_maps_ddl(dialect: str = "mysql") -> str:
"""The build-maps table added by v2."""
return next(
statement
for statement in _split_statements(retrieval_index_ddl(dialect))
if "retrieval_build_maps" in statement
)
async def _033_retrieval_index(cursor: Any) -> None:
for statement in _split_statements(retrieval_index_ddl(_dialect(cursor))):
await cursor.execute(statement)
@@ -1,7 +1,7 @@
"""A3-01: the HTTP entry to the unified write path.
`POST /api/v1/memory-writes` accepts a write and returns a request id;
`GET /api/v1/memory-writes/{id}` reports its outcome.
`POST /api/v2/memory-writes` accepts OpenAI-style messages and returns a
request id; `GET /api/v2/memory-writes/{id}` reports its outcome.
Two rules shape this endpoint, both consequences of A0-03's measurement that HY
has no idempotency key:
@@ -45,17 +45,6 @@ from one_memory.memory_server.registry import WriteRegistry, WriteState
from one_memory.memory_server.submit import WriteConflictError
class MemoryWriteInput(BaseModel):
model_config = ConfigDict(extra="forbid")
internal_session_id: str = Field(min_length=1, max_length=128)
first_round: int = Field(ge=1)
last_round: int = Field(ge=1)
evidence_ids: list[str] = Field(min_length=1)
priority: Literal["live", "normal"] = "normal"
idempotency_key: str | None = Field(default=None, max_length=256)
class MemoryWriteReceipt(BaseModel):
request_id: str
state: WriteState
@@ -121,67 +110,6 @@ def _registry(
return cast(WriteRegistry | DurableWriteRegistry, registry)
def router(dependencies: list[Any] | None = None) -> APIRouter:
instance = APIRouter(
prefix="/api/v1/memory-writes",
tags=["memory-writes"],
dependencies=dependencies or [],
)
@instance.post("", response_model=MemoryWriteReceipt, status_code=202)
def submit(payload: MemoryWriteInput, request: Request) -> MemoryWriteReceipt:
registry = _registry(request)
batch = BatchIdentity(
internal_session_id=payload.internal_session_id,
first_round=payload.first_round,
last_round=payload.last_round,
)
request_contract = WriteRequest(
internal_session_id=payload.internal_session_id,
batch_identity=batch,
evidence_ids=payload.evidence_ids,
idempotency_key=payload.idempotency_key or batch.batch_id,
priority=payload.priority,
)
try:
record, enqueued = registry.submit(request_contract)
except WriteConflictError as error:
raise HTTPException(409, str(error)) from error
except LookupError as error:
# Evidence-first: the write points at Evidence the business
# database does not hold. Rejecting it here is a caller error, not
# a server fault, and it must not create a queue row.
raise HTTPException(422, str(error)) from error
return MemoryWriteReceipt(
request_id=record.request_id,
state=record.state,
created_at=record.created_at,
updated_at=record.updated_at,
detail="" if enqueued else "already accepted; not enqueued again",
)
@instance.get("/{request_id}", response_model=MemoryWriteReceipt)
def status(request_id: str, request: Request) -> MemoryWriteReceipt:
registry = _registry(request, write=False)
try:
record = registry.get(request_id)
except DurableQueueSchemaUnavailableError as error:
raise HTTPException(503, str(error)) from error
if record is None:
raise HTTPException(404, "unknown request id")
return MemoryWriteReceipt(
request_id=record.request_id,
state=record.state,
created_at=record.created_at,
updated_at=record.updated_at,
detail=record.detail,
raw_ids=record.raw_ids,
memory_ids=record.memory_ids,
)
return instance
def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
instance = APIRouter(
prefix="/api/v2/memory-writes",
@@ -64,7 +64,6 @@ EXEMPT_PATHS = frozenset(
"memory_server/sqlite_semantics.py",
"memory_tasks.py",
"sdk_data_plane_migration.py",
"sdk_migration_inventory.py",
"target_schema.py",
# D93 A: the visibility cache is SQLite-only by construction. It is a
# derived table with triggers on the six tables the read check reads —
@@ -9,9 +9,8 @@ Evidence→queue crash window stayed open.
This adapter is deliberately small. It turns one committed Evidence record
into the one-record complete-turn request, submits it once under an evidence
derived idempotency key, and returns the durable record. The existing
``DurableWriteDrain`` remains the only consumer; it calls
``process_records``, which preserves the complete-turn/debounce/settle rules
in ``MemoryProcessingSupervisor``.
``DurableWriteDrain`` remains the only consumer; it calls the SDK write
coordinator once and persists only the public receipt ids.
"""
from __future__ import annotations
@@ -477,7 +477,7 @@ class MemoryTaskStore:
#: Statuses whose input snapshot nothing reads back afterwards.
#:
#: Deliberately narrow. A `completed` task's inputs are read after the fact:
#: `memory_processing` compares them against the evidence actually sent to
#: SDK write processing compares them against the evidence actually sent to
#: HY to prove coverage, and `matters/discovery` reads them to report how
#: many records HY really saw. Clearing those would silently turn a
#: verification into a pass and a report into a lie, which is far worse
@@ -10,7 +10,6 @@ from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass
from datetime import UTC, datetime
from importlib import import_module
from pathlib import Path
from typing import TypedDict
from uuid import uuid4
@@ -46,14 +45,7 @@ def audit_precipitation_request(request_id: str) -> Iterator[None]:
def _current_prompt_stage() -> str:
try:
resolver = getattr( # noqa: B009 - optional SDK version boundary
import_module("hy_memory.utils.model_call_context"),
"current_model_call_stage",
)
except (ImportError, AttributeError):
return "other"
return str(resolver())
return "other"
class ModelRequestAuditStore:
@@ -1,623 +0,0 @@
"""Read-only process: serves the same read routes as Core without write access.
Opens the database in `mode=ro` with `PRAGMA query_only=ON`; every store
operation here goes through `main_database().read()`, which already enforces
this. No HY SDK, no Chroma, no ONNX runtime, no keychain: the imports that
would pull them in are simply not reached, and the routes below never call
into a write path.
Reads that need HY (semantic search, graph overlay) are delegated to the
Core's internal endpoints; the Core is the only process that owns the
vector stores. Side effects (call events, feedback) are queued and sent
asynchronously, never blocking the read response.
"""
from __future__ import annotations
import asyncio
import builtins
import contextlib
import logging
import os
import sqlite3
import time
from collections.abc import AsyncIterator, Collection, Mapping, Sequence
from contextlib import asynccontextmanager
from datetime import UTC, datetime
from pathlib import Path
from typing import Any, cast
from fastapi import Depends, FastAPI, HTTPException, Query, Request
from pydantic import BaseModel, ConfigDict, SecretStr
from one_memory.database_access import main_database
from one_memory.home_summary import (
HomeSummary,
HomeSummaryUnavailable,
read_home_summary,
)
from one_memory.memory_governance import GovernedMemory, MemoryStatus
from one_memory.memory_library import MemoryLibraryPage, read_memory_library
from one_memory.recall import RecallRequest, _KeywordCandidates
from one_memory.semantic_deadline import SemanticMemoryHit
from one_memory.source_read_access import SourceReadAccess
logger = logging.getLogger(__name__)
MAX_READER_RSS_BYTES = 250 * 1024 * 1024
_PROBE_QUERY = "__om_probe__"
def _check_imports() -> None:
"""Fail fast if any forbidden module is already loaded."""
forbidden = ("hy_memory", "chromadb", "onnxruntime", "keyring")
import sys
loaded = [name for name in forbidden if name in sys.modules]
if loaded:
raise ImportError(f"reader_process_loaded_forbidden_modules: {', '.join(loaded)}")
def _memory_readable_sql() -> str:
return """
m.user_id = ? AND m.status = 'confirmed' AND m.deleted_at IS NULL
AND NOT EXISTS (
SELECT 1 FROM governed_memory_evidence link
JOIN evidence e ON e.evidence_id = link.evidence_id
JOIN source_grants g ON g.user_id=e.user_id AND g.source_id=e.source_id
WHERE link.memory_id=m.memory_id AND +link.active=1
AND g.revoked_at IS NOT NULL
)
AND NOT EXISTS (
SELECT 1 FROM governed_memory_evidence link
LEFT JOIN evidence e ON e.evidence_id = link.evidence_id
WHERE link.memory_id=m.memory_id AND +link.active=1
AND (e.evidence_id IS NULL OR e.user_id != m.user_id)
)
"""
class ReaderSettings(BaseModel):
model_config = ConfigDict(frozen=True)
data_dir: Path
host: str = "127.0.0.1"
port: int = 0 # 0 = ephemeral
api_token: SecretStr
core_port: int
core_api_token: str
class SideEffectForwarder:
"""Queues read-path side effects and forwards them to Core.
Reads in the Reader must stay read-only: the database is opened with
`query_only`, so a call event cannot be written here. Instead each read
that would have recorded one enqueues it, and a background task posts it
to Core's internal endpoint. The response never waits for it; a Core that
is unavailable drops it and counts the drop.
"""
def __init__(self, settings: ReaderSettings) -> None:
self.settings = settings
self.queue: asyncio.Queue[dict[str, object]] = asyncio.Queue(maxsize=256)
self.dropped = 0
self.task: asyncio.Task[None] | None = None
def enqueue(self, event: dict[str, object]) -> None:
try:
self.queue.put_nowait(event)
except asyncio.QueueFull:
self.dropped += 1
def start(self) -> None:
if self.task is None or self.task.done():
self.task = asyncio.ensure_future(self._drain())
def stop(self) -> None:
if self.task is not None and not self.task.done():
self.task.cancel()
async def _drain(self) -> None:
import httpx
while True:
event = await self.queue.get()
try:
async with httpx.AsyncClient(
headers={
"Authorization": f"Bearer {self.settings.core_api_token}",
"Content-Type": "application/json",
},
timeout=5.0,
proxy=None,
) as client:
await client.post(
f"http://127.0.0.1:{self.settings.core_port}/internal/call-event",
json=event,
)
except Exception:
# A dropped side effect is never a failed read: recall already
# answered. Counted so a persistently unavailable Core is
# visible rather than silent.
self.dropped += 1
async def _core_get(
settings: ReaderSettings,
path: str,
timeout_s: float = 5.0,
params: Mapping[
str, str | int | float | bool | None | Sequence[str | int | float | bool | None]
]
| None = None,
) -> Any:
"""Call the Core's internal read endpoint for data only Core can serve.
`params` is passed to httpx rather than interpolated into the path: a
query containing `&`, `#` or a space would otherwise be cut short or
read as a fragment, silently searching for the wrong thing.
"""
import httpx
url = f"http://127.0.0.1:{settings.core_port}{path}"
async with httpx.AsyncClient(
headers={"Authorization": f"Bearer {settings.core_api_token}"},
timeout=timeout_s,
proxy=None,
) as client:
response = await client.get(url, params=params)
response.raise_for_status()
return response.json()
class _ReaderGovernance:
"""Minimal read interface over the governance store for recall."""
def __init__(self, database_path: Path) -> None:
self.database_path = database_path
def list(
self,
*,
user_id: str,
status: MemoryStatus | None = None,
query: str | None = None,
limit: int = 100,
offset: int = 0,
) -> list[GovernedMemory]:
"""One page of memories, read through the Core's own read function.
`MemoryGovernanceStore.list` cannot be used directly: constructing the
store runs schema DDL on the write connection, and the Reader opens
with `query_only`. `read_memory_library` is the same read expressed as
a static SQL read (revoked sources excluded, per-row freshness checked
for the rows it returns, cheap total for the rest), so the Reader and
the Core cannot drift on what "readable" means.
"""
page = read_memory_library(
self.database_path,
user_id=user_id,
query=query,
status=status,
offset=offset,
limit=limit,
)
return list(page.items)
def find_confirmed_matching_terms(
self,
*,
user_id: str,
terms: Collection[str],
limit: int = 200,
scopes: Collection[str] | None = None,
) -> tuple[builtins.list[GovernedMemory], bool]:
if not 1 <= limit <= 2_000:
raise ValueError("invalid candidate limit")
cleaned = tuple(dict.fromkeys(term.strip() for term in terms if term.strip()))
if not cleaned or scopes is not None and not scopes:
return [], False
clauses = ["m.user_id=?", "m.status='confirmed'", "m.deleted_at IS NULL"]
values: builtins.list[object] = [user_id]
matches = []
for term in cleaned[:32]:
matches.append("m.content LIKE ? ESCAPE '\\'")
escaped = term.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
values.append(f"%{escaped}%")
clauses.append(f"({' OR '.join(matches)})")
if scopes is not None:
clauses.append(
"EXISTS (SELECT 1 FROM json_each(m.scopes_json) "
f"WHERE json_each.value IN ({','.join('?' for _ in scopes)}))"
)
values.extend(scopes)
values.append(limit + 1)
with main_database(self.database_path).read(row_factory=sqlite3.Row) as db:
rows = db.execute(
f"SELECT m.* FROM governed_memories m WHERE {' AND '.join(clauses)} "
"ORDER BY m.updated_at DESC,m.memory_id LIMIT ?",
values,
).fetchall()
items = self._memories_from_rows(db, rows[:limit])
return items, len(rows) > limit or len(cleaned) > 32
def get(self, memory_id: str, *, user_id: str) -> GovernedMemory | None:
memory = _get_memory_direct(self.database_path, memory_id, user_id)
return memory if isinstance(memory, GovernedMemory) else None
def sources_behind_memories(
self,
*,
user_id: str,
memory_ids: Collection[str],
) -> dict[str, frozenset[str]]:
if not memory_ids:
return {}
identifiers = tuple(memory_ids)
placeholders = ",".join("?" for _ in identifiers)
with main_database(self.database_path).read(row_factory=sqlite3.Row) as db:
rows = db.execute(
"SELECT link.memory_id,e.source_id FROM governed_memory_evidence link "
"JOIN evidence e ON e.evidence_id=link.evidence_id "
f"WHERE link.active=1 AND e.user_id=? AND link.memory_id IN ({placeholders})",
(user_id, *identifiers),
).fetchall()
found: dict[str, set[str]] = {}
for row in rows:
found.setdefault(str(row["memory_id"]), set()).add(str(row["source_id"]))
return {key: frozenset(values) for key, values in found.items()}
@staticmethod
def _memories_from_rows(
db: sqlite3.Connection, rows: Sequence[sqlite3.Row]
) -> builtins.list[GovernedMemory]:
from one_memory.memory_governance import MemoryGovernanceStore
return MemoryGovernanceStore.memories_from_rows(db, builtins.list(rows))
def _conversation_hit(row: Any) -> dict[str, Any]:
"""One stored row as a search hit.
The author travels with the row. A client that is not told who wrote a row
has to invent an author, and the invention is always "the user", which
files an assistant turn under the person who was talking to it.
"""
return {
"evidence_id": row["evidence_id"],
"conversation_id": row["conversation_id"],
"source_id": row["source_id"],
"workspace": row["workspace"],
"project": row["project"],
"created_at": row["created_at"],
"excerpt": row["verbatim"][:200] if row["verbatim"] else "",
# Both columns are NOT NULL on `evidence`, so a stored row always
# knows; a summary of a whole conversation has no single author and
# never reaches here.
"role": row["role"],
"written_by": row["written_by"],
}
def create_reader_app(settings: ReaderSettings) -> FastAPI:
"""A read-only FastAPI application for the Reader process."""
_check_imports()
database_path = settings.data_dir / "one-memory.sqlite3"
if not database_path.exists():
raise FileNotFoundError(f"reader_database_missing: {database_path}")
# Validate schema version without writing.
from one_memory.memory_server.migrations import SCHEMA_TABLE
from one_memory.schema_gate import BUILD_SCHEMA_VERSION
with main_database(database_path).read() as db:
tables = {
str(row[0]) for row in db.execute("SELECT name FROM sqlite_master WHERE type='table'")
}
if SCHEMA_TABLE in tables:
row = db.execute(f"SELECT MAX(version) FROM {SCHEMA_TABLE}").fetchone()
found = int(row[0]) if row and row[0] is not None else 0
if found > BUILD_SCHEMA_VERSION:
raise RuntimeError(f"reader_schema_too_new: {found} > {BUILD_SCHEMA_VERSION}")
elif not tables:
raise RuntimeError("reader_database_empty")
access = SourceReadAccess(database_path)
forwarder = SideEffectForwarder(settings)
governance = _ReaderGovernance(database_path)
@asynccontextmanager
async def lifespan(_: FastAPI) -> AsyncIterator[None]:
monitor = asyncio.get_running_loop().create_task(_memory_monitor())
forwarder.start()
yield
monitor.cancel()
forwarder.stop()
async def _memory_monitor() -> None:
import resource
import sys
while True:
await asyncio.sleep(30)
usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
rss_bytes = usage if sys.platform == "darwin" else usage * 1024
if rss_bytes > MAX_READER_RSS_BYTES:
logger.error(
"reader_memory_limit_exceeded: %d bytes > %d",
rss_bytes,
MAX_READER_RSS_BYTES,
)
os._exit(1)
app = FastAPI(title="MemHub Reader", lifespan=lifespan)
app.state.settings = settings
app.state.database_path = database_path
expected_token = settings.api_token.get_secret_value()
def require_auth(request: Request) -> None:
import hmac
origin = request.headers.get("origin")
if origin is not None and origin not in (
"one-memory://app",
"http://localhost:5173",
):
raise HTTPException(403, "origin_not_allowed")
authorization = request.headers.get("authorization", "")
scheme, _, token = authorization.partition(" ")
if not (scheme.lower() == "bearer" and hmac.compare_digest(token, expected_token)):
raise HTTPException(401, "invalid_local_token")
protected = [Depends(require_auth)]
@app.get("/api/v1/home-summary", response_model=HomeSummary, dependencies=protected)
def home_summary() -> HomeSummary:
try:
return read_home_summary(database_path)
except HomeSummaryUnavailable:
# Belt and braces: read_home_summary answers stale itself, but if
# that ever changes, never let the homepage 500.
return HomeSummary(
memory_count=0, latest=None, observed_at=datetime.now(UTC), stale=True
)
@app.get("/api/v1/memory-library", response_model=MemoryLibraryPage, dependencies=protected)
def memory_library(
query: str | None = Query(default=None, max_length=512),
status: str | None = None,
active_only: bool = False,
source: str | None = Query(default=None, min_length=1),
offset: int = Query(default=0, ge=0),
limit: int = Query(default=100, ge=1, le=500),
) -> MemoryLibraryPage:
from one_memory.memory_governance import MemoryStatus as MS
ms = MS(status) if status else None
return read_memory_library(
database_path,
user_id="local",
query=query,
status=ms,
active_only=active_only,
source=source,
offset=offset,
limit=limit,
)
@app.get("/api/v1/memories", dependencies=protected)
def list_memories(
status: str | None = None,
query: str | None = None,
limit: int = Query(default=100, ge=1, le=500),
offset: int = Query(default=0, ge=0),
) -> list[dict[str, Any]]:
from one_memory.memory_governance import MemoryStatus as MS
ms = MS(status) if status else None
memories = governance.list(
user_id="local", status=ms, query=query, limit=limit, offset=offset
)
visible = access.memories(memories)
return [m.model_dump(mode="json") for m in visible]
@app.get("/api/v1/memories/graph", dependencies=protected)
async def memory_graph(
limit: int = Query(default=160, ge=1, le=300),
) -> dict[str, Any]:
memories = governance.list(user_id="local", status=MemoryStatus.CONFIRMED, limit=500)
projected = access.memories(memories[:limit])
nodes: dict[str, dict[str, Any]] = {}
edges: list[dict[str, Any]] = []
source_ids: set[str] = set()
with main_database(database_path).read(row_factory=sqlite3.Row) as db:
for memory in projected:
node_id = f"memory:{memory.memory_id}"
nodes[node_id] = {
"node_id": node_id,
"node_type": "memory",
"label": memory.content,
"memory_id": memory.memory_id,
"kind": memory.kind.value
if hasattr(memory.kind, "value")
else str(memory.kind),
"status": memory.status.value
if hasattr(memory.status, "value")
else str(memory.status),
}
evidence_ids = list(memory.evidence_ids)
if evidence_ids:
placeholders = ",".join("?" for _ in evidence_ids)
sql = (
f"SELECT DISTINCT source_id FROM evidence"
f" WHERE evidence_id IN ({placeholders})"
)
for row in db.execute(
sql,
evidence_ids,
):
sid = str(row["source_id"])
source_ids.add(sid)
for sid in sorted(source_ids):
source_node_id = f"source:{sid}"
nodes[source_node_id] = {
"node_id": source_node_id,
"node_type": "source",
"label": sid,
"source_id": sid,
}
for memory in projected:
if any(_source_for_evidence(db, eid) == sid for eid in memory.evidence_ids):
memory_node_id = f"memory:{memory.memory_id}"
edges.append(
{
"edge_id": f"{source_node_id}->{memory_node_id}",
"source_node_id": source_node_id,
"target_node_id": memory_node_id,
}
)
# Fetch HY graph overlay from Core (not available in Reader).
hy_overlay = None
with contextlib.suppress(Exception):
hy_overlay = await _core_get(
settings,
"/internal/graph-snapshot",
timeout_s=5.0,
params={"limit": limit},
)
if hy_overlay and not hy_overlay.get("consulted", False):
hy_overlay = None
return {
"limit": limit,
"total": len(projected),
"nodes": tuple(nodes.values()),
"edges": edges,
**({"hy_overlay": hy_overlay} if hy_overlay else {}),
}
@app.get("/api/v1/memories/{memory_id}", dependencies=protected)
def get_memory(memory_id: str) -> dict[str, Any]:
memory = _get_memory_direct(database_path, memory_id, "local")
if memory is None:
raise HTTPException(404, "memory_not_found")
if not access.memories([memory]):
raise HTTPException(403, "memory_source_revoked")
return cast(dict[str, Any], memory.model_dump(mode="json"))
@app.post("/api/v1/recall", response_model=None, dependencies=protected)
async def recall(request: RecallRequest) -> dict[str, Any]:
from one_memory.recall import GovernedRecallTarget
started = time.monotonic()
target = GovernedRecallTarget(
governance=governance,
semantic_search=None,
resolve_workspace=None,
filter_memories=access.memories,
)
if request.query == _PROBE_QUERY:
return target.recall(request, user_id="local").model_dump(mode="json")
deadline = time.monotonic() + request.timeout_ms / 1_000
semantic_ids: tuple[str, ...] = ()
try:
remaining = max(0.1, deadline - time.monotonic())
result = await asyncio.wait_for(
_core_get(
settings,
"/internal/semantic-ids",
timeout_s=remaining,
params={"query": request.query, "limit": request.limit},
),
timeout=remaining,
)
semantic_ids = tuple(str(value) for value in result.get("ids", []) if value)
except Exception:
pass
readable = target._readable_scopes(request)
# Keyword candidates from read-only SQL.
keywords = target._keyword_candidates(
request, user_id="local", readable=readable, deadline=deadline
)
receipt = target._rank(
request,
user_id="local",
readable=readable,
semantic_hits={
memory_id: SemanticMemoryHit(
memory_id,
1.0 - index / max(1, len(semantic_ids)),
)
for index, memory_id in enumerate(semantic_ids)
},
keywords=keywords
if isinstance(keywords, _KeywordCandidates)
else _KeywordCandidates((), False, False),
deadline=deadline,
timed_out=False,
)
forwarder.enqueue(
{
"tool": "om_recall",
"query": request.query,
"scope": request.scope,
"state": receipt.state.value,
"memory_count": len(receipt.hits),
"elapsed_ms": (time.monotonic() - started) * 1000,
}
)
return receipt.model_dump(mode="json")
@app.get("/api/v1/conversations/search", dependencies=protected)
def search_conversations(
query: str = Query(min_length=1, max_length=512),
limit: int = Query(default=20, ge=1, le=100),
) -> dict[str, Any]:
if not database_path.exists():
raise HTTPException(503, "conversation_search_unavailable")
with main_database(database_path).read(row_factory=sqlite3.Row) as db:
try:
rows = db.execute(
"""SELECT * FROM evidence
WHERE user_id = 'local' AND verbatim LIKE ? ESCAPE '\\'
ORDER BY imported_at DESC LIMIT ?""",
(f"%{query}%", limit),
).fetchall()
except Exception:
raise HTTPException(503, "conversation_search_unavailable") from None
hits = [_conversation_hit(row) for row in rows]
return {
"hits": hits,
"empty_reason": None if hits else "no_matches",
"imported_messages": len(hits),
}
@app.get("/health")
def health() -> dict[str, str]:
return {"status": "ok", "service": "one-memory-reader"}
return app
def _get_memory_direct(database_path: Path, memory_id: str, user_id: str) -> Any:
from one_memory.memory_governance import _memory_from_row, _memory_links
with main_database(database_path).read(row_factory=sqlite3.Row) as db:
row = db.execute(
"SELECT * FROM governed_memories WHERE memory_id = ? AND user_id = ?",
(memory_id, user_id),
).fetchone()
if row is None:
return None
links = _memory_links(db, [memory_id])
return _memory_from_row(row, *(tuple(vals) for vals in links[memory_id]))
def _source_for_evidence(db: sqlite3.Connection, evidence_id: str) -> str | None:
row = db.execute(
"SELECT source_id FROM evidence WHERE evidence_id = ?", (evidence_id,)
).fetchone()
return str(row["source_id"]) if row else None
+11 -2
View File
@@ -12,7 +12,7 @@ from typing import Any, Literal, Protocol
from pydantic import BaseModel, ConfigDict, Field
from one_memory.hy_memory_bridge import HySdkMemoryRecord
from one_memory.hy_sdk_types import HySdkMemoryRecord
from one_memory.memory_governance import GovernedMemory, MemoryStatus
from one_memory.semantic_deadline import (
SemanticMemoryHit,
@@ -81,7 +81,11 @@ class RecallHit(BaseModel):
score: float = Field(ge=0, le=1)
#: HY already rendered memory time/evolution/capsule context into content.
#: Hosts must not append their own governance timestamp in this case.
content_format: Literal["plain", "hy-sdk"] = "plain"
#: "raw" is an excerpt of an imported conversation, not a memory.
content_format: Literal["plain", "hy-sdk", "raw"] = "plain"
#: Raw excerpts only: the app and conversation the excerpt came from.
source: str | None = None
session_id: str | None = None
class RecallReceipt(BaseModel):
@@ -103,6 +107,11 @@ class RecallReceipt(BaseModel):
#: whether the answer is that nothing was found or that reading was not
#: allowed.
excluded_sources: tuple[str, ...] = ()
#: "raw" when recall searched the imported conversations instead of
#: memories, because no generation model is configured or no memory
#: exists yet (`fallback_reason`). Raw hits are excerpts, not memories.
mode: Literal["memory", "raw"] = "memory"
fallback_reason: Literal["model_unavailable", "no_memories"] | None = None
class RecallTarget(Protocol):
@@ -0,0 +1,149 @@
"""Recall that degrades to the original conversations instead of answering empty.
MemHub_recall searches the memories the SDK distilled from conversations. That
needs a configured generation model to have produced any, and a library that
holds some. Without either, the memory search can only come back empty, so
recall searches the SDK's raw layer (the imported conversations themselves)
instead: plain retrieval over what was imported, the same lane
`/api/v2/context/search` reads, under the same raw-source authority.
"""
from __future__ import annotations
import asyncio
import time
from collections.abc import Awaitable, Callable
from datetime import UTC, datetime
from typing import Literal
from one_memory.hy_sdk_types import HySdkRawHit
from one_memory.recall import (
PROBE_QUERY,
RecallHit,
RecallReceipt,
RecallRequest,
RecallState,
TruncationBudget,
)
FallbackReason = Literal["model_unavailable", "no_memories"]
#: Raw sources this caller may read, or None for every source (a trusted local
#: caller). An empty set means the caller may read no raw conversation.
RawSources = frozenset[str] | None
async def recall_with_fallback(
request: RecallRequest,
*,
model_ready: Callable[[], Awaitable[bool]],
recall_memories: Callable[[RecallRequest], Awaitable[RecallReceipt]],
has_memories: Callable[[], Awaitable[bool]],
raw_sources: Callable[[], Awaitable[RawSources]],
search_raw: Callable[[str, int, RawSources], Awaitable[tuple[HySdkRawHit, ...]]],
) -> RecallReceipt:
"""Memory recall when a model and memories exist, otherwise the raw layer.
The library check runs only after an empty memory search: with memories
present and nothing relevant, the answer stays an honest empty memory
result rather than a different kind of hit.
"""
if request.query == PROBE_QUERY:
return await recall_memories(request)
started = time.monotonic()
if await model_ready():
receipt = await recall_memories(request)
if receipt.hits or await has_memories():
return receipt
reason: FallbackReason = "no_memories"
else:
reason = "model_unavailable"
remaining = request.timeout_ms / 1_000 - (time.monotonic() - started)
return await _raw_recall(
request,
reason=reason,
remaining_seconds=remaining,
raw_sources=raw_sources,
search_raw=search_raw,
)
async def _raw_recall(
request: RecallRequest,
*,
reason: FallbackReason,
remaining_seconds: float,
raw_sources: Callable[[], Awaitable[RawSources]],
search_raw: Callable[[str, int, RawSources], Awaitable[tuple[HySdkRawHit, ...]]],
) -> RecallReceipt:
def receipt(
hits: tuple[RecallHit, ...] = (),
*,
used: int = 0,
truncated_by: TruncationBudget | None = None,
unexamined: int = 0,
) -> RecallReceipt:
return RecallReceipt(
state=RecallState.GROUNDED if hits else RecallState.EMPTY,
query=request.query,
scope=request.scope,
hits=hits,
used_characters=used,
truncated=truncated_by is not None,
truncated_by=truncated_by,
unexamined_count=unexamined,
mode="raw",
fallback_reason=reason,
)
sources = await raw_sources()
if sources is not None and not sources:
# The caller may read no original conversation: an empty answer, not
# a wider grant than recall had.
return receipt()
if remaining_seconds <= 0:
return receipt(truncated_by=TruncationBudget.TIMEOUT)
try:
found = await asyncio.wait_for(
search_raw(request.query, request.limit, sources),
timeout=remaining_seconds,
)
except TimeoutError:
return receipt(truncated_by=TruncationBudget.TIMEOUT)
hits: list[RecallHit] = []
used = 0
truncated_by: TruncationBudget | None = None
for index, hit in enumerate(found):
if sources is not None and hit.source not in sources:
continue
if len(hits) >= request.limit:
truncated_by = TruncationBudget.COUNT
return receipt(
tuple(hits),
used=used,
truncated_by=truncated_by,
unexamined=len(found) - index,
)
if used + len(hit.content) > request.character_budget:
truncated_by = TruncationBudget.CHARACTER
continue
hits.append(
RecallHit(
memory_id=hit.raw_id,
content=hit.content,
kind="raw",
scopes=(),
evidence_ids=(),
updated_at=(
datetime.fromtimestamp(hit.memory_at, tz=UTC)
if hit.memory_at is not None
else datetime(1970, 1, 1, tzinfo=UTC)
),
score=hit.score,
content_format="raw",
source=hit.source,
session_id=hit.session_id,
)
)
used += len(hit.content)
return receipt(tuple(hits), used=used, truncated_by=truncated_by)
@@ -22,6 +22,13 @@ class RetiredEvidenceStore:
del limit
return 0
def release_sdk_staging(self, evidence_ids: tuple[str, ...]) -> int:
# The write drain releases staged Evidence after every SDK write. On
# SDK authority nothing is staged; without this, `__getattr__` raised
# past `getattr(..., None)` and every completed write logged an error.
del evidence_ids
return 0
def get(self, _identifier: str) -> None:
return None
@@ -1,167 +0,0 @@
"""Single-Evidence identity chunking; no reads, persistence, normalization or grants.
Callers must first establish eligibility/sanitization/permission. This module
does not strip wrappers or secrets and must not be used as a content safety gate.
"""
from dataclasses import dataclass
from typing import Protocol
from one_memory.embedding_assets import BUILTIN_EMBEDDING_MAX_TOKENS
from one_memory.local_embedding import TokenizedText, TokenOffset
class OffsetTokenizer(Protocol):
def tokenize_with_offsets(self, text: str) -> TokenizedText: ...
class ChunkingError(ValueError):
"""Invalid offsets, uncovered text or insufficient encoding budget; never truncate."""
@dataclass(frozen=True)
class TextChunk:
ordinal: int
start_cp: int
end_cp: int
token_count: int
text: str
@dataclass(frozen=True)
class _Span:
start: int
end: int
tokens: int
def _spans(text: str, encoded: TokenizedText) -> tuple[_Span, ...]:
if not isinstance(encoded, TokenizedText) or len(encoded.token_ids) != len(encoded.offsets):
raise ChunkingError("chunk_token_offset_count_mismatch")
spans: list[_Span] = []
previous_start = previous_end = 0
for token_id, offset in zip(encoded.token_ids, encoded.offsets, strict=True):
if type(token_id) is not int or token_id < 0 or not isinstance(offset, TokenOffset):
raise ChunkingError("chunk_invalid_token_offset")
if (
type(offset.start_cp) is not int
or type(offset.end_cp) is not int
or type(offset.special) is not bool
):
raise ChunkingError("chunk_invalid_token_offset")
start, end = offset.start_cp, offset.end_cp
if offset.special:
if start != 0 or end != 0:
raise ChunkingError("chunk_invalid_special_offset")
continue
if not 0 <= start < end <= len(text) or start < previous_start or end < previous_end:
raise ChunkingError("chunk_invalid_offset_order_or_bounds")
if start > previous_end and not text[previous_end:start].isspace():
raise ChunkingError("chunk_uncovered_nonwhitespace")
# Overlapping subpieces (including several pieces for one code point)
# form an indivisible interval; slicing inside it loses token identity.
if spans and start < spans[-1].end:
prior = spans[-1]
spans[-1] = _Span(prior.start, end, prior.tokens + 1)
else:
spans.append(_Span(start, end, 1))
previous_start, previous_end = start, end
if text[previous_end:] and not text[previous_end:].isspace():
raise ChunkingError("chunk_uncovered_nonwhitespace")
return tuple(spans)
def _encoding(text: str, tokenizer: OffsetTokenizer) -> tuple[TokenizedText, tuple[_Span, ...]]:
try:
result = tokenizer.tokenize_with_offsets(text)
except Exception as error:
raise ChunkingError("chunk_tokenization_failed") from error
return result, _spans(text, result)
def chunk_evidence(
text: str,
tokenizer: OffsetTokenizer,
*,
prefix: str = "",
target_tokens: int = 384,
overlap_tokens: int = 64,
max_tokens: int = BUILTIN_EMBEDDING_MAX_TOKENS,
) -> tuple[TextChunk, ...]:
"""Chunk one original with code point [start,end), preserving exact slices.
target/overlap count original non-special tokens. Overlap is up to the
requested token count at indivisible offset boundaries; forward progress is
mandatory. token_count is re-tokenized prefix + chunk.text INCLUDING specials,
never an additive estimate. Prefix is caller-provided, unchanged, and omitted
from chunk.text/offsets. Empty/whitespace-only inputs have no index chunks.
"""
if not isinstance(text, str) or not isinstance(prefix, str):
raise ChunkingError("chunk_requires_text")
if (
any(type(value) is not int for value in (target_tokens, overlap_tokens, max_tokens))
or not 0 <= overlap_tokens < target_tokens
or not 1 <= max_tokens <= BUILTIN_EMBEDDING_MAX_TOKENS
or target_tokens > max_tokens
):
raise ChunkingError("chunk_invalid_token_budget")
if not text or text.isspace():
return ()
_, spans = _encoding(text, tokenizer)
if not spans:
raise ChunkingError("chunk_uncovered_nonwhitespace")
chunks: list[TextChunk] = []
start_index = 0
previous_end_cp = 0
while start_index < len(spans):
end_index = start_index
body_tokens = 0
while end_index < len(spans):
candidate = body_tokens + spans[end_index].tokens
if end_index > start_index and candidate > target_tokens:
break
body_tokens = candidate
end_index += 1
# Prefer the last paragraph boundary in the latter half of the target.
# Never trade away an entire overlap window or split a token interval.
if end_index < len(spans):
count = 0
preferred = None
for boundary in range(start_index + 1, end_index + 1):
count += spans[boundary - 1].tokens
if count >= max(target_tokens // 2, overlap_tokens + 1):
left, right = spans[boundary - 1].end, spans[boundary].start
if "\n" in text[max(spans[boundary - 1].start, left - 1) : right]:
preferred = boundary
if preferred is not None:
end_index = preferred
start_cp = 0 if start_index == 0 else spans[start_index].start
while end_index > start_index:
# Include gaps verbatim, including whitespace and combining marks;
# source coverage is never reconstructed by joining decoded tokens.
end_cp = spans[end_index].start if end_index < len(spans) else len(text)
encoded, _ = _encoding(prefix + text[start_cp:end_cp], tokenizer)
token_count = len(encoded.token_ids)
if token_count <= max_tokens:
break
end_index -= 1
else:
raise ChunkingError("chunk_indivisible_span_exceeds_token_budget")
if end_cp <= previous_end_cp or start_cp > previous_end_cp:
raise ChunkingError("chunk_coverage_cannot_progress")
chunks.append(TextChunk(len(chunks), start_cp, end_cp, token_count, text[start_cp:end_cp]))
previous_end_cp = end_cp
if end_index == len(spans):
break
next_start = end_index
overlap = 0
while next_start > start_index + 1:
candidate = overlap + spans[next_start - 1].tokens
if candidate > overlap_tokens:
break
next_start -= 1
overlap = candidate
start_index = next_start
if previous_end_cp != len(text):
raise ChunkingError("chunk_incomplete_coverage")
return tuple(chunks)
Loaded 100 of 456 files, more files were not shown because too many files have changed in this diff. Show more