Compare commits

...
Author SHA1 Message Date
alvinfeiandClaude Opus 5.5 b1b3e0a291 fix(sdk): drop the user-turn filter for basic_info
Owner decision on #248: the writer no longer filters basic_info against
the user turns. The placeholder guard (with healing on read) and the
prompt hard rules stay; the guard alone keeps an assistant quoting
"name is 用户" out of the profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 08:18:32 +08:00
alvinfeiandClaude Opus 5.5 8895fea775 release: hy-memory 2.0.9
Placeholder values no longer overwrite the user's basic profile (L0). Not
yet published to PyPI.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 04:56:01 +08:00
alvinfeiandClaude Opus 5.5 78882f3300 fix(sdk): placeholder values never overwrite the basic profile
A weak extraction model filled basic_info with "用户", "The user" or "N/A"
when the user stated nothing, and each placeholder superseded the real L0
value. upsert_basic_profile now drops placeholder values and writes no new
version when nothing is left; profile assembly skips stored placeholders.
The writer keeps a basic_info field only when its value appears in a user
message of the request. The zh and en extraction prompts add a hard rule
and a small-talk counter-example.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 04:56:01 +08:00
feitianxiang 988667cd87 Merge pull request #224 from Tencent-Hunyuan/chore/build-36-docs
chore(release): bump MemHub build number to 36 for v1.0.5
2026-10-28 22:07:22 +08:00
alvinfeiandClaude Opus 5.5 72968f831f chore(release): bump MemHub build number to 36
The macOS v1.0.5 artifacts were built from release/memhub-build-36-main
(dbba403): main's code at a621262 with build 36. Main moves to the same
build so the Linux Runtimes built from the v1.0.5 tag match them. The
release docs name build 36.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 22:02:27 +08:00
feitianxiang af25f51f9e Merge pull request #223 from Tencent-Hunyuan/docs/release-1.0.5
docs: prepare MemHub v1.0.5 (news, build 35, macOS arm64 only)
2026-10-28 21:59:59 +08:00
alvinfeiandClaude Opus 5.5 b3f13e130f docs: MemHub 1.0.5 news, build 35, macOS arm64 only
The README news line announces v1.0.5 briefly; details live in the release
notes. The release facts said build 14 and promised a macOS Intel build that
this version does not ship. CHANGELOG's Unreleased section becomes MemHub
1.0.5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 21:58:28 +08:00
kristinhe1111 a621262fac Merge pull request #222 from Tencent-Hunyuan/fix/memory-detail-structure-and-stability
fix(memhub): stabilize memory detail navigation
2026-10-28 20:44:13 +08:00
kristinhe e027635d3f Merge remote-tracking branch 'origin/main' into fix/memory-detail-structure-and-stability
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	memhub/apps/web/src/MemoryMap.test.tsx
2026-10-28 20:36:06 +08:00
feitianxiang 4cb33b7108 Merge pull request #217 from Tencent-Hunyuan/fix/memory-source-filter-sync
fix(memhub): 同步记忆筛选范围并优化首次启动状态
2026-10-28 20:32:24 +08:00
alvinfeiandClaude Opus 5.5 b50c9f33eb fix(memhub): give the map its own graph-loading state
The map hid the previous scope's graph by passing stale into the library
view, which also blanked the list status, pagination and empty states.
A separate graphLoading flag now hides the old graph and shows
"正在读取记忆地图…" while stale keeps its original meaning and text.
The client-side revoked-source graph filter is dropped to match Core.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:24:39 +08:00
alvinfeiandClaude Opus 5.5 545b7d38bf fix(memhub): keep revoked-source memories on the memory map
Memories already distilled from a revoked source stay visible, so the
graph endpoint no longer checks revocation before or after the SDK read,
no longer filters memories by revoked source and reports the SDK total
again. The map and the list now agree. The scene="all" and
statuses=["active"] scope stays.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:24:39 +08:00
feitianxiang 5f78d67363 Merge pull request #221 from Tencent-Hunyuan/fix/linux-lifecycle-queue-check
fix(memhub): Linux lifecycle checks the write queue, not recall without a model
2026-10-28 20:17:45 +08:00
alvinfei 33d7713d4e Merge remote-tracking branch 'github/main' into fix/memory-source-filter-sync 2026-10-28 20:14:52 +08:00
kristinheandCursor 68a4b546f9 fix(memhub): stabilize memory detail navigation
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 20:11:12 +08:00
kristinhe cbcc50f35e Merge remote-tracking branch 'origin/main' into fix/memory-source-filter-sync
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	memhub/apps/web/src/MemorySpace.tsx
2026-10-28 20:09:35 +08:00
alvinfeiandClaude Opus 5.5 528d2c9fd7 fix(memhub): Linux lifecycle checks the write queue, not recall without a model
The lifecycle verification imported a conversation on a runner with no
generation model and then expected recall to find it. Under SDK authority
nothing reaches the SDK, not even its raw layer, until a model is active,
so the check could never pass and blocked the Linux release archive.

It now asserts what the runner can prove: the imported and appended turns
reach Core's durable write queue, and survive an idempotent upgrade.
docs/TODO-raw-recall-without-model.md records the product gap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:05:44 +08:00
kristinhe1111 0eb9a2844e Merge pull request #216 from Tencent-Hunyuan/fix/onboarding-language-menu-clipping
fix(memhub): keep option menus inside viewport
2026-10-28 20:01:37 +08:00
feitianxiang 8e56bf7a37 Merge pull request #218 from Tencent-Hunyuan/fix/sdk-client-race-and-unknown-jobs
fix(memhub): serialize SDK client startup, log failure frames, settle pre-upgrade unknown writes
2026-10-28 19:49:24 +08:00
alvinfeiandClaude Opus 5.5 352fffb3da fix(memhub): serialize SDK client startup, log failure frames, settle pre-upgrade unknown writes
- hy_sdk_runtime: construct one HyMemoryClient at a time per home. Two
  first-time Chroma opens on a fresh vector dir race Chroma's unlocked
  System cache; on a clean install the drain's and the UI's clients both
  failed and the first write job was lost as unknown.
- sdk_write_processor: a client that fails to construct raises
  hy_memory_client_unavailable before HY is invoked, which parks the job
  (waiting_for_model) instead of leaving it unknown.
- sdk_write_coordinator: a retired profile's handle closes only after its
  last in-flight use (defensive; a profile revision change used to close
  the client under a running write).
- durable_queue: durable_write_job_failed/parked log the exception chain as
  file:line:function frames, never the message.
- sdk_migration_runner: the cutover settles pre-cutover unknown (and
  invoked running) write jobs as cancelled/abandoned_at_cutover and counts
  them as abandoned.unknown_write_jobs; never retried.
- durable_queue/api: retry-unknown and the reconcile helpers no longer
  assume the dropped legacy task tables; the endpoint uses the startup
  queue so it does not recreate memory_write_job_evidence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:39:13 +08:00
kristinheandCursor 5103492762 fix(memhub): harden memory overview recovery
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 19:26:01 +08:00
kristinheandCursor 082aeee1f2 fix(memhub): align graph scope and startup states
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 19:16:48 +08:00
kristinheandCursor 201901d35e fix(memhub): keep option menus inside viewport
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 19:04:11 +08:00
feitianxiang 42c0ca1d14 Merge pull request #215 from Tencent-Hunyuan/feat/memhub-sdk-logs-quiet
feat(memhub): HY pipeline log off, SDK text logs only in hy_memory.log (hy-memory 2.0.8)
2026-10-28 15:46:40 +08:00
alvinfeiandClaude Opus 5.5 0ca2e56ccc chore(release): bump MemHub build number to 35
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:38:05 +08:00
alvinfeiandClaude Opus 5.5 f328cdbf8e docs: record hy-memory 2.0.8 as published
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:38:05 +08:00
alvinfeiandClaude Opus 5.5 3f57a3c978 build(memhub): lock hy-memory 2.0.8 from PyPI
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:37:51 +08:00
alvinfeiandClaude Opus 5.5 19a7af0e71 feat(memhub): remove the HY pipeline section from the Debug panel
MemHub no longer writes the HY pipeline JSONL, so the Desktop diagnostics
reader stops scanning hy-memory/governed-v2/logs and the renderer drops the
"hy-memory" service and "pipeline" source. A test covers leftover pipeline
files and hy_memory.log staying out of the panel.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:36:50 +08:00
alvinfeiandClaude Opus 5.5 5791c0d1e7 feat(memhub): HY pipeline log off, SDK text logs only in hy_memory.log
- MEMORY_PIPELINE_LOG_ENABLED and MEMORY_PIPELINE_TRACE_ENABLED are false
  at every diagnostic level, both at startup and on a live level change.
  MEMORY_LOG_LLM_CONTENT keeps its debug-only setting: it still decides
  whether HY's DEBUG lines in hy_memory.log carry full prompts/responses.
- After each HyMemoryClient construction (success or failure), Core drops
  the SDK's stdout/stderr StreamHandlers, which under launchd duplicated
  every SDK line into service.log/service-error.log, and forwards SDK
  WARNING+ to Core's core.log handler so the Debug panel still shows SDK
  problems.
- A live level change also sets the level of HY's hy_memory.log handler,
  which the SDK otherwise fixes at the level of the first client.
- log_file.py no longer claims core.log covers the HY SDK at every level.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:36:50 +08:00
alvinfeiandClaude Opus 5.5 bf92105210 build(memhub): pin hy-memory 2.0.8
2.0.8 adds MEMORY_PIPELINE_LOG_ENABLED and keeps conversation text out of
INFO logs. apps/core/uv.lock is not updated: 2.0.8 is not on PyPI yet, so
the lock is regenerated (uv lock --directory apps/core) once it is
published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:36:50 +08:00
feitianxiang 3387247842 Merge pull request #214 from Tencent-Hunyuan/feat/sdk-pipeline-log-switch
feat(sdk): switch to turn the pipeline log off, keep conversation text out of INFO logs (2.0.8)
2026-10-28 15:34:41 +08:00
alvinfeiandClaude Opus 5.5 791285f4c9 fix(sdk): keep model output out of the reconcile parse-failure warning
The final RECONCILE JSON parse failure warning embedded the raw response
whenever MEMORY_LOG_LLM_CONTENT was on. It now reports the length; the raw
output is already logged per attempt at DEBUG.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:06:09 +08:00
alvinfeiandClaude Opus 5.5 dc762df78b feat(sdk): switch to turn the pipeline log off, keep conversation text out of INFO logs (2.0.8)
MEMORY_PIPELINE_LOG_ENABLED (default true) turns off the pipeline JSONL
log. With the DB trace also off, store_pipeline_log becomes a no-op that
does no timing, redaction, background work or database write. The init line
states the real on/off state.

Lines that carried conversation or memory text at INFO/WARNING now log ids,
counts and lengths; the text goes to DEBUG: [add] input, search queries and
rewrites, coding tasks and fact statements/subjects, reconcile op detail,
extracted info, embedding inputs and unparsable model output.

Bump hy-memory to 2.0.8 (not yet published).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:03:21 +08:00
feitianxiang 8277a72aea Merge pull request #206 from rahul05ranjan/fix/runtime-stale-lease-boot-clean
fix(runtime): 清理早于本次开机的服务锁,异常关机后不再崩溃循环
2026-10-28 14:42:46 +08:00
feitianxiang 5eed0f81fb Merge pull request #200 from u7-u7/fix/literal-special-token-counting
fix(tokens): count literal special-token spellings as ordinary text
2026-10-28 14:36:16 +08:00
feitianxiang 59ca5afca6 Merge pull request #199 from u7-u7/fix/guarded-thread-context
fix(async): preserve context in guarded thread calls
2026-10-28 14:36:11 +08:00
feitianxiang 2648012ad5 Merge pull request #203 from u7-u7/fix/migration-log-write-order
fix(memhub): preserve migration log write order
2026-10-28 14:36:05 +08:00
alvinfeiandClaude Opus 5.5 63a80092c2 fix(runtime): identify stale leases by boot id and publish them atomically
Comparing the lock directory's mtime with Date.now() - os.uptime() breaks
when the wall clock jumps forward after a lease is taken (no RTC, VM
resume, NTP or manual correction): a live lease looks older than the boot
and is cleared, so a second Runtime can open the same data dir.

owner.json now records the kernel boot identity (kern.bootsessionuuid on
macOS, /proc/sys/kernel/random/boot_id on Linux, null elsewhere), read
once per process. removeStaleLease clears a lease whose boot_id differs
from the current one, which still fixes the recycled-PID crash loop of
#184 without consulting the clock; when either id is unknown the PID
probe decides as before. bootTime, BOOT_SLACK_MS and defaultBootTime are
gone; the boot id source is injectable via RuntimeServiceOptions.bootId.

The lease is published in one step: owner.json is written into a
runtime-service.lock.tmp-<uuid> staging dir that is then renamed onto the
lock, so a power cut can no longer leave a lock without an owner. Because
rename() replaces an existing empty directory, an existing lock of any
kind is reported as taken first. releaseLease also removes the lock with
a single rename, so this version never shows an owner-less lock; staging
and cleared-lease debris is swept once the lease is held.

An owner-less lock can therefore only come from an older version. It is
cleared when its mtime predates the kernel-reported boot time
(kern.boottime, /proc/stat btime) and otherwise keeps failing as
runtime_service_lock_invalid, as before.

Refs #184

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 14:33:33 +08:00
feitianxiang 6ba49bf112 Merge pull request #209 from Tencent-Hunyuan/fix/memory-map-node-navigation
fix(memhub): open source memories from map nodes
2026-10-28 14:10:28 +08:00
feitianxiang cf410aa9cf Merge pull request #210 from Tencent-Hunyuan/fix/harness-refresh-on-upgrade
fix(memhub): refresh harness skills and MCP entry after an upgrade
2026-10-28 14:08:01 +08:00
kristinheandCursor 81ed1d816f fix(memhub): open source memories from map nodes
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 12:00:26 +08:00
alvinfeiandClaude Opus 5.5 30c0b1d5ff fix(memhub): refresh harness skills and MCP entry after an upgrade
After an upgrade, hosts kept the call-rule skill and MCP entry an older
release wrote, so models were still told to call MemHub_ask and the other
removed tools, and WorkBuddy kept listing them from its tool-list cache.

- Stamp every managed MCP entry with ONE_MEMORY_MCP_TOOLSET, a tool
  contract version kept beside the tool list (mcp-toolset.ts). WorkBuddy
  fingerprints the stdio transport, env included, so a new stamp makes it
  restart the server and drop the cached list. An entry without the
  current stamp reads as stale in skill drift; verify still passes it.
- HostPermissions.refreshInstalledHosts runs the existing repoint and
  stale-sync planners for every host this data dir installed, leaves
  user-edited skills alone, emits allowlisted host_refresh.* diagnostics
  (host ids and counts only), and records the build it completed in
  host-permissions/host-refresh.json so it runs once per build.
- The Runtime calls it as it starts (covering App and `memhub upgrade`),
  and Desktop calls it after an embedded Runtime is up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:58:21 +08:00
feitianxiang 1086f4d582 Merge pull request #208 from Tencent-Hunyuan/feat/memhub-experience-memory-at
fix(memhub): experiences show their conversation time (hy-memory 2.0.7)
2026-10-28 11:50:41 +08:00
alvinfeiandClaude Opus 5.5 32539e7a82 chore(release): bump MemHub build number to 34
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:44:42 +08:00
alvinfeiandClaude Opus 5.5 cfae020a31 test(memhub): install docs name hy-memory 2.0.7
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:44:42 +08:00
alvinfeiandClaude Opus 5.5 5d51513ec8 docs: record hy-memory 2.0.7 as published
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:37:30 +08:00
alvinfeiandClaude Opus 5.5 9db5b49ccb build(memhub): lock hy-memory 2.0.7 from PyPI
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:37:23 +08:00
alvinfeiandClaude Opus 5.5 cb7df2a022 fix(memhub): metadata repair adds the SDK's memory_at column itself
The repair wrote memory_at only when the coding store already had the
column, which depended on the cutover readback having opened the store
through a 2.0.7 client first. On a migrated 1.0.4 home it now brings each
coding.sqlite3 / coding_cv.sqlite3 to the installed SDK's schema before
repairing, by running CodingMemoryStore's own SQLite schema step (its
initialize() would also open Chroma, which must stay closed offline). The
SDK's DDL is not repeated here.

The test now starts from a 2.0.6-shaped store without the column and
expects the bundle's conversation time in it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:37:13 +08:00
alvinfeiandClaude Opus 5.5 812d1c3846 build(memhub): pin hy-memory 2.0.7
2.0.7 stores and returns the conversation time of working/coding
experiences. apps/core/uv.lock is not updated: 2.0.7 is not on PyPI yet,
so the lock is regenerated (uv lock --directory apps/core) once it is
published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:36:56 +08:00
alvinfeiandClaude Opus 5.5 c8b660651e fix(memhub): experiences show their conversation time
After the SDK upgrade 124 working/coding experiences read 时间未知: the SDK
never stored their conversation time. hy-memory 2.0.7 stores memory_at for
experiences and facts; Core now maps it from list, get and search, so the
memory library, detail, map list and home summary carry it.

An experience stored before 2.0.7 has none. The list labels its write time
写入于 and the detail 写入时间, instead of 时间未知 or calling it an import.

Metadata repair also writes the conversation time recovered from the
migration bundle into coding_memory_meta / working_fact when the store has
the memory_at column. The repair version is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:36:56 +08:00
feitianxiang bdd2d9cf84 Merge pull request #207 from Tencent-Hunyuan/feat/sdk-coding-memory-at
feat(sdk): coding and working experiences keep their conversation time (2.0.7)
2026-10-28 11:33:29 +08:00
alvinfeiandClaude Opus 5.5 fc2e51a8cc release: hy-memory 2.0.7
Coding and working experiences and facts keep their conversation time
(memory_at) and return it from list, get and search. Not yet published to
PyPI.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:09:54 +08:00
alvinfeiandClaude Opus 5.5 d4fca0639e feat(sdk): coding and working experiences keep their conversation time
client.add passed memory_at only to the chat path; the productivity writer
dropped it and coding_memory_meta / working_fact had no column for it, so
every experience had only its write time and MemHub showed it as 时间未知.

Both tables gain a nullable memory_at column (epoch seconds), added to
existing coding/working databases on open. The writer stamps it on drafts
and new facts, defaulting to the request time as chat does; an UPDATE keeps
the later of the stored and incoming times. list_memories (coding, working,
all), get and search return memory_at in chat's format.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:09:54 +08:00
Rahul Ranjanandrahul05ranjan a17230dadb fix(runtime): clear leases that predate the current boot
A forced power-off leaves runtime-service.lock behind with no chance to
release it (#184). After the reboot the liveness probe cannot tell the
dead owner from the process that reused its PID, so launchd's KeepAlive
restarts the wrapper into runtime_service_already_running without end;
and a power cut inside the mkdir -> privateWrite window leaves the lock
without owner.json at all, which fails as runtime_service_lock_invalid
forever.

A lease created before the current boot can never have a living owner:
every process from that boot is gone. removeStaleLease now compares the
lock directory's mtime with the boot time (os.uptime(), injectable via
bootTime for tests) and clears such leases before it ever trusts owner
or PID. A two-minute slack absorbs forward clock steps taken since
boot, so a lease taken right after boot is not mistaken for stale.
Same-boot leases keep the old behavior exactly: live owner still
refused, dead owner still reclaimed, corrupt same-boot lease still
refused.

Fixes #184

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
2026-10-28 01:00:05 +05:30
feitianxiang f7961f340a Merge pull request #198 from Tencent-Hunyuan/fix/memory-detail-layout
fix(memhub): 稳定记忆页并优化设置交互
2026-10-28 00:54:42 +08:00
feitianxiang cfebae118d Merge pull request #202 from Tencent-Hunyuan/fix/sdk-write-pipeline
fix(memhub): SDK write pipeline — turn grouping, thinking off, manual adds, home count
2026-10-28 00:54:05 +08:00
u7-u7 d76bfa6474 fix(memhub): preserve migration log write order 2026-10-28 00:49:54 +08:00
alvinfeiandClaude Opus 5.5 b169906fcb chore(release): bump MemHub build number to 33
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 00:38:34 +08:00
alvinfeiandClaude Opus 5.5 f3b417ced4 fix(memhub): write conversations by one turn rule on every path
File import made one job per file-read increment per conversation: a
transcript read mid-turn gave tool-only jobs, which the SDK refused with
400 "content or messages is required" and Core recorded unknown (16
times); a first import gave one giant job per conversation. Hook capture
and catch-up wrote each completed turn alone, splitting long turns into
pieces that repeated the question.

turn_batches (conversation_processing) is now the one rule:
- a turn is a user message and everything up to the next one; records
  before the first user message join the first turn, and are dropped if
  no user message ever comes;
- a turn is complete when the next user message arrives, or on flush;
- every 5 complete turns are one batch; before 5, the turn that would
  pass 8000 characters starts the next batch; one turn over 8000 is a
  batch of its own and is never split;
- characters are what HY reads (tool records capped at
  _HY_TOOL_CHARACTERS);
- a short batch is written only on flush; no batch lacks a user message.

File import keeps its assembling row per (file, conversation) and now
writes only what the rule completes, carrying the rest to the next
increment. History flushes at end of file. Live polling and hook capture
(which now stage into the same rows) flush after 10 minutes without a
new message, swept on the drain's maintenance timer. A record a job
already holds is not staged again, so a re-read creates no second job;
batch keys stay content hashes. Legacy catch-up groups by the same rule.

Backstop: SdkWriteProcessor completes a batch with no user/assistant
message as nothing to remember, without calling the SDK.

The unused _merge_historical_chunks and the old window code
(_conversation_chunks, segmenting, ONE_MEMORY_SPLIT_TURNS) are removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 00:38:06 +08:00
u7-u7 e9698a293c fix(tokens): count literal special-token spellings as ordinary text 2026-10-28 00:28:14 +08:00
alvinfeiandClaude Opus 5.5 aaebe64244 fix(memhub): metadata repair never runs beside an SDK client
After an upgrade 76/76 memories showed source=default: the startup
metadata repair failed with a KeyError. It ran after readiness and
opened its own Chroma client on the SDK's vector store. Chroma keeps one
process-wide System per path with no lock, and the repair's close() (or
a failed client init) released it while the first library request was
opening the SDK client, whose next lookup raised KeyError.

- sdk-migrate now runs the repair offline right after the cutover
  commits and writes the receipt, so startup normally finds it and does
  nothing. A repair failure there is logged and left to startup.
- Startup keeps the repair as a fallback but holds the home's SDK stores
  offline from before readiness until the repair thread ends; every SDK
  client is built through acquire_client, which waits meanwhile.
  Readiness is not delayed; the first request waits for the repair.
  Homes migrated with a failed repair have no receipt, so they are
  repaired on the next launch.
- A failed recovery step logs its exception chain as file:line:function
  frames (core.log) and safe_stack (diagnostic event), never messages.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 23:53:18 +08:00
alvinfeiandClaude Opus 5.5 f3d13c4351 fix(memhub): a hand-added memory no longer waits Pending forever
On SDK authority the add path submits straight to the intake and writes
no one-memory-manual grant row. The intake admitted the record, but the
drain's claim gate required the grant receipt, so the job was never
eligible: attempts=0 forever while later jobs ran.

_source_writes_on now treats a missing manual grant as allowed and only
an explicit revocation as refused. Intake, claim and execute all ask it,
so jobs already stuck become claimable on the next drain tick. The SDK
add path refuses a revoked manual source with memory_add_source_revoked,
like the legacy path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 23:53:00 +08:00
kristinheandCursor 3e8a006f65 fix(memhub): satisfy full lint gate
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 23:51:51 +08:00
kristinheandCursor a70830970d fix(memhub): polish settings interactions
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 23:45:08 +08:00
kristinheandCursor 556d95df84 fix(memhub): stabilize memory detail and pagination
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 23:45:00 +08:00
alvinfeiandClaude Opus 5.5 ad05303d23 fix(memhub): turn model thinking off on the SDK write path
Profiles default to disable_thinking, but build_sdk_config never told
the SDK. DeepSeek Flash spent the SDK's 2000 agent_max_tokens thinking
and answered finish_reason=length with no content (EMPTY_RESPONSE).

The provider mapping moves into disable_thinking_params, shared by
attach_disable_thinking and the SDK's llm.extra_body. The SDK sends
extra_body on every chat call, and the coding/working writers build
their LLMProvider from the same llm config.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 23:37:09 +08:00
alvinfeiandClaude Opus 5.5 423cb00dbf fix(memhub): home memory count matches the memory page
The home card counted chat memories only (list_memories without a
scene), so it showed 55 while the memory page showed 94. It now reads
memory_stats total (chat + working + coding experiences, facts not
counted) and takes the newest memory from list_memories(scene="all").

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 23:33:19 +08:00
feitianxiang e09fbd16f9 Merge pull request #197 from Tencent-Hunyuan/feat/cli-simplify
feat(memhub): simplify the CLI and drop the notes concept
2026-10-27 20:21:20 +08:00
alvinfeiandClaude Opus 5.5 dd3809a34b chore(release): bump MemHub build number to 32
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 20:16:09 +08:00
alvinfeiandClaude Opus 5.5 efe7df4048 feat(memhub): delete-data and backup restore need the path typed twice
`--yes` alone no longer deletes data or restores a backup: the target
directory has to be repeated with `--confirm <path>`. Without it the
command refuses with exit 4 and shows the exact command to run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 19:49:30 +08:00
alvinfeiandClaude Opus 5.5 072e0e5e90 feat(memhub): simplify the CLI and drop the notes concept
API: adding one memory is POST /api/v2/memories with its status at
GET /api/v2/memories/additions/{request_id}; /api/v1/memory-notes is
removed. Error codes memory_note_* become memory_add_*, contracts types
MemoryAddInput/MemoryAddReceipt, broker /memories, MCP MemHub_write and
the web composer follow. Stored names (memory_note_requests,
one-memory-manual, direct_memory_note) stay. /api/v2/memory-library
takes an optional scene filter.

CLI:
- No plan files. memories add/add-status, models add/test/receipt and
  sources connect execute directly; async receipts stay.
- One --yes, only for irreversible operations (memories delete, models
  remove, delete-data, backup restore, graph-width rebuild); without it
  they preview and exit 0. --confirm is a hidden alias.
- Removed notes, calls, setup plan, sources import, models
  plan/configure/probe*, and the unreachable processing, extraction and
  import-file commands, with their dead allow-list entries.
- memories list gets --scene/--source/--query; show/update/delete take
  --kind/--variant (and --task). All memories JSON is schema_version 2.
- Fixes from the CLI review: global --data-dir, real
  --confirm-overrides and help that matches init, server failures exit 1
  not 2, catalog exit codes match, JSON errors on stdout, bad filters are
  rejected instead of "busy", write submit prints its request_id for
  retries, recall labels raw excerpts, stop says it disables login
  auto-start, backup help says memories are not included yet.
- Hidden aliases: path -> scan-roots, task -> write status,
  graph width -> graph-width, init --source -> sources connect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 19:21:34 +08:00
feitianxiang a0221bc434 Merge pull request #196 from Tencent-Hunyuan/fix/migration-failure-freeze
fix(memhub): restore and freeze the home when an upgrade migration fails
2026-10-27 19:18:19 +08:00
alvinfeiandClaude Opus 5.5 754555e3c3 fix(memhub): restore and freeze the home when an upgrade migration fails
A failed SDK migration left a half-migrated home (schema migrations,
moved legacy work, renamed Chroma collections) that the next run
resumed on top of, the maintenance fence re-enabled the LaunchAgent so
launchd retried at once, a deferred service idled forever, and capture
hooks kept writing into the legacy database.

Restore: before the first mutation the run records a restore point
(the pre-cutover bundle, or a fresh bundle when resuming an old ledger
row). Any failure before SDK authority is committed restores the
management database (with WAL), config.json and hy-memory/governed-v2
through a verified staging copy and a journaled rename swap; the failed
state is kept aside in backups/sdk-migration-failed/. A failure after
the commit, or an unreadable authority, never restores. An interrupted
swap is finished on the next start; meanwhile status reports required
and Core refuses to serve. The migration's own code stays the result;
the restore outcome is logged and reported separately.

Freeze: a deferred migration turns the service enable marker off, makes
the runtime service exit instead of idling, and leaves a frozen capture
fence that hooks honour. Desktop launch, memhub install/upgrade and now
start/restart re-run the migration and unfreeze on success, restoring
the previous auto-start.

The failure dialog says when data was restored, and its GitHub issue now
prefills the body: from/to version, code, failing step, whether data was
restored, macOS version and the log path shown with ~. migrate.invoked
records the pre-upgrade version. sdk_data_plane_* codes now reach the
dialog too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 19:11:04 +08:00
feitianxiang b43c967baa Merge pull request #195 from Tencent-Hunyuan/feat/memory-panel-all-scenes
feat(memhub): memory panel lists and searches chat, working and coding
2026-10-27 18:04:32 +08:00
alvinfeiandClaude Opus 5.5 dac2c15b3f feat(memhub): memory panel lists and searches chat, working and coding
Pin hy-memory 2.0.6 and use its scene-aware APIs in the memory panel.

- Empty query: list_memories(scene="all") mixes the three scenes newest
  first, 20 per page, with per-scene totals and the source filter.
- Search: one SDK search(scene="all", rerank=False, limit=200) per
  query; Core keeps the keyword matches in a server-side result window
  (window_id, 300s idle TTL, 32 windows, ids only) and every page is cut
  from it, so pages never repeat or skip. An expired window re-searches
  and returns page 1 with window_reset. Pages are read fresh, so edits
  and deletes show.
- Items carry scene, kind, variant, task/solution and times; the panel
  shows 对话 / 工作 / 编码 badges. Detail, edit and delete pass
  kind/variant (experience edits the solution, optionally the task);
  an ambiguous id is 409, a task on a non-experience 422.
- memhub memories list/show print scene/kind/variant.
- SDK errors are re-exported through hy_sdk_runtime to keep the SDK
  import boundary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 16:54:08 +08:00
feitianxiang 0b775d7d61 Merge pull request #194 from Tencent-Hunyuan/release/hy-memory-2.0.6
release: hy-memory 2.0.6
2026-10-27 15:59:35 +08:00
alvinfeiandClaude Opus 5.5 4e217f845e release: hy-memory 2.0.6
Published hy-memory==2.0.6 to PyPI: search rerank switch, size-gated
MMR and near-duplicate demotion, scene="all" search and list, per-scene
list/memory_stats, productivity update/delete, reconcile time order,
HTTP 409/400 for ambiguous or bad namespaces, and the stats UTC fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 15:55:00 +08:00
feitianxiang 2102a298d4 Merge pull request #193 from Tencent-Hunyuan/fix/sdk-near-dup-gate
fix(sdk): gate near-duplicate demotion with the MMR top-k threshold
2026-10-27 15:42:38 +08:00
alvinfeiandClaude Opus 5.5 e540386db1 fix(sdk): gate near-duplicate demotion with the MMR top-k threshold
near_dup_demote compares each candidate with every earlier
non-duplicate by Jaccard, quadratic when candidates differ. Its pool
grows with limit (about max(3*limit, 30) plus keyword hits), so a
limit-200 page could cost ~180k comparisons. It now runs only when the
caller's limit is below MMR_MAX_TOP_K, like MMR; large pages want every
relevant item, similar ones included. The read reports
near_dup_applied next to mmr_applied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 15:34:41 +08:00
feitianxiang 1374ac26b7 Merge pull request #192 from Tencent-Hunyuan/feat/sdk-search-scenes
feat(sdk): rerank switch, size-gated MMR, scene="all" search and list, productivity edit/delete
2026-10-27 15:22:41 +08:00
alvinfeiandClaude Opus 5.5 47dafbb102 feat(sdk): rerank switch, size-gated MMR, scene="all" search and list, productivity edit/delete
search:
- rerank=True (default) keeps today's behavior; rerank=False skips the
  local cross-encoder, graph rerank and the coding facts' BM25 rerank for
  that call (carried on ReadRequest, so concurrent calls are independent).
- MMR now runs only when the caller's limit is below MMR_MAX_TOP_K (20).
  It is O(k^2 * pool) Jaccard and made large result pages slow.
- scene="all" searches chat, working and coding concurrently with one
  shared embedding and fuses them by Reciprocal Rank Fusion (k=60) over
  each lane's own rank, so no lane is buried by its score scale. Items
  carry scene, kind, memory_id, content, fused score, lane score/rank,
  source and times. A failed or disabled lane is reported in "errors".

list_memories:
- scene="coding"|"working" honour include_sources/exclude_sources with
  the filtered total.
- scene="all" lists the three scenes newest first with limit/offset over
  the merged order, per-scene totals and the same unified item fields.

update/delete:
- kind/variant disambiguate like get(); an experience update replaces
  its solution (task= also replaces the task and its search vectors), a
  fact update replaces the statement; delete removes experiences and
  facts with their vectors. Chat calls are unchanged.
- The HTTP server maps an ambiguous id to 409 and a bad namespace to 400
  instead of 500, passes kind/variant through, and strips the query
  string from PUT/DELETE paths, which leaked it into the memory id.

memory_stats now reads productivity times (written as naive UTC) as UTC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 15:13:12 +08:00
feitianxiang 49ad1b3086 Merge pull request #185 from Tencent-Hunyuan/fix/migration-chroma-settings
fix: list and count every memory scene, and migrate without a Chroma clash
2026-10-27 14:35:45 +08:00
alvinfeiandClaude Opus 5.5 3064515b43 Merge main into fix/migration-chroma-settings
Keep both CHANGELOG entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 14:30:30 +08:00
feitianxiang e5c8cc3819 Merge pull request #190 from Tencent-Hunyuan/feat/mcp-three-tools
feat(memhub): cut the MCP server to recall, write and diagnostic trace
2026-10-27 14:13:53 +08:00
alvinfeiandClaude Opus 5.5 27f64f1b7a chore(memhub): bump build to 31 and teach new-harness skills the three tools
The custom-Harness connection Skill covered pairing and the MCP config
but never told the model what the tools are, and not every Harness
reads the MCP initialize instructions. It now has a "Using MemHub after
connecting" section for MemHub_recall (including raw-mode results),
MemHub_write and MemHub_diagnostic_trace. The command-only Skill (Pi)
says recall can return original-conversation excerpts and that the
host has no write path.

Build number 30 -> 31.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 14:07:09 +08:00
kristinhe1111 3c5451db24 Merge pull request #183 from Tencent-Hunyuan/fix/home-recent-update-metadata
fix(memhub): stabilize recent updates and memory details
2026-10-27 13:22:45 +08:00
alvinfeiandClaude Opus 5.5 a08f69a747 feat(memhub): cut the MCP server to recall, write and diagnostic trace
MCP hosts now see three tools: MemHub_recall, MemHub_write and
MemHub_diagnostic_trace. MemHub_ask, MemHub_conversations,
MemHub_context, MemHub_memory_write and MemHub_memory_write_status are
removed, with their broker routes (/ask, /conversations, /memory-writes)
and client functions. Broker /context stays: the WorkBuddy injection
hook, recall-cli --context and the Hermes provider still use it.

MemHub_write saves one hand-added memory exactly like the app's
"添加记忆": broker POST /memory-notes authenticates the installation and
calls Core /api/v1/memory-notes with trusted headers, so the note takes
the direct-write lane ahead of history and is reconciled through
add_extracted. It takes {text, request_id?} and returns the note receipt
with the request_id to retry with.

MemHub_recall falls back to the SDK raw layer when no generation model
is configured, or when the memory search is empty and the library holds
no memory yet. The receipt says so with mode "raw" and fallback_reason;
raw hits are conversation excerpts (content_format "raw", source,
session_id). Raw reads keep the /context raw-lane authority: a host
needs om_conversations and its granted, unrevoked sources, so recall's
own grant never widens to original conversations.

Stored grants holding om_ask or om_conversations still load; new
pairings get om_recall, om_conversations and om_diagnostic_trace.
Skill text (SKILL_VERSION 19), connection prompt, docs and tests follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 13:16:55 +08:00
kristinheandCursor 099f5ae7c0 test(memhub): update memory search affordance
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 13:13:55 +08:00
kristinhe1111 e7625032d8 Merge pull request #187 from Tencent-Hunyuan/fix/memory-filter-clear-query
fix(memhub): 清空搜索框时移除已提交的记忆关键词
2026-10-27 13:11:07 +08:00
kristinhe1111 3aa3b5957d Merge pull request #186 from Tencent-Hunyuan/fix/hy-debug-pipeline-logs
fix(memhub): surface HY pipeline logs in Debug
2026-10-27 12:24:15 +08:00
kristinheandCursor 409ab9a9b4 fix(memhub): clear submitted memory query with input
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 12:22:55 +08:00
alvinfeiandClaude Opus 5.5 c2f50dce06 fix: claim hand-added memories first, and never let older memories win
MemHub: "添加记忆" (add a memory) went into the same durable queue as
history, which is claimed through a ten-minute event-time window, oldest
first. A note is stamped "now", so with a backlog it ran only after the
whole history. Jobs from one-memory-manual are now claimed ahead of the
window, still one at a time per source and in session order; history
keeps its strict timeline.

SDK reconcile: memories are not written in the order they happened, so a
new memory can be older than an existing one. Both reconcile prompts now
say only memory_at decides which is newer, that an older memory never
supersedes or overwrites a newer one (it is added as a past state), and
that an unknown time is not newer. The Chinese prompt now lists memory_at
among the new-memory fields, which it had left out. enforce_time_order
makes this hold in code: a SUPERSEDE or UPDATE aimed at a memory with a
later memory_at becomes an ADD.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 12:21:01 +08:00
kristinheandCursor 2735d8693a fix(memhub): prioritize memory list space
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 12:19:49 +08:00
kristinheandCursor 47bbfb4b27 fix(memhub): simplify memory note copy
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 12:11:54 +08:00
kristinheandCursor 3f169c9951 style(memhub): format language selector
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 12:01:58 +08:00
kristinheandCursor cb97aa5822 fix(memhub): show attributed app calls on home
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:43:25 +08:00
alvinfeiandClaude Opus 5.5 38a4da6849 fix: list and count every memory scene, and migrate without a Chroma clash
SDK: list_memories(scene="chat"|"coding"|"working") lists one kind of
memory. coding and working list the scene's whole store, newest update
first, with workspace_id still narrowing it, and total is the real count
instead of the page size. memory_stats() returns each scene's count and
latest created/updated time plus their total, reading no memory text.
Without a workspace, coding and working memories could not be listed or
counted, so a 45-memory library listed 22.

MemHub migration: reattaching legacy vectors opened Chroma with its own
settings and never closed it, so the readback that followed in the same
process could not open the store ("An instance of Chroma already exists
... with different settings"). Every 1.0.4 home with legacy vectors
failed its first upgrade launch. Offline code now opens Chroma as the SDK
does and closes it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-27 11:41:10 +08:00
kristinheandCursor 48a5d70026 fix(memhub): stabilize language setting layout
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:20:53 +08:00
kristinheandCursor b21c929728 fix(memhub): stabilize memory detail layout
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:12:47 +08:00
kristinheandCursor b9b3c587ff fix(memhub): hide recent update internals
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-27 11:01:14 +08:00
u7-u7 6718ba8bdf fix(async): preserve context in guarded thread calls 2026-10-27 10:41:13 +08:00
282 changed files with 23319 additions and 11668 deletions

No files matched your search

+146
View File
@@ -4,6 +4,39 @@ Hy Memory and MemHub are versioned and released independently.
## Unreleased
## MemHub 1.0.5 — 2026-10-28
Build 36, on `hy-memory==2.0.8`. Linux x64/arm64 Runtime and macOS arm64
App and Runtime. Upgrading from 1.0.4 migrates the existing library to SDK
authority on first launch.
- MemHub turns off the HY pipeline log and trace at every diagnostic level
(hy-memory 2.0.8 `MEMORY_PIPELINE_LOG_ENABLED=false`,
`MEMORY_PIPELINE_TRACE_ENABLED=false`), so the per-step JSONL files under
`hy-memory/governed-v2/logs/` and the `pipeline_logs` rows stop growing.
The Debug panel no longer shows the HY pipeline section. Existing files are
left in place.
- HY SDK text logs are written once, to `hy_memory.log`. They are no longer
copied to `service.log` and `service-error.log` through the SDK's
stdout/stderr streams. SDK warnings and errors also go to `core.log`, so the
Debug panel still shows them. A live switch of the diagnostic level now also
changes the level of `hy_memory.log`.
- MemHub shows when working and coding experiences were talked about. After
the SDK upgrade every one of them read "时间未知" (time unknown) because the
SDK never stored their conversation time; hy-memory 2.0.7 stores it and
Core now passes it through the memory list, detail, search and home
summary. An experience stored before 2.0.7 shows its write time, labelled
"写入于" / "写入时间", instead of "时间未知". Metadata repair also writes
the conversation time it recovers from the migration bundle into the SDK's
coding/working stores when they already have the column.
- The SDK migration no longer fails its first run on a home with legacy
vectors: reattaching them left a Chroma instance open with its own settings,
and the readback that followed could not open the same store
("An instance of Chroma already exists ... with different settings").
- MemHub "添加记忆" (add a memory) no longer waits behind unprocessed history.
Notes are claimed ahead of the event-time window that keeps history strictly
old to new; they still run one at a time and in order within their session.
- The SDK memory page now reads its list, count, source and time labels and
graph from one page response for one source filter, so rows, totals and the
map no longer disagree. Memories with no recorded source are shown and
@@ -61,6 +94,119 @@ Hy Memory and MemHub are versioned and released independently.
`hy_node_permissions` proof table. Evidence, governed memories, model
profiles, SDK stores, and source/host grants are preserved.
## Hy Memory 2.0.9 — 2026-11-01
- Placeholder values no longer overwrite the user's basic profile (L0). A
small extraction model filled `basic_info` with `{"name": "用户"}`,
`The user` or `N/A` when the user had said nothing about themselves, and
each placeholder superseded the real value, so the L0 chain read
费天翔 → Fei → 用户 → The user. Two defences:
- `upsert_basic_profile` drops placeholder values: empty, `用户`,
`the user`, `user`, `n/a`, `na`, `none`, `null`, `unknown`, `未提供`,
`未知`, `不详`, `-`, `?` and the field's own name, compared
case-insensitively with whitespace and punctuation stripped. When nothing
is left it writes no new version and no new L0 head. Placeholders already
stored in the chain no longer override earlier real values when the
profile is assembled, so `get_user_profile` of an affected user shows the
last real value again without any migration. This also stops an
assistant quoting an old memory ("The user's name is 用户.") from being
written back.
- The extraction prompts (zh and en) state the rule: only what the user
said in a `user` turn, never the field name or a placeholder, `{}`
otherwise, with a small-talk counter-example.
## Hy Memory 2.0.8 — 2026-10-28
- Published
[`hy-memory==2.0.8`](https://pypi.org/project/hy-memory/2.0.8/).
- New `MEMORY_PIPELINE_LOG_ENABLED` (default `true`) turns the pipeline JSONL
log (`{MEMORY_LOG_DIR}/<subdir>/<date>.log`) off. With `false` no JSONL
file or directory is written. When `MEMORY_PIPELINE_TRACE_ENABLED` is also
`false`, `store_pipeline_log` becomes a no-op that returns `True` at once:
no timing, redaction, background task or database write. The init log line
now states whether the JSONL log is on or off.
- Conversation and memory text no longer appears in INFO or WARNING logs. The
`[add]` input, search queries and rewrites, coding experience tasks, fact
statements and subjects, reconcile op details, extracted info, embedding
inputs and unparsable model output moved to DEBUG. INFO lines keep ids,
counts and character lengths; parse-failure warnings remain but report only
the length of the output, e.g. `failed to parse output (812 chars)`. The
`[embed]` INFO line now carries `request.input_chars` instead of
`request.input`. The `JSON_PARSE_FAILED` error message from extraction no
longer embeds the model output either.
## Hy Memory 2.0.7 — 2026-10-28
- Published
[`hy-memory==2.0.7`](https://pypi.org/project/hy-memory/2.0.7/).
- Coding and working experiences and facts now keep their conversation time.
`add(..., memory_at=...)` reached only the chat path before; the
productivity path dropped it, so every experience had only its write time
and MemHub showed them as "时间未知" (time unknown). `coding_memory_meta`
and `working_fact` gain a nullable `memory_at` column (epoch seconds) in
both the working and coding stores; existing databases get it on open, and
their old rows stay `None`. Without `memory_at` a write uses the time the
request arrived, as chat does. When an experience is updated by a later
conversation it keeps the later of the two times.
- `list_memories` (`scene="coding"`, `"working"` and `"all"`), `get` and
`search` return `memory_at` for experiences and facts in the same field and
format as chat (epoch seconds, `None` when unknown). Experience search
results now also carry `created_at`.
## Hy Memory 2.0.6 — 2026-10-27
- Published
[`hy-memory==2.0.6`](https://pypi.org/project/hy-memory/2.0.6/).
- `search(..., rerank=False)` skips every reranking stage for that one
call (the local cross-encoder, the memory-graph rerank, and the BM25 rerank of
working/coding facts), so results come back in retrieval-score order. The
flag travels on the read request, so concurrent searches do not affect each
other. The default (`True`) keeps current behavior; `/api/v1/search` accepts
it too.
- MMR diversity selection now runs only when the requested `limit` is
below 20 (`MMR_MAX_TOP_K`) and is skipped for larger limits, where its
quadratic token-overlap pass is slow and gains little. Previously it ran for
every limit. `MEMORY_HYBRID_V2_MMR_LAMBDA` still sets the weight.
Near-duplicate demotion follows the same gate: it compares each candidate
with every earlier non-duplicate, which is quadratic in a pool that grows
with `limit`, and large result pages should show similar items anyway.
- `search(scene="all")` searches chat, working and coding at the same time
and merges them with Reciprocal Rank Fusion (k=60) over each scene's own
ranking, so no scene is crowded out by the other scenes' score scales. Each
result says which scene and kind it came from. A scene that fails or is turned
off is reported in `errors` and does not fail the search. Source filters apply
to all three.
- `list_memories(scene="all")` lists chat, working and coding memories
together, newest update first, with `limit`/`offset` paging over the merged
order and `total` summing all three. `scene="coding"` and `"working"` now
honor `include_sources` / `exclude_sources`, and `total` counts only the
filtered memories.
- `update()` and `delete()` now work on working/coding experiences and
facts, not just chat memories. They take the same optional `kind` / `variant`
as `get()`, and an ID found in more than one place raises
`AmbiguousMemoryIdError` instead of guessing. Updating an experience replaces
its solution and keeps its task, unless a new `task=` is also passed; updating
a fact replaces its statement. Deleting removes the stored row and its vectors.
- `list_memories(scene="chat" | "coding" | "working")` lists one kind of
memory. `coding` and `working` list that scene's whole store, newest update
first (`workspace_id` still narrows it), and `total` is now the real count
rather than the page size. The new `memory_stats()` returns the count and the
latest created/updated time for each scene and their total, without reading
any memory text. Previously coding and working memories could not be listed
or counted without a workspace, so a library of 45 memories listed 22.
- Reconcile keeps time order. Both reconcile prompts now state that only
`memory_at` decides which memory is newer, that an older memory must never
supersede or overwrite a newer one, and that an unknown time is not newer;
the Chinese prompt now lists `memory_at` among the new-memory fields. The
reconciler also enforces this in code: a SUPERSEDE or UPDATE aimed at a memory
with a later `memory_at` becomes an ADD, so the newer memory stays current.
- The HTTP server returns 409 `ambiguous_memory_id` and 400 for a bad
`kind`/`variant` instead of 500, passes `kind`/`variant` (and `task` on PUT)
to get/update/delete, and no longer reads a PUT/DELETE query string as part
of the memory id.
- `memory_stats()` reads working/coding times, which are stored as naive UTC,
as UTC rather than local time.
## Hy Memory 2.0.5 — 2026-10-22
- Published
+1
View File
@@ -33,6 +33,7 @@ English | [中文](README.zh.md)
## Latest News
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** quantized **FP8** (**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**) and **GGUF** (**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**) versions of **Hy-Memory-30B-A3B** on Hugging Face!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **Released [MemHub v1.0.5](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.5)** with Chinese, English and Spanish interfaces, one-time pairing codes for more custom Harnesses, and smoother browsing, search, startup and recovery, on **`hy-memory==2.0.8`**. See the release notes for details.
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **Released [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** and **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**, with a signed macOS desktop app and verified macOS/Linux Runtimes!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.4`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.4/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** on Hugging Face along with its [technical report](https://arxiv.org/abs/2610.22196) on attributed reinforcement learning for long-term memory writing!
+1
View File
@@ -33,6 +33,7 @@
## 最新动态
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **Hy-Memory-30B-A3B** 的 **FP8**(**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**)与 **GGUF**(**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**)量化版本模型!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **正式发布 [MemHub v1.0.5](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.5)**:支持中文、英文和西班牙语界面,可通过一次性配对码连接更多自定义 Harness,并优化了记忆浏览、搜索、启动和异常恢复;底层使用 **`hy-memory==2.0.8`**。详见发布说明。
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **正式发布 [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** 与 **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**,提供已签名的 macOS 桌面应用及经过验证的 macOS/Linux Runtime!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.4`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.4/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** 长期记忆写入模型及[技术报告](https://arxiv.org/abs/2610.22196)!
+1
View File
@@ -191,6 +191,7 @@ SDK 不设置 Rerank Search 并发上限。开启后模型在 `HyMemoryClient`
| `MEMORY_LOG_FORMAT` | `text` | `text` / `json` |
| `MEMORY_LOG_DIR` | — | 统一日志根目录 |
| `MEMORY_TRACE_LOG_DIR` | — | JSONL 日志目录的兼容旧名;优先使用 `MEMORY_LOG_DIR` |
| `MEMORY_PIPELINE_LOG_ENABLED` | `true` | 是否写 pipeline JSONL 日志(`{MEMORY_LOG_DIR}/<subdir>/<date>.log`);`false` 完全不写 |
| `MEMORY_PIPELINE_TRACE_ENABLED` | `true` | 是否写入 cache 的 pipeline trace 表 |
| `MEMORY_MEMORY_OPERATIONS_ENABLED` | `false` | 是否写入 memory operation 审计记录 |
| `MEMORY_HISTORY_ENABLE` | `true` | 是否启用历史记录 |
+37 -3
View File
@@ -682,6 +682,23 @@ Output:
```
John's occupation and location are HIS attributes, so they go into `memory` with "John" as the subject. They must NEVER land in `basic_info.occupation` / `basic_info.location` — `basic_info` is exclusively the user's own profile. Only `name` (Tim, from the user message's `name` field) belongs there.
## Example 11 — Counter-example: small talk has no basic_info
New Messages:
[{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello! The user's name is The user. How can I help?"}]
Memory Date: 2025-09-03
Output:
```json
{
"memory": [],
"intentions": [],
"basic_info": {}
}
```
The user stated nothing about themselves. The assistant's line is not a user turn, and "The user" is a placeholder, not a name. Output `{}` — never `{"name": "The user"}` or `{"name": "user"}`.
---
"""
@@ -749,6 +766,14 @@ Rules:
fields (e.g. name, age, location, occupation, employer) **explicitly stated**
in the NEW user messages. Never infer, never invent. If no field is explicitly
stated, output `{{}}`.
* HARD RULE: fill a field ONLY if the user stated it in a `user` turn.
Anything said only by the assistant (including the assistant quoting an
earlier memory such as "The user's name is X") does NOT count. If nothing
qualifies, output `"basic_info": {{}}`.
* NEVER write the field name itself or a placeholder as a value ("user",
"The user", "用户", "name", "N/A", "unknown", "none", "-"). Omit the field.
* Negative example: small talk such as `[user]: Hi` / `[assistant]: How can
I help?`, or `[user]: Reply with exactly: PONG` → `"basic_info": {{}}`.
* Exception for `name`: the `name` field on the user's own message, or a leading
line "The user's name is X" / "用户名叫X", is an authoritative source for
`name` — even though the user never said it out loud.
@@ -817,6 +842,14 @@ Rules:
fields (e.g. name, age, location, occupation, employer) **explicitly stated**
in the NEW user messages. Never infer, never invent. If no field is explicitly
stated, output `{{}}`.
* HARD RULE: fill a field ONLY if the user stated it in a `user` turn.
Anything said only by the assistant (including the assistant quoting an
earlier memory such as "The user's name is X") does NOT count. If nothing
qualifies, output `"basic_info": {{}}`.
* NEVER write the field name itself or a placeholder as a value ("user",
"The user", "用户", "name", "N/A", "unknown", "none", "-"). Omit the field.
* Negative example: small talk such as `[user]: Hi` / `[assistant]: How can
I help?`, or `[user]: Reply with exactly: PONG` → `"basic_info": {{}}`.
* Exception for `name`: the `name` field on the user's own message, or a leading
line "The user's name is X" / "用户名叫X", is an authoritative source for
`name` — even though the user never said it out loud.
@@ -1177,7 +1210,7 @@ class Extractor:
if not tc_id:
logger.error(
f"[extractor] tool_call_id is empty! "
f"raw tc: {json.dumps(tc_raw, ensure_ascii=False, default=str)[:500]}"
f"raw_keys={list(tc_raw.keys())}"
)
try:
t_res = await tool_registry.dispatch(
@@ -1311,11 +1344,12 @@ class Extractor:
if parsed is None:
if not all_tool_calls:
logger.warning(
f"[extractor] JSON parse failed: {raw_text[:200]}"
f"[extractor] JSON parse failed ({len(raw_text)} chars)"
)
logger.debug(f"[extractor] unparsable output: {raw_text[:200]!r}")
return ExtractResult(
success=False,
error=f"JSON parse failed: {raw_text[:200]}",
error=f"JSON parse failed ({len(raw_text)} chars)",
error_code="JSON_PARSE_FAILED",
raw_response=raw_text,
tokens_used=total_tokens,
+1 -2
View File
@@ -129,8 +129,7 @@ def _parse_tool_calls_list(raw_calls: list, _json) -> List[Dict[str, Any]]:
tc_id = tc.get("id") or ""
if not tc_id:
logger.error(
f"[llm] tool_call missing 'id'! raw: "
f"{_json.dumps(tc, ensure_ascii=False, default=str)[:500]}"
f"[llm] tool_call missing 'id'! keys={list(tc.keys())}"
)
# 保留原始 tc 完整结构,只规范化 arguments
+1 -1
View File
@@ -234,7 +234,7 @@ class MemAgent:
f"identity={len(identity)} facts={len(facts)} "
f"basic_info={basic_field_count}fields"
)
logger.info(f"MemAgent: extracted_info={extract_result.info}")
logger.debug(f"MemAgent: extracted_info={extract_result.info}")
# 3. 冲突检测(如果有已有记忆)
if mode == ProcessMode.FULL and existing_memories:
+45 -7
View File
@@ -365,6 +365,13 @@ EXTRACT_PROMPT_ZH = """## 最近 k 条消息
* basic_info:**只装用户本人的画像**。
仅当 NEW user messages 中**明确**陈述稳定画像字段(如 姓名、年龄、所在地、职业、雇主)时才填写。
绝不推断、绝不编造。没有任何字段被明确陈述时输出 `{{}}`。
* **硬性规则**:只有用户在 `user` 回合里**明确说出**的字段才能填写。只在
assistant 回合出现的内容(包括 assistant 引用旧记忆,如"The user's name is X"
/ "用户名是X")一律不算。没有符合条件的字段时,必须输出 `"basic_info": {{}}`。
* **绝不**把字段名本身或占位符当值写入("用户"、"The user"、"user"、"name"、
"姓名"、"N/A"、"unknown"、"未知"、"未提供"、"-"),宁可省略该字段。
* 负例:寒暄如 `[user]: 你好` / `[assistant]: 我可以帮你。`,或
`[user]: Reply with exactly: PONG` → `"basic_info": {{}}`。
* `name` 的例外:用户自己那条消息上的 `name` 字段,或文首形如
"用户名叫X" / "The user's name is X" 的那一行,都是 `name` 的权威来源
——即使用户本人从未开口自报姓名。
@@ -440,6 +447,13 @@ EXTRACT_PROMPT_ZH_NO_INTENTION = """## 最近 k 条消息
* basic_info:**只装用户本人的画像**。
仅当 NEW user messages 中**明确**陈述稳定画像字段(如 姓名、年龄、所在地、职业、雇主)时才填写。
绝不推断、绝不编造。没有任何字段被明确陈述时输出 `{{}}`。
* **硬性规则**:只有用户在 `user` 回合里**明确说出**的字段才能填写。只在
assistant 回合出现的内容(包括 assistant 引用旧记忆,如"The user's name is X"
/ "用户名是X")一律不算。没有符合条件的字段时,必须输出 `"basic_info": {{}}`。
* **绝不**把字段名本身或占位符当值写入("用户"、"The user"、"user"、"name"、
"姓名"、"N/A"、"unknown"、"未知"、"未提供"、"-"),宁可省略该字段。
* 负例:寒暄如 `[user]: 你好` / `[assistant]: 我可以帮你。`,或
`[user]: Reply with exactly: PONG` → `"basic_info": {{}}`。
* `name` 的例外:用户自己那条消息上的 `name` 字段,或文首形如
"用户名叫X" / "The user's name is X" 的那一行,都是 `name` 的权威来源
——即使用户本人从未开口自报姓名。
@@ -669,6 +683,23 @@ Memory Date:2025-09-02
```
职业和所在地是 **John 的**属性,所以进 `memory` 并以"John"作主体,**绝不能**落到 `basic_info` 的 `occupation` / `location` —— `basic_info` 专属于用户本人。只有 `name`(Tim,来自用户消息的 `name` 字段)才该放进去。注意输出语言随输入语言(此例为英文)。
## 示例 11 —— 反例:寒暄没有 basic_info
新消息:
[{"role": "user", "content": "你好"},
{"role": "assistant", "content": "你好!我可以帮你。之前记得 The user's name is 用户."}]
Memory Date:2025-09-03
输出:
```json
{
"memory": [],
"intentions": [],
"basic_info": {}
}
```
用户没有说任何关于自己的信息。assistant 那句不是 user 回合,"用户" 也是占位符而不是名字。必须输出 `{}`——**绝不能**输出 `{"name": "用户"}`。
---
"""
@@ -743,6 +774,7 @@ RECONCILE_PROMPT_ZH = """你是一个记忆整合系统(Memory Reconciler)
* content
* owner
* memory_at(发生时间)
* tags
new memories 没有 ID。
@@ -791,6 +823,7 @@ new memories 没有 ID。
* 无损合并
* 保留所有有效信息
* 不要产生重复 memory
* 两者在某个值上不一致时,保留 memory_at 更晚的那个值;绝不用更早的 memory 覆盖更新的值
### 同一次事件的补充 vs 另一次发生
@@ -823,6 +856,7 @@ new memories 没有 ID。
* 新 memory 与旧 memory 在同一维度冲突
* 两者不能同时为真
* 旧 memory 已不再代表当前状态
* 新 memory 的 memory_at 不早于旧 memory 的 memory_at
例如:
@@ -837,6 +871,7 @@ new memories 没有 ID。
* wording 差异
* 同义表达
* 可共存偏好
* 新 memory 的 memory_at 早于旧 memory:它是更早的状态,绝不能取代更新的状态(见「时间规则」)
要求:
@@ -859,7 +894,7 @@ new memories 没有 ID。
### 把多条旧记忆折叠进同一条链
如果**两条或更多**已有记忆描述的是**同一维度**,而新记忆是该维度上的最新变化,
如果**两条或更多**已有记忆描述的是**同一维度**,而新记忆是该维度上的最新变化(它的 memory_at 晚于所有被折叠的记忆),
你可以把它们折叠进同一条演化链。此时用 `memory_ids`(有序列表,**旧→新**)
代替单个 `memory_id`:
@@ -951,13 +986,16 @@ SUPERSEDE 不需要复制旧内容,因为旧节点仍存在。
## 时间规则
`memory_at: null`
每条 memory(新的和已有的)都带 memory_at,表示它发生的时间。记忆**不一定按发生顺序写入**:
历史可能晚导入,导入还没处理完时用户也可能手动添加记忆。所以一条 new memory 可能**比已有记忆更早**。
表示时间未知。
不要编造时间顺序。
不要仅因为“看起来更新”就 supersede。
* 新旧**只看 memory_at**。"New memories" 只表示"正在写入",不表示"更近"。
* 更早的 memory 绝不能取代更新的 memory:不能 SUPERSEDE 一条更新的记忆,也不能用 UPDATE 拿更早的值覆盖更新的值。
* 如果 new memory 比与它冲突的已有记忆更早:保留已有记忆作为当前状态;把 new memory 以"过去状态"ADD
(用过去时,并带上时间,例如"2023 年用户住在上海"),没有新信息就不输出。
* `memory_at: null` 表示时间未知。未知**不算更新**:不要用 memory_at 为 null 的 memory 去 SUPERSEDE 一条有时间的记忆。
* 不要编造时间顺序。
* 不要仅因为“看起来更新”就 supersede。
---
+116 -12
View File
@@ -50,7 +50,7 @@ def is_canonical_state_node(node: Any) -> bool:
def _log_ops_detail(ops: List["ReconcileOp"]) -> None:
"""以单条 INFO(JSON list)输出 reconcile 的逐条结果,便于线上 debug。"""
"""以单条 DEBUG(JSON list)输出 reconcile 的逐条结果;含记忆正文,不进 INFO。"""
if not ops:
return
view = [
@@ -68,7 +68,7 @@ def _log_ops_detail(ops: List["ReconcileOp"]) -> None:
}
for o in ops
]
logger.info(
logger.debug(
f"[reconciler] ops detail: {json.dumps(view, ensure_ascii=False, default=str)}"
)
@@ -221,6 +221,8 @@ Rules:
* merge losslessly
* preserve all useful information
* do NOT create duplicates
* if the two disagree on a value, keep the value from the memory with the later
`memory_at`; never let an older memory overwrite a newer value
### Same occurrence vs. another occurrence
@@ -255,6 +257,7 @@ Use ONLY when:
* the new memory contradicts an existing memory
* both cannot be true at the same time
* the old memory would now give the wrong current-state answer
* the new memory's `memory_at` is the same as or LATER than the existing memory's `memory_at`
Examples:
@@ -268,6 +271,8 @@ Do NOT use for:
* compatible preferences
* wording differences
* semantic equivalence
* a new memory whose `memory_at` is EARLIER than the existing memory's — it is an
older state and must never replace a newer one (see "Time order")
Rules:
@@ -289,7 +294,8 @@ Rules:
### Folding several existing memories into one chain
If TWO OR MORE existing memories track the SAME dimension and the new memory is
the latest change on that dimension, you may fold them into a single evolution
the latest change on that dimension — its `memory_at` is later than every folded
memory's `memory_at` — you may fold them into a single evolution
chain. Use `memory_ids` (an ordered list, OLDEST → NEWEST) instead of a single
`memory_id`:
@@ -318,8 +324,7 @@ older memories' text into it; they remain queryable in the chain history.
* Separate dated occurrences stay separate ADDs; use `same_series` only for the
same subject and same concrete activity/event type.
* Keep each memory focused on one coherent topic.
* Do not invent chronology.
* `memory_at: null` means unknown time.
* Follow "Time order" below.
* **NEVER merge or supersede across different subjects.** The subject is whoever
the content is about: "The user ..." vs "John ..." vs "Mary ...". Memories about
different people are independent facts even when the sentence is otherwise
@@ -332,6 +337,27 @@ older memories' text into it; they remain queryable in the chain history.
---
# Time order
Every memory, new or existing, has `memory_at`: when it happened. Memories are
NOT always written in the order they happened — history can be imported late,
and a memory can be added while older history is still being processed. A new
memory can therefore be OLDER than an existing one.
* Decide which memory is newer ONLY by `memory_at`. "New memories" means "being
written now", not "more recent".
* An older memory must never replace a newer one: no SUPERSEDE of a newer
memory, and no UPDATE that overwrites a newer value with an older one.
* If a new memory is older than an existing memory it conflicts with, keep the
existing memory as the current state. ADD the new one as a past state, worded
in the past with its time (for example "In 2023 the user lived in Shanghai"),
or output nothing if it adds no information.
* `memory_at: null` means unknown time. Unknown is NOT newer: never SUPERSEDE a
memory that has a known `memory_at` with one whose `memory_at` is null.
* Do not invent chronology.
---
# Optional: `related`
On any op you MAY add a `related` array recording how the memory being written
@@ -441,6 +467,81 @@ def append_reconcile_structure_prompt(
return prompt + RECONCILE_STRUCTURE_METADATA_PROMPT_EN
def _as_aware(value: Any) -> Optional[datetime]:
"""memory_at as a comparable aware datetime; None when unknown or unreadable."""
if isinstance(value, str) and value:
try:
value = datetime.fromisoformat(value)
except (TypeError, ValueError):
return None
if not isinstance(value, datetime):
return None
# Stored nodes rebuild memory_at from epoch seconds as naive local time
# (MemoryNode.from_dict), so a naive value is local, not UTC.
return value if value.tzinfo is not None else value.astimezone()
def enforce_time_order(
ops: List["ReconcileOp"],
new_memory_at: Any,
existing: Dict[str, Any],
) -> List["ReconcileOp"]:
"""An older memory must never replace a newer one, whatever the model says.
Memories are not always written in the order they happened: history is
imported late, and a person can add a memory while older history is still
being processed. The prompt tells the model to compare `memory_at`; this
makes it hold. A SUPERSEDE or UPDATE aimed at an existing memory whose
`memory_at` is later than the memory being written is turned into an ADD,
so the newer memory stays the current state. Unknown times are not
compared. A content-less UPDATE (the legacy DELETE mapping) is dropped.
"""
written_at = _as_aware(new_memory_at)
if written_at is None:
return ops
kept: List[ReconcileOp] = []
for op in ops:
if op.op not in ("SUPERSEDE", "UPDATE"):
kept.append(op)
continue
targets = list(op.memory_ids) if op.memory_ids else [op.memory_id]
newer = [
target for target in targets
if target and (node := existing.get(target)) is not None
and (target_at := _as_aware(getattr(node, "memory_at", None))) is not None
and target_at > written_at
]
if not newer:
kept.append(op)
continue
logger.warning(
"[reconciler] %s blocked: memory_at %s is older than existing %s; "
"kept the newer memory",
op.op, written_at.isoformat(), ",".join(newer),
)
if not op.content:
continue
kept.append(ReconcileOp(
op="ADD",
content=op.content,
layer=op.layer,
owner=op.owner,
tags=list(op.tags),
tag_ners=list(op.tag_ners),
topic_paths=list(op.topic_paths),
event_facet=dict(op.event_facet),
topic_assignments=list(op.topic_assignments),
structure_references=list(op.structure_references),
topic_paths_provided=op.topic_paths_provided,
event_facet_provided=op.event_facet_provided,
topic_assignments_provided=op.topic_assignments_provided,
speculate=op.speculate,
related=list(op.related),
reason=f"time_order: older than {','.join(newer)}; {op.reason}"[:500],
))
return kept
def build_reconcile_prompt(
*,
new_memories: List[str],
@@ -1399,6 +1500,7 @@ class MemoryReconciler:
_tok_total += response.tokens_used
raw_response = response.content
ops = self._parse_ops(raw_response)
ops = enforce_time_order(ops, current_time, candidate_map)
ops = self._inherit_exact_input_structure(
ops, new_memories_with_meta,
)
@@ -1412,9 +1514,10 @@ class MemoryReconciler:
logger.warning(
f"[reconciler] RECONCILE JSON parse failed "
f"(attempt {_attempt+1}/{_MAX_PARSE_ATTEMPTS}), "
f"retrying: raw="
f"{raw_response[:200] if _log_llm_content_enabled() else '<redacted>'}"
f"retrying: raw({len(raw_response or '')} chars)"
)
if _log_llm_content_enabled():
logger.debug(f"[reconciler] unparsable raw={raw_response[:200]!r}")
recon_elapsed = (_time.perf_counter() - t_recon) * 1000
@@ -1426,8 +1529,8 @@ class MemoryReconciler:
error_code = "LLM_UNAVAILABLE"
elif raw_response.strip():
error_msg = (
f"RECONCILE JSON parse failed after {_n_attempts} attempts: "
f"{raw_response[:200] if _log_llm_content_enabled() else '<redacted>'}"
f"RECONCILE JSON parse failed after {_n_attempts} attempts "
f"({len(raw_response)} chars)"
)
error_code = "JSON_PARSE_FAILED"
else:
@@ -1704,7 +1807,8 @@ class MemoryReconciler:
except Exception as _re:
logger.debug(f"[reconciler] json_repair failed: {_re}")
if data is None:
logger.warning(f"[reconciler] failed to parse ops JSON: {text[:200]}")
logger.warning(f"[reconciler] failed to parse ops JSON ({len(text)} chars)")
logger.debug(f"[reconciler] unparsable ops output: {text[:200]!r}")
return []
if not isinstance(data, list):
@@ -1916,11 +2020,11 @@ class MemoryReconciler:
if clash is not None:
logger.warning(
f"[reconciler] memory_id {clash} already touched by '{touched[clash]}', "
f"current op '{op.op} ({op.reason[:60]})' dropped"
f"current op '{op.op}' dropped"
)
continue
for t in target_ids:
touched[t] = f"{op.op}({op.reason[:40]})"
touched[t] = op.op
validated.append(op)
return validated
+3 -1
View File
@@ -211,7 +211,9 @@ def parse_tool_calls_from_json(raw: Any) -> List[ToolCall]:
try:
args = json.loads(args_raw) if args_raw.strip() else {}
except json.JSONDecodeError:
logger.warning(f"[tools] failed to parse arguments as JSON: {args_raw[:120]}")
logger.warning(
f"[tools] failed to parse arguments as JSON ({len(args_raw)} chars)"
)
args = {}
elif isinstance(args_raw, dict):
args = args_raw
+77
View File
@@ -26,6 +26,8 @@ from __future__ import annotations
import asyncio
import logging
import re
import unicodedata
import uuid
import weakref
from dataclasses import dataclass
@@ -208,6 +210,76 @@ def _sanitize_kv(
return result
# ================================================================
# 占位符守卫(P0-1)
# ================================================================
# 弱模型(如 qwen3.5-2b)在用户什么都没说时,会把 schema 字段说明里的词当值
# 填进 basic_info:{"name": "用户"} / "The user" / "N/A"。每个占位符都会
# supersede 掉真实值,演化链变成 费天翔 → Fei → 用户 → The user。
# 规则:占位符永远不入库;读侧组装 KV 时占位符也不覆盖已有真实值。
_PLACEHOLDER_VALUES = frozenset({
"用户", "该用户", "本用户", "用户本人",
"the user", "the user's", "user", "a user", "this user",
"n/a", "na", "none", "null", "nil", "unknown", "unspecified",
"not provided", "not specified", "not mentioned",
"未提供", "未知", "不详", "未说明", "未提及", "无",
"-", "?",
})
# 内置字段的中文标签:弱模型也会把字段说明的中文词当值。
_FIELD_LABELS: Dict[str, tuple] = {
"name": ("姓名", "名字", "用户名", "用户姓名", "用户名字"),
"age": ("年龄", "用户年龄"),
"location": ("所在地", "地点", "城市", "用户所在地"),
"occupation": ("职业", "工作", "用户职业"),
"employer": ("雇主", "公司", "用户雇主"),
}
# 去掉标点/符号后再比较,让 "The user." / "用户。" / "N/A" 都命中。
# "-" / "?" 本身就是纯标点,剥完为空串,同样判为占位符。
_PUNCT_RE = re.compile(r"[\s\.,;:!?'\"`~()\[\]{}<>/\\|_\-\u3000-\u303f\uff00-\uff0f\uff1a-\uff20]+")
def _placeholder_key(value: str) -> str:
s = unicodedata.normalize("NFKC", str(value or "")).strip().lower()
s = s.replace("\u2019", "'")
s = _PUNCT_RE.sub(" ", s)
return " ".join(s.split())
_PLACEHOLDER_KEYS = frozenset(_placeholder_key(v) for v in _PLACEHOLDER_VALUES)
def is_placeholder_value(field_name: str, value: Any) -> bool:
"""value 是否为占位符(空 / 用户 / The user / N/A / 字段名本身 等)。
大小写不敏感、首尾空白与标点不敏感。数字(如 age=30)永远不是占位符。
"""
if value is None:
return True
if isinstance(value, bool):
return False
if isinstance(value, (int, float)):
return False
key = _placeholder_key(value)
if not key:
return True
if key in _PLACEHOLDER_KEYS:
return True
field_key = _placeholder_key(field_name)
if field_key and key in (field_key, field_key.replace("_", " ")):
return True
for label in _FIELD_LABELS.get(field_name, ()):
if key == _placeholder_key(label):
return True
return False
def _drop_placeholders(kv: Dict[str, Any]) -> Dict[str, Any]:
return {k: v for k, v in kv.items() if not is_placeholder_value(k, v)}
# ================================================================
# 公共入口:upsert_basic_profile
# ================================================================
@@ -287,6 +359,8 @@ async def _upsert_basic_profile_unlocked(
# L0 Basic Info 从输入、唯一性到落盘都只属于 user_id。
new_kv = _sanitize_kv(kv or {}, allowed_fields=allowed_fields)
# 单调性守卫:占位符永远不覆盖已有值,过滤后为空就不写新版本、不产生新链头。
new_kv = _drop_placeholders(new_kv)
if not new_kv:
return BasicProfileUpsertResult(
success=True, changed=False, reason="no valid fields after sanitize"
@@ -513,5 +587,8 @@ def _assemble_full_kv_from_nodes(
for k, v in kv.items():
if allowed_set is not None and k not in allowed_set:
continue
# 历史上已落盘的占位符版本(如 name=用户)不覆盖更早的真实值。
if is_placeholder_value(k, v):
continue
full_kv[k] = v
return full_kv
+1081 -91
View File
File diff suppressed because it is too large. Load diff
+4 -1
View File
@@ -1166,7 +1166,10 @@ class CodingMemoryExtractor:
obj = None
if not isinstance(obj, dict):
logger.warning(f"[coding-extract] failed to parse output: {text!r}")
logger.warning(
f"[coding-extract] failed to parse output ({len(text or '')} chars)"
)
logger.debug(f"[coding-extract] unparsable output: {text!r}")
return [], []
# "memories" 是双层化之前的 key,老 prompt 缓存 / 旧模型可能还这么回
+20 -7
View File
@@ -454,9 +454,10 @@ class WorkingFactReconciler:
workspace_id: Optional[str] = None,
branch: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
) -> Tuple[List[WorkingFact], List[FactReconcileOp]]:
"""
执行 reconcile + 持久化。
执行 reconcile + 持久化。memory_at 是这段对话的时间,记到新事实上。
Returns:
(真正落库的新事实, 执行过的 ops)。LLM 挂掉时退化为全 INSERT 不覆盖,
@@ -493,6 +494,7 @@ class WorkingFactReconciler:
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
)
logger.info(
f"[fact-reconcile] drafts={len(drafts)} inserted={len(facts)} "
@@ -568,9 +570,12 @@ class WorkingFactReconciler:
kept.append(fact)
if blocked:
logger.info(
f"[fact-reconcile] subject gate blocked {len(blocked)} candidate(s) "
f"for subject={draft.subject!r}: "
+ ", ".join(f"{f.fact_id}({f.subject!r})" for f in blocked)
f"[fact-reconcile] subject gate blocked {len(blocked)} candidate(s): "
+ ", ".join(f.fact_id for f in blocked)
)
logger.debug(
f"[fact-reconcile] subject gate: draft subject={draft.subject!r} "
f"blocked subjects={[f.subject for f in blocked]!r}"
)
result.append(kept)
return result
@@ -743,7 +748,10 @@ class WorkingFactReconciler:
obj = None
if not isinstance(obj, dict):
logger.warning(f"[fact-reconcile] failed to parse ops output: {text!r}")
logger.warning(
f"[fact-reconcile] failed to parse ops output ({len(text or '')} chars)"
)
logger.debug(f"[fact-reconcile] unparsable ops output: {text!r}")
return {}
ops_raw = obj.get("ops")
if not isinstance(ops_raw, list):
@@ -781,6 +789,7 @@ class WorkingFactReconciler:
workspace_id: Optional[str],
branch: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
) -> Tuple[List[WorkingFact], List[FactReconcileOp]]:
inserted: List[WorkingFact] = []
ops: List[FactReconcileOp] = []
@@ -801,6 +810,7 @@ class WorkingFactReconciler:
subject=draft.subject,
details=draft.details,
status="active",
memory_at=memory_at,
)
allowed = {c.fact_id for c in (candidates[idx] if idx < len(candidates) else [])}
@@ -831,8 +841,11 @@ class WorkingFactReconciler:
# 别的主体的状态不是。闸门关掉时这道复查一并跳过,否则闸门就没关干净。
if self.subject_gate and _subject_conflict(draft.subject, old.subject):
logger.warning(
f"[fact-reconcile] refused cross-subject SUPERSEDE: "
f"{old.fact_id}({old.subject!r}) by draft subject {draft.subject!r}"
f"[fact-reconcile] refused cross-subject SUPERSEDE: {old.fact_id}"
)
logger.debug(
f"[fact-reconcile] cross-subject: old subject={old.subject!r} "
f"draft subject={draft.subject!r}"
)
continue
try:
+21 -10
View File
@@ -195,8 +195,10 @@ async def classify_messages_scene(
result = _parse_scene_json(resp.content)
if result is None:
logger.warning(
f"[coding-judge] failed to parse classify_messages output: {resp.content!r}; defaulting to chat"
f"[coding-judge] failed to parse classify_messages output "
f"({len(resp.content or '')} chars); defaulting to chat"
)
logger.debug(f"[coding-judge] unparsable classify_messages output: {resp.content!r}")
await _store_coding_log(
cache, request_id=request_id, user_id=user_id, agent_id=agent_id,
session_id=session_id, step="CODING_JUDGE", prompt=prompt,
@@ -207,9 +209,8 @@ async def classify_messages_scene(
return "chat"
scene = result["scene"]
parsed = {**result, "is_coding": scene != "chat"}
logger.info(
f"[coding-judge] scene={scene} reason={result.get('reason', '')!r}"
)
logger.info(f"[coding-judge] scene={scene}")
logger.debug(f"[coding-judge] scene={scene} reason={result.get('reason', '')!r}")
await _store_coding_log(
cache, request_id=request_id, user_id=user_id, agent_id=agent_id,
session_id=session_id, step="CODING_JUDGE", prompt=prompt,
@@ -295,8 +296,10 @@ async def classify_queries_scene(
result = _parse_scene_json(resp.content)
if result is None:
logger.warning(
f"[coding-judge] failed to parse classify_queries output: {resp.content!r}; defaulting to chat"
f"[coding-judge] failed to parse classify_queries output "
f"({len(resp.content or '')} chars); defaulting to chat"
)
logger.debug(f"[coding-judge] unparsable classify_queries output: {resp.content!r}")
return "chat"
scene = result["scene"]
logger.info(f"[coding-judge] queries scene={scene}")
@@ -422,15 +425,18 @@ async def classify_and_rewrite_queries(
scene = _parse_scene_label(head)
if scene is None:
logger.warning(
f"[coding-judge] classify_and_rewrite unrecognized head={head!r}; falling back to chat"
f"[coding-judge] classify_and_rewrite unrecognized head "
f"({len(head)} chars); falling back to chat"
)
logger.debug(f"[coding-judge] classify_and_rewrite unrecognized head={head!r}")
return {"scene": "chat", "is_coding": False, "rewrite_query": rewrite, "ok": False}
is_coding = scene != "chat"
logger.info(
f"[coding-judge] classify_and_rewrite scene={scene} "
f"rewrite={rewrite!r}"
f"rewrite({len(rewrite)} chars)"
)
logger.debug(f"[coding-judge] classify_and_rewrite rewrite={rewrite!r}")
return {"scene": scene, "is_coding": is_coding, "rewrite_query": rewrite, "ok": True}
@@ -569,8 +575,9 @@ async def analyze_queries(
parsed_scene = _parse_scene_label(head)
if parsed_scene is None:
logger.warning(
f"[coding-judge] analyze_queries unrecognized head={head!r}; ok=False"
f"[coding-judge] analyze_queries unrecognized head ({len(head)} chars); ok=False"
)
logger.debug(f"[coding-judge] analyze_queries unrecognized head={head!r}")
return {**fallback, "since_raw": ""}
scene = parsed_scene
idx = 1
@@ -589,9 +596,10 @@ async def analyze_queries(
created_after = _since_to_created_after(since_raw, current_time)
logger.info(
f"[coding-judge] analyze scene={scene} rewrite={rewrite!r} "
f"[coding-judge] analyze scene={scene} rewrite({len(rewrite)} chars) "
f"since={since_raw!r} created_after={created_after}"
)
logger.debug(f"[coding-judge] analyze rewrite={rewrite!r}")
return {
"scene": scene, "is_coding": scene != "chat", "rewrite_query": rewrite,
"created_after": created_after, "since_raw": since_raw, "ok": True,
@@ -614,7 +622,10 @@ def _since_to_created_after(since_raw: str, current_time: str) -> Optional[float
return dt.timestamp()
except (ValueError, TypeError):
continue
logger.warning(f"[coding-judge] analyze_queries unparsable SINCE={since_raw!r}")
logger.warning(
f"[coding-judge] analyze_queries unparsable SINCE ({len(since_raw)} chars)"
)
logger.debug(f"[coding-judge] analyze_queries unparsable SINCE={since_raw!r}")
return None
+22 -10
View File
@@ -388,7 +388,10 @@ class CodingMemoryReconciler:
obj = None
if not obj or not isinstance(obj, dict):
logger.warning(f"[coding-reconcile] failed to parse ops output: {text!r}")
logger.warning(
f"[coding-reconcile] failed to parse ops output ({len(text or '')} chars)"
)
logger.debug(f"[coding-reconcile] unparsable ops output: {text!r}")
return []
ops_raw = obj.get("ops") or []
if not isinstance(ops_raw, list):
@@ -441,10 +444,10 @@ class CodingMemoryReconciler:
for op in ops:
if op.action == "ADD":
if op.draft_idx is None or not (0 <= op.draft_idx < len(drafts)):
logger.warning(f"[coding-reconcile] drop invalid ADD: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop invalid ADD: {_op_ref(op)}")
continue
if op.draft_idx in consumed:
logger.warning(f"[coding-reconcile] drop ADD on consumed draft: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop ADD on consumed draft: {_op_ref(op)}")
continue
d = drafts[op.draft_idx]
memory = self._draft_to_new_memory(d)
@@ -458,18 +461,18 @@ class CodingMemoryReconciler:
elif op.action == "UPDATE":
if op.draft_idx is None or not (0 <= op.draft_idx < len(drafts)):
logger.warning(f"[coding-reconcile] drop invalid UPDATE: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop invalid UPDATE: {_op_ref(op)}")
continue
if not op.target_experience_id:
logger.warning(f"[coding-reconcile] drop UPDATE missing target: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop UPDATE missing target: {_op_ref(op)}")
continue
if not self._target_was_offered(op, candidates_per_draft):
logger.warning(
f"[coding-reconcile] drop UPDATE outside authority scope: {op.to_dict()}"
f"[coding-reconcile] drop UPDATE outside authority scope: {_op_ref(op)}"
)
continue
if op.draft_idx in consumed:
logger.warning(f"[coding-reconcile] drop UPDATE on consumed draft: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop UPDATE on consumed draft: {_op_ref(op)}")
continue
d = drafts[op.draft_idx]
memory = self._draft_to_new_memory(d, experience_id=op.target_experience_id)
@@ -488,17 +491,17 @@ class CodingMemoryReconciler:
elif op.action == "DELETE":
if not op.target_experience_id:
logger.warning(f"[coding-reconcile] drop DELETE missing target: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop DELETE missing target: {_op_ref(op)}")
continue
if not self._target_was_offered(op, candidates_per_draft):
logger.warning(
f"[coding-reconcile] drop DELETE outside authority scope: {op.to_dict()}"
f"[coding-reconcile] drop DELETE outside authority scope: {_op_ref(op)}"
)
continue
# DELETE 需要知道 user_id;通过 draft_idx(如有)或第一个 draft 推断
user_id = self._infer_user_id(op, drafts)
if not user_id:
logger.warning(f"[coding-reconcile] drop DELETE no user_id context: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop DELETE no user_id context: {_op_ref(op)}")
continue
try:
ok = await self.store.delete(op.target_experience_id, user_id=user_id)
@@ -577,6 +580,7 @@ class CodingMemoryReconciler:
confidence=d.confidence,
source=d.source,
extraction_source=d.extraction_source,
memory_at=d.memory_at,
)
@@ -584,6 +588,14 @@ class CodingMemoryReconciler:
# Helpers
# ================================================================
def _op_ref(op: ReconcileOp) -> str:
"""op 的日志引用:只带动作与编号,不带 LLM 写的 reason。"""
return (
f"action={op.action} draft_idx={op.draft_idx} "
f"target={op.target_experience_id}"
)
def _truncate(s: str, n: int) -> str:
if not s:
return ""
+210 -11
View File
@@ -86,6 +86,7 @@ CREATE TABLE IF NOT EXISTS coding_memory_meta (
confidence REAL NOT NULL DEFAULT 0.7,
source TEXT NOT NULL DEFAULT 'default',
extraction_source TEXT NOT NULL DEFAULT 'auto_extract',
memory_at INTEGER,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
@@ -109,6 +110,7 @@ CREATE TABLE IF NOT EXISTS working_fact (
derived_experience_ids TEXT NOT NULL DEFAULT '[]',
status TEXT NOT NULL DEFAULT 'active',
fact_chain_summary TEXT NOT NULL DEFAULT '',
memory_at INTEGER,
created_at TEXT NOT NULL
);
-- 反查前驱(谁的 fact_link 指向我)走这个索引,写演变总结时要顺着链往回走
@@ -164,6 +166,8 @@ _ADDED_COLUMNS = {
"coding_memory_meta": {
"source_fact_ids": "TEXT NOT NULL DEFAULT '[]'",
"extraction_source": "TEXT NOT NULL DEFAULT 'auto_extract'",
# 对话时间,秒级 epoch(同 chat)。老库里的行是 NULL:那时没存过。
"memory_at": "INTEGER",
},
# 早期版本存过一列 fact_chain(机械累积的历代说法),已废弃:同样的信息顺着
# fact_link 反查就有。老库里那一列留着不管,有默认值不影响插入。
@@ -177,6 +181,7 @@ _ADDED_COLUMNS = {
"subject": "TEXT NOT NULL DEFAULT ''",
"details": "TEXT NOT NULL DEFAULT ''",
"source": "TEXT NOT NULL DEFAULT 'default'",
"memory_at": "INTEGER",
},
}
@@ -217,6 +222,21 @@ def _in_clause(column: str, values: Optional[List[str]]) -> Tuple[str, List[Any]
return f"{column} IN ({placeholders})", list(values)
def _source_where(
include_sources: Optional[List[str]], exclude_sources: Optional[List[str]],
) -> Tuple[str, List[Any]]:
"""Caller-source filter as SQL (``1`` when unfiltered); exclusion wins."""
clauses: List[str] = []
params: List[Any] = []
if include_sources:
clauses.append(f"source IN ({','.join('?' for _ in include_sources)})")
params.extend(include_sources)
if exclude_sources:
clauses.append(f"source NOT IN ({','.join('?' for _ in exclude_sources)})")
params.extend(exclude_sources)
return (" AND ".join(clauses) or "1"), params
def _scope_matches(
row_workspace_id: Optional[str],
row_branch: Optional[str],
@@ -275,6 +295,29 @@ def _parse_dt(s: Optional[str]) -> Optional[datetime]:
return None
def _from_epoch(value: Any) -> Optional[datetime]:
"""memory_at 列(秒级 epoch)→ datetime;NULL 或坏值返回 None。"""
if value is None:
return None
try:
return datetime.fromtimestamp(int(value))
except Exception:
return None
def _to_epoch(value: Optional[datetime]) -> Optional[int]:
return int(value.timestamp()) if isinstance(value, datetime) else None
def _later(a: Optional[datetime], b: Optional[datetime]) -> Optional[datetime]:
"""两个对话时间取较晚的;一边缺就取另一边。"""
if a is None:
return b
if b is None:
return a
return a if _to_epoch(a) >= _to_epoch(b) else b
def _row_to_memory(row: sqlite3.Row) -> CodingMemory:
"""SQLite row → CodingMemory"""
return CodingMemory(
@@ -293,6 +336,7 @@ def _row_to_memory(row: sqlite3.Row) -> CodingMemory:
confidence=row["confidence"],
source=row["source"],
extraction_source=row["extraction_source"],
memory_at=_from_epoch(row["memory_at"]),
created_at=_parse_dt(row["created_at"]),
updated_at=_parse_dt(row["updated_at"]),
)
@@ -319,6 +363,7 @@ def _row_to_fact(row: sqlite3.Row) -> WorkingFact:
# 老库刚 ADD COLUMN 时可能取不到这一列,用 keys() 兜一下
fact_chain_summary=(row["fact_chain_summary"] or ""
if "fact_chain_summary" in row.keys() else ""),
memory_at=_from_epoch(row["memory_at"]),
created_at=_parse_dt(row["created_at"]),
)
@@ -560,7 +605,11 @@ class CodingMemoryStore:
await self._upsert_vec_keys(memory)
logger.info(
f"[coding-store] inserted experience_id={memory.experience_id} "
f"task={memory.task!r} keys={1 + len(memory.search_keys)}"
f"keys={1 + len(memory.search_keys)}"
)
logger.debug(
f"[coding-store] inserted experience_id={memory.experience_id} "
f"task={memory.task!r}"
)
async def update(self, memory: CodingMemory) -> None:
@@ -582,6 +631,8 @@ class CodingMemoryStore:
memory.updated_at = datetime.utcnow()
if memory.created_at is None:
memory.created_at = prev.created_at
# 合并后的经验讲的是两段对话,对话时间取较晚的那段
memory.memory_at = _later(prev.memory_at, memory.memory_at)
await _run_in_pool(self._upsert_sql_sync, memory)
# 删旧 VDB
await self._delete_vec_keys_by_count(memory.experience_id, prev_key_count)
@@ -603,6 +654,86 @@ class CodingMemoryStore:
logger.info(f"[coding-store] deleted experience_id={experience_id}")
return True
async def edit_experience(
self,
experience_id: str,
*,
user_id: Optional[str] = None,
task: Optional[str] = None,
solution: Optional[str] = None,
search_keys: Optional[List[str]] = None,
) -> Optional[CodingMemory]:
"""调用方手动改一条经验(client.update 用)。返回改后的经验;不存在返回 None。
与 update() 的区别:update 是写入链路的覆盖式重写,带写授权校验(同
agent/session/workspace 才能改);这里是用户对已有条目的直接编辑,只动
给定的字段,其余原样保留,因此不做授权校验。
向量行只嵌 task + search_keys(solution 不进向量),所以只改 solution 时
不重嵌、不碰向量库;改了 task / search_keys 才删旧 key 行重写。
updated_at 总是刷新。
"""
prev = await self.get_by_id(experience_id, user_id=user_id)
if prev is None:
return None
updated = deepcopy(prev)
if task is not None:
updated.task = task
if solution is not None:
updated.solution = solution
if search_keys is not None:
updated.search_keys = list(search_keys)
updated.updated_at = datetime.utcnow()
await _run_in_pool(self._upsert_sql_sync, updated)
if updated.task != prev.task or updated.search_keys != prev.search_keys:
await self._delete_vec_keys_by_count(experience_id, 1 + len(prev.search_keys))
await self._upsert_vec_keys(updated)
logger.info(f"[coding-store] edited experience_id={experience_id}")
return updated
async def edit_fact(
self,
fact_id: str,
*,
statement: str,
user_id: Optional[str] = None,
) -> Optional[WorkingFact]:
"""调用方手动改一条事实的 statement。返回改后的事实;不存在返回 None。
statement 是事实层的检索文本,所以 active 事实会重嵌并覆盖它那一行向量
(node_id 就是 fact_id,upsert 即覆盖)。superseded 事实的向量行早已撤掉,
这里不把它加回去,否则会让过期结论重新占召回名额。
"""
prev = await self.get_fact(fact_id, user_id=user_id)
if prev is None:
return None
updated = deepcopy(prev)
updated.statement = statement
await _run_in_pool(self._upsert_fact_sql_sync, updated)
if updated.status == "active":
await self._upsert_fact_vec(updated)
logger.info(f"[coding-store] edited fact_id={fact_id}")
return updated
async def delete_fact(self, fact_id: str, user_id: Optional[str] = None) -> bool:
"""物理删除一条事实(SQLite 行 + 事实向量行),返回是否真有删到。
别处留下的引用(经验的 source_fact_ids、后继事实的 fact_link)不清:
召回侧本来就按存在性过滤死链(get_many / get_facts 查不到自然掉队)。
"""
prev = await self.get_fact(fact_id, user_id=user_id)
if prev is None:
logger.info(f"[coding-store] delete fact skipped (not found): {fact_id}")
return False
if self._fact_vec is not None:
try:
await self._fact_vec.delete(fact_id)
except Exception as e:
logger.debug(f"[coding-store] delete fact vec {fact_id} ignored: {e}")
await _run_in_pool(self._delete_fact_sql_sync, fact_id, prev.user_id)
logger.info(f"[coding-store] deleted fact_id={fact_id}")
return True
# ------------------------------------------------------------
# 读取
# ------------------------------------------------------------
@@ -687,29 +818,84 @@ class CodingMemoryStore:
branch: Optional[str] = None,
limit: int = 100,
offset: int = 0,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
) -> List["CodingMemory"]:
"""按 workspace_id(+可选 branch)纯过滤列出该 user 的完整 coding memory。
作用域规则见 _scope_where(与 search 路径共用同一套)。无语义排序,按
updated_at 倒序(最近修改优先)。
updated_at 倒序(最近修改优先)。include_sources / exclude_sources 按
source 列在 SQL 里过滤(排除优先),所以 LIMIT/OFFSET 在过滤之后生效。
"""
return await _run_in_pool(
self._list_by_workspace_sync, user_id, workspace_id, branch, limit, offset
self._list_by_workspace_sync, user_id, workspace_id, branch, limit, offset,
include_sources, exclude_sources,
)
def _list_by_workspace_sync(
self, user_id: str, workspace_id: str, branch: Optional[str],
limit: int, offset: int,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
) -> List["CodingMemory"]:
scope_sql, scope_params = _scope_where(workspace_id, branch)
source_sql, source_params = _source_where(include_sources, exclude_sources)
with self._lock:
assert self._conn is not None
sql = (f"SELECT * FROM coding_memory_meta WHERE user_id = ? AND {scope_sql} "
f"ORDER BY updated_at DESC LIMIT ? OFFSET ?")
params: List[Any] = [user_id] + scope_params + [limit, offset]
f"AND {source_sql} "
f"ORDER BY updated_at DESC, experience_id ASC LIMIT ? OFFSET ?")
params: List[Any] = ([user_id] + scope_params + source_params
+ [limit, offset])
cur = self._conn.execute(sql, params)
return [_row_to_memory(r) for r in cur.fetchall()]
async def stats(
self,
user_id: str,
workspace_id: Optional[str] = None,
branch: Optional[str] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
) -> Dict[str, Any]:
"""经验数、最新创建/修改时间、active 事实数。纯聚合,不读正文。
作用域与 list_by_workspace 相同:workspace_id 为空表示该 user 全部;
来源过滤同样与 list_by_workspace 一致,count 因此恒等于分页可翻到的总数。
"""
return await _run_in_pool(
self._stats_sync, user_id, workspace_id, branch,
include_sources, exclude_sources,
)
def _stats_sync(
self, user_id: str, workspace_id: Optional[str], branch: Optional[str],
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
) -> Dict[str, Any]:
scope_sql, scope_params = _scope_where(workspace_id, branch)
source_sql, source_params = _source_where(include_sources, exclude_sources)
with self._lock:
assert self._conn is not None
row = self._conn.execute(
f"SELECT COUNT(*) AS n, MAX(created_at) AS created, "
f"MAX(updated_at) AS updated FROM coding_memory_meta "
f"WHERE user_id = ? AND {scope_sql} AND {source_sql}",
[user_id] + scope_params + source_params,
).fetchone()
facts = self._conn.execute(
f"SELECT COUNT(*) AS n FROM working_fact "
f"WHERE user_id = ? AND status = 'active' AND {scope_sql} "
f"AND {source_sql}",
[user_id] + scope_params + source_params,
).fetchone()
return {
"count": int(row["n"] or 0),
"latest_created_at": _parse_dt(row["created"]) if row["created"] else None,
"latest_updated_at": _parse_dt(row["updated"]) if row["updated"] else None,
"facts": int(facts["n"] or 0),
}
# ------------------------------------------------------------
# 召回
# ------------------------------------------------------------
@@ -820,7 +1006,8 @@ class CodingMemoryStore:
fact.created_at = datetime.utcnow()
await _run_in_pool(self._upsert_fact_sql_sync, fact)
await self._upsert_fact_vec(fact)
logger.info(
logger.info(f"[coding-store] inserted fact_id={fact.fact_id}")
logger.debug(
f"[coding-store] inserted fact_id={fact.fact_id} "
f"statement={fact.statement!r}"
)
@@ -1049,8 +1236,8 @@ class CodingMemoryStore:
experience_id, user_id, agent_id, task, search_keys, solution,
boundary_envs, workspace_id, branch, session_id,
files, source_fact_ids, confidence, source, extraction_source,
created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
memory_at, created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(experience_id) DO UPDATE SET
user_id = excluded.user_id,
agent_id = excluded.agent_id,
@@ -1066,6 +1253,7 @@ class CodingMemoryStore:
confidence = excluded.confidence,
source = excluded.source,
extraction_source = excluded.extraction_source,
memory_at = excluded.memory_at,
updated_at = excluded.updated_at
""",
(
@@ -1076,6 +1264,7 @@ class CodingMemoryStore:
json.dumps(memory.files, ensure_ascii=False),
json.dumps(memory.source_fact_ids, ensure_ascii=False),
memory.confidence, memory.source, memory.extraction_source,
_to_epoch(memory.memory_at),
(memory.created_at or datetime.utcnow()).isoformat(),
(memory.updated_at or datetime.utcnow()).isoformat(),
),
@@ -1088,6 +1277,14 @@ class CodingMemoryStore:
(experience_id, user_id),
)
def _delete_fact_sql_sync(self, fact_id: str, user_id: str) -> None:
with self._lock:
assert self._conn is not None
self._conn.execute(
"DELETE FROM working_fact WHERE fact_id = ? AND user_id = ?",
(fact_id, user_id),
)
def _get_by_id_sync(
self, experience_id: str, user_id: Optional[str]
) -> Optional[CodingMemory]:
@@ -1208,8 +1405,8 @@ class CodingMemoryStore:
fact_id, user_id, agent_id, session_id, workspace_id, branch,
source, statement, files, subject, details,
fact_link, derived_experience_ids,
status, fact_chain_summary, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
status, fact_chain_summary, memory_at, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(fact_id) DO UPDATE SET
user_id = excluded.user_id,
agent_id = excluded.agent_id,
@@ -1224,7 +1421,8 @@ class CodingMemoryStore:
fact_link = excluded.fact_link,
derived_experience_ids = excluded.derived_experience_ids,
status = excluded.status,
fact_chain_summary = excluded.fact_chain_summary
fact_chain_summary = excluded.fact_chain_summary,
memory_at = excluded.memory_at
""",
(
fact.fact_id, fact.user_id, fact.agent_id,
@@ -1237,6 +1435,7 @@ class CodingMemoryStore:
json.dumps(fact.derived_experience_ids, ensure_ascii=False),
fact.status,
fact.fact_chain_summary or "",
_to_epoch(fact.memory_at),
(fact.created_at or datetime.utcnow()).isoformat(),
),
)
+14
View File
@@ -25,6 +25,11 @@ FactStatus = Literal["active", "superseded"]
MAX_FACT_CHAIN_ENTRIES = 8
def _epoch(value: Optional[datetime]) -> Optional[int]:
"""memory_at 对外是秒级 epoch,与 chat 一致。"""
return int(value.timestamp()) if isinstance(value, datetime) else None
# ================================================================
# 草稿(LLM extract 阶段输出)
# ================================================================
@@ -53,6 +58,8 @@ class CodingMemoryDraft:
files: List[str] = field(default_factory=list)
source: str = "default"
extraction_source: str = "auto_extract"
# 对话时间(SDK 注入,同 chat 的 memory_at)
memory_at: Optional[datetime] = None
def __post_init__(self) -> None:
from ..utils.source_filter import normalize_source
@@ -73,6 +80,7 @@ class CodingMemoryDraft:
"files": list(self.files),
"source": self.source,
"extraction_source": self.extraction_source,
"memory_at": _epoch(self.memory_at),
}
@@ -111,6 +119,9 @@ class CodingMemory:
extraction_source: str = "auto_extract"
# 时间
# memory_at 是对话时间(同 chat),created_at / updated_at 是写库时间。
# UPDATE 合并时取两者中较晚的那个对话时间,见 store.update。
memory_at: Optional[datetime] = None
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
@@ -135,6 +146,7 @@ class CodingMemory:
"confidence": self.confidence,
"source": self.source,
"extraction_source": self.extraction_source,
"memory_at": _epoch(self.memory_at),
"created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None,
}
@@ -249,6 +261,7 @@ class WorkingFact:
fact_chain_summary: str = ""
# 时间
memory_at: Optional[datetime] = None # 对话时间(同 chat),没传时是收到请求的时间
created_at: Optional[datetime] = None # 产生时间,新的优先
def __post_init__(self) -> None:
@@ -272,6 +285,7 @@ class WorkingFact:
"derived_experience_ids": list(self.derived_experience_ids),
"status": self.status,
"fact_chain_summary": self.fact_chain_summary,
"memory_at": _epoch(self.memory_at),
"created_at": self.created_at.isoformat() if self.created_at else None,
}
+8 -1
View File
@@ -18,6 +18,7 @@ CodingWriter - 写入端协调器
import logging
import time
import uuid
from datetime import datetime
from typing import Any, Dict, List, Optional
from ..agent.llm_provider import LLMProvider
@@ -79,9 +80,11 @@ class CodingWriter:
session_id: Optional[str] = None,
request_id: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
) -> Dict[str, Any]:
"""
working/coding 段写入端到端。返回响应 dict(兼容 client.add 返回结构):
working/coding 段写入端到端。memory_at 是对话时间,记到这次写出的经验和
事实上;不传则用收到请求的时间(与 chat 链路一致)。返回响应 dict(兼容 client.add 返回结构):
{
"success": True,
"scene": "productivity",
@@ -101,6 +104,8 @@ class CodingWriter:
"""
request_id = request_id or str(uuid.uuid4())
t0 = time.perf_counter()
if memory_at is None:
memory_at = datetime.now()
if not messages:
return self._empty_response(request_id, t0)
@@ -172,6 +177,7 @@ class CodingWriter:
# extractor 只负责内容;调用来源由 SDK 入口注入,不能交给 LLM 生成。
for draft in drafts:
draft.source = source
draft.memory_at = memory_at
# 3. 经验层 reconcile + 持久化
ops: List[ReconcileOp] = (
@@ -197,6 +203,7 @@ class CodingWriter:
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
)
experience_ids = [op.target_experience_id for op in ops
+15 -4
View File
@@ -80,7 +80,7 @@ def _count_tokens(text: str) -> int:
try:
import tiktoken
return len(tiktoken.get_encoding("cl100k_base").encode(text))
return len(tiktoken.get_encoding("cl100k_base").encode(text, disallowed_special=()))
except Exception:
return len(text) // _CHARS_PER_TOKEN_FALLBACK + 1
@@ -98,7 +98,8 @@ def _truncate_for_embedding(text: str, limit: int = _EMBED_MAX_TOKENS) -> str:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode(text)
# Input is ordinary user text, not a tokenizer control-token stream.
ids = enc.encode(text, disallowed_special=())
if len(ids) <= limit:
return text
dropped = len(ids) - limit
@@ -753,16 +754,22 @@ class EmbedService:
"""
打一条 INFO 级 embedding 请求日志(含 request_id / provider / model / api / 入参 / 出参)。
出参只含 vector 维度,不含向量数值。request_id 同时写入 message,
出参只含 vector 维度,不含向量数值。入参的 input 是待向量化的原文
(对话 / 记忆 / 查询),INFO 只记每条的字符数(input_chars),原文仅在
DEBUG 单独输出。request_id 同时写入 message,
并依赖 log formatter 的 %(request_id)s(setup_logging 后生效)。
"""
texts = request_payload.get("input")
info_request = {k: v for k, v in request_payload.items() if k != "input"}
if isinstance(texts, list):
info_request["input_chars"] = [len(str(t)) for t in texts]
payload = {
"request_id": get_request_id(),
"status": status,
"provider": self._embed_config.provider,
"model": self._embed_config.model,
"api": api_url,
"request": request_payload,
"request": info_request,
"response": response_payload,
}
try:
@@ -770,6 +777,10 @@ class EmbedService:
except Exception:
body = str(payload)
logger.info(f"[embed] {body}")
if texts is not None and logger.isEnabledFor(logging.DEBUG):
logger.debug(
f"[embed] input: {json.dumps(texts, ensure_ascii=False, default=str)}"
)
async def _embed_openai(
self, texts: List[str]
@@ -160,6 +160,28 @@ def near_dup_demote(
return out
# MMR 只在调用方请求的 top-k(search 的 limit)小于这个值时自动启用。
# mmr_select 的贪心是 O(k^2 * pool) 次 Jaccard,k 大时代价陡增,而大 k 本身已
# 足够容纳多样性,收益很小。权重仍由 MEMORY_HYBRID_V2_MMR_LAMBDA 控制。
MMR_MAX_TOP_K = 20
def mmr_applies(top_k: int, lambda_: float) -> bool:
"""本次检索是否走 MMR:λ 在有效区间内,且 top_k < MMR_MAX_TOP_K。"""
return 0.0 < float(lambda_) < 0.999 and int(top_k) < MMR_MAX_TOP_K
def near_dup_applies(top_k: int) -> bool:
"""本次检索是否走近重复降权:与 MMR 同一个 top-k 门槛。
near_dup_demote 让每个候选和此前所有"非重复"候选两两算 Jaccard,候选
互不相似时是 O(pool^2);池子随 limit 放大(约 max(3*limit, 30) + 关键词
命中),limit 达到 MMR_MAX_TOP_K 时代价同样陡增。大 k 的调用方(如记忆
面板翻页)要的是全部相关项,相似项也应照常出现,降权没有收益。
"""
return int(top_k) < MMR_MAX_TOP_K
def mmr_select(
hits: Sequence[Dict[str, Any]],
limit: int,
@@ -115,11 +115,16 @@ class ReconcileHybridRetriever:
fused_max = max(fused_all) if fused_all else 0.0
base = (
f"[reconcile-hybrid][{req}] pool={pool_size} q_terms={q_terms!r} "
f"[reconcile-hybrid][{req}] pool={pool_size} q_terms={len(q_terms)} "
f"raw_bm25 max={raw_max:.4f} nonzero={raw_nonzero}/{len(raw_bm25)} "
f"bm25_norm max={norm_max:.4f} nonzero={norm_nonzero}/{len(bm25_norms)} "
f"sem_max={sem_max:.4f} fused_max={fused_max:.4f} "
f"kept={kept}/{len(fused_all)} min_score={self._cfg.min_score} "
f"query({len(query_text or '')} chars)"
)
# query 是待对账的新记忆原文,词项与预览只进 DEBUG
logger.debug(
f"[reconcile-hybrid][{req}] q_terms={q_terms!r} "
f"query={q_preview!r} lemmatized={lem_preview!r}"
)
+5
View File
@@ -401,6 +401,11 @@ class ReadRequest:
intention_limit: int = 0 # Intention 路的 topk,默认 0 关闭
# 重排开关(按次生效)。False 时跳过全部重排阶段:本地 cross-encoder
# (recall.local_rerank_enabled) 与记忆关系图重排 (_apply_graph_rerank),
# 结果按召回分排序。走请求对象而非环境变量,并发请求互不影响。
rerank: bool = True
# 时间过滤
created_after: Optional[float] = None # Unix timestamp (float),只返回 gmt_created >= 此值的记忆
+2 -1
View File
@@ -692,9 +692,10 @@ class CrossDomainSweeper:
)
logger.info(
f"[sweeper] Created L6 core {core_id[:12]}: '{core_content[:60]}' "
f"[sweeper] Created L6 core {core_id[:12]} "
f"from {len(cluster_ids)} basics, evidence_inherited={len(inherited_evidence_ids)}"
)
logger.debug(f"[sweeper] L6 core {core_id[:12]} content={core_content[:60]!r}")
return 1, 0, [core_id]
except Exception as e:
+20 -6
View File
@@ -57,7 +57,9 @@ from ._retrieval.ner import (
memory_matches_time_hints,
)
from ._retrieval.diversity import (
near_dup_applies,
near_dup_demote,
mmr_applies,
mmr_select,
social_boilerplate_demote,
pool_df_demote,
@@ -1279,7 +1281,10 @@ class BaseReadPipeline(ReadPipeline):
SAME_SERIES_TIMELINE_ON = bool(
getattr(self.config.recall, "same_series_timeline_enabled", False)
)
LOCAL_RERANK_ON = bool(
# 按次的重排开关(search(rerank=False)):关掉本地 cross-encoder 与图重排,
# 结果按召回分排序。只读请求对象,不碰共享状态,并发请求互不影响。
RERANK_ON = bool(getattr(request, "rerank", True))
LOCAL_RERANK_ON = RERANK_ON and bool(
getattr(self.config.recall, "local_rerank_enabled", False)
)
LOCAL_RERANK_POOL_FACTOR = max(1, int(
@@ -1431,6 +1436,11 @@ class BaseReadPipeline(ReadPipeline):
MIN_SCORE = _resolve_normal_min_score(request.min_score)
final_limit = request.limit if request.limit > 0 else 10
# MMR 只在调用方的 top-k(final_limit,即 search 的 limit)< MMR_MAX_TOP_K
# 时启用;池子(LOCAL_RERANK 下是 final_limit 的若干倍)不参与判定。
MMR_ON = mmr_applies(final_limit, MMR_LAMBDA)
# 近重复降权同样是候选两两比较,用同一个 top-k 门槛。
NEAR_DUP_ON = near_dup_applies(final_limit)
vdb_sem_limit = max(final_limit * 3, 30)
vdb_kw_limit = max(final_limit * 5, 80)
graph_limit = max(
@@ -2462,7 +2472,7 @@ class BaseReadPipeline(ReadPipeline):
# 只把已选出的 top-N 换个顺序收益很小。
relation_paths: List[Dict[str, Any]] = []
_graph_stats: Dict[str, Any] = {}
if self._expansion_on():
if RERANK_ON and self._expansion_on():
_t_graph = datetime.now()
_pool_before = len(normal_results)
normal_results, relation_paths, _graph_stats = await self._apply_graph_rerank(
@@ -2525,7 +2535,7 @@ class BaseReadPipeline(ReadPipeline):
df_thr=POOL_DF_THR,
penalty=POOL_DF_PENALTY,
)
if NEAR_DUP_THR > 0 and NEAR_DUP_PENALTY > 0:
if NEAR_DUP_ON and NEAR_DUP_THR > 0 and NEAR_DUP_PENALTY > 0:
normal_results = near_dup_demote(
normal_results,
jaccard_thr=NEAR_DUP_THR,
@@ -2538,7 +2548,7 @@ class BaseReadPipeline(ReadPipeline):
))
if LOCAL_RERANK_ON else final_limit
)
if 0.0 < MMR_LAMBDA < 0.999 and len(normal_results) > _pre_rerank_limit:
if MMR_ON and len(normal_results) > _pre_rerank_limit:
normal_results = mmr_select(
normal_results,
_pre_rerank_limit,
@@ -2715,7 +2725,7 @@ class BaseReadPipeline(ReadPipeline):
reverse=True,
)
if (
0.0 < MMR_LAMBDA < 0.999
MMR_ON
and len(multi_seed_pool) > _pre_rerank_limit
):
multi_seed_pool = mmr_select(
@@ -3544,6 +3554,9 @@ class BaseReadPipeline(ReadPipeline):
"episode_capsule_render_mode": EPISODE_CAPSULE_RENDER_MODE,
"same_series_timeline_enabled": SAME_SERIES_TIMELINE_ON,
"local_rerank_enabled": LOCAL_RERANK_ON,
"rerank": RERANK_ON,
"mmr_applied": MMR_ON,
"near_dup_applied": NEAR_DUP_ON,
"local_rerank_model": getattr(
self.config.recall,
"local_rerank_model",
@@ -3574,8 +3587,9 @@ class BaseReadPipeline(ReadPipeline):
f"hidden_l0={len(hidden_l0_hits)} fwd_l6={len(forward_l6)} rev_l6={len(reverse_l6)} "
f"fused_l6={len(fused_l6)} graph_intent={len(graph_intention_hits)} "
f"returned={len(final_results)} "
f"query='{request.query[:80]}'"
f"query({len(request.query or '')} chars)"
)
logger.debug(f"[read/{self.VERSION}] query={request.query[:80]!r}")
response.success = True
+2 -1
View File
@@ -2516,7 +2516,8 @@ def _parse_operations_json(text: str) -> Optional[List[Dict[str, Any]]]:
return result
return None
except json.JSONDecodeError:
logger.warning(f"[S2-agent] JSON parse failed: {json_str[:200]}")
logger.warning(f"[S2-agent] JSON parse failed ({len(json_str)} chars)")
logger.debug(f"[S2-agent] unparsable output: {json_str[:200]!r}")
return None
+45 -7
View File
@@ -33,9 +33,28 @@ import threading
from datetime import datetime
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Any, Dict, Optional
from urllib.parse import parse_qs, urlparse
from .retrieval import AmbiguousMemoryIdError
logger = logging.getLogger("hy_memory.server")
def _namespace_args(values: Dict[str, Any]) -> Dict[str, Any]:
"""kind / variant from a query string (lists) or a JSON body (scalars).
They disambiguate an id that exists in more than one namespace (chat,
experience, fact; working or coding). Absent means "resolve it".
"""
args: Dict[str, Any] = {}
for key in ("kind", "variant"):
value = values.get(key)
if isinstance(value, list):
value = value[0] if value else None
if value:
args[key] = str(value)
return args
# Lazy-init global client
_client = None
_client_lock = None
@@ -148,7 +167,6 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
if path == "/api/v1/metrics":
minutes = 5
if "?" in self.path:
from urllib.parse import parse_qs, urlparse
qs = parse_qs(urlparse(self.path).query)
minutes = int(qs.get("minutes", ["5"])[0])
client = _get_client()
@@ -161,7 +179,8 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
if m:
memory_id = m.group(1)
client = _get_client()
result = client.get(memory_id)
query = parse_qs(urlparse(self.path).query)
result = client.get(memory_id, **_namespace_args(query))
if result is None:
_json_response(self, 404, {"error": "not_found", "memory_id": memory_id})
else:
@@ -170,6 +189,10 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
_json_response(self, 404, {"error": "not_found", "path": path})
except AmbiguousMemoryIdError as e:
_json_response(self, 409, {"error": "ambiguous_memory_id", "detail": str(e)})
except ValueError as e:
_json_response(self, 400, {"error": str(e)})
except Exception as e:
logger.error(f"[server] GET {path} error: {e}", exc_info=True)
_json_response(self, 500, {"error": str(e)})
@@ -213,7 +236,7 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
_json_response(self, 500, {"error": str(e)})
def do_PUT(self):
path = self.path.rstrip("/")
path = self.path.split("?")[0].rstrip("/")
body = _read_json_body(self)
if body is None:
_json_response(self, 400, {"error": "invalid JSON body"})
@@ -228,29 +251,43 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
_json_response(self, 400, {"error": "content is required"})
return
client = _get_client()
result = client.update(memory_id, content)
result = client.update(
memory_id,
content,
**_namespace_args(body),
**({"task": body["task"]} if body.get("task") else {}),
)
_json_response(self, 200, result)
return
_json_response(self, 404, {"error": "not_found", "path": path})
except AmbiguousMemoryIdError as e:
_json_response(self, 409, {"error": "ambiguous_memory_id", "detail": str(e)})
except ValueError as e:
_json_response(self, 400, {"error": str(e)})
except Exception as e:
logger.error(f"[server] PUT {path} error: {e}", exc_info=True)
_json_response(self, 500, {"error": str(e)})
def do_DELETE(self):
path = self.path.rstrip("/")
path = self.path.split("?")[0].rstrip("/")
try:
m = re.match(r"^/api/v1/memories/([^/]+)$", path)
if m:
memory_id = m.group(1)
client = _get_client()
result = client.delete(memory_id)
query = parse_qs(urlparse(self.path).query)
result = client.delete(memory_id, **_namespace_args(query))
_json_response(self, 200, result)
return
_json_response(self, 404, {"error": "not_found", "path": path})
except AmbiguousMemoryIdError as e:
_json_response(self, 409, {"error": "ambiguous_memory_id", "detail": str(e)})
except ValueError as e:
_json_response(self, 400, {"error": str(e)})
except Exception as e:
logger.error(f"[server] DELETE {path} error: {e}", exc_info=True)
_json_response(self, 500, {"error": str(e)})
@@ -455,7 +492,8 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
if key in body:
kwargs[key] = body[key]
for key in ("limit", "min_score", "schema_min_score", "schema_limit", "reader"):
for key in ("limit", "min_score", "schema_min_score", "schema_limit", "reader",
"rerank"):
if key in body:
kwargs[key] = body[key]
+4 -1
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
import asyncio
import concurrent.futures
import contextvars
import functools
from concurrent.futures import Executor
from typing import Any, Callable, Optional
@@ -39,7 +40,9 @@ async def run_in_executor_guarded(
async def to_thread_guarded(func: Callable[..., Any], *args: Any, **kwargs: Any) -> Any:
"""Guarded equivalent of ``asyncio.to_thread`` using the default pool."""
return await run_in_executor_guarded(None, func, *args, **kwargs)
context = contextvars.copy_context()
call = functools.partial(func, *args, **kwargs)
return await run_in_executor_guarded(None, context.run, call)
def concurrent_result_guarded(
+2 -1
View File
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
"""
Pipeline Log Writer — 环节级 JSONL 日志(始终开启,与 SQLite Trace 分离)。
Pipeline Log Writer — 环节级 JSONL 日志(默认开启,MEMORY_PIPELINE_LOG_ENABLED=false 关闭;
与 SQLite Trace 分离)。
目录: {log_dir}/{subdir}/{date}.log
每行: request_id, namespace, user_id, agent_id, step, prompt, content, parsed,
+71 -30
View File
@@ -1,16 +1,20 @@
# -*- coding: utf-8 -*-
"""
Pipeline 可观测性:Log(文件 JSONL,始终开启)与 Trace(SQLite pipeline_logs,可关闭)。
Pipeline 可观测性:Log(文件 JSONL,默认开启,可关闭)与 Trace(SQLite pipeline_logs,可关闭)。
- Log:运维 tail/grep,每个环节即时一行,不依赖后续步骤成功与否。
- Trace:Inspector / 调试,落库 pipeline_logs(需 MEMORY_CACHE_BACKEND=sqlite 等持久化后端)。
环境变量:
MEMORY_PIPELINE_LOG_ENABLED — Log 写 JSONL 文件,默认 true;false 时完全不写
MEMORY_PIPELINE_TRACE_ENABLED — Trace 落库,默认 true;OpenClaw C 端可设 false
MEMORY_MEMORY_OPERATIONS_ENABLED — memory_operations 审计表落库,默认 false
MEMORY_LOG_DIR — Log 目录(优先)
MEMORY_TRACE_LOG_DIR — Log 目录(兼容旧名)
Log 与 Trace 都关闭时不包装 store_pipeline_log,而是换成立即返回 True 的
空操作:热路径不做任何计时、脱敏或后台任务,也不写库。
子目录({log_dir}/{subdir}/{date}.log)由 PipelineLogWriter 从每条记录的
user_id 前缀(第一个 "__" 左侧)切分;无前缀回落 "default"。该前缀是
通用 namespace,可用于项目、部署或数据分桶。
@@ -94,6 +98,16 @@ def is_pipeline_trace_enabled() -> bool:
)
def is_pipeline_log_enabled() -> bool:
"""是否把 pipeline step 写成 JSONL 文件({log_dir}/<subdir>/<date>.log)。默认开启。"""
return os.getenv("MEMORY_PIPELINE_LOG_ENABLED", "true").strip().lower() in (
"true",
"1",
"yes",
"on",
)
def is_memory_operations_enabled() -> bool:
"""是否将知识库变动写入 memory_operations 审计表。默认关闭(false)。"""
return os.getenv("MEMORY_MEMORY_OPERATIONS_ENABLED", "false").lower() in (
@@ -125,7 +139,7 @@ def redact_content(value: str) -> str:
def resolve_pipeline_log_dir() -> str:
"""
JSONL pipeline log 根目录(始终使用,有默认值)。
JSONL pipeline log 根目录(MEMORY_PIPELINE_LOG_ENABLED 开启时使用,有默认值)。
优先级: MEMORY_LOG_DIR > MEMORY_TRACE_LOG_DIR > {MEMORY_DATA_DIR}/logs/pipeline
"""
@@ -228,6 +242,7 @@ def wrap_store_pipeline_log(
) -> Callable[..., Any]:
"""包装 cache.store_pipeline_log:计时同步、JSONL/DB 异步落盘,不堵主请求。
- log_writer 为 None 时不写 JSONL(MEMORY_PIPELINE_LOG_ENABLED=false)
- step-to-step elapsed_ms / event_at_ms 在调用瞬间打点(与业务墙钟对齐)
- event_at_ms:调用方传入非 0 值则直接复用(App 已在自己被调用的那一刻
同步打点,比这里更接近真实事件时刻);传 0 / 不传才用本地 time.time()
@@ -410,7 +425,8 @@ def wrap_store_pipeline_log(
try:
loop = asyncio.get_running_loop()
# JSONL 用线程池,与 loop 无亲和,留在当前 loop。
_track_task(loop.create_task(_persist_jsonl()))
if log_writer is not None:
_track_task(loop.create_task(_persist_jsonl()))
# DB trace 用 cache pool(可能 aiomysql,绑定 owner_loop)。若当前不在
# owner_loop(如 search 走独立 loop),派发回 owner_loop 执行,避免跨 loop。
if trace_enabled:
@@ -446,25 +462,26 @@ def wrap_store_pipeline_log(
)
except RuntimeError:
# 无 running loop(极少见):退化为同步写,保证不丢日志
try:
log_writer.write_step(
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
step=step,
prompt=prompt,
response=response,
parsed=parsed,
memory_ids=mem_ids,
elapsed_ms=step_elapsed_ms,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=total_tokens,
event_at_ms=event_at_ms,
)
except Exception:
pass
if log_writer is not None:
try:
log_writer.write_step(
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
step=step,
prompt=prompt,
response=response,
parsed=parsed,
memory_ids=mem_ids,
elapsed_ms=step_elapsed_ms,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=total_tokens,
event_at_ms=event_at_ms,
)
except Exception:
pass
if trace_enabled:
try:
result = original_store(
@@ -496,6 +513,11 @@ def wrap_store_pipeline_log(
return _store_pipeline_observability
async def _noop_store_pipeline_log(*args: Any, **kwargs: Any) -> bool:
"""Log 与 Trace 都关闭时的 store_pipeline_log:不计时、不写文件、不写库。"""
return True
def install_pipeline_observability(
cache: Any,
*,
@@ -503,6 +525,9 @@ def install_pipeline_observability(
) -> None:
"""给 cache 安装 JSONL + Trace 包装(fire-and-forget)。
MEMORY_PIPELINE_LOG_ENABLED=false 时不写 JSONL;若 Trace 也关闭,
store_pipeline_log 被换成空操作(不包装、不写库)。
在 cache 所属 loop(write loop / _LoopThread)上调用,本函数会捕获该 loop
作为 owner_loop 传给包装器:search 走独立 loop 后,从 search loop 触发的 DB
落库会被派发回这个 owner_loop,避免跨 loop 用 aiomysql 污染连接池。
@@ -514,15 +539,31 @@ def install_pipeline_observability(
except RuntimeError:
owner_loop = None
log_dir = resolve_pipeline_log_dir()
log_writer = PipelineLogWriter(log_dir)
log_enabled = is_pipeline_log_enabled()
trace_enabled = is_pipeline_trace_enabled()
logger.info(
f"[{log_label}] Pipeline log (JSONL) enabled: {log_dir}/<subdir>/<date>.log "
f"(subdir from user_id prefix before '__', else default) ; "
f"pipeline trace (DB)={'on' if trace_enabled else 'off'} ; "
f"persist=async(fire-and-forget)"
)
if not log_enabled and not trace_enabled:
logger.info(
f"[{log_label}] Pipeline log (JSONL) disabled ; pipeline trace (DB)=off ; "
f"store_pipeline_log=no-op"
)
cache.store_pipeline_log = _noop_store_pipeline_log
return
log_writer = None
if log_enabled:
log_dir = resolve_pipeline_log_dir()
log_writer = PipelineLogWriter(log_dir)
logger.info(
f"[{log_label}] Pipeline log (JSONL) enabled: {log_dir}/<subdir>/<date>.log "
f"(subdir from user_id prefix before '__', else default) ; "
f"pipeline trace (DB)={'on' if trace_enabled else 'off'} ; "
f"persist=async(fire-and-forget)"
)
else:
logger.info(
f"[{log_label}] Pipeline log (JSONL) disabled ; "
f"pipeline trace (DB)=on ; persist=async(fire-and-forget)"
)
cache.store_pipeline_log = wrap_store_pipeline_log(
cache.store_pipeline_log,
log_writer=log_writer,
+2 -1
View File
@@ -36,7 +36,8 @@ def count_tokens(text: str) -> int:
enc = _get_encoder()
if enc:
try:
return len(enc.encode(text))
# Quoted control-token spellings are ordinary input text.
return len(enc.encode(text, disallowed_special=()))
except Exception as e:
logger.debug(f"[token_count] encode failed, char fallback: {e}")
return max(1, len(text) // 4)
+14
View File
@@ -172,6 +172,20 @@ result = client.add_extracted(
该入口仍走 SDK reconcile/持久化;MemHub 只做管理面门禁与可选
`memory_id` 记录。
应用的「添加记忆」、MCP `MemHub_write`(经 broker `POST /memories` 以本机可信身份转发)
共用同一个本机 HTTP 合同,直接新增一条记忆:
```text
POST /api/v2/memories {request_id, text, scope: "all-apps"}
GET /api/v2/memories/additions/{request_id} # 该次新增的状态
```
两者返回同一回执(`evidence_id`、`text`、`scope`、`created_at`、`state`、`task_id`、
`memory_ids`、`error`)。同一 `request_id` 重放返回原回执;内容不同返回 409
`memory_add_conflict`,未知 `request_id` 返回 404 `memory_add_not_found`。新增走持久
写入队列的优先通道,最终调用 `add_extracted` + reconcile。带外部读取信封的请求
一律拒绝(403 `external_endpoint_denied`)。
## Search:搜索记忆
```python
+6 -4
View File
@@ -29,10 +29,12 @@ prompt/response、密钥、访问令牌或完整本地路径。结构化日志
文件。详细合同见
[`docs/contracts/diagnostic-logging.md`](docs/contracts/diagnostic-logging.md)。
用户显式把诊断级别切换为 `debug` 时,Core 会启用 HY pipeline trace,并将
`MEMORY_LOG_LLM_CONTENT` 设为 `true`。该模式会把完整模型 prompt/response 写入本地调试
日志,其中可能包含获授权的对话正文和模型输出;它只应用于有意识的故障排查,不应长期
开启,也不应在未经检查和脱敏时分享相关日志。切回 `info` 会停止后续正文记录,但不会
HY pipeline 日志与 trace 在任何级别下都关闭。HY SDK 的文本日志只写入
`hy-memory/governed-v2/logs/hy_memory.log`,其中的 WARNING 及以上同时写入 `core.log`;
SDK 不在 INFO 及以上级别记录对话或记忆正文。用户显式把诊断级别切换为 `debug` 时,Core 将
`MEMORY_LOG_LLM_CONTENT` 设为 `true`。该模式会把完整模型 prompt/response 写入
`hy_memory.log` 的 DEBUG 行,其中可能包含获授权的对话正文和模型输出;它只应用于有意识的
故障排查,不应长期开启,也不应在未经检查和脱敏时分享相关日志。切回 `info` 会停止后续正文记录,但不会
自动删除此前已写入的调试文件。
具体行为以当前源码、网络请求实现和发布说明为准。发现实现与本说明不一致时,请按
+7 -6
View File
@@ -18,13 +18,13 @@ If you run into a problem, please open an Issue with the version, platform,repro
## 发行与平台合同
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.5`。MemHub 源码版本为 `1.0.5`
(build 14),GitHub 仓库为
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.8`。MemHub 源码版本为 `1.0.5`
(build 36),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
`v1.0.5` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
`v1.0.5` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64 App 和 Runtime。
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
@@ -109,15 +109,16 @@ Node、Host Bridge 或模型。用户显式执行 `memhub install` 后,CLI 才
Python Core 和运行所需的 Python 依赖位于 MemHub 受管 Runtime,不写入系统 Python。
MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime 版本目录中。
`npm uninstall` 只移除 npm CLI;`memhub uninstall` 移除服务、受管 Runtime 和 CLI 链接,
默认保留 Memory。只有独立的 `memhub delete-data` 在 Runtime 停止且二次确认精确路径后
才会删除数据。
默认保留 Memory。只有独立的 `memhub delete-data` 在 Runtime 停止、显式加 `--yes` 并用
`--confirm <数据目录>` 再写一遍同一个目录后才会删除数据(不加参数只预览要删除的目录)。`memhub stop` 停止受管服务时也会关闭
登录自动启动,再次 `memhub start` 会恢复。
当前平台能力和安装边界以源码、CI 与发布说明为准。未完成真机验收的平台不会仅凭跨平台
测试宣称可用。
## Hy Memory
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.5`。SDK 配置、权限边界及 search
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.8`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+1 -1
View File
@@ -2352,7 +2352,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.5",
"versionInfo": "2.0.8",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+1 -1
View File
@@ -9,7 +9,7 @@ Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.5
## hy-memory 2.0.8
License: Apache-2.0
@@ -30,9 +30,7 @@ const REQUIRED_COMMAND_SURFACE = [
"sync",
"memories",
"profile",
"calls",
"notes",
"graph",
"graph-width",
"models",
"host",
"doctor",
@@ -50,19 +48,17 @@ const REQUIRED_COMMAND_SURFACE = [
"uninstall",
"delete-data",
"recall",
"setup",
"scan",
"progress",
"jobs",
"path",
"scan-roots",
"adapter",
"write",
"task",
"serve",
];
const REQUIRED_MARKERS = [
"backup_restore_confirmation_required",
"backup_restore_confirmation_mismatch",
"backup_restore_runtime_state_unknown",
"backup_restore_maintenance_identity_invalid",
"runtime_service_activation_unverified",
@@ -79,16 +75,18 @@ const REQUIRED_MARKERS = [
* exact literal the catalog usage string contains.
*/
const REQUIRED_SUBCOMMANDS = [
{ command: "calls", marker: "memhub calls clear" },
{ command: "notes", marker: "memhub notes add" },
{ command: "notes", marker: "memhub notes process" },
{ command: "memories", marker: "memhub memories add" },
{ command: "memories", marker: "memhub memories add-status" },
{ command: "memories", marker: "memhub memories update" },
{ command: "memories", marker: "memhub memories delete" },
{ command: "write", marker: "memhub write submit" },
{ command: "write", marker: "memhub write status" },
{ command: "graph", marker: "memhub graph width" },
{ command: "graph-width", marker: "memhub graph-width --decision" },
{ command: "models", marker: "memhub models add" },
{ command: "models", marker: "memhub models test" },
{ command: "models", marker: "memhub models edit" },
{ command: "models", marker: "memhub models activate" },
{ command: "sources", marker: "memhub sources connect" },
{ command: "jobs", marker: "memhub jobs start" },
{ command: "jobs", marker: "memhub jobs <pause|resume|cancel>" },
];
@@ -100,57 +98,30 @@ const REQUIRED_SUBCOMMANDS = [
* counterexamples on the packaged Runtime") had no machine-checkable probe.
* These run without `--lifecycle`, so they are safe against any package.
*/
const PROFILE_ID = "44444444-4444-4444-8444-444444444444";
const STOPPED_DESTRUCTIVE_PROBES = [
{ name: "memories_add", args: ["memories", "add", "synthetic memory"] },
{
name: "calls_clear_preview",
args: ["calls", "clear", "--event", "11111111-1111-4111-8111-111111111111"],
},
{
name: "calls_clear_apply",
args: [
"calls",
"clear",
"--event",
"11111111-1111-4111-8111-111111111111",
"--confirm",
],
},
{
name: "calls_clear_all_apply",
args: ["calls", "clear", "--all", "--confirm"],
},
{
name: "notes_process_apply",
args: [
"notes",
"process",
"33333333-3333-4333-8333-333333333333",
"--confirm",
],
name: "memories_delete_apply",
args: ["memories", "delete", "m-1", "--yes"],
},
{
name: "graph_width_apply",
args: ["graph", "width", "--decision", "keep_existing", "--confirm"],
args: ["graph-width", "--decision", "keep_existing"],
},
{
name: "models_activate_apply",
args: [
"models",
"activate",
"44444444-4444-4444-8444-444444444444",
"--confirm",
],
name: "graph_width_rebuild_apply",
args: ["graph-width", "--decision", "rebuild_with_new", "--yes"],
},
{ name: "models_activate_apply", args: ["models", "activate", PROFILE_ID] },
{ name: "models_test", args: ["models", "test", PROFILE_ID] },
{
name: "models_remove_apply",
args: ["models", "remove", PROFILE_ID, "--yes"],
},
{
name: "models_edit_apply",
args: [
"models",
"edit",
"44444444-4444-4444-8444-444444444444",
"--config-file",
"/nonexistent/patch.json",
"--confirm",
],
name: "sources_connect_import",
args: ["sources", "connect", "codex", "--import"],
},
];
@@ -119,6 +119,7 @@ describe("A19 batch execution", () => {
readerFor([]),
operations,
planned.notFound,
{ confirmOverrides: false },
);
// Exactly one request for two sources — the batch endpoint, per contract.
expect(post).toHaveBeenCalledTimes(1);
@@ -165,6 +166,7 @@ describe("A19 batch execution", () => {
readerFor([]),
operations,
planned.notFound,
{ confirmOverrides: false },
);
expect(document.state).toBe("partial");
const byId = Object.fromEntries(
@@ -206,6 +208,7 @@ describe("A19 batch execution", () => {
readerFor([]),
operations,
[],
{ confirmOverrides: false },
);
expect(document.sources.every((row) => row.status === "failed")).toBe(true);
expect(
@@ -249,6 +252,7 @@ describe("A19 batch execution", () => {
readerFor([]),
operations,
[],
{ confirmOverrides: false },
);
const busy = document.sources.find((row) => row.source_id === "cursor");
expect(document.state).toBe("partial");
@@ -288,6 +292,7 @@ describe("A19 batch execution", () => {
readerFor([]),
operations,
planned.notFound,
{ confirmOverrides: false },
);
const notFoundRow = document.sources.find(
(row) => row.source_id === "cursor",
@@ -296,3 +301,82 @@ describe("A19 batch execution", () => {
expect(document.not_found_source_ids).toContain("cursor");
});
});
describe("environment-override sources need --confirm-overrides", () => {
it("leaves them out and names them unless confirmed", async () => {
const planned = await planBatchImport(
readerFor([
found("codex"),
found("dsh", {
candidate_kind: "environment_override",
requires_override_confirmation: true,
}),
]),
);
if (planned.state !== "preview") throw new Error("expected preview");
const post = vi.fn(
async (_path: string, body: Record<string, unknown>) => ({
state: "committed" as const,
data: {
request_id: body.request_id,
accepted_source_ids: body.source_ids,
failed_sources: {},
jobs: [],
},
}),
);
const operations = { post } as unknown as CoreOperations;
const skipped = await executeBatchImport(
planned.candidates,
readerFor([]),
operations,
planned.notFound,
{ confirmOverrides: false },
);
expect(post.mock.calls[0]![1]).toMatchObject({
source_ids: ["codex"],
confirm_environment_overrides: false,
});
expect(skipped.override_required_source_ids).toEqual(["dsh"]);
expect(renderBatchScanText(skipped)).toContain("--confirm-overrides");
const confirmed = await executeBatchImport(
planned.candidates,
readerFor([]),
operations,
planned.notFound,
{ confirmOverrides: true },
);
expect(post.mock.calls[1]![1]).toMatchObject({
source_ids: ["codex", "dsh"],
confirm_environment_overrides: true,
});
expect(confirmed.override_required_source_ids).toBeUndefined();
});
it("refuses without a request when every detected source needs confirmation", async () => {
const planned = await planBatchImport(
readerFor([
found("dsh", {
candidate_kind: "environment_override",
requires_override_confirmation: true,
}),
]),
);
if (planned.state !== "preview") throw new Error("expected preview");
const post = vi.fn();
const document = await executeBatchImport(
planned.candidates,
readerFor([]),
{ post } as unknown as CoreOperations,
planned.notFound,
{ confirmOverrides: false },
);
expect(post).not.toHaveBeenCalled();
expect(document).toMatchObject({
state: "rejected",
code: "environment_override_confirmation_required",
});
expect(batchScanExitCode(document)).toBe(1);
});
});
+47 -9
View File
@@ -1,4 +1,5 @@
/** A19: one-shot scan/import of every *detected* source.
/** A19: one-shot connect/import of every *detected* source
* (`sources connect --all-detected --import`).
*
* The engine already takes many sources in a single `POST /api/v1/sources/connect`
* (`source_ids` is a 1..32 tuple) and answers per source. The CLI used to reject
@@ -37,13 +38,16 @@ export interface BatchSourceRow {
export interface BatchScanDocument {
schema_version: 1;
product: string;
command: "scan" | "init";
command: "sources connect";
state: "committed" | "partial" | "rejected" | "unknown" | "no_sources";
request_id: string | null;
accepted_source_ids: string[];
sources: BatchSourceRow[];
/** `location_state != found` rows are listed so the user can see what was skipped. */
not_found_source_ids: string[];
/** Detected sources whose location comes from an environment override and
* that were left out because `--confirm-overrides` was not given. */
override_required_source_ids?: string[];
code?: string;
}
@@ -118,12 +122,39 @@ export async function planBatchImport(reader: CoreReader): Promise<
/** Run the batch connect once and fold the per-source receipt into rows. */
export async function executeBatchImport(
candidates: readonly SourceConnectionCandidate[],
detected: readonly SourceConnectionCandidate[],
reader: CoreReader,
operations: CoreOperations,
notFound: readonly string[],
options: { confirmOverrides: boolean },
): Promise<BatchScanDocument> {
const requestId = randomUUID();
// An environment-override location is a choice the user has to confirm:
// without `--confirm-overrides` those sources are left out and named.
const skipped = options.confirmOverrides
? []
: detected
.filter((candidate) => candidate.requires_override_confirmation)
.map((candidate) => candidate.source_id);
const overrideFields =
skipped.length > 0 ? { override_required_source_ids: skipped } : {};
const candidates = detected.filter(
(candidate) => !skipped.includes(candidate.source_id),
);
if (candidates.length === 0) {
return {
schema_version: 1,
product: PRODUCT,
command: "sources connect",
state: "rejected",
request_id: null,
accepted_source_ids: [],
sources: [],
not_found_source_ids: [...notFound],
...overrideFields,
code: "environment_override_confirmation_required",
};
}
// Same shared builder as the single-source import and the Web console, so a
// batch connect cannot drift from either.
const request = buildSourceImportRequest({
@@ -141,9 +172,9 @@ export async function executeBatchImport(
: [[candidate.source_id, candidate.candidate_revision]],
),
),
confirmEnvironmentOverrides: candidates.some(
(candidate) => candidate.requires_override_confirmation,
),
confirmEnvironmentOverrides:
options.confirmOverrides &&
candidates.some((candidate) => candidate.requires_override_confirmation),
fullRescan: false,
onboarding: true,
enableSync: true,
@@ -154,7 +185,7 @@ export async function executeBatchImport(
return {
schema_version: 1,
product: PRODUCT,
command: "scan",
command: "sources connect",
state: result.state,
request_id: requestId,
accepted_source_ids: [],
@@ -168,6 +199,7 @@ export async function executeBatchImport(
failure_reason: result.code,
})),
not_found_source_ids: [...notFound],
...overrideFields,
code: result.code,
};
}
@@ -254,7 +286,7 @@ export async function executeBatchImport(
return {
schema_version: 1,
product: PRODUCT,
command: "scan",
command: "sources connect",
state:
failedCount === 0
? "committed"
@@ -270,6 +302,7 @@ export async function executeBatchImport(
accepted_source_ids: accepted,
sources: rows,
not_found_source_ids: [...notFound],
...overrideFields,
};
}
@@ -301,5 +334,10 @@ export function renderBatchScanText(document: BatchScanDocument): string {
: document.state === "partial"
? `部分完成:${document.accepted_source_ids.length} 个成功,其余见上方`
: `未完成:${document.code ?? "unknown"}`;
return `${summary}\n${lines.join("\n")}\n`;
const overrides = document.override_required_source_ids ?? [];
const hint =
overrides.length > 0
? `\n未接入(目录来自环境变量覆盖,需确认):${overrides.join("、")}\n确认目录无误后加 --confirm-overrides 重新运行。`
: "";
return `${summary}\n${lines.join("\n")}${hint}\n`;
}
@@ -0,0 +1,371 @@
// Contract fixes from the CLI review: exit codes, where JSON goes, the global
// `--data-dir`, and the write idempotency key.
import { mkdtemp, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
import { COMMANDS } from "./command-catalog.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
request: vi.fn(),
}));
vi.mock("axios", () => ({
default: {
create: vi.fn(() => transport),
get: transport.get,
post: transport.post,
request: transport.request,
isAxiosError: () => false,
},
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
inspectBackgroundHealth: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
const ATTACHED = {
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
};
beforeEach(() => {
vi.clearAllMocks();
for (const method of Object.values(transport)) method.mockReset();
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue(ATTACHED);
runtime.inspectBackgroundHealth.mockResolvedValue(null);
});
describe("--data-dir is a global option", () => {
it("points a Runtime-attached command at that data directory", async () => {
transport.get.mockResolvedValue({
status: 200,
data: { request_id: "r", profile: null, elapsed_ms: 1 },
});
const result = await capturedRun(
["profile", "--data-dir", "/synthetic/other", "--json"],
{ ONE_MEMORY_DATA_DIR: "/synthetic/default" },
);
expect(result.code).toBe(0);
expect(runtime.inspectRuntimeStatus).toHaveBeenCalledWith(
expect.objectContaining({ dataDir: "/synthetic/other" }),
);
});
it("is accepted by every command the catalog used to advertise it on", async () => {
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "stopped",
code: "runtime_not_running",
});
runtime.discoverRuntimeAttachment.mockResolvedValue(null);
for (const argv of [
["profile"],
["scan-roots", "list"],
["adapter", "list"],
["scan", "--source", "codex"],
["write", "status", "r-1"],
["backup", "create", "--target", "/synthetic/data/backups/a"],
["memories", "list"],
]) {
const result = await capturedRun([
...argv,
"--data-dir",
"/synthetic/other",
"--json",
]);
expect(result.code, argv.join(" ")).toBe(3);
}
});
it("refuses a relative or repeated --data-dir", async () => {
for (const argv of [
["profile", "--data-dir", "relative"],
["profile", "--data-dir", "/a", "--data-dir", "/b"],
["profile", "--data-dir"],
]) {
expect((await capturedRun(argv)).code).toBe(2);
}
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
});
});
describe("Core failures are exit 1, not usage errors", () => {
it("adapter probe reports a Core refusal with its code on stdout", async () => {
transport.request.mockResolvedValue({
status: 409,
data: { detail: "source_not_authorized" },
});
const result = await capturedRun([
"adapter",
"probe",
"--source",
"codex",
"--json",
]);
expect(result.code).toBe(1);
expect(result.stderr).toBe("");
expect(JSON.parse(result.stdout)).toMatchObject({
command: "adapter probe",
state: "failed",
code: "source_not_authorized",
exit_code: 1,
});
});
it("adapter probe prints readable text without --json", async () => {
transport.request.mockResolvedValue({
status: 200,
data: { source: "codex", available: true },
});
const result = await capturedRun(["adapter", "probe", "--source", "codex"]);
expect(result.code).toBe(0);
expect(result.stdout).toContain("来源 codex 的探测结果");
expect(result.stdout).toContain("available: true");
});
it("scan-roots add posts directly and reports a refusal as exit 1", async () => {
transport.request.mockResolvedValue({
status: 409,
data: { detail: "scan_root_conflict" },
});
const result = await capturedRun([
"scan-roots",
"add",
"--root",
"/synthetic/root",
"--source",
"codex",
"--json",
]);
expect(result.code).toBe(1);
expect(transport.request).toHaveBeenCalledWith(
expect.objectContaining({
method: "post",
url: "http://127.0.0.1:8765/api/v1/scan-roots",
data: { root: "/synthetic/root", source: "codex", recursive: true },
}),
);
expect(JSON.parse(result.stdout).code).toBe("scan_root_conflict");
});
it("the old `path add` spelling still works", async () => {
transport.request.mockResolvedValue({ status: 200, data: { ok: true } });
const result = await capturedRun([
"path",
"remove",
"--root",
"/synthetic/root",
"--yes",
]);
expect(result.code).toBe(0);
expect(transport.request.mock.calls[0][0]).toMatchObject({
method: "delete",
data: { root: "/synthetic/root" },
});
});
it("scan --source runs directly", async () => {
transport.post.mockResolvedValue({
status: 200,
data: { job_id: "j1", source_id: "codex", status: "pending" },
});
const result = await capturedRun(["scan", "--source", "codex", "--json"]);
expect(result.code).toBe(0);
expect(JSON.parse(result.stdout)).toMatchObject({
command: "scan",
state: "committed",
scan: { job_id: "j1" },
});
// A scan without a source is a usage error, not a preview.
expect((await capturedRun(["scan"])).code).toBe(2);
});
});
describe("write submit keeps its idempotency key", () => {
async function messagesFile(): Promise<string> {
const directory = await mkdtemp(path.join(tmpdir(), "memhub-write-"));
const file = path.join(directory, "messages.json");
await writeFile(file, JSON.stringify([{ role: "user", content: "hi" }]));
return file;
}
it("generates a request id, sends it and prints it with a retry hint", async () => {
transport.post.mockImplementation(
async (_url: string, body: { request_id: string }) => ({
status: 200,
data: {
request_id: body.request_id,
state: "accepted",
created_at: "T",
updated_at: "T",
},
}),
);
const result = await capturedRun([
"write",
"submit",
"--session",
"s1",
"--messages-file",
await messagesFile(),
]);
expect(result.code).toBe(0);
const sent = transport.post.mock.calls[0][1].request_id as string;
expect(sent).toMatch(/^[0-9a-f-]{36}$/);
expect(result.stdout).toContain(`request_id: ${sent}`);
expect(result.stdout).toContain(`--request-id ${sent}`);
});
it("reports a 409 conflict as JSON on stdout with the request id", async () => {
transport.post.mockResolvedValue({
status: 409,
data: { detail: "memory_write_conflict" },
});
const result = await capturedRun([
"write",
"submit",
"--session",
"s1",
"--messages-file",
await messagesFile(),
"--request-id",
"req-1",
"--json",
]);
expect(result.code).toBe(1);
expect(result.stderr).toBe("");
expect(JSON.parse(result.stdout)).toMatchObject({
command: "write submit",
state: "rejected",
request_id: "req-1",
code: "memory_write_conflict",
});
});
it("write status of an unknown id is exit 1 with state not_found", async () => {
transport.get.mockResolvedValue({ status: 404, data: {} });
const error = Object.assign(new Error("404"), {
response: { status: 404 },
});
transport.get.mockRejectedValue(error);
const result = await capturedRun(["task", "--id", "nope", "--json"]);
// `axios.isAxiosError` is mocked false, so the read error surfaces as a
// failed read; either way it is exit 1, never a silent exit 0.
expect(result.code).toBe(1);
});
});
describe("doctor and ui use exit 3 for a stopped Runtime", () => {
it("doctor exits 3 when the Runtime is stopped", async () => {
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "stopped",
code: "runtime_not_running",
});
const result = await capturedRun(["doctor", "--json"]);
expect(result.code).toBe(3);
expect(JSON.parse(result.stdout).schema_version).toBe(1);
});
});
describe("catalog exit codes", () => {
it("lists exit 2 for every command, since every parser can reject", () => {
for (const command of COMMANDS) {
expect(command.exit_codes, command.name).toContain(2);
}
});
it("lists exit 3 for every command that attaches to the Runtime", () => {
for (const name of [
"status",
"doctor",
"init",
"sources",
"sync",
"progress",
"jobs",
"models",
"memories",
"profile",
"graph-width",
"host",
"recall",
"scan-roots",
"adapter",
"scan",
"write",
"backup",
"ui",
"onboarding",
]) {
const command = COMMANDS.find((entry) => entry.name === name);
expect(command?.exit_codes, name).toContain(3);
}
});
it("does not claim exit 3 or 4 for lifecycle commands that never return them", () => {
for (const name of ["start", "stop", "restart"]) {
const command = COMMANDS.find((entry) => entry.name === name);
expect(command?.exit_codes).toEqual([0, 1, 2]);
}
});
it("says stop turns login auto-start off for a managed service", () => {
const stop = COMMANDS.find((entry) => entry.name === "stop");
expect(stop?.summary).toContain("登录时也不会自动启动");
});
});
async function capturedRun(
arguments_: string[],
environment: NodeJS.ProcessEnv = { ONE_MEMORY_DATA_DIR: "/synthetic/data" },
) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return { code: await run(arguments_, environment), stdout, stderr };
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
+317 -22
View File
@@ -1,8 +1,16 @@
import { mkdtemp, readdir, readFile, realpath, rm } from "node:fs/promises";
import {
mkdir,
mkdtemp,
readdir,
readFile,
realpath,
rm,
} from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import {
claimRuntimeServiceForApp,
clearSdkMigrationBlock,
readRuntimeServiceOwner,
recordSdkMigrationBlock,
resolveRuntimeCliConfig,
@@ -922,6 +930,38 @@ describe("public One Memory CLI", () => {
target_path: "/tmp/install/current/bin/one-memory",
};
it("says the data was restored when a failed migration put it back", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce(presentCliLink);
runtimeMocks.runSdkMigration.mockResolvedValueOnce({
state: "deferred",
code: "sdk_migration_memory_unreadable",
dataRestored: true,
} as never);
const streams = captureStreams();
let code: number;
try {
code = await run(
upgradeArguments,
{},
{ waitForRuntimeReady: async () => true },
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
const failure = JSON.parse(streams.stderr());
expect(failure).toMatchObject({
code: "sdk_migration_memory_unreadable",
data_restored: true,
log_file: "/tmp/custom data/logs/sdk-migration.log",
});
expect(failure.message).toContain("记忆数据已恢复到升级前的状态。");
expect(failure.message).toContain(
"/tmp/custom data/logs/sdk-migration.log",
);
});
it("migrates the home with the new Core before restarting the Runtime", async () => {
vi.clearAllMocks();
runtimeMocks.planCliLink.mockResolvedValueOnce(presentCliLink);
@@ -1971,16 +2011,11 @@ describe("stopped Runtime is exit 3, not a usage error", () => {
["path", "list", "--json"],
["adapter", "list", "--json"],
["task", "--id", "nope", "--json"],
[
"write",
"submit",
"--session",
"s",
"--messages-file",
"/tmp/messages.json",
"--json",
],
["backup", "create", "--target", "/tmp/x", "--yes", "--json"],
["write", "status", "nope", "--json"],
["backup", "create", "--target", "/tmp/x", "--json"],
["scan-roots", "add", "--root", "/tmp/x", "--json"],
["scan", "--source", "codex", "--json"],
["onboarding", "reset", "--json"],
])("%s ... --json exits 3 with runtime_not_running", async (...argv) => {
const streams = captureStreams();
let code: number;
@@ -1990,7 +2025,9 @@ describe("stopped Runtime is exit 3, not a usage error", () => {
streams.restore();
}
expect(code!).toBe(3);
const document = JSON.parse(streams.stderr());
// A result, not an error: on stdout like every other --json document.
expect(streams.stderr()).toBe("");
const document = JSON.parse(streams.stdout());
expect(document).toMatchObject({
code: "runtime_not_running",
exit_code: 3,
@@ -1999,6 +2036,48 @@ describe("stopped Runtime is exit 3, not a usage error", () => {
expect(document.status).not.toBe("usage_error");
});
it("write submit validates its messages file before attaching, then exits 3", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "memhub-write-"));
const file = path.join(directory, "messages.json");
await import("node:fs/promises").then(({ writeFile }) =>
writeFile(file, JSON.stringify([{ role: "user", content: "hi" }])),
);
const streams = captureStreams();
let missing: number;
let code: number;
try {
missing = await run(
[
"write",
"submit",
"--session",
"s",
"--messages-file",
path.join(directory, "absent.json"),
"--json",
],
{},
);
code = await run(
[
"write",
"submit",
"--session",
"s",
"--messages-file",
file,
"--json",
],
{},
);
} finally {
streams.restore();
await rm(directory, { recursive: true, force: true });
}
expect(missing!).toBe(2);
expect(code!).toBe(3);
});
it.each(["pause", "resume"] as const)(
"parses sync %s as the direct sync-set command",
async (action) => {
@@ -2134,6 +2213,94 @@ describe("lifecycle commands report a migration block", () => {
expect(output.value()).not.toContain("runtime_not_ready_within_budget");
});
it("restart on a frozen home retries the migration in the foreground first", async () => {
const dataDir = await mkdtemp(path.join(tmpdir(), "memhub-frozen-"));
await recordSdkMigrationBlock(
dataDir,
"sdk_migration_memory_unreadable",
"desktop",
{ frozen: { service_was_enabled: true }, data_restored: true },
);
const controls = {
...blockedControls(),
migrateSdkData: vi.fn(async () => ({
state: "deferred" as const,
code: "sdk_migration_memory_unreadable",
dataRestored: true,
})),
};
const output = captureStdout();
let code: number;
try {
code = await run(
["restart", "--json"],
{ ONE_MEMORY_DATA_DIR: dataDir },
controls,
);
} finally {
output.restore();
await rm(dataDir, { recursive: true, force: true });
}
expect(code!).toBe(1);
expect(controls.migrateSdkData).toHaveBeenCalledWith(
expect.objectContaining({ dataDir, caller: "cli-restart" }),
);
// Still frozen: the service is not started to retry on its own.
expect(controls.restartManagedRuntime).not.toHaveBeenCalled();
expect(controls.ensureRuntime).not.toHaveBeenCalled();
expect(JSON.parse(output.value())).toMatchObject({
command: "restart",
state: "blocked",
code: "sdk_migration_memory_unreadable",
data_restored: true,
});
});
it("restart on a frozen home restarts the Runtime once the retry succeeds", async () => {
const dataDir = await mkdtemp(path.join(tmpdir(), "memhub-frozen-"));
await recordSdkMigrationBlock(
dataDir,
"sdk_migration_memory_unreadable",
"desktop",
{ frozen: { service_was_enabled: false } },
);
const order: string[] = [];
const migrateSdkData = vi.fn(async () => {
order.push("migrate");
await clearSdkMigrationBlock(dataDir);
return { state: "cutover" as const };
});
const output = captureStreams();
let code: number;
try {
code = await run(
["restart", "--json"],
{ ONE_MEMORY_DATA_DIR: dataDir },
{
migrateSdkData,
waitForRuntimeReady: vi.fn(async () => true),
restartManagedRuntime: vi.fn(async () => {
order.push("restart");
return "restarted" as const;
}),
stopRuntime: vi.fn(async () => "stopped" as const),
ensureRuntime: vi.fn(async () => {
order.push("restart");
return { pid: 1 } as never;
}),
},
);
} finally {
output.restore();
await rm(dataDir, { recursive: true, force: true });
}
expect(code!).toBe(0);
expect(migrateSdkData).toHaveBeenCalledWith(
expect.objectContaining({ caller: "cli-restart" }),
);
expect(order).toEqual(["migrate", "restart"]);
});
it("restart in text mode names the log file and GitHub issues", async () => {
const dataDir = await blockedHome();
const output = captureStdout();
@@ -2583,7 +2750,8 @@ describe("ui wires the Core session contract without token leakage", () => {
} finally {
output.restore();
}
expect(code!).toBe(4);
// A stopped Runtime is exit 3, like every other command.
expect(code!).toBe(3);
const document = JSON.parse(output.value());
expect(document).toMatchObject({
schema_version: 1,
@@ -2612,6 +2780,120 @@ describe("ui wires the Core session contract without token leakage", () => {
// refuse while the Runtime owns the data dir, stop+call Core from a
// maintenance process, and never report success from a malformed response.
describe("backup restore drives the Core managed-restore contract", () => {
it("previews without --yes, exits 0 and starts nothing", async () => {
const maintenance = vi.fn();
const output = captureStdout();
let code: number;
try {
code = await run(
["backup", "restore", "--source", "/tmp/x", "--json"],
{},
{ runMaintenanceRestore: maintenance },
);
} finally {
output.restore();
}
expect(code!).toBe(0);
expect(maintenance).not.toHaveBeenCalled();
expect(JSON.parse(output.value())).toMatchObject({
command: "backup restore",
state: "preview",
target: "/tmp/x",
applied: false,
next_step: "memhub backup restore --source /tmp/x --yes --confirm /tmp/x",
});
});
it("deletes data only when --confirm repeats the data directory", async () => {
const home = await mkdtemp(path.join(tmpdir(), "memhub-delete-home-"));
const dataDir = path.join(home, ".one-memory");
await mkdir(path.join(dataDir, "logs"), { recursive: true });
const deleteData = async (...extra: string[]) => {
const output = captureStdout();
try {
const code = await run(
["delete-data", "--data-dir", dataDir, ...extra, "--json"],
{ HOME: home },
);
return { code, document: JSON.parse(output.value()) };
} finally {
output.restore();
}
};
try {
const refused = await deleteData("--yes");
expect(refused.code).toBe(4);
expect(refused.document).toMatchObject({
state: "blocked",
code: "data_deletion_confirmation_required",
});
expect(await readdir(dataDir)).toEqual(["logs"]);
const deleted = await deleteData("--yes", "--confirm", dataDir);
expect(deleted.code).toBe(0);
expect(await readdir(dataDir).catch(() => [])).toEqual([]);
} finally {
await rm(home, { recursive: true, force: true });
}
});
it("refuses --yes without repeating the backup path", async () => {
const maintenance = vi.fn();
const output = captureStdout();
let code: number;
try {
code = await run(
["backup", "restore", "--source", "/tmp/x", "--yes", "--json"],
{},
{ runMaintenanceRestore: maintenance },
);
} finally {
output.restore();
}
expect(code!).toBe(4);
expect(maintenance).not.toHaveBeenCalled();
expect(JSON.parse(output.value())).toMatchObject({
state: "rejected",
code: "backup_restore_confirmation_required",
next_step:
"恢复需要再写一遍备份路径:memhub backup restore --source /tmp/x --yes --confirm /tmp/x",
});
});
it("restores with --yes and the repeated backup path", async () => {
const maintenance = vi.fn(async () => ({
state: "restored" as const,
result: {
state: "restored" as const,
restoredBytes: 1,
previousBackupPath: "/tmp/pre.sqlite3",
sourceSchemaVersion: 34,
},
}));
const output = captureStdout();
let code: number;
try {
code = await run(
[
"backup",
"restore",
"--source",
"/tmp/x",
"--yes",
"--confirm",
"/tmp/x",
"--json",
],
{},
{ runMaintenanceRestore: maintenance },
);
} finally {
output.restore();
}
expect(code!).toBe(0);
expect(maintenance).toHaveBeenCalledTimes(1);
});
it("rejects while the Runtime is running, with a distinct exit code", async () => {
runtimeMocks.inspectRuntimeStatus.mockResolvedValueOnce({
state: "running",
@@ -2728,7 +3010,7 @@ describe("backup restore drives the Core managed-restore contract", () => {
});
});
it("requires --confirm to repeat the exact source path", async () => {
it("refuses a legacy --confirm that does not repeat the source path", async () => {
const maintenance = vi.fn();
const output = captureStdout();
let code: number;
@@ -2756,7 +3038,7 @@ describe("backup restore drives the Core managed-restore contract", () => {
command: "backup",
verb: "restore",
state: "rejected",
code: "backup_restore_confirmation_required",
code: "backup_restore_confirmation_mismatch",
});
});
@@ -2908,7 +3190,7 @@ describe("init onboarding", () => {
// Boolean flags were idempotent, but accepting them twice hid a duplicated
// argv fragment that could also have duplicated a value-bearing flag.
it.each(["--import", "--full-rescan", "--confirm-overrides", "--dry-run"])(
it.each(["--import", "--confirm-overrides"])(
"refuses a repeated init %s flag",
async (flag) => {
expect(await run(["init", "--source", "codex", flag, flag], {})).toBe(2);
@@ -2981,12 +3263,25 @@ describe("init onboarding", () => {
},
);
it("accepts --dry-run together with --import", async () => {
// Parsing must accept the pair; it stops at the Runtime check, which is
// not a usage failure.
expect(
await run(["init", "--source", "codex", "--import", "--dry-run"], {}),
).not.toBe(2);
// `--dry-run` and `--full-rescan` never did anything (P0-2/P0-3): they are
// gone rather than accepted as no-ops.
it.each(["--dry-run", "--full-rescan"])(
"refuses the removed no-op %s flag",
async (flag) => {
expect(
await run(["init", "--source", "codex", "--import", flag], {}),
).toBe(2);
},
);
it("still parses the old `init --source --import` spelling", async () => {
// It stops at the Runtime check (exit 3), which is not a usage failure.
const streams = captureStreams();
try {
expect(await run(["init", "--source", "codex", "--import"], {})).toBe(3);
} finally {
streams.restore();
}
});
});
+1664 -2042
View File
File diff suppressed because it is too large. Load diff
+226 -195
View File
@@ -58,6 +58,12 @@ const CONFIRM_OPTION: CommandOption = {
flag: "--yes",
summary: "确认执行;不加该参数时只输出计划,不做任何改动。",
};
/** The one confirmation convention, for irreversible operations only. */
const DESTRUCTIVE_CONFIRM_OPTION: CommandOption = {
flag: "--yes",
summary:
"确认这次不可逆操作;不加时只说明将要做什么,不做任何改动,exit 0。(旧的 --confirm 仍作为别名接受。)",
};
const JSON_OPTION: CommandOption = {
flag: "--json",
summary: "以 JSON 输出到 stdout,供脚本与 Agent 消费。",
@@ -70,21 +76,6 @@ const INTERVAL_OPTION: CommandOption = {
flag: "--interval <seconds>",
summary: "刷新间隔,默认 5 秒,范围 1..300;必须与 --watch 一起使用。",
};
const PLAN_FILE_OPTION: CommandOption = {
flag: "--plan-file <path>",
summary: "读取先前 --json 保存的有界计划文件。",
};
const EXPLICIT_CONFIRM_OPTION: CommandOption = {
flag: "--confirm",
summary: "确认执行计划文件逐字描述的操作;缺少时绝不写入。",
};
// Same flag, but for commands that name their target directly instead of
// going through a saved plan file; the plan-file wording would be wrong here.
const DIRECT_CONFIRM_OPTION: CommandOption = {
flag: "--confirm",
summary: "确认执行上面预览的这次操作;缺少时绝不写入。",
};
const SERVICE_OPTIONS: readonly CommandOption[] = [
PLATFORM_OPTION,
HOME_OPTION,
@@ -154,55 +145,77 @@ const INSTALL_OPTIONS: readonly CommandOption[] = [
const INSTALL_USAGE =
"[--source <path> | --release-manifest-url <https-url>] [--channel <stable|prerelease>] [--install-root <path>] [--home <path>] [--bin-dir <path>] [--data-dir <path>] [--uid <id>] [--xdg-config-home <path>] [--no-shell-path] [--shell-profile <path>] [--preserve-data-dir] [--allow-downgrade] [--yes]";
const USAGE_EXITS = [EXIT_OK, EXIT_FAILED, EXIT_USAGE] as const;
const RUNTIME_EXITS = [
EXIT_OK,
EXIT_FAILED,
EXIT_USAGE,
EXIT_RUNTIME_STOPPED,
] as const;
const KIND_OPTION: CommandOption = {
flag: "--kind <chat|experience|fact>",
summary:
"记忆所在的命名空间;工作/编码场景的经验和事实需要同时给 --variant。list 输出里每条都带 kind/variant。",
};
const VARIANT_OPTION: CommandOption = {
flag: "--variant <working|coding>",
summary:
"经验或事实属于工作还是编码场景;必须和 --kind experience|fact 一起用。",
};
export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "serve",
summary: "在前台启动完整 Runtime(Core 与 Host Gateway),不注册系统服务。",
usage: "memhub serve",
options: [],
exit_codes: [EXIT_OK, EXIT_FAILED],
exit_codes: USAGE_EXITS,
},
{
name: "start",
summary: "启动本机 Runtime;已有通过身份校验的实例时只连接,不重复启动。",
summary:
"启动本机 Runtime;已有通过身份校验的实例时只连接,不重复启动。受管服务会同时恢复登录时自动启动。",
usage: "memhub start [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_RUNTIME_STOPPED],
exit_codes: USAGE_EXITS,
},
{
name: "stop",
summary: "停止通过身份校验的本机 Runtime;身份不明确时不发送信号。",
summary:
"停止通过身份校验的本机 Runtime;身份不明确时不发送信号。受管服务(launchd/systemd、App)停止后登录时也不会自动启动,直到再次运行 memhub start。",
usage: "memhub stop [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED],
exit_codes: USAGE_EXITS,
},
{
name: "restart",
summary: "先停止再启动本机 Runtime,两步串行执行。",
usage: "memhub restart [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED],
exit_codes: USAGE_EXITS,
},
{
name: "status",
summary: "只读查询 Runtime 运行状态;不启动服务、不迁移数据、不改配置。",
usage: "memhub status [--watch] [--interval <seconds>] [--json]",
options: [WATCH_OPTION, INTERVAL_OPTION, JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "doctor",
summary: "按诊断口径做同样的只读检查;默认不修复任何东西。",
summary:
"按诊断口径做只读检查并给出修复建议;默认不修复任何东西。Runtime 未运行时 exit 3。",
usage: "memhub doctor [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "logs",
summary:
"读取本机脱敏诊断日志;logs export 会把排查包直接存到下载目录。默认显示最近一次会话的 200 条事件。",
"读取本机脱敏诊断日志;logs export 会把排查包(含系统崩溃报告)直接存到下载目录。默认显示最近一次会话的 200 条事件。",
usage:
"memhub logs export [--json]\nmemhub logs [--since <15m|2h|7d|RFC3339>] [--session <id>] [--trace <id>] [--incident <id>] [--operation <id>] [--request <id>] [--job <id>] [--task <id>] [--component <name>] [--event <name>] [--errors-only] [--tail <1..5000>] [--follow] [--json]",
"memhub logs export [--json]\n memhub logs [--since <15m|2h|7d|RFC3339>] [--session <id>] [--trace <id>] [--incident <id>] [--operation <id>] [--request <id>] [--job <id>] [--task <id>] [--component <name>] [--event <name>] [--errors-only] [--tail <1..5000>] [--follow] [--json]",
options: [
{
flag: "--since <time>",
@@ -226,64 +239,46 @@ export const COMMANDS: readonly CommandDescriptor[] = [
"以 NDJSON 输出到 stdout(每行一个脱敏事件;非单 JSON 文档),供脚本与 Agent 消费。",
},
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE],
exit_codes: USAGE_EXITS,
},
{
name: "init",
summary:
"首次接入:选择来源 → 选择是否导入历史。接入即开启该 Harness 的处理;后续可用 sync pause/resume 冻结或恢复。在终端且不带 --source 时进入交互引导(默认全选);非终端或加 --json 时不读 stdin、只列出可用来源和下一步。--all-detected 一次接入所有已检测来源。",
"首次接入引导。在终端里(未加 --json)进入交互向导:选择来源 → 是否开启处理 → 配置模型。非终端或加 --json 时不读 stdin,只列出可接入的来源和下一步命令。脚本接入请用 sources connect。",
usage: "memhub init [--json]",
options: [JSON_OPTION],
exit_codes: RUNTIME_EXITS,
},
{
name: "sources",
summary:
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。",
usage:
"memhub init [--json]\n memhub init --source <id> [--source <id> ...] [--import] [--dry-run] [--full-rescan] [--confirm-overrides] [--json]\n memhub init --all-detected [--import] [--dry-run] [--json]",
"memhub sources list [--json]\n memhub sources connect <id> [<id> ...] [--import] [--confirm-overrides] [--request-id <uuid>] [--json]\n memhub sources connect --all-detected [--import] [--confirm-overrides] [--json]\n memhub sources receipt <request-id> [--json]",
options: [
{
flag: "--source <id>",
summary: "要接入的来源,可重复;终端下不给则进入交互引导。",
flag: "--import",
summary:
"接入后立即导入历史会话并开启持续同步;不加时只授权,处理保持冻结。",
},
{
flag: "--all-detected",
summary:
"接入并(加 --import 时)导入全部 location_state=found 的来源。",
},
{
flag: "--import",
summary: "接入后立即导入该来源的历史会话(默认只接入不导入)。",
},
{
flag: "--dry-run",
summary: "只出计划,不接入也不导入;用于先看会做什么。",
},
{
flag: "--full-rescan",
summary: "导入时忽略既有游标做全量重扫。",
"接入全部已检测到(location_state=found)的来源;与具体来源互斥。",
},
{
flag: "--confirm-overrides",
summary: "确认接受环境覆盖类来源(如 DSH_HOME 指定的目录)。",
summary:
"确认接受目录来自环境变量覆盖的来源(如 DSH_HOME 指定的目录);不加时这些来源被拒绝(指定来源时)或跳过并列出(--all-detected 时)。",
},
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
},
{
name: "setup",
summary: "生成无头首次导入计划;只读取已有来源,不探测目录、不写授权。",
usage: "memhub setup plan --source <id> [--source <id> ...] [--json]",
options: [
{
flag: "--source <id>",
summary: "计划导入的来源,可重复 1..32 次。",
flag: "--request-id <uuid>",
summary:
"--import 的幂等键;用同一个 id 重试只会读回第一次的回执,不会重复导入。",
},
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
},
{
name: "sources",
summary: "列出来源、执行已确认的导入计划。",
usage:
"memhub sources list [--json]\n memhub sources import --plan-file <path> --confirm [--json]\n memhub sources receipt <request-id> [--json]",
options: [PLAN_FILE_OPTION, EXPLICIT_CONFIRM_OPTION, JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "sync",
@@ -297,7 +292,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
},
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "progress",
@@ -318,17 +313,17 @@ export const COMMANDS: readonly CommandDescriptor[] = [
INTERVAL_OPTION,
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "jobs",
summary:
"启动或控制单个导入任务:start 创建 history/sync 作业;pause/resume/cancel 控制已有作业。所有写操作先出计划,必须显式 --confirm;list/show 为只读。",
"查看或控制导入任务:start 创建 history/sync 任务;pause/resume/cancel 控制已有任务。都直接执行。",
usage:
"memhub jobs list [--json]\n" +
" memhub jobs show <job-id> [--json]\n" +
" memhub jobs start --source <id> [--operation <history|sync>] [--confirm] [--json]\n" +
" memhub jobs <pause|resume|cancel> --job <id> [--confirm] [--json]",
" memhub jobs start --source <id> [--operation <history|sync>] [--json]\n" +
" memhub jobs <pause|resume|cancel> --job <id> [--json]",
options: [
{
flag: "--source <id>",
@@ -336,123 +331,116 @@ export const COMMANDS: readonly CommandDescriptor[] = [
},
{
flag: "--operation <history|sync>",
summary: "start 的导入类型,默认 history。",
summary:
"start 的导入类型,默认 history;sync 与 scan --source 相同,立即增量同步一次。",
},
{
flag: "--job <id>",
summary: "pause/resume/cancel 的目标导入任务。",
},
DIRECT_CONFIRM_OPTION,
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "models",
summary:
"交互配置模型,或生成配置/探测计划、执行已确认操作与查询回执;脚本模式的凭据只从受控文件读取。",
"管理生成模型配置。add/test 直接提交并返回操作 ID(异步,用 receipt 查结果);edit/activate 直接执行;remove 不可逆,需 --yes。<profile-id>/<operation-id> 都是 UUID;脚本模式的凭据只从受控文件读取。",
usage:
"memhub models list [--json]\n memhub models show <profile-id> [--json]\n memhub models plan --config-file <path> [--json]\n memhub models configure --plan-file <path> --confirm [--api-key-file <path>] [--json]\n memhub models receipt <operation-id> [--json]\n memhub models edit <profile-id> --config-file <path> [--confirm] [--api-key-file <path>] [--json]\n memhub models activate <profile-id> [--confirm] [--json]\n memhub models probe-plan <profile-id> [--json]\n memhub models probe --plan-file <path> --confirm [--json]\n memhub models probe-receipt <operation-id> [--json]\n memhub models remove <profile-id> [--confirm] [--json]",
"memhub models list [--json]\n memhub models show <profile-id> [--json]\n memhub models add --config-file <path> [--api-key-file <path>] [--activate] [--json]\n memhub models test <profile-id> [--json]\n memhub models receipt <operation-id> [--json]\n memhub models edit <profile-id> --config-file <path> [--api-key-file <path>] [--json]\n memhub models activate <profile-id> [--json]\n memhub models remove <profile-id> [--yes] [--json]",
alternative_usage: "memhub models setup",
options: [
{
flag: "--config-file <path>",
summary: "不含 api_key 的有界 JSON 模型配置。",
summary:
"有界 JSON 模型配置,不能含 api_key;edit 的补丁必须带 expected_revision。",
},
PLAN_FILE_OPTION,
EXPLICIT_CONFIRM_OPTION,
{
flag: "--api-key-file <path>",
summary: "从 regular file 读取凭据;内容不进入参数、输出或错误。",
summary: "从属主专用的普通文件读取凭据;内容不进入参数、输出或错误。",
},
{
flag: "--activate",
summary: "add 成功后把新配置设为当前使用。",
},
DESTRUCTIVE_CONFIRM_OPTION,
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "memories",
summary: "通过 SDK 数据面列出、查看、修改或物理删除记忆。",
summary:
"列出、搜索、查看、添加、修改或物理删除记忆(对话、工作、编码三个场景)。add 和 App 的「添加记忆」是同一个入口;delete 不可逆,需 --yes。",
usage:
"memhub memories list [--status <state>] [--offset <n>] [--limit <n>] [--json]\n memhub memories show <memory-id> [--json]\n memhub memories update <memory-id> --content-file <path> [--confirm] [--json]\n memhub memories delete <memory-id> [--confirm] [--json]",
"memhub memories list [--scene <all|chat|working|coding>] [--source <id>] [--query <text>] [--status <state>] [--offset <n>] [--limit <n>] [--json]\n memhub memories show <memory-id> [--kind <kind>] [--variant <variant>] [--json]\n memhub memories add <text> [--request-id <uuid>] [--json]\n memhub memories add --content-file <path> [--request-id <uuid>] [--json]\n memhub memories add-status <request-id> [--json]\n memhub memories update <memory-id> --content-file <path> [--kind <kind>] [--variant <variant>] [--task <text>] [--json]\n memhub memories delete <memory-id> [--kind <kind>] [--variant <variant>] [--yes] [--json]",
options: [
{
flag: "--scene <all|chat|working|coding>",
summary: "只列一个场景,默认 all。working/coding 列的是该场景的经验。",
},
{ flag: "--source <id>", summary: "只列某个来源写入的记忆。" },
{
flag: "--query <text>",
summary:
"关键词搜索(只搜 active 记忆,在最相关的一批里匹配所有关键词,输出 search_window 时表示可能少计)。",
},
{
flag: "--status <state>",
summary: `记忆状态,取值 ${MEMORY_STATUSES.join("、")}。`,
summary: `记忆状态,取值 ${MEMORY_STATUSES.join("、")},默认 active。`,
},
{ flag: "--offset <n>", summary: "跳过前 n 条,默认 0。" },
{
flag: "--limit <n>",
summary: "本页最多返回多少条,默认 100,上限 500。",
},
KIND_OPTION,
VARIANT_OPTION,
{
flag: "--content-file <path>",
summary:
"从有界普通文件读取 update 的新正文;预览只显示摘要,--confirm 时才写入。",
"从有界普通文件读取正文:add 的新记忆(≤12000 字),或 update 的新内容(经验为 solution)。",
},
DIRECT_CONFIRM_OPTION,
{
flag: "--task <text>",
summary: "update 经验(--kind experience)时同时改写 task。",
},
{
flag: "--request-id <uuid>",
summary:
"add 的幂等键;用同一个 id 重试只会读回第一次的结果,不会重复添加。不给时自动生成并输出。",
},
DESTRUCTIVE_CONFIRM_OPTION,
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "profile",
summary: "读取 SDK 组装后的用户长期画像(L0 + 已启用的长期画像补充)。",
usage: "memhub profile [--json]",
options: [DATA_DIR_OPTION, JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
options: [JSON_OPTION],
exit_codes: RUNTIME_EXITS,
},
{
name: "calls",
name: "graph-width",
summary:
"清除本机调用记录;不可逆删除,默认只出计划,必须显式 --confirm 才执行。",
"嵌入维度变化、图谱宽度不匹配时的处置决定。keep_existing 与 export_first 直接执行;rebuild_with_new 会归档旧图谱并重建,需 --yes。",
usage:
"memhub calls clear --event <id> [--event <id> ...] [--confirm] [--json]\n memhub calls clear --all [--confirm] [--json]",
options: [
{
flag: "--event <id>",
summary: "要清除的调用记录事件 ID,可重复,最多 100 条。",
},
{
flag: "--all",
summary: "清除全部调用记录;与 --event 互斥。",
},
DIRECT_CONFIRM_OPTION,
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
},
{
name: "notes",
summary:
"端内新增记忆(memory-notes):从普通文件新增,或显式确认后处理已有笔记。",
usage:
"memhub notes add --content-file <path> [--json]\n memhub notes apply --plan-file <path> --content-file <path> --confirm [--json]\n memhub notes process <note-id> [--confirm] [--json]",
options: [
{
flag: "--content-file <path>",
summary: "从有界普通文件读取新增正文;计划只保存摘要,执行时重新校验。",
},
PLAN_FILE_OPTION,
DIRECT_CONFIRM_OPTION,
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
},
{
name: "graph",
summary: "图谱宽度不匹配时的用户决定;默认只出计划,--confirm 后才执行。",
usage:
"memhub graph width --decision <keep_existing|rebuild_with_new|export_first> [--confirm] [--json]",
"memhub graph-width --decision <keep_existing|export_first|rebuild_with_new> [--yes] [--json]",
options: [
{
flag: "--decision <decision>",
summary:
"keep_existing(保持)、export_first(先导出)、rebuild_with_new(导出后重建)。",
"keep_existing(保持现有图谱)、export_first(先导出)、rebuild_with_new(导出后归档旧图谱并按新维度重建)。",
},
{
...DESTRUCTIVE_CONFIRM_OPTION,
summary: "rebuild_with_new 必须加;其它决定不需要。",
},
DIRECT_CONFIRM_OPTION,
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "host",
@@ -480,12 +468,12 @@ export const COMMANDS: readonly CommandDescriptor[] = [
},
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "recall",
summary:
"从当前授权的原文与已沉淀记忆统一检索上下文;本地只读,不调用模型、不触发补索引。",
"从原始对话与已沉淀记忆统一检索上下文;本地只读,不调用模型、不触发补索引。文本输出标明每条是「原始对话摘录」还是「记忆」(JSON 里是 kind: raw|memory)。",
usage:
"memhub recall <query> [--kind <raw|memory>] [--source <id>] [--limit <n>] [--character-budget <n>] [--timeout-ms <n>] [--json]\n memhub recall --query-file <path> [...]\n memhub recall --stdin [...]",
options: [
@@ -516,21 +504,21 @@ export const COMMANDS: readonly CommandDescriptor[] = [
},
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_RUNTIME_STOPPED],
exit_codes: RUNTIME_EXITS,
},
{
name: "service-plan",
summary: "生成用户级服务计划;只输出计划,不写文件也不执行命令。",
usage: `memhub service-plan ${SERVICE_USAGE}`,
options: SERVICE_OPTIONS,
exit_codes: [EXIT_OK, EXIT_PLAN_BLOCKED],
exit_codes: [EXIT_OK, EXIT_USAGE, EXIT_PLAN_BLOCKED],
},
{
name: "service-install",
summary: "预览用户级服务安装;只有 --yes 才写入服务文件并注册。",
usage: `memhub service-install ${SERVICE_USAGE} [--yes]`,
options: [...SERVICE_OPTIONS, CONFIRM_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_PLAN_BLOCKED],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
},
{
name: "service-uninstall",
@@ -538,15 +526,15 @@ export const COMMANDS: readonly CommandDescriptor[] = [
"预览用户级服务卸载;只有 --yes 才注销并删除逐字匹配的受管服务文件,不删数据。",
usage: `memhub service-uninstall ${SERVICE_USAGE} [--yes]`,
options: [...SERVICE_OPTIONS, CONFIRM_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_PLAN_BLOCKED],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
},
{
name: "install",
summary:
"预览发行目录安装、memhub/memhub-cli/one-memory 短链接、PATH 条目与用户服务计划;只有 --yes 才执行并在失败时回滚。",
"预览发行目录安装、memhub/memhub-cli/one-memory 短链接、PATH 条目与用户服务计划;只有 --yes 才执行。记忆数据迁移开始前失败会回滚;迁移开始后失败保留新版本、拒绝启动,并输出迁移日志与反馈地址。",
usage: `memhub install ${INSTALL_USAGE}`,
options: INSTALL_OPTIONS,
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_PLAN_BLOCKED],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
},
{
name: "uninstall",
@@ -559,24 +547,31 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "upgrade",
summary:
"预览升级;只有 --yes 才切换受管版本并重启服务,失败时恢复旧版本,不删数据。",
"预览升级;只有 --yes 才切换受管版本并重启服务,不删数据。记忆数据迁移开始前失败会恢复旧版本;迁移开始后失败保留新版本、拒绝启动,并输出迁移日志与反馈地址(不会回滚到旧版本)。",
usage: `memhub upgrade ${INSTALL_USAGE}`,
options: INSTALL_OPTIONS,
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_PLAN_BLOCKED],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
},
{
name: "delete-data",
summary:
"独立、二次确认地删除 One Memory Home(数据)。这是唯一会删除数据的命令;uninstall 不会。默认只出预览;需 Runtime 已停止,并显式 --confirm <path> 复述该目录。",
"删除 MemHub 数据目录(记忆、来源、配置),不可逆。这是唯一会删除数据的命令;uninstall 不会。需要 Runtime 已停止;不加参数时只预览要删除的目录;删除需要 --yes 并用 --confirm 再写一遍同一个目录。",
usage:
"memhub delete-data [--data-dir <path>] [--yes --confirm <path>] [--json]",
options: [
DATA_DIR_OPTION,
CONFIRM_OPTION,
{
flag: "--data-dir <path>",
summary: "要删除的数据目录绝对路径;默认当前使用的数据目录。",
},
{
...DESTRUCTIVE_CONFIRM_OPTION,
summary:
"删除时必须加,且要同时给出 --confirm <path>;不加时只预览,exit 0。",
},
{
flag: "--confirm <path>",
summary:
"第二次确认:逐字复述预览里的数据目录;缺少它时即使 --yes 也不删除。",
"删除时必须加:再写一遍要删除的数据目录,与预览中的目录完全一致才执行。",
},
JSON_OPTION,
],
@@ -588,113 +583,149 @@ export const COMMANDS: readonly CommandDescriptor[] = [
"onboarding reset 回到首次运行的引导流程。只清除引导完成状态:不删记忆数据,也不需要卸载重装。",
usage: "memhub onboarding reset [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE],
exit_codes: RUNTIME_EXITS,
},
{
name: "help",
summary: "显示总帮助或单个命令的用法。",
usage: "memhub help [command] [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK],
exit_codes: [EXIT_OK, EXIT_USAGE],
},
{
name: "version",
summary: "显示 CLI 版本。",
usage: "memhub version [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK],
exit_codes: [EXIT_OK, EXIT_USAGE],
},
{
name: "path",
name: "scan-roots",
summary:
"查看或登记被扫描的目录(scan roots)。list 只读;add/remove 预览后加 --yes 才提交;均已接入 Runtime 的 scan-roots 端点。",
"查看或登记被扫描的目录(scan roots)。add/remove 直接执行。(旧名 path 仍可用。)",
usage:
"memhub path list [--json]\n" +
" memhub path add --root <path> [--source <id>] [--no-recursive] [--yes]\n" +
" memhub path remove --root <path> [--yes]",
options: [DATA_DIR_OPTION, JSON_OPTION, CONFIRM_OPTION],
exit_codes: [EXIT_OK, EXIT_USAGE, EXIT_FAILED, EXIT_PLAN_BLOCKED],
"memhub scan-roots list [--json]\n" +
" memhub scan-roots add --root <path> [--source <id>] [--no-recursive] [--json]\n" +
" memhub scan-roots remove --root <path> [--json]",
options: [
{ flag: "--root <path>", summary: "要登记或移除的目录。" },
{
flag: "--source <id>",
summary: "把目录绑定到某个来源;不给时由 Runtime 自动识别。",
},
{ flag: "--no-recursive", summary: "只扫描该目录本身,不进入子目录。" },
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
},
{
name: "adapter",
summary:
"查看 Harness Adapter 的就绪状态与最近一次探测结果(list),或现场探测某个来源(probe --source <id>);两者均为只读,已接入 Runtime 的 adapters/探测端点。",
"查看 Harness Adapter 的就绪状态与最近一次探测结果(list),或现场探测某个来源(probe --source <id>);两者均为只读。",
usage:
"memhub adapter list [--json]\n" +
" memhub adapter probe --source <id> [--json]",
options: [DATA_DIR_OPTION, JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_USAGE, EXIT_FAILED],
options: [
{ flag: "--source <id>", summary: "probe 的目标来源。" },
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
},
{
name: "scan",
summary:
"立即扫描;默认只输出计划,加 --yes 才真正执行。指定 --source 扫描单个来源,--all 一次扫描/导入全部已检测来源(Runtime 按来源寻址,不接受裸路径)。",
usage:
"memhub scan [--source <id>] [--full] [--yes] [--json]\n memhub scan --all [--yes] [--json]",
"立即增量扫描一个已连接并开启同步的来源,返回导入任务(与 jobs start --operation sync 相同)。接入全部来源请用 sources connect --all-detected --import。",
usage: "memhub scan --source <id> [--json]",
options: [
{
flag: "--source <id>",
summary: "按单个来源扫描;与 --all 互斥。",
summary: "要扫描的来源。",
},
{
flag: "--all",
summary:
"处理全部 location_state=found 的来源,按来源逐条回执;部分成功返回非 0。",
},
{
flag: "--full",
summary: "忽略既有游标做全量重扫;默认只读取增量。",
},
DATA_DIR_OPTION,
JSON_OPTION,
CONFIRM_OPTION,
],
exit_codes: [EXIT_OK, EXIT_USAGE, EXIT_FAILED, EXIT_PLAN_BLOCKED],
exit_codes: RUNTIME_EXITS,
},
{
name: "write",
summary: "提交 OpenAI messages 到 SDK 写入队列,或按 request id 查询状态。",
usage:
"memhub write submit --messages-file <path> --session <id> [--source <id>] [--request-id <id>] [--memory-at <RFC3339>] [--live] [--json]\n memhub write status --id <request-id> [--json]",
options: [DATA_DIR_OPTION, JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_USAGE, EXIT_FAILED],
},
{
name: "task",
summary:
"兼容别名:按 request id 查询写入状态;新脚本请使用 write status。",
usage: "memhub task --id <request-id> [--json]",
options: [DATA_DIR_OPTION, JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_USAGE, EXIT_FAILED],
"提交一段 OpenAI 格式的对话(messages)到写入队列,异步加工成记忆;或按 request id 查询状态。request id 是幂等键:不给时自动生成并输出,重试请带上同一个。新增一条结论式记忆请用 memories add。",
usage:
"memhub write submit --messages-file <path> --session <id> [--source <id>] [--request-id <id>] [--memory-at <RFC3339>] [--live] [--json]\n memhub write status <request-id> [--json]",
options: [
{
flag: "--messages-file <path>",
summary:
"OpenAI messages 数组(或 {messages:[…]}),1..200 条,role 为 user/assistant/tool,至少一条 user。",
},
{ flag: "--session <id>", summary: "会话 ID。" },
{ flag: "--source <id>", summary: "来源应用,默认 memhub-cli。" },
{
flag: "--request-id <id>",
summary:
"幂等键:同 id 同内容不会重复入队,同 id 不同内容被拒绝(409)。",
},
{ flag: "--memory-at <RFC3339>", summary: "对话发生时间,必须带时区。" },
{ flag: "--live", summary: "优先处理。" },
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
},
{
name: "backup",
summary:
"备份与恢复 MemHub 数据(两阶段)。create 调用 Runtime 的备份端点;restore 会覆盖现有数据,要求 Runtime 已停止且 `--confirm` 逐字重复 `--source`,随后由非 Runtime 维护态 Core 完成受管校验、保护性备份和原子替换。",
"备份与恢复主库(来源、写入队列、配置)。注意:不包含记忆本身——SDK 记忆库 hy-memory/ 不在备份里,restore 也不会替换它。create 的目标要放在 <数据目录>/backups/ 下才能被 restore 使用;restore 覆盖现有主库,需 Runtime 已停止,并加 --yes 与 --confirm <同一个备份路径>。",
usage:
"memhub backup create --target <path> [--yes] [--json]\n" +
" memhub backup restore --source <path> --yes --confirm <path> [--json]",
"memhub backup create --target <path> [--json]\n" +
" memhub backup restore --source <path> [--yes --confirm <path>] [--json]",
options: [
DATA_DIR_OPTION,
JSON_OPTION,
CONFIRM_OPTION,
{
flag: "--target <path>",
summary: "create 的备份文件路径,必须在数据目录内。",
},
{
flag: "--source <path>",
summary:
"restore 的备份文件,必须是 <数据目录>/backups/ 下的受管备份。",
},
{
...DESTRUCTIVE_CONFIRM_OPTION,
summary:
"restore 必须加,且要同时给出 --confirm <path>;create 不需要。",
},
{
flag: "--confirm <path>",
summary: "restore 要求逐字重复 --source 路径;不一致时不做任何改动。",
summary:
"restore 必须加:再写一遍 --source 的备份路径,完全一致才执行。",
},
JSON_OPTION,
],
exit_codes: [EXIT_OK, EXIT_USAGE, EXIT_FAILED, EXIT_PLAN_BLOCKED],
exit_codes: [...RUNTIME_EXITS, EXIT_PLAN_BLOCKED],
},
{
name: "ui",
summary:
"用系统默认浏览器打开 Runtime 托管的 Web Console:先向 Core 换取一次性 loopback 登录码,再交给系统浏览器由 Core 置 HttpOnly/SameSite=Strict session;CLI 不向浏览器暴露裸 Core token。Core 未提供该合同时如实返回 ui_session_contract_unavailable(exit 4)。",
"用系统默认浏览器打开 Runtime 托管的 Web Console:先向 Core 换取一次性 loopback 登录码,再交给系统浏览器由 Core 置 HttpOnly/SameSite=Strict session;CLI 不向浏览器暴露裸 Core token。Runtime 未运行时 exit 3;Core 未提供该合同时如实返回 ui_session_contract_unavailable(exit 4)。",
usage: "memhub ui [--json]",
options: [JSON_OPTION],
exit_codes: [EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_PLAN_BLOCKED],
exit_codes: [...RUNTIME_EXITS, EXIT_PLAN_BLOCKED],
},
];
/** Old names that still run but are not listed: `path` → `scan-roots`,
* `task` → `write status`, `graph width` → `graph-width`. */
export const HIDDEN_COMMAND_ALIASES: Readonly<Record<string, string>> = {
path: "scan-roots",
task: "write",
graph: "graph-width",
};
export function findCommand(name: string): CommandDescriptor | undefined {
return COMMANDS.find((command) => command.name === name);
}
/** A listed command, or the command a hidden alias now runs as. */
export function findCommandOrAlias(
name: string,
): CommandDescriptor | undefined {
return findCommand(HIDDEN_COMMAND_ALIASES[name] ?? name);
}
+114 -30
View File
@@ -69,7 +69,6 @@ const COMMANDS = [
"doctor",
"logs",
"init",
"setup",
"sources",
"sync",
"progress",
@@ -77,10 +76,7 @@ const COMMANDS = [
"models",
"memories",
"profile",
// C6 App BLOCK (25f4ecf): UI write operations that had no CLI equivalent.
"calls",
"notes",
"graph",
"graph-width",
"host",
"recall",
"service-plan",
@@ -93,19 +89,20 @@ const COMMANDS = [
"onboarding",
"help",
"version",
// A6 product commands. Appended, not inserted: the catalog is a stable
// contract, so existing names must not shift position.
"path",
"scan-roots",
"adapter",
"scan",
"write",
"task",
"backup",
// `ui` is declared but blocked on a Core session contract; it is listed so
// `help` documents it and the blocked shape is machine-readable.
"ui",
] as const;
/** Old names that still run and still answer `--help`, but are not listed. */
const HIDDEN_ALIASES = [
["path", "scan-roots"],
["task", "write"],
] as const;
describe("CLI-22 shared output contract", () => {
describe("help", () => {
it.each([["help"], ["--help"], ["-h"]])(
@@ -146,6 +143,56 @@ describe("CLI-22 shared output contract", () => {
},
);
it.each(HIDDEN_ALIASES)(
"documents the hidden alias %s as %s and keeps it out of the listing",
async (alias, target) => {
const streams = captureStreams();
try {
expect(await run(["help", alias], {})).toBe(0);
expect(await run([alias, "--help"], {})).toBe(0);
} finally {
streams.restore();
}
expect(streams.stdout()).toContain(`${PRODUCT} ${target}`);
const listing = captureStreams();
try {
expect(await run(["help", "--json"], {})).toBe(0);
} finally {
listing.restore();
}
const names = JSON.parse(listing.stdout()).commands.map(
(entry: { name: string }) => entry.name,
);
expect(names).not.toContain(alias);
},
);
it("lists no removed command", async () => {
const streams = captureStreams();
try {
expect(await run(["help", "--json"], {})).toBe(0);
} finally {
streams.restore();
}
const catalog = streams.stdout();
for (const removed of [
'"name":"setup"',
'"name":"notes"',
'"name":"calls"',
'"name":"graph"',
"--plan-file",
"probe-plan",
"probe-receipt",
"models configure",
"sources import",
"--full-rescan",
"--dry-run",
"--full]",
]) {
expect(catalog).not.toContain(removed);
}
});
it("never executes the documented command", async () => {
for (const command of COMMANDS) {
for (const argv of [
@@ -201,27 +248,63 @@ describe("CLI-22 shared output contract", () => {
expect(payload.commands).toBeUndefined();
});
it("documents the destructive restore confirmation explicitly", async () => {
it("documents the one --yes convention for destructive commands", async () => {
const general = captureStreams();
try {
expect(await run(["help"], {})).toBe(0);
} finally {
general.restore();
}
expect(general.stdout()).toContain("确认约定");
for (const [command, usage] of [
[
"backup",
"memhub backup restore --source <path> [--yes --confirm <path>]",
],
["memories", "memhub memories delete <memory-id>"],
["models", "memhub models remove <profile-id> [--yes]"],
[
"delete-data",
"memhub delete-data [--data-dir <path>] [--yes --confirm <path>]",
],
["graph-width", "[--yes]"],
] as const) {
const streams = captureStreams();
try {
expect(await run(["help", command], {})).toBe(0);
} finally {
streams.restore();
}
expect(streams.stdout()).toContain(usage);
// Only the two commands with no way back repeat the target path.
if (command !== "backup" && command !== "delete-data") {
expect(streams.stdout()).not.toContain("--confirm <path>");
}
}
});
it("says backup does not include the memories themselves", async () => {
const streams = captureStreams();
try {
expect(await run(["help", "backup"], {})).toBe(0);
} finally {
streams.restore();
}
expect(streams.stdout()).toContain(
"memhub backup restore --source <path> --yes --confirm <path>",
);
expect(streams.stdout()).toContain("逐字重复 --source");
expect(streams.stdout()).toContain("不包含记忆本身");
});
it("does not attach restore's confirmation flag to path", async () => {
const streams = captureStreams();
try {
expect(await run(["help", "path"], {})).toBe(0);
} finally {
streams.restore();
it("does not promise a rollback after the migration started", async () => {
for (const command of ["install", "upgrade"]) {
const streams = captureStreams();
try {
expect(await run(["help", command], {})).toBe(0);
} finally {
streams.restore();
}
expect(streams.stdout()).toContain("迁移开始后失败保留新版本");
expect(streams.stdout()).not.toContain("失败时恢复旧版本");
expect(streams.stdout()).not.toContain("失败时回滚");
}
expect(streams.stdout()).not.toContain("--confirm <path>");
});
});
@@ -545,22 +628,23 @@ describe("documented commands", () => {
models: [
"list",
"show",
"plan",
"configure",
"add",
"test",
"receipt",
"probe-plan",
"probe",
"probe-receipt",
"edit",
"activate",
"remove",
],
// Source processing is one Freeze/Thaw switch under `sync`; the obsolete
// independent read/write `sources access` command must not return here.
sources: ["list", "import", "receipt"],
sources: ["list", "connect", "receipt"],
jobs: ["list", "show", "start", "<pause|resume|cancel>"],
// `sync <pause|resume>` is documented as one alternation, so it is
// checked as an entry rather than as two literal subcommands.
sync: ["status", "<pause|resume>"],
memories: ["list", "show", "update", "delete"],
memories: ["list", "show", "add", "add-status", "update", "delete"],
"scan-roots": ["list", "add", "remove"],
write: ["submit", "status"],
host: ["list", "status", "install", "uninstall", "sync"],
};
@@ -133,6 +133,20 @@ describe("unified context search", () => {
expect(output).not.toMatch(/^- raw |memory-1|raw-1|采用状态/m);
});
it("labels raw hits as original-conversation excerpts, not memories", async () => {
const streams = captureStreams();
try {
expect(await run(["recall", "synthetic question"])).toBe(0);
} finally {
streams.restore();
}
const output = streams.stdout();
expect(output).toContain(
"- [原始对话摘录 · codex] Synthetic raw citation.",
);
expect(output).toContain("- [记忆] Synthetic SDK memory.");
});
it("queries the SDK context endpoint directly and preserves item ids", async () => {
const streams = captureStreams();
try {
+11 -1
View File
@@ -23,6 +23,7 @@ export interface ContextSearchDocument {
| "unavailable"
| "denied"
| "not_found"
| "rejected"
| "invalid";
code?: string;
request_id?: string;
@@ -97,6 +98,9 @@ export function renderContextText(document: ContextSearchDocument): string {
if (document.state === "unavailable") return "暂时无法查记忆,请稍后重试。\n";
if (document.state === "denied") return "当前应用没有查记忆权限。\n";
if (document.state === "invalid") return "查记忆返回了无法识别的结果。\n";
if (document.state === "rejected") {
return `查记忆被拒绝(${document.code}),请检查参数。\n`;
}
const lines = [
document.state === "empty"
? "没有查到相关记忆。"
@@ -104,8 +108,14 @@ export function renderContextText(document: ContextSearchDocument): string {
? "查到以下内容,部分内容还在整理中:"
: "查到以下相关内容:",
];
// A raw hit is an excerpt of an original conversation, not a distilled
// memory; say which one each line is so it is not read as a conclusion.
for (const item of document.items) {
lines.push(`- ${String(item.content)}`);
const label =
item.kind === "raw"
? `[原始对话摘录 · ${String(item.source)}]`
: "[记忆]";
lines.push(`- ${label} ${String(item.content)}`);
}
if (document.truncated) lines.push("结果已按请求预算截断。");
return `${lines.join("\n")}\n`;
+125 -76
View File
@@ -18,6 +18,9 @@ const READ_ONLY_PATHS = [
"/api/v2/user-profile",
] as const;
const SDK_MEMORY_DETAIL = /^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}$/;
/** Status of one `POST /api/v2/memories` addition, by its request id. */
const MEMORY_ADDITION =
/^\/api\/v2\/memories\/additions\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
/** Contract (C0, H25): the one-time loopback code exchange for `memhub ui`.
* It is a POST because it mints a short-lived code; the CLI sends the bearer
@@ -79,9 +82,15 @@ export const RESERVED_MEMORY_SEGMENTS = ["ask", "graph"] as const;
/** A response larger than this is refused before it is buffered. */
const MAX_RESPONSE_BYTES = 4 * 1024 * 1024;
/** `rejected` is Core refusing the request itself (400/409/422: a bad filter,
* an ambiguous id). It is not "Core is busy" — only a transport failure or a
* 5xx is `unavailable`. */
export type ReadOutcome<T> =
| { state: "ok"; data: T }
| { state: "denied" | "unavailable" | "not_found"; code: string };
| {
state: "denied" | "unavailable" | "not_found" | "rejected";
code: string;
};
export type CoreReader = <T>(
path: string,
@@ -140,10 +149,27 @@ export function createCoreReader(attachment: {
if (response.status === 404) {
return { state: "not_found" as const, code: "http_404" };
}
if ([400, 409, 422].includes(response.status)) {
return {
state: "rejected" as const,
code: detailCode(response.data, response.status),
};
}
return { state: "unavailable" as const, code: `http_${response.status}` };
};
}
/** Core's stable `detail` code when it sent one, else the bare status. */
function detailCode(value: unknown, status: number): string {
const detail =
value && typeof value === "object" && !Array.isArray(value)
? (value as { detail?: unknown }).detail
: undefined;
return typeof detail === "string" && /^[a-z0-9_.:-]{1,128}$/.test(detail)
? detail
: `http_${status}`;
}
/**
* Context search is semantically read-only but intentionally uses POST so the
* private query and filters are carried in a bounded body, never in a URL.
@@ -192,6 +218,7 @@ export function isReadOnlyPath(path: string): boolean {
if ((READ_ONLY_PATHS as readonly string[]).includes(path)) return true;
if (IMPORT_JOB.test(path)) return true;
if (OPERATION_RECEIPT.test(path)) return true;
if (MEMORY_ADDITION.test(path)) return true;
if (!SDK_MEMORY_DETAIL.test(path)) return false;
const segment = path.slice("/api/v2/memories/".length);
return !(RESERVED_MEMORY_SEGMENTS as readonly string[]).includes(segment);
@@ -199,8 +226,10 @@ export function isReadOnlyPath(path: string): boolean {
/** A write accepted by the unified Memory Server entry.
*
* `request_id` is what the caller keeps and queries later. Retrying is unsafe:
* HY has no idempotency key, so a resubmit duplicates the memory.
* `request_id` is the idempotency key: Core keys the write on it, so a retry
* with the same id and the same messages is answered from the first write, and
* the same id with different messages is refused (409). A retry without the
* original id is a new write.
*/
export interface MemoryWriteReceipt {
request_id: string;
@@ -267,6 +296,11 @@ function baseUrlOf(attachment: RuntimeAttachmentLike): string {
return `http://${attachment.host}:${attachment.port}`;
}
export type MemoryWriteOutcome =
| { state: "accepted"; receipt: MemoryWriteReceipt }
| { state: "rejected"; code: string; http_status: number }
| { state: "unknown"; code: string };
export async function submitMemoryWrite(
attachment: RuntimeAttachmentLike,
request: {
@@ -277,28 +311,48 @@ export async function submitMemoryWrite(
memory_at?: string;
priority?: "live" | "normal";
},
): Promise<MemoryWriteReceipt> {
const response = await axios.post<MemoryWriteReceipt>(
`${baseUrlOf(attachment)}/api/v2/memory-writes`,
{
request_id: request.request_id,
session_id: request.session,
source_id: request.source,
messages: [...request.messages],
...(request.memory_at === undefined
? {}
: { memory_at: request.memory_at }),
...(request.priority === undefined ? {} : { priority: request.priority }),
},
{
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout: 15_000,
proxy: false,
maxRedirects: 0,
maxContentLength: MAX_RESPONSE_BYTES,
},
);
return response.data;
): Promise<MemoryWriteOutcome> {
let response: { status: number; data: unknown };
try {
response = await axios.post<MemoryWriteReceipt>(
`${baseUrlOf(attachment)}/api/v2/memory-writes`,
{
request_id: request.request_id,
session_id: request.session,
source_id: request.source,
messages: [...request.messages],
...(request.memory_at === undefined
? {}
: { memory_at: request.memory_at }),
...(request.priority === undefined
? {}
: { priority: request.priority }),
},
{
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout: 15_000,
proxy: false,
maxRedirects: 0,
maxContentLength: MAX_RESPONSE_BYTES,
validateStatus: () => true,
},
);
} catch {
// The request may or may not have reached Core; the same request id makes
// a retry safe.
return { state: "unknown", code: "transport_result_unknown" };
}
if (response.status >= 200 && response.status < 300) {
return { state: "accepted", receipt: response.data as MemoryWriteReceipt };
}
if (response.status >= 400 && response.status < 500) {
return {
state: "rejected",
code: detailCode(response.data, response.status),
http_status: response.status,
};
}
return { state: "unknown", code: `http_${response.status}_result_unknown` };
}
/** Ask Runtime to scan one source now, returning the import job it started.
@@ -495,48 +549,52 @@ export async function readScanRoots(
}
}
/** A direct Core write whose refusal carries Core's stable code. */
export type CoreWriteResult =
| { ok: true; data: unknown }
| { ok: false; code: string };
async function coreWrite(
attachment: RuntimeAttachmentLike,
method: "post" | "delete",
pathname: string,
body: Record<string, unknown>,
timeout = 10_000,
): Promise<CoreWriteResult> {
let response: { status: number; data: unknown };
try {
response = await axios.request({
method,
url: `${baseUrlOf(attachment)}${pathname}`,
data: body,
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout,
proxy: false,
maxRedirects: 0,
maxContentLength: MAX_RESPONSE_BYTES,
validateStatus: () => true,
});
} catch {
return { ok: false, code: "transport_failed" };
}
if (response.status >= 200 && response.status < 300) {
return { ok: true, data: response.data };
}
return { ok: false, code: detailCode(response.data, response.status) };
}
export async function addScanRoot(
attachment: RuntimeAttachmentLike,
body: { root: string; source?: string; recursive?: boolean },
): Promise<unknown | null> {
try {
const response = await axios.post<unknown>(
`${baseUrlOf(attachment)}/api/v1/scan-roots`,
body,
{
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout: 10_000,
proxy: false,
maxRedirects: 0,
maxContentLength: MAX_RESPONSE_BYTES,
},
);
return response.data;
} catch (error) {
return { _error: error instanceof Error ? error.message : String(error) };
}
): Promise<CoreWriteResult> {
return coreWrite(attachment, "post", "/api/v1/scan-roots", body);
}
export async function removeScanRoot(
attachment: RuntimeAttachmentLike,
body: { root: string },
): Promise<unknown | null> {
try {
const response = await axios.delete<unknown>(
`${baseUrlOf(attachment)}/api/v1/scan-roots`,
{
headers: { Authorization: `Bearer ${attachment.apiToken}` },
data: body,
timeout: 10_000,
proxy: false,
maxRedirects: 0,
maxContentLength: MAX_RESPONSE_BYTES,
},
);
return response.data;
} catch (error) {
return { _error: error instanceof Error ? error.message : String(error) };
}
): Promise<CoreWriteResult> {
return coreWrite(attachment, "delete", "/api/v1/scan-roots", body);
}
export async function readAdapters(
@@ -562,21 +620,12 @@ export async function readAdapters(
export async function probeAdapter(
attachment: RuntimeAttachmentLike,
source: string,
): Promise<unknown | null> {
try {
const response = await axios.post<unknown>(
`${baseUrlOf(attachment)}/api/v1/adapters/probe`,
{ source },
{
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout: 15_000,
proxy: false,
maxRedirects: 0,
maxContentLength: MAX_RESPONSE_BYTES,
},
);
return response.data;
} catch (error) {
return { _error: error instanceof Error ? error.message : String(error) };
}
): Promise<CoreWriteResult> {
return coreWrite(
attachment,
"post",
"/api/v1/adapters/probe",
{ source },
15_000,
);
}
+9 -94
View File
@@ -4,9 +4,11 @@ const WRITE_PATHS = [
"/api/v1/sources/connect",
"/api/v1/model-profiles",
] as const;
const MEMORY_WRITE_PATH =
/^\/api\/v1\/memories(?:\/[A-Za-z0-9][\w.-]{0,127}\/(?:correct|scopes|actions\/delete))?$/;
const SDK_MEMORY_PATH = /^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}$/;
/** `POST /api/v2/memories` adds one memory (the app's 添加记忆). */
const MEMORY_ADD_PATH = "/api/v2/memories";
/** One SDK memory; delete names its namespace in the query string. */
const SDK_MEMORY_PATH =
/^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}(?:\?kind=(?:chat|experience|fact)(?:&variant=(?:working|coding))?)?$/;
const MODEL_PROBE_PATH =
/^\/api\/v1\/model-profiles\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\/probe$/;
const MODEL_PROFILE_PATH =
@@ -22,18 +24,7 @@ const SOURCE_REVOKE_PATH =
const SOURCE_IMPORT_JOB_PATH = "/api/v1/import-jobs";
const IMPORT_JOB_ACTION_PATH =
/^\/api\/v1\/import-jobs\/[A-Za-z0-9][\w.-]{0,127}\/(?:pause|resume|cancel)$/;
const CALL_EVENTS_CLEAR_PATH = "/api/v1/call-events/clear";
const CALL_EVENTS_CLEAR_ALL_PATH = "/api/v1/call-events/clear-all";
const IMPORT_FILE_COMMIT =
/^\/api\/v1\/import-files\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\/commit$/;
const IMPORT_FILE =
/^\/api\/v1\/import-files\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const MEMORY_NOTES_PATH = "/api/v1/memory-notes";
const MEMORY_NOTE_PROCESS =
/^\/api\/v1\/memory-notes\/[A-Za-z0-9][\w.:-]{0,127}\/process$/;
const EXTRACTION_SETTINGS_PATH = "/api/v1/extraction-settings";
const GRAPH_WIDTH_PATH = "/api/v1/graph-width";
const MANUAL_EXTRACT_PATH = "/api/v1/memory-processing/manual-extract";
const ONBOARDING_COMPLETE_PATH = "/api/v1/onboarding/complete";
const MAX_BYTES = 4 * 1024 * 1024;
@@ -70,18 +61,6 @@ export interface CorePatchOperations {
): Promise<OperationOutcome<T>>;
}
export interface CoreUploadOperations {
uploadImportFile<T>(
input: {
bytes: Uint8Array;
filename: string;
inputKind: "raw_conversation" | "memory_content";
operationId: string;
},
signal?: AbortSignal,
): Promise<OperationOutcome<T>>;
}
export function createCoreOperations(attachment: {
host: string;
port: number;
@@ -90,8 +69,7 @@ export function createCoreOperations(attachment: {
}): CoreOperations &
CoreDeleteOperations &
CorePutOperations &
CorePatchOperations &
CoreUploadOperations {
CorePatchOperations {
const client = axios.create({
baseURL: `http://${attachment.host}:${attachment.port}`,
headers: { Authorization: `Bearer ${attachment.apiToken}` },
@@ -102,18 +80,6 @@ export function createCoreOperations(attachment: {
maxBodyLength: 128 * 1024,
validateStatus: () => true,
});
// Uploads carry up to 128 MiB of raw octet-stream, so they need a separate
// body limit and a deadline that leaves room for parsing on the Core side.
const uploadClient = axios.create({
baseURL: `http://${attachment.host}:${attachment.port}`,
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout: 120_000,
proxy: false,
maxRedirects: 0,
maxContentLength: MAX_BYTES,
maxBodyLength: 128 * 1024 * 1024,
validateStatus: () => true,
});
return {
async post<T>(
path: string,
@@ -123,19 +89,13 @@ export function createCoreOperations(attachment: {
if (
!(WRITE_PATHS as readonly string[]).includes(path) &&
path !== SOURCE_IMPORT_JOB_PATH &&
!MEMORY_WRITE_PATH.test(path) &&
path !== MEMORY_ADD_PATH &&
!MODEL_PROBE_PATH.test(path) &&
!SOURCE_SYNC_PATH.test(path) &&
!IMPORT_JOB_ACTION_PATH.test(path) &&
!SOURCE_AUTHORIZE_PATH.test(path) &&
!SOURCE_REVOKE_PATH.test(path) &&
path !== CALL_EVENTS_CLEAR_PATH &&
path !== CALL_EVENTS_CLEAR_ALL_PATH &&
!IMPORT_FILE_COMMIT.test(path) &&
path !== MEMORY_NOTES_PATH &&
!MEMORY_NOTE_PROCESS.test(path) &&
path !== GRAPH_WIDTH_PATH &&
path !== MANUAL_EXTRACT_PATH &&
path !== ONBOARDING_COMPLETE_PATH &&
!MODEL_PROFILE_ACTIVATE.test(path)
) {
@@ -171,7 +131,7 @@ export function createCoreOperations(attachment: {
body: Record<string, unknown>,
signal?: AbortSignal,
) {
if (path !== EXTRACTION_SETTINGS_PATH && !SDK_MEMORY_PATH.test(path)) {
if (!SDK_MEMORY_PATH.test(path) || path.includes("?")) {
throw new Error("core_operation_path_not_allowed");
}
let response: { status: number; data: unknown };
@@ -233,7 +193,7 @@ export function createCoreOperations(attachment: {
};
},
async delete<T>(path: string, signal?: AbortSignal) {
if (!IMPORT_FILE.test(path) && !SDK_MEMORY_PATH.test(path)) {
if (!SDK_MEMORY_PATH.test(path) && !MODEL_PROFILE_PATH.test(path)) {
throw new Error("core_operation_path_not_allowed");
}
let response: { status: number; data: unknown };
@@ -257,51 +217,6 @@ export function createCoreOperations(attachment: {
code: `http_${response.status}_result_unknown`,
};
},
async uploadImportFile<T>(
input: {
bytes: Uint8Array;
filename: string;
inputKind: "raw_conversation" | "memory_content";
operationId: string;
},
signal?: AbortSignal,
) {
const query = new URLSearchParams({
filename: input.filename,
input_kind: input.inputKind,
operation_id: input.operationId,
});
let response: { status: number; data: unknown };
try {
response = await uploadClient.post(
`/api/v1/import-files?${query.toString()}`,
input.bytes,
{
headers: {
"Content-Type": "application/octet-stream",
"Content-Length": String(input.bytes.byteLength),
},
...(signal ? { signal } : {}),
},
);
} catch {
return { state: "unknown", code: "transport_result_unknown" };
}
if (response.status >= 200 && response.status < 300) {
return { state: "committed", data: response.data as T };
}
if (response.status >= 400 && response.status < 500) {
return {
state: "rejected",
code: safeErrorCode(response.data, response.status),
http_status: response.status,
};
}
return {
state: "unknown",
code: `http_${response.status}_result_unknown`,
};
},
};
}
+11 -24
View File
@@ -23,39 +23,26 @@ function accountHomeDirectory(): string | undefined {
async function main(): Promise<number> {
const arguments_ = process.argv.slice(2);
const command = arguments_[0];
const action = arguments_[1];
// A preview (`--yes` absent) must not create or write any diagnostic file:
// the contract is that it only emits a plan. Diagnostics for these commands
// are initialized only once the caller has confirmed the mutation.
const confirmed = arguments_.includes("--yes")
? true
: arguments_.includes("--confirm");
const confirmed = arguments_.includes("--yes");
const installLike = [
"install",
"upgrade",
"service-install",
"service-uninstall",
].includes(command ?? "");
const planFileApply =
(command === "sources" && action === "import") ||
(command === "models" && (action === "configure" || action === "probe")) ||
(command === "processing" && action === "apply") ||
(command === "memories" && action === "apply");
const isPreview = (installLike || planFileApply) && !confirmed;
const mutating =
[
"start",
"stop",
"restart",
"install",
"upgrade",
"service-install",
"service-uninstall",
].includes(command ?? "") ||
(command === "sources" && action === "import") ||
(command === "models" && ["configure", "probe"].includes(action ?? "")) ||
(command === "processing" && action === "apply") ||
(command === "memories" && action === "apply");
const isPreview = installLike && !confirmed;
const mutating = [
"start",
"stop",
"restart",
"install",
"upgrade",
"service-install",
"service-uninstall",
].includes(command ?? "");
let diagnostics: DiagnosticLogger | null = null;
if (mutating && !isPreview) {
try {
+93
View File
@@ -0,0 +1,93 @@
/**
* `memhub graph-width`: the user's decision when the embedding width changed
* and the memory graph no longer matches it (`POST /api/v1/graph-width`).
*
* `keep_existing` and `export_first` change nothing irreversible and run
* directly. `rebuild_with_new` archives the current graph and rebuilds it, so
* the CLI asks for `--yes` before sending it.
*/
import { parseGraphWidthDecisionReceipt } from "@one-memory/operations";
import type { CoreOperations } from "./core-operations.js";
import { PRODUCT } from "./version.js";
export const GRAPH_WIDTH_DECISIONS = [
"keep_existing",
"rebuild_with_new",
"export_first",
] as const;
export type GraphWidthDecision = (typeof GRAPH_WIDTH_DECISIONS)[number];
/** The one decision that cannot be taken back. */
export function graphWidthDecisionIsDestructive(
decision: GraphWidthDecision,
): boolean {
return decision === "rebuild_with_new";
}
export type GraphWidthDocument =
| {
schema_version: 1;
product: string;
command: "graph-width";
state: "committed";
decision: GraphWidthDecision;
receipt: Record<string, unknown>;
}
| {
schema_version: 1;
product: string;
command: "graph-width";
state: "rejected" | "unknown";
decision: GraphWidthDecision;
code: string;
};
export async function decideGraphWidth(
decision: GraphWidthDecision,
operations: CoreOperations,
): Promise<GraphWidthDocument> {
const base = {
schema_version: 1 as const,
product: PRODUCT,
command: "graph-width" as const,
decision,
};
const result = await operations.post<unknown>("/api/v1/graph-width", {
decision,
});
if (result.state !== "committed") {
return { ...base, state: result.state, code: result.code };
}
const receipt = parseGraphWidthDecisionReceipt(result.data);
if (receipt === null) {
return { ...base, state: "unknown", code: "graph_width_receipt_invalid" };
}
return {
...base,
state: "committed",
receipt: {
archived_path: receipt.archived_path ?? null,
exported_path: receipt.exported_path ?? null,
mismatch: receipt.mismatch ?? null,
},
};
}
export function graphWidthExitCode(document: GraphWidthDocument): number {
return document.state === "committed" ? 0 : 1;
}
export function renderGraphWidthText(document: GraphWidthDocument): string {
if (document.state === "committed") {
const lines = [`已执行图谱宽度决定:${document.decision}`];
if (document.receipt.exported_path) {
lines.push(`导出:${document.receipt.exported_path}`);
}
if (document.receipt.archived_path) {
lines.push(`旧图谱已归档:${document.receipt.archived_path}`);
}
return `${lines.join("\n")}\n`;
}
return `图谱宽度决定未完成:${document.state}(${document.code})\n`;
}
File diff suppressed because it is too large. Load diff
+82 -287
View File
@@ -1,4 +1,4 @@
import { createHash, randomUUID } from "node:crypto";
import { randomUUID } from "node:crypto";
import type { ImportJob, ModelProfile } from "@one-memory/contracts";
import {
buildModelProfileCreateRequest,
@@ -26,45 +26,7 @@ const PROVIDERS = [
"local_onnx",
] as const;
interface SourceImportPlanBody {
schema_version: 1;
kind: "source_import";
request_id: string;
source_ids: string[];
source_roots: Record<string, string>;
source_candidate_revisions: Record<string, number>;
confirm_environment_overrides: boolean;
full_rescan: false;
onboarding: true;
}
export type SourceImportPlan = SourceImportPlanBody & { digest: string };
interface ModelCreatePlanBody {
schema_version: 1;
kind: "model_profile_create";
operation_id: string;
config: Record<string, unknown>;
}
export type ModelCreatePlan = ModelCreatePlanBody & { digest: string };
/** A removal only needs an id: there is no config to review, and inventing one
* to fit the create shape would misrepresent what the command does.
*/
export interface ModelRemovePlan {
schema_version: 1;
kind: "model_profile_remove";
target: string;
}
export type FlowDocument =
| {
schema_version: 1;
product: string;
state: "preview";
plan: SourceImportPlan | ModelCreatePlan | ModelRemovePlan;
}
| {
schema_version: 1;
product: string;
@@ -133,10 +95,24 @@ function classifySourceImport(
return committed(operationId, receipt);
}
export async function planSourceImport(
/**
* Connect sources and import their history in one direct request.
*
* Reads the current candidates, refuses a source whose location comes from an
* environment override unless the caller confirmed overrides, and posts one
* `/sources/connect` (grant + history import + live sync). `requestId` is the
* idempotency key: re-running with the same id reads the first receipt.
*/
export async function connectSourcesWithImport(
reader: CoreReader,
sourceIds: readonly string[],
operations: CoreOperations,
input: {
sourceIds: readonly string[];
confirmOverrides: boolean;
requestId?: string;
},
): Promise<FlowDocument> {
const sourceIds = input.sourceIds;
if (
sourceIds.length < 1 ||
sourceIds.length > 32 ||
@@ -145,6 +121,8 @@ export async function planSourceImport(
) {
return invalid("invalid_source_selection");
}
const requestId = input.requestId ?? randomUUID();
if (!UUID.test(requestId)) return invalid("invalid_request_id");
const outcome = await reader<unknown>("/api/v1/sources");
if (outcome.state !== "ok") return fromReadFailure(outcome);
const candidates = validateSourceConnectionCandidates(outcome.data);
@@ -156,69 +134,59 @@ export async function planSourceImport(
return invalid("source_unavailable");
}
const items = selected.map(([candidate]) => candidate);
const body: SourceImportPlanBody = {
schema_version: 1,
kind: "source_import",
request_id: randomUUID(),
source_ids: [...sourceIds],
source_roots: Object.fromEntries(
items.map((item) => [item.source_id, item.candidate_root_id]),
),
source_candidate_revisions: Object.fromEntries(
items.flatMap((item) =>
item.candidate_revision === undefined
? []
: [[item.source_id, item.candidate_revision]],
),
),
confirm_environment_overrides: items.some(
(item) => item.requires_override_confirmation,
),
full_rescan: false,
onboarding: true,
};
return preview(withDigest(body));
}
export async function executeSourceImport(
value: unknown,
reader: CoreReader,
operations: CoreOperations,
): Promise<FlowDocument> {
const plan = validateSourcePlan(value);
if (!plan) return invalid("source_plan_invalid");
const needsOverride = items
.filter((item) => item.requires_override_confirmation)
.map((item) => item.source_id);
if (needsOverride.length > 0 && !input.confirmOverrides) {
return {
schema_version: 1,
product: PRODUCT,
state: "rejected",
operation_id: requestId,
code: "environment_override_confirmation_required",
receipt: { override_required_source_ids: needsOverride },
};
}
// The shared builder owns the connect body's field set and coalescing, so
// the CLI and the Web console send the same request by construction.
let request: ReturnType<typeof buildSourceImportRequest>;
try {
request = buildSourceImportRequest({
sourceIds: plan.source_ids,
sourceRoots: plan.source_roots,
sourceCandidateRevisions: plan.source_candidate_revisions,
confirmEnvironmentOverrides: plan.confirm_environment_overrides,
fullRescan: plan.full_rescan,
onboarding: plan.onboarding,
sourceIds: [...sourceIds],
sourceRoots: Object.fromEntries(
items.map((item) => [item.source_id, item.candidate_root_id]),
),
sourceCandidateRevisions: Object.fromEntries(
items.flatMap((item) =>
item.candidate_revision === undefined
? []
: [[item.source_id, item.candidate_revision]],
),
),
confirmEnvironmentOverrides: needsOverride.length > 0,
fullRescan: false,
onboarding: true,
enableSync: true,
requestId: plan.request_id,
requestId,
});
} catch (error) {
if (error instanceof SourceImportInputError) {
return invalid("source_plan_invalid");
return invalid("invalid_source_selection");
}
throw error;
}
const result = await operations.post<unknown>(request.path, request.body);
if (result.state === "rejected") {
return rejected(plan.request_id, result.code);
return rejected(requestId, result.code);
}
if (result.state === "committed") {
const receipt = validateSourceReceipt(result.data, plan.request_id);
if (!receipt) return unknown(plan.request_id, "source_receipt_invalid");
return classifySourceImport(plan.request_id, receipt);
const receipt = validateSourceReceipt(result.data, requestId);
if (!receipt) return unknown(requestId, "source_receipt_invalid");
return classifySourceImport(requestId, receipt);
}
const recovered = await readSourceReceipt(reader, plan.request_id);
const recovered = await readSourceReceipt(reader, requestId);
if (recovered.state === "unknown" || recovered.state === "not_found") {
return unknown(plan.request_id, result.code);
return unknown(requestId, result.code);
}
// A recovered receipt must be classified the same way as a fresh one: an
// all-rejected or all-busy replay is still not success. Only a terminal
@@ -262,26 +230,23 @@ export async function readSourceReceipt(
return committed(requestId, receipt);
}
export function planModelProfile(value: unknown): FlowDocument {
const config = validateModelConfig(value);
if (!config) return invalid("model_config_invalid");
const body: ModelCreatePlanBody = {
schema_version: 1,
kind: "model_profile_create",
operation_id: randomUUID(),
config,
};
return preview(withDigest(body));
}
export async function executeModelProfile(
/**
* Create a model profile directly from a bounded config (never containing the
* key) plus an optional key read from a controlled file.
*
* `operationId` is Core's creation idempotency key; the receipt can be read
* later with `readModelReceipt`.
*/
export async function createModelProfile(
value: unknown,
apiKey: string | undefined,
reader: CoreReader,
operations: CoreOperations,
operationId: string = randomUUID(),
): Promise<FlowDocument> {
const plan = validateModelPlan(value);
if (!plan) return invalid("model_plan_invalid");
const config = validateModelConfig(value);
if (!config) return invalid("model_config_invalid");
if (!UUID.test(operationId)) return invalid("invalid_operation_id");
// The shared create builder owns the body's allow-list, bounds and the
// creation_operation_id/api_key handling, so the CLI and the Web console
// send the same create request.
@@ -289,31 +254,31 @@ export async function executeModelProfile(
try {
request = buildModelProfileCreateRequest(
{
...(plan.config as Record<string, unknown>),
...config,
...(apiKey === undefined ? {} : { api_key: apiKey }),
} as Parameters<typeof buildModelProfileCreateRequest>[0],
plan.operation_id,
operationId,
);
} catch (error) {
if (error instanceof ModelProfileInputError) {
return invalid("model_plan_invalid");
return invalid("model_config_invalid");
}
throw error;
}
const result = await operations.post<unknown>(request.path, request.body);
if (result.state === "rejected") {
return rejected(plan.operation_id, result.code);
return rejected(operationId, result.code);
}
if (result.state === "committed") {
const profile = validateModelProfile(result.data);
return profile
? committed(plan.operation_id, projectProfile(profile))
: unknown(plan.operation_id, "model_receipt_invalid");
? committed(operationId, projectProfile(profile))
: unknown(operationId, "model_receipt_invalid");
}
const recovered = await readModelReceipt(reader, plan.operation_id);
const recovered = await readModelReceipt(reader, operationId);
return recovered.state === "committed"
? { ...recovered, state: "recovered" }
: unknown(plan.operation_id, result.code);
: unknown(operationId, result.code);
}
export async function readModelReceipt(
@@ -362,27 +327,6 @@ export function parseApiKey(bytes: Buffer | null): string | undefined {
}
export function renderFlowText(document: FlowDocument): string {
if (document.state === "preview") {
const plan = document.plan;
// A removal has no config digest or operation id to confirm: the target id
// *is* the thing being reviewed. Rendering fields it does not have would
// print "undefined".
if (plan.kind === "model_profile_remove") {
return [
`预览:${plan.kind}`,
`目标配置:${plan.target}`,
"未执行任何写操作。加 --confirm 才会真正删除。",
"",
].join("\n");
}
return [
`预览:${plan.kind}`,
`操作 ID:${"request_id" in plan ? plan.request_id : plan.operation_id}`,
`确认摘要:${plan.digest}`,
"未执行任何写操作。请把 --json 输出保存为 plan 文件,再用对应命令加 --confirm。",
"",
].join("\n");
}
const lines = [
`操作状态:${document.state}`,
`操作 ID:${document.operation_id ?? "无"}`,
@@ -407,98 +351,20 @@ export function renderFlowText(document: FlowDocument): string {
if (document.state === "partial" && Array.isArray(accepted)) {
lines.push(` 已接入:${accepted.join("、")}`);
}
const overrides = document.receipt.override_required_source_ids;
if (Array.isArray(overrides)) {
lines.push(
` 这些来源的目录来自环境变量覆盖,需要确认:${overrides.join("、")}`,
" 确认目录无误后加 --confirm-overrides 重新运行。",
);
}
}
lines.push("");
return lines.join("\n");
}
export function flowExitCode(document: FlowDocument): number {
return ["preview", "committed", "recovered"].includes(document.state) ? 0 : 1;
}
export function inspectExecutionPlan(value: unknown): {
kind: "source_import" | "model_profile_create";
operationId: string;
digest: string;
} | null {
const source = validateSourcePlan(value);
if (source) {
return {
kind: source.kind,
operationId: source.request_id,
digest: source.digest,
};
}
const model = validateModelPlan(value);
return model
? {
kind: model.kind,
operationId: model.operation_id,
digest: model.digest,
}
: null;
}
function validateSourcePlan(value: unknown): SourceImportPlan | null {
const plan = record(value);
const requestId = asString(plan?.request_id);
if (
plan?.schema_version !== 1 ||
plan.kind !== "source_import" ||
!UUID.test(requestId) ||
!Array.isArray(plan.source_ids) ||
plan.source_ids.length < 1 ||
plan.source_ids.length > 32 ||
plan.source_ids.some(
(id) => typeof id !== "string" || !SOURCE_ID.test(id),
) ||
new Set(plan.source_ids).size !== plan.source_ids.length ||
plan.full_rescan !== false ||
plan.onboarding !== true ||
typeof plan.confirm_environment_overrides !== "boolean" ||
typeof plan.digest !== "string"
) {
return null;
}
const sourceIds = plan.source_ids as string[];
const roots = stringMap(plan.source_roots, sourceIds);
const revisions = numberMap(plan.source_candidate_revisions, sourceIds);
if (!roots || !revisions) return null;
const body: SourceImportPlanBody = {
schema_version: 1,
kind: "source_import",
request_id: requestId,
source_ids: [...sourceIds],
source_roots: roots,
source_candidate_revisions: revisions,
confirm_environment_overrides: plan.confirm_environment_overrides,
full_rescan: false,
onboarding: true,
};
const expected = withDigest(body);
return timingSafeText(expected.digest, plan.digest) ? expected : null;
}
function validateModelPlan(value: unknown): ModelCreatePlan | null {
const plan = record(value);
const operationId = asString(plan?.operation_id);
if (
plan?.schema_version !== 1 ||
plan.kind !== "model_profile_create" ||
!UUID.test(operationId) ||
typeof plan.digest !== "string"
) {
return null;
}
const config = validateModelConfig(plan.config);
if (!config) return null;
const expected = withDigest<ModelCreatePlanBody>({
schema_version: 1,
kind: "model_profile_create",
operation_id: operationId,
config,
});
return timingSafeText(expected.digest, plan.digest) ? expected : null;
return ["committed", "recovered"].includes(document.state) ? 0 : 1;
}
function validateModelConfig(value: unknown): Record<string, unknown> | null {
@@ -644,37 +510,6 @@ function projectJob(job: ImportJob): Record<string, unknown> {
};
}
function withDigest<T extends object>(body: T): T & { digest: string } {
return { ...body, digest: digest(body) };
}
function digest(value: unknown): string {
return createHash("sha256").update(canonical(value)).digest("hex");
}
function canonical(value: unknown): string {
if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`;
if (value && typeof value === "object") {
return `{${Object.keys(value)
.sort()
.map(
(key) =>
`${JSON.stringify(key)}:${canonical((value as Record<string, unknown>)[key])}`,
)
.join(",")}}`;
}
return JSON.stringify(value);
}
function preview(plan: SourceImportPlan | ModelCreatePlan): FlowDocument {
return {
schema_version: 1,
product: PRODUCT,
state: "preview",
plan,
};
}
function committed(
operationId: string,
receipt: Record<string, unknown>,
@@ -767,43 +602,3 @@ function optionalInteger(
(value as number) <= maximum)
);
}
function stringMap(
value: unknown,
keys: readonly string[],
): Record<string, string> | null {
const map = record(value);
if (
!map ||
Object.keys(map).length !== keys.length ||
keys.some((key) => !boundedText(map[key], 256))
) {
return null;
}
return Object.fromEntries(keys.map((key) => [key, map[key] as string]));
}
function numberMap(
value: unknown,
allowedKeys: readonly string[],
): Record<string, number> | null {
const map = record(value);
if (
!map ||
Object.entries(map).some(
([key, revision]) =>
!allowedKeys.includes(key) ||
!Number.isSafeInteger(revision) ||
(revision as number) < 0,
)
) {
return null;
}
return Object.fromEntries(
Object.entries(map).map(([key, revision]) => [key, revision as number]),
);
}
function timingSafeText(expected: string, actual: unknown): boolean {
return typeof actual === "string" && expected === actual;
}
@@ -1,64 +0,0 @@
import path from "node:path";
import { describe, expect, it } from "vitest";
import { initPlanAttemptPath } from "./cli.js";
import { interactiveInitAttemptPath } from "./init-attempt.js";
// Regression: `init --source <id> --import` used to pass the literal string
// `"init"` to `claimPlanAttempt`, which resolved to `<cwd>/init.one-memory-attempt`.
// Two consequences: the fence was shared across unrelated runs in the same
// directory, and any unrelated file named `init` could make a valid import
// report a misleading usage error (exit 2) instead of reading its receipt.
describe("init import attempt fence", () => {
it("lives under the data dir, not the current working directory", () => {
const fence = initPlanAttemptPath(
"/tmp/example data",
"11111111-2222-4333-8444-555555555555",
);
expect(fence).toBe(
path.join(
"/tmp/example data",
".cli-init-attempt-11111111-2222-4333-8444-555555555555",
),
);
expect(path.isAbsolute(fence)).toBe(true);
expect(fence).not.toBe("init");
// Directly under the data dir: no nested directory that could fail to be
// created and turn a valid import into a usage error.
expect(path.dirname(fence)).toBe("/tmp/example data");
});
it("is per data dir and per operation, so installs/runs cannot collide", () => {
const a = initPlanAttemptPath("/tmp/data-a", "op-1");
const b = initPlanAttemptPath("/tmp/data-b", "op-1");
const c = initPlanAttemptPath("/tmp/data-a", "op-2");
expect(new Set([a, b, c]).size).toBe(3);
});
});
// QA-CLI R3 F1: the interactive wizard passed a CWD-relative singleton
// (`init` or `<journalDir>/init`) to `claimPlanAttempt`. A stale
// `init.one-memory-attempt` in the working directory then made every later
// `memhub init` fail with "交互计划无效 / exit 2".
describe("interactive init attempt fence", () => {
it("lives under the data dir and is unique per operation", () => {
const fence = interactiveInitAttemptPath(
"/tmp/data",
"11111111-2222-4333-8444-555555555555",
);
expect(fence).toBe(
path.join(
"/tmp/data",
".cli-init-attempt-11111111-2222-4333-8444-555555555555",
),
);
expect(path.isAbsolute(fence)).toBe(true);
expect(fence).not.toBe("init");
});
it("does not reuse one fence for two distinct plans", () => {
const first = interactiveInitAttemptPath("/tmp/data", "op-1");
const second = interactiveInitAttemptPath("/tmp/data", "op-2");
expect(first).not.toBe(second);
});
});
-16
View File
@@ -1,16 +0,0 @@
import path from "node:path";
/** Where the interactive `init` wizard keeps its durable attempt fence.
*
* The wizard originally passed a CWD-relative singleton (`init`) to
* `claimPlanAttempt`. That shared one fence across unrelated runs and let a
* stale `init.one-memory-attempt` in the working directory turn every later
* wizard run into "交互计划无效 / exit 2". Keep the fence under the Runtime's
* data dir, keyed by the operation id, exactly like `init --source --import`.
*/
export function interactiveInitAttemptPath(
dataDir: string,
operationId: string,
): string {
return path.join(dataDir, `.cli-init-attempt-${operationId}`);
}
+6 -6
View File
@@ -418,9 +418,9 @@ describe("A2 dual-mode init", () => {
});
// QA-CLI R3 F1: a stale CWD `init.one-memory-attempt` used to poison every
// later wizard run. The fence now lives under the isolated data dir and is
// keyed by the operation id.
it("ignores a stale CWD init fence and keeps its fence under the data dir", async () => {
// later wizard run. There are no plan files or attempt fences any more: the
// connect request id is the idempotency key Core keeps.
it("ignores a stale CWD init fence and writes no fence files", async () => {
transport.get.mockImplementation(async (url: string) =>
url === "/api/v1/sources"
? { status: 200, data: [candidate("codex", "Codex")] }
@@ -471,7 +471,7 @@ describe("A2 dual-mode init", () => {
const dataEntries = await readdir(dataDir);
expect(
dataEntries.some((entry) => entry.startsWith(".cli-init-attempt-")),
).toBe(true);
).toBe(false);
});
it("keeps selected Harnesses frozen without creating history jobs", async () => {
@@ -506,7 +506,7 @@ describe("A2 dual-mode init", () => {
});
});
describe("A19 init --all-detected", () => {
describe("A19 sources connect --all-detected (old spelling: init --all-detected)", () => {
it("connects every found source in one batch call without prompting", async () => {
transport.get.mockImplementation(async (url: string) =>
url === "/api/v1/sources"
@@ -556,7 +556,7 @@ describe("A19 init --all-detected", () => {
expect(prompter.asked).toBe(0);
expect(code).toBe(0);
const document = JSON.parse(streams.value());
expect(document.command).toBe("init");
expect(document.command).toBe("sources connect");
expect(document.state).toBe("committed");
expect(
document.sources.map((row: { source_id: string }) => row.source_id),
+23 -50
View File
@@ -4,18 +4,11 @@ import type {
CoreDeleteOperations,
CoreOperations,
} from "./core-operations.js";
import {
executeModelProfile,
type FlowDocument,
planModelProfile,
} from "./headless-flows.js";
import { createModelProfile, type FlowDocument } from "./headless-flows.js";
import { approvalFingerprint, listHosts, readHost } from "./host-flow.js";
import { parseSelection, type WizardPrompter } from "./init-wizard.js";
import { executeModelProbe, planModelProbe } from "./model-probe-flow.js";
import {
executeModelProfileActivate,
planModelProfileActivate,
} from "./model-write-flows.js";
import { startModelProbe } from "./model-probe-flow.js";
import { activateModelProfile } from "./model-write-flows.js";
type ProviderPreset = {
id: string;
@@ -359,30 +352,26 @@ export async function runInteractiveModelSetup(
);
})
: new Map<string, string>();
const planned = planModelProfile({
name: preset.label,
provider_kind: preset.provider_kind,
base_url: baseUrl,
model,
embedding_provider_kind: "local_onnx",
embedding_model: "bge-small-zh-v1.5",
embedding_dimensions: 512,
timeout_seconds: 90,
max_concurrency: 4,
disable_thinking: true,
});
if (
planned.state !== "preview" ||
planned.plan.kind !== "model_profile_create"
) {
return { state: "failed", code: "model_config_invalid" };
}
const created = await executeModelProfile(
planned.plan,
const created = await createModelProfile(
{
name: preset.label,
provider_kind: preset.provider_kind,
base_url: baseUrl,
model,
embedding_provider_kind: "local_onnx",
embedding_model: "bge-small-zh-v1.5",
embedding_dimensions: 512,
timeout_seconds: 90,
max_concurrency: 4,
disable_thinking: true,
},
apiKey,
clients.reader,
clients.operations,
);
if (created.state === "invalid") {
return { state: "failed", code: "model_config_invalid" };
}
const profile = receiptProfile(created);
if (profile === null) {
prompter.write("模型配置未保存。");
@@ -391,18 +380,11 @@ export async function runInteractiveModelSetup(
}
try {
const probePlan = await planModelProbe(
const probed = await startModelProbe(
clients.reader,
clients.operations,
profile.profile_id,
);
const probed =
probePlan.state === "preview"
? await executeModelProbe(
probePlan.plan,
clients.reader,
clients.operations,
)
: probePlan;
if (!probePassed(probed)) {
await clients.operations.delete(
`/api/v1/model-profiles/${encodeURIComponent(profile.profile_id)}`,
@@ -421,17 +403,8 @@ export async function runInteractiveModelSetup(
throw error;
}
const activation = planModelProfileActivate({
profileId: profile.profile_id,
});
if (
activation.state !== "preview" ||
activation.plan.kind !== "model_profile_activate"
) {
return { state: "failed", code: "model_activate_plan_invalid" };
}
const activated = await executeModelProfileActivate(
activation.plan,
const activated = await activateModelProfile(
profile.profile_id,
clients.operations,
);
if (activated.state !== "committed") {
+13 -13
View File
@@ -101,7 +101,7 @@ describe("memhub jobs", () => {
expect(transport.post).not.toHaveBeenCalled();
});
it("start previews without a POST", async () => {
it("start posts directly, without a preview step", async () => {
const output = captureStdout();
let code: number;
try {
@@ -110,14 +110,14 @@ describe("memhub jobs", () => {
output.restore();
}
expect(code!).toBe(0);
expect(transport.post).not.toHaveBeenCalled();
expect(JSON.parse(output.value())).toMatchObject({
state: "preview",
plan: { source_id: "codex", operation: "history" },
});
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/import-jobs",
{ source_id: "codex", operation: "history" },
undefined,
);
});
it("start --confirm posts the exact Core request", async () => {
it("start still accepts a legacy --confirm", async () => {
const output = captureStdout();
let code: number;
try {
@@ -146,7 +146,7 @@ describe("memhub jobs", () => {
});
it.each(["pause", "resume", "cancel"] as const)(
"%s previews without a POST",
"%s posts the action endpoint directly",
async (action) => {
const output = captureStdout();
let code: number;
@@ -156,11 +156,11 @@ describe("memhub jobs", () => {
output.restore();
}
expect(code!).toBe(0);
expect(transport.post).not.toHaveBeenCalled();
expect(JSON.parse(output.value())).toMatchObject({
state: "preview",
plan: { job_id: "job-1", action },
});
expect(transport.post).toHaveBeenCalledWith(
`/api/v1/import-jobs/job-1/${action}`,
{},
undefined,
);
},
);
+583
View File
@@ -0,0 +1,583 @@
import { mkdtemp, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
}));
vi.mock("axios", () => ({
default: { create: vi.fn(() => transport), get: vi.fn() },
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
const REQUEST = "33333333-3333-4333-8333-333333333333";
const EVIDENCE = "44444444-4444-4444-8444-444444444444";
function addReceipt(state = "pending", memoryIds: string[] = []) {
return {
evidence_id: EVIDENCE,
text: "Prefers tea",
scope: "all-apps",
created_at: "2026-10-27T00:00:00Z",
state,
task_id: null,
memory_ids: memoryIds,
error: null,
};
}
function memoryRecord(overrides: Record<string, unknown> = {}) {
return {
memory_id: "exp-1",
content: "deploy\nrun make",
layer: "experience",
status: "active",
is_latest: true,
supersedes: [],
superseded_by: [],
tags: [],
owner: null,
source_raw_ids: [],
user_id: "local",
agent_id: "",
session_id: null,
source: "codebuddy",
memory_at: null,
gmt_created: 1_790_000_000,
scene: "working",
kind: "experience",
variant: "working",
task: "deploy",
solution: "run make",
...overrides,
};
}
beforeEach(() => {
vi.clearAllMocks();
transport.get.mockReset();
transport.post.mockReset();
transport.put.mockReset();
transport.patch.mockReset();
transport.delete.mockReset();
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue({
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
});
});
describe("memories add", () => {
it("adds one memory directly and prints the receipt with its request id", async () => {
transport.post.mockResolvedValue({ status: 201, data: addReceipt() });
const result = await capturedRun([
"memories",
"add",
"Prefers tea",
"--json",
]);
expect(result.code).toBe(0);
const [url, body] = transport.post.mock.calls[0];
expect(url).toBe("/api/v2/memories");
expect(body).toMatchObject({ text: "Prefers tea", scope: "all-apps" });
expect(body.request_id).toMatch(/^[0-9a-f-]{36}$/);
expect(JSON.parse(result.stdout)).toMatchObject({
schema_version: 2,
command: "memories add",
state: "accepted",
request_id: body.request_id,
receipt: { evidence_id: EVIDENCE, state: "pending" },
});
});
it("reuses --request-id so a retry is idempotent, and says so in text", async () => {
transport.post.mockResolvedValue({ status: 201, data: addReceipt() });
const result = await capturedRun([
"memories",
"add",
"Prefers tea",
"--request-id",
REQUEST,
]);
expect(result.code).toBe(0);
expect(transport.post.mock.calls[0][1].request_id).toBe(REQUEST);
expect(result.stdout).toContain(`request_id: ${REQUEST}`);
expect(result.stdout).toContain(`memhub memories add-status ${REQUEST}`);
});
it("reads the text from a bounded file", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "om-memory-add-"));
const file = path.join(directory, "memory.txt");
await writeFile(file, "Prefers tea\n", { mode: 0o600 });
transport.post.mockResolvedValue({ status: 201, data: addReceipt() });
const result = await capturedRun([
"memories",
"add",
"--content-file",
file,
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post.mock.calls[0][1].text).toBe("Prefers tea");
});
it("refuses bad input before contacting Core", async () => {
for (const argv of [
["memories", "add"],
["memories", "add", "a", "--content-file", "/tmp/x"],
["memories", "add", "a", "--request-id", "not-a-uuid"],
["memories", "add", " "],
["memories", "add", "x".repeat(12_001)],
]) {
expect((await capturedRun(argv)).code).toBe(2);
}
expect(transport.post).not.toHaveBeenCalled();
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
});
it("reports a Core conflict as rejected, on stdout, exit 1", async () => {
transport.post.mockResolvedValue({
status: 409,
data: { detail: "memory_add_conflict" },
});
const result = await capturedRun([
"memories",
"add",
"Other text",
"--request-id",
REQUEST,
"--json",
]);
expect(result.code).toBe(1);
expect(JSON.parse(result.stdout)).toMatchObject({
state: "rejected",
code: "memory_add_conflict",
request_id: REQUEST,
});
});
it("never reports a malformed success body as accepted", async () => {
transport.post.mockResolvedValue({
status: 201,
data: { evidence_id: "x" },
});
const result = await capturedRun(["memories", "add", "tea", "--json"]);
expect(result.code).toBe(1);
expect(JSON.parse(result.stdout).state).toBe("unknown");
});
it("answers a stopped Runtime with exit 3 and writes nothing", async () => {
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "stopped",
code: "runtime_not_running",
});
const result = await capturedRun(["memories", "add", "tea", "--json"]);
expect(result.code).toBe(3);
expect(JSON.parse(result.stdout).code).toBe("runtime_not_running");
expect(transport.post).not.toHaveBeenCalled();
});
});
describe("memories add-status", () => {
it("reads the addition by request id", async () => {
transport.get.mockResolvedValue({
status: 200,
data: addReceipt("ready", ["m-1"]),
});
const result = await capturedRun([
"memories",
"add-status",
REQUEST,
"--json",
]);
expect(result.code).toBe(0);
expect(transport.get).toHaveBeenCalledWith(
`/api/v2/memories/additions/${REQUEST}`,
undefined,
);
expect(JSON.parse(result.stdout)).toMatchObject({
command: "memories add-status",
state: "accepted",
receipt: { state: "ready", memory_ids: ["m-1"] },
});
});
it("reports an unknown request id as not_found, exit 1", async () => {
transport.get.mockResolvedValue({
status: 404,
data: { detail: "memory_add_not_found" },
});
const result = await capturedRun(["memories", "status", REQUEST]);
expect(result.code).toBe(1);
expect(result.stdout).toContain("未找到该 request id");
});
it("refuses a non-UUID request id", async () => {
expect((await capturedRun(["memories", "add-status", "abc"])).code).toBe(2);
expect(transport.get).not.toHaveBeenCalled();
});
});
describe("memories update / delete pass the namespace", () => {
it("updates an experience with kind, variant and task, directly", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "om-memory-update-"));
const file = path.join(directory, "solution.txt");
await writeFile(file, "run make deploy\n", { mode: 0o600 });
transport.put.mockResolvedValue({ status: 200, data: memoryRecord() });
const result = await capturedRun([
"memories",
"update",
"exp-1",
"--content-file",
file,
"--kind",
"experience",
"--variant",
"working",
"--task",
"deploy",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.put).toHaveBeenCalledWith(
"/api/v2/memories/exp-1",
{
content: "run make deploy",
kind: "experience",
variant: "working",
task: "deploy",
},
undefined,
);
expect(JSON.parse(result.stdout)).toMatchObject({
schema_version: 2,
state: "committed",
operation: "update",
});
});
it("refuses --task on anything but an experience, and a lone --variant", async () => {
for (const argv of [
["memories", "update", "m-1", "--content-file", "/tmp/x", "--task", "t"],
[
"memories",
"update",
"m-1",
"--content-file",
"/tmp/x",
"--kind",
"fact",
"--variant",
"coding",
"--task",
"t",
],
["memories", "delete", "m-1", "--variant", "coding"],
["memories", "delete", "m-1", "--kind", "chat", "--variant", "coding"],
["memories", "show", "m-1", "--kind", "nope"],
]) {
expect((await capturedRun(argv)).code).toBe(2);
}
expect(transport.put).not.toHaveBeenCalled();
expect(transport.delete).not.toHaveBeenCalled();
});
it("previews a delete without --yes and changes nothing", async () => {
const result = await capturedRun([
"memories",
"delete",
"fact-1",
"--kind",
"fact",
"--variant",
"coding",
"--json",
]);
expect(result.code).toBe(0);
expect(JSON.parse(result.stdout)).toMatchObject({
schema_version: 2,
command: "memories delete",
state: "preview",
target: "fact-1",
applied: false,
});
expect(JSON.parse(result.stdout).next_step).toContain(
"--kind fact --variant coding --yes",
);
expect(transport.delete).not.toHaveBeenCalled();
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
});
it("deletes with --yes (or the legacy --confirm) and names the namespace", async () => {
transport.delete.mockResolvedValue({ status: 204, data: "" });
for (const flag of ["--yes", "--confirm"]) {
const result = await capturedRun([
"memories",
"delete",
"fact-1",
"--kind",
"fact",
"--variant",
"coding",
flag,
"--json",
]);
expect(result.code).toBe(0);
expect(JSON.parse(result.stdout)).toMatchObject({
schema_version: 2,
state: "committed",
operation: "delete",
});
}
expect(transport.delete).toHaveBeenCalledWith(
"/api/v2/memories/fact-1?kind=fact&variant=coding",
undefined,
);
});
it("explains an ambiguous id instead of a bare code", async () => {
transport.delete.mockResolvedValue({
status: 409,
data: { detail: "memory_id_ambiguous" },
});
const result = await capturedRun(["memories", "delete", "dup", "--yes"]);
expect(result.code).toBe(1);
expect(result.stdout).toContain("--kind");
});
it("shows a memory in its namespace", async () => {
transport.get.mockResolvedValue({ status: 200, data: memoryRecord() });
const result = await capturedRun([
"memories",
"show",
"exp-1",
"--kind",
"experience",
"--variant",
"working",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.get).toHaveBeenCalledWith("/api/v2/memories/exp-1", {
params: { kind: "experience", variant: "working" },
});
const document = JSON.parse(result.stdout);
expect(document.schema_version).toBe(2);
expect(document).not.toHaveProperty("evidence");
expect(document).not.toHaveProperty("history");
expect(document.memory).toMatchObject({
kind: "experience",
variant: "working",
task: "deploy",
});
});
});
describe("memories list filters", () => {
const page = {
memories: [memoryRecord()],
total: 1,
limit: 100,
offset: 0,
search_window: 200,
window_id: "w1",
window_reset: false,
};
it("sends --scene, --source and --query to the library", async () => {
transport.get.mockResolvedValue({ status: 200, data: page });
const result = await capturedRun([
"memories",
"list",
"--scene",
"working",
"--source",
"codebuddy",
"--query",
"deploy",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.get).toHaveBeenCalledWith("/api/v2/memory-library", {
params: {
offset: 0,
limit: 100,
scene: "working",
source: "codebuddy",
query: "deploy",
},
});
const document = JSON.parse(result.stdout);
expect(document.schema_version).toBe(2);
expect(document.page).toMatchObject({
state: "ok",
scene: "working",
query: "deploy",
search_window: 200,
});
expect(document.memories[0]).toMatchObject({
kind: "experience",
variant: "working",
});
});
it("omits scene for the default all", async () => {
transport.get.mockResolvedValue({ status: 200, data: page });
await capturedRun(["memories", "list", "--scene", "all", "--json"]);
expect(transport.get.mock.calls[0][1].params).not.toHaveProperty("scene");
});
it("validates --status, --scene and --query locally instead of reporting a busy backend", async () => {
for (const argv of [
["memories", "list", "--status", "bogus"],
["memories", "list", "--scene", "normal"],
["memories", "list", "--query", "x", "--status", "archived"],
["memories", "list", "--source", "Bad Source"],
]) {
expect((await capturedRun(argv)).code).toBe(2);
}
expect(transport.get).not.toHaveBeenCalled();
});
it("reports a Core 422 as rejected, not as 后台正忙", async () => {
transport.get.mockResolvedValue({
status: 422,
data: { detail: "sdk_search_supports_active_status_only" },
});
const result = await capturedRun(["memories", "list"]);
expect(result.code).toBe(1);
expect(result.stdout).not.toContain("后台正忙");
expect(result.stdout).toContain("sdk_search_supports_active_status_only");
});
});
describe("graph-width", () => {
it("runs a non-destructive decision directly", async () => {
transport.post.mockResolvedValue({
status: 200,
data: { archived_path: null, exported_path: "/tmp/x", mismatch: null },
});
const result = await capturedRun([
"graph-width",
"--decision",
"export_first",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/graph-width",
{ decision: "export_first" },
undefined,
);
expect(JSON.parse(result.stdout).state).toBe("committed");
});
it("previews rebuild_with_new without --yes and applies it with --yes", async () => {
const preview = await capturedRun([
"graph-width",
"--decision",
"rebuild_with_new",
"--json",
]);
expect(preview.code).toBe(0);
expect(JSON.parse(preview.stdout)).toMatchObject({
state: "preview",
applied: false,
});
expect(transport.post).not.toHaveBeenCalled();
transport.post.mockResolvedValue({
status: 200,
data: { archived_path: "/tmp/old", exported_path: null, mismatch: null },
});
const applied = await capturedRun([
"graph",
"width",
"--decision",
"rebuild_with_new",
"--yes",
"--json",
]);
expect(applied.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/graph-width",
{ decision: "rebuild_with_new" },
undefined,
);
});
});
describe("removed commands", () => {
it.each([
["notes", "add", "--content-file", "/tmp/x"],
["notes", "process", REQUEST, "--confirm"],
["calls", "clear", "--all", "--confirm"],
["processing", "extract", "--evidence", REQUEST, "--confirm"],
["extraction", "set", "--window", "6m", "--confirm"],
["import-file", "upload", "--file", "/tmp/x", "--confirm"],
])("%s is an unknown command that writes nothing", async (...argv) => {
const result = await capturedRun(argv as string[]);
expect(result.code).toBe(2);
expect(result.stderr).toContain("未知命令");
expect(transport.post).not.toHaveBeenCalled();
});
});
async function capturedRun(arguments_: string[]) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return {
code: await run(arguments_, {
ONE_MEMORY_DATA_DIR: "/synthetic/data",
}),
stdout,
stderr,
};
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
+244 -21
View File
@@ -1,5 +1,17 @@
import type { SdkMemoryPage, SdkMemoryRecord } from "@one-memory/contracts";
import { randomUUID } from "node:crypto";
import type {
MemoryAddReceipt,
SdkMemoryPage,
SdkMemoryRecord,
} from "@one-memory/contracts";
import {
buildMemoryAddRequest,
MemoryAddInputError,
memoryAddStatusPath,
parseMemoryAddReceipt,
} from "@one-memory/operations";
import { type CoreReader, RESERVED_MEMORY_SEGMENTS } from "./core-client.js";
import type { CoreOperations } from "./core-operations.js";
import { PRODUCT } from "./version.js";
/** Output budget: a long memory is shortened visibly, never dropped silently. */
@@ -38,27 +50,52 @@ export function parseMemoryContent(bytes: Buffer): string | null {
: null;
}
export const MEMORY_SCENES = ["all", "chat", "working", "coding"] as const;
export type MemoryScene = (typeof MEMORY_SCENES)[number];
export const MEMORY_KINDS = ["chat", "experience", "fact"] as const;
export type MemoryKind = (typeof MEMORY_KINDS)[number];
export const MEMORY_VARIANTS = ["working", "coding"] as const;
export type MemoryVariant = (typeof MEMORY_VARIANTS)[number];
/** Which SDK namespace a memory lives in. A chat memory needs neither; a
* working/coding experience or fact needs both so Core does not have to guess
* (an id present in two namespaces is refused as ambiguous). */
export interface MemoryNamespace {
kind?: MemoryKind;
variant?: MemoryVariant;
}
export function namespaceParameters(
namespace: MemoryNamespace,
): Record<string, string> {
return {
...(namespace.kind === undefined ? {} : { kind: namespace.kind }),
...(namespace.variant === undefined ? {} : { variant: namespace.variant }),
};
}
export interface MemoryQuery {
query?: string;
status?: string;
source?: string;
scene?: MemoryScene;
offset: number;
limit: number;
}
/** Every `memories` document is schema 2: list/show share it with
* add/update/delete, and the v1-era `evidence`/`history` fields are gone. */
export interface MemoryPageDocument {
schema_version: 1;
schema_version: 2;
product: string;
page: Record<string, unknown>;
memories: Record<string, unknown>[];
}
export interface MemoryDetailDocument {
schema_version: 1;
schema_version: 2;
product: string;
memory: Record<string, unknown>;
evidence: Record<string, unknown>[];
history: Record<string, unknown>[];
}
export async function readMemoryPage(
@@ -70,10 +107,15 @@ export async function readMemoryPage(
limit: query.limit,
};
if (query.status !== undefined) parameters.statuses = query.status;
if (query.source !== undefined) parameters.source = query.source;
if (query.query !== undefined) parameters.query = query.query;
if (query.scene !== undefined && query.scene !== "all") {
parameters.scene = query.scene;
}
const outcome = await reader<unknown>("/api/v2/memory-library", parameters);
if (outcome.state !== "ok") {
return {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
page: { state: outcome.state, code: outcome.code },
memories: [],
@@ -82,7 +124,7 @@ export async function readMemoryPage(
const page = sdkMemoryPage(outcome.data);
if (page === null) {
return {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
page: { state: "invalid", code: "response_invalid" },
memories: [],
@@ -90,15 +132,21 @@ export async function readMemoryPage(
}
const items = page.memories;
return {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
page: {
state: "ok",
scene: query.scene ?? "all",
...(query.query === undefined ? {} : { query: query.query }),
total: page.total,
offset: page.offset,
limit: page.limit,
returned: items.length,
has_more: page.offset + items.length < page.total,
// A keyword search only looks among this many most relevant memories.
...(typeof page.search_window === "number"
? { search_window: page.search_window }
: {}),
},
memories: items.map((item) => describeMemory(item)),
};
@@ -107,42 +155,195 @@ export async function readMemoryPage(
export async function readMemoryDetail(
reader: CoreReader,
memoryId: string,
namespace: MemoryNamespace = {},
): Promise<MemoryDetailDocument> {
const outcome = await reader<unknown>(`/api/v2/memories/${memoryId}`);
const parameters = namespaceParameters(namespace);
const outcome = await reader<unknown>(
`/api/v2/memories/${memoryId}`,
Object.keys(parameters).length > 0 ? parameters : undefined,
);
if (outcome.state !== "ok") {
return {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
memory: { state: outcome.state, code: outcome.code },
evidence: [],
history: [],
};
}
const memory = sdkMemory(outcome.data);
if (memory === null || memory.memory_id !== memoryId) {
return {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
memory: { state: "invalid", code: "response_invalid" },
evidence: [],
history: [],
};
}
return {
schema_version: 1,
schema_version: 2,
product: PRODUCT,
memory: { state: "ok", ...describeMemory(memory) },
evidence: [],
history: [],
};
}
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
/** The same bound the app's 添加记忆 box applies. */
export const MEMORY_ADD_MAX_CHARACTERS = 12_000;
export interface MemoryAddDocument {
schema_version: 2;
product: string;
command: "memories add" | "memories add-status";
state: "accepted" | "rejected" | "unknown" | "not_found" | "invalid";
request_id: string;
code?: string;
receipt?: Record<string, unknown>;
}
export function isRequestId(value: string): boolean {
return UUID.test(value);
}
/** A memory text: non-blank, bounded, no control characters except tab and
* newline. */
export function validMemoryText(text: string): boolean {
return (
text.trim().length > 0 &&
[...text].length <= MEMORY_ADD_MAX_CHARACTERS &&
![...text].some((character) => {
const code = character.charCodeAt(0);
return code <= 8 || (code >= 11 && code <= 31) || code === 127;
})
);
}
function addReceipt(receipt: MemoryAddReceipt): Record<string, unknown> {
return {
evidence_id: receipt.evidence_id,
state: receipt.state,
created_at: receipt.created_at,
task_id: receipt.task_id,
memory_ids: receipt.memory_ids,
error: receipt.error,
};
}
/**
* Add one memory directly, the same request as the app's 添加记忆.
*
* `requestId` is Core's idempotency key: retrying with the same id and text
* returns the first receipt; the same id with other text is refused.
*/
export async function addMemory(
operations: CoreOperations,
text: string,
requestId: string = randomUUID(),
): Promise<MemoryAddDocument> {
const base = {
schema_version: 2 as const,
product: PRODUCT,
command: "memories add" as const,
request_id: requestId,
};
if (!validMemoryText(text) || !UUID.test(requestId)) {
return { ...base, state: "invalid", code: "memory_add_invalid" };
}
let request: ReturnType<typeof buildMemoryAddRequest>;
try {
request = buildMemoryAddRequest({
request_id: requestId,
text,
scope: "all-apps",
});
} catch (error) {
if (error instanceof MemoryAddInputError) {
return { ...base, state: "invalid", code: "memory_add_invalid" };
}
throw error;
}
const result = await operations.post<unknown>(request.path, request.body);
if (result.state !== "committed") {
return { ...base, state: result.state, code: result.code };
}
const receipt = parseMemoryAddReceipt(result.data);
return receipt === null
? { ...base, state: "unknown", code: "memory_add_receipt_invalid" }
: { ...base, state: "accepted", receipt: addReceipt(receipt) };
}
/** Read how one addition is doing, by the request id it was submitted with. */
export async function readMemoryAddStatus(
reader: CoreReader,
requestId: string,
): Promise<MemoryAddDocument> {
const base = {
schema_version: 2 as const,
product: PRODUCT,
command: "memories add-status" as const,
request_id: requestId,
};
if (!UUID.test(requestId)) {
return { ...base, state: "invalid", code: "invalid_request_id" };
}
const outcome = await reader<unknown>(memoryAddStatusPath(requestId));
if (outcome.state === "not_found") {
return { ...base, state: "not_found", code: "memory_add_not_found" };
}
if (outcome.state !== "ok") {
return {
...base,
state: outcome.state === "rejected" ? "rejected" : "unknown",
code: outcome.code,
};
}
const receipt = parseMemoryAddReceipt(outcome.data);
return receipt === null
? { ...base, state: "unknown", code: "memory_add_receipt_invalid" }
: { ...base, state: "accepted", receipt: addReceipt(receipt) };
}
export function memoryAddExitCode(document: MemoryAddDocument): number {
return document.state === "accepted" ? 0 : 1;
}
export function renderMemoryAddText(document: MemoryAddDocument): string {
if (document.state === "accepted" && document.receipt) {
return [
`request_id: ${document.request_id}`,
`evidence_id: ${document.receipt.evidence_id}`,
`state: ${document.receipt.state}`,
...((document.receipt.memory_ids as string[]).length > 0
? [`memory_ids: ${(document.receipt.memory_ids as string[]).join(",")}`]
: []),
document.command === "memories add"
? `查询进度:memhub memories add-status ${document.request_id};重试请带上同一个 --request-id,不会重复添加。`
: "",
"",
]
.filter((line, index, all) => line !== "" || index === all.length - 1)
.join("\n");
}
if (document.state === "not_found") {
return `未找到该 request id:${document.request_id}\n`;
}
return [
`添加记忆未完成:${document.state}(${document.code})`,
document.state === "unknown"
? `结果未知;可以用同一个 --request-id ${document.request_id} 安全重试。`
: "",
"",
]
.filter((line, index, all) => line !== "" || index === all.length - 1)
.join("\n");
}
export function renderMemoryPageText(document: MemoryPageDocument): string {
const page = document.page;
if (page.state !== "ok") {
if (page.state === "unavailable") {
return "后台正忙,记忆列表这次没查完,不代表没有记忆,请稍后再试\n";
}
if (page.state === "rejected") {
return `记忆查询被拒绝(${page.code}),请检查参数。\n`;
}
return `记忆查询未完成:${page.state}(${page.code})\n`;
}
const lines = [
@@ -150,9 +351,16 @@ export function renderMemoryPageText(document: MemoryPageDocument): string {
page.has_more ? ",还有更多,请用 --offset 继续" : ""
}`,
];
if (typeof page.search_window === "number") {
lines.push(`(搜索只在最相关的 ${page.search_window} 条里匹配关键词)`);
}
for (const memory of document.memories) {
const namespace =
memory.kind === "chat"
? ""
: ` kind=${memory.kind} variant=${memory.variant}`;
lines.push(
`- ${memory.memory_id} ${memory.status}/${memory.layer} source=${memory.source ?? "未知"}`,
`- ${memory.memory_id} [${memory.scene}] ${memory.status}/${memory.layer} source=${memory.source ?? "未知"}${namespace}`,
` ${memory.content}${memory.content_truncated ? ` …(共 ${memory.content_characters} 字)` : ""}`,
);
}
@@ -167,10 +375,14 @@ export function renderMemoryDetailText(document: MemoryDetailDocument): string {
if (memory.state === "unavailable") {
return "后台正忙,记忆详情这次没查完,不代表没有这条记忆,请稍后再试\n";
}
if (memory.code === "memory_id_ambiguous") {
return "这个 ID 同时存在于多个场景;请加 --kind(以及 --variant)指明是哪一条。\n";
}
return `记忆读取未完成:${memory.state}(${memory.code})\n`;
}
const lines = [
`记忆 ${memory.memory_id}`,
`场景:${memory.scene} 类型:${memory.kind}${memory.variant ? `(${memory.variant})` : ""}`,
`状态:${memory.status} 层:${memory.layer} 来源:${memory.source ?? "未知"}`,
`owner:${memory.owner ?? "未知"} latest:${memory.is_latest}`,
`raw IDs:${(memory.source_raw_ids as string[]).join(",") || "无"}`,
@@ -188,6 +400,10 @@ function describeMemory(memory: SdkMemoryRecord): Record<string, unknown> {
const budgeted = budget(memory.content ?? "");
return {
memory_id: memory.memory_id,
// Pre-2.0.6 Core listed chat memories only and named no scene.
scene: memory.scene ?? "chat",
kind: memory.kind ?? "chat",
variant: memory.variant ?? null,
status: memory.status,
layer: memory.layer,
is_latest: memory.is_latest,
@@ -199,6 +415,9 @@ function describeMemory(memory: SdkMemoryRecord): Record<string, unknown> {
tags: memory.tags ?? [],
memory_at: memory.memory_at,
gmt_created: memory.gmt_created,
...(memory.task === undefined || memory.task === null
? {}
: { task: memory.task }),
content: budgeted.text,
content_truncated: budgeted.truncated,
content_characters: budgeted.characters,
@@ -216,6 +435,9 @@ function sdkMemoryPage(value: unknown): SdkMemoryPage | null {
total: value.total,
limit: value.limit,
offset: value.offset,
...(integer(value.search_window)
? { search_window: value.search_window }
: {}),
};
}
@@ -231,8 +453,9 @@ function sdkMemory(value: unknown): SdkMemoryRecord | null {
!strings(value.superseded_by) ||
!strings(value.tags) ||
!strings(value.source_raw_ids) ||
!text(value.user_id) ||
!text(value.agent_id)
// Working/coding memories may carry an empty user or agent id.
typeof value.user_id !== "string" ||
typeof value.agent_id !== "string"
)
return null;
return value as unknown as SdkMemoryRecord;
+88 -63
View File
@@ -3,9 +3,9 @@ import { describe, expect, it, vi } from "vitest";
import type { CoreReader } from "./core-client.js";
import type { CoreOperations } from "./core-operations.js";
import {
executeModelProbe,
planModelProbe,
modelProbeExitCode,
readModelProbeReceipt,
startModelProbe,
} from "./model-probe-flow.js";
const PROFILE_ID = "22222222-2222-4222-8222-222222222222";
@@ -83,67 +83,101 @@ function completed(operationId: string) {
};
}
describe("model probe operation flow", () => {
it("plans against the exact current profile revision", async () => {
const reader = vi.fn().mockResolvedValue({ state: "ok", data: [PROFILE] });
const document = await planModelProbe(reader as CoreReader, PROFILE_ID);
expect(document.state).toBe("preview");
if (document.state !== "preview") throw new Error("expected preview");
expect(document.plan).toMatchObject({
kind: "model_profile_probe",
profile_id: PROFILE_ID,
expected_profile_revision: 3,
});
expect(document.plan.operation_id).toMatch(
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/,
);
expect(document.plan.digest).toMatch(/^[0-9a-f]{64}$/);
});
const OPERATION_ID = "33333333-3333-4333-8333-333333333333";
it("proves operation support with schema-1 not_found before posting", async () => {
const planned = await planModelProbe(
vi.fn().mockResolvedValue({ state: "ok", data: [PROFILE] }) as CoreReader,
PROFILE_ID,
);
if (planned.state !== "preview") throw new Error("expected preview");
const operationId = planned.plan.operation_id;
const reader = vi
.fn()
.mockResolvedValueOnce({ state: "ok", data: notFound(operationId) });
/** Reader that answers the profile list, then the queued operation reads. */
function readerWith(...operationReads: unknown[]) {
const reader = vi.fn(async (path: string) => {
if (path === "/api/v1/model-profiles") {
return { state: "ok", data: [PROFILE] };
}
return operationReads.shift() ?? { state: "not_found", code: "http_404" };
});
return reader;
}
describe("model probe operation flow", () => {
it("probes the exact current profile revision once support is proven", async () => {
const reader = readerWith({ state: "ok", data: notFound(OPERATION_ID) });
const post = vi
.fn()
.mockResolvedValue({ state: "committed", data: completed(operationId) });
.mockResolvedValue({ state: "committed", data: completed(OPERATION_ID) });
const result = await executeModelProbe(
planned.plan,
reader as CoreReader,
const result = await startModelProbe(
reader as unknown as CoreReader,
{ post } as CoreOperations,
PROFILE_ID,
OPERATION_ID,
);
expect(result.state).toBe("completed");
expect(reader).toHaveBeenCalledWith(
`/api/v1/model-probe-operations/${operationId}`,
`/api/v1/model-probe-operations/${OPERATION_ID}`,
);
expect(post).toHaveBeenCalledWith(
`/api/v1/model-profiles/${PROFILE_ID}/probe`,
{ operation_id: operationId, expected_profile_revision: 3 },
{ operation_id: OPERATION_ID, expected_profile_revision: 3 },
);
});
it("does not post when the Core lacks the exact operation contract", async () => {
const planned = await planModelProbe(
vi.fn().mockResolvedValue({ state: "ok", data: [PROFILE] }) as CoreReader,
it("generates its own operation id and treats a pending probe as submitted", async () => {
const post = vi.fn(
async (_path: string, body: Record<string, unknown>) => ({
state: "committed",
data: {
...notFound(String(body.operation_id)),
profile_id: PROFILE_ID,
expected_profile_revision: 3,
state: "pending",
stale: false,
},
}),
);
const reader = vi.fn(async (path: string) =>
path === "/api/v1/model-profiles"
? { state: "ok", data: [PROFILE] }
: {
state: "ok",
data: notFound(path.split("/").at(-1) ?? ""),
},
);
const result = await startModelProbe(
reader as unknown as CoreReader,
{ post } as unknown as CoreOperations,
PROFILE_ID,
);
if (planned.state !== "preview") throw new Error("expected preview");
expect(result.state).toBe("pending");
expect(result.operation_id).toMatch(
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/,
);
expect(modelProbeExitCode(result)).toBe(0);
});
it("refuses an unknown profile without posting", async () => {
const post = vi.fn();
const result = await executeModelProbe(
planned.plan,
vi.fn().mockResolvedValue({
const result = await startModelProbe(
readerWith() as unknown as CoreReader,
{ post } as CoreOperations,
"44444444-4444-4444-8444-444444444444",
OPERATION_ID,
);
expect(result).toMatchObject({
state: "not_found",
code: "model_profile_not_found",
});
expect(post).not.toHaveBeenCalled();
});
it("does not post when the Core lacks the exact operation contract", async () => {
const post = vi.fn();
const result = await startModelProbe(
readerWith({
state: "not_found",
code: "http_404",
}) as CoreReader,
}) as unknown as CoreReader,
{ post } as CoreOperations,
PROFILE_ID,
OPERATION_ID,
);
expect(result).toMatchObject({
state: "disabled",
@@ -153,19 +187,15 @@ describe("model probe operation flow", () => {
});
it("does not post when a legacy Core returns a non-operation body", async () => {
const planned = await planModelProbe(
vi.fn().mockResolvedValue({ state: "ok", data: [PROFILE] }) as CoreReader,
PROFILE_ID,
);
if (planned.state !== "preview") throw new Error("expected preview");
const post = vi.fn();
const result = await executeModelProbe(
planned.plan,
vi.fn().mockResolvedValue({
const result = await startModelProbe(
readerWith({
state: "ok",
data: { detail: "legacy_route_shape" },
}) as CoreReader,
}) as unknown as CoreReader,
{ post } as CoreOperations,
PROFILE_ID,
OPERATION_ID,
);
expect(result).toMatchObject({
state: "disabled",
@@ -175,24 +205,19 @@ describe("model probe operation flow", () => {
});
it("reconciles an unknown POST by GET and never treats not_found as replay permission", async () => {
const planned = await planModelProbe(
vi.fn().mockResolvedValue({ state: "ok", data: [PROFILE] }) as CoreReader,
PROFILE_ID,
const reader = readerWith(
{ state: "ok", data: notFound(OPERATION_ID) },
{ state: "ok", data: notFound(OPERATION_ID) },
);
if (planned.state !== "preview") throw new Error("expected preview");
const operationId = planned.plan.operation_id;
const reader = vi
.fn()
.mockResolvedValueOnce({ state: "ok", data: notFound(operationId) })
.mockResolvedValueOnce({ state: "ok", data: notFound(operationId) });
const post = vi.fn().mockResolvedValue({
state: "unknown",
code: "transport_result_unknown",
});
const result = await executeModelProbe(
planned.plan,
reader as CoreReader,
const result = await startModelProbe(
reader as unknown as CoreReader,
{ post } as CoreOperations,
PROFILE_ID,
OPERATION_ID,
);
expect(result).toMatchObject({
state: "unknown",
+42 -124
View File
@@ -1,11 +1,10 @@
import { createHash, randomUUID } from "node:crypto";
import { randomUUID } from "node:crypto";
import type { CoreReader, ReadOutcome } from "./core-client.js";
import type { CoreOperations } from "./core-operations.js";
import { validateModelProfiles } from "./core-responses.js";
import { PRODUCT } from "./version.js";
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const DIGEST = /^[0-9a-f]{64}$/;
const RECEIPT_KEYS = new Set([
"schema_version",
"operation_id",
@@ -59,23 +58,7 @@ const CODES = new Set([
"embedding_unavailable",
]);
interface ModelProbePlanBody {
schema_version: 1;
kind: "model_profile_probe";
operation_id: string;
profile_id: string;
expected_profile_revision: number;
}
export type ModelProbePlan = ModelProbePlanBody & { digest: string };
export type ModelProbeFlowDocument =
| {
schema_version: 1;
product: string;
state: "preview";
plan: ModelProbePlan;
}
| {
schema_version: 1;
product: string;
@@ -91,11 +74,25 @@ export type ModelProbeFlowDocument =
code: string;
};
export async function planModelProbe(
/**
* Start one capability probe of a model profile (one real model request).
*
* Reads the profile's current revision so Core can refuse a probe of a
* profile that changed meanwhile, checks that this Core tracks probe
* operations (an old Core would start an untracked request), then submits it.
* The probe runs asynchronously: the answer is usually `pending`, and
* `models receipt <operation-id>` reads the result.
*/
export async function startModelProbe(
reader: CoreReader,
operations: CoreOperations,
profileId: string,
operationId: string = randomUUID(),
): Promise<ModelProbeFlowDocument> {
if (!UUID.test(profileId)) return failure("invalid", "invalid_profile_id");
if (!UUID.test(operationId)) {
return failure("invalid", "invalid_probe_operation_id");
}
const outcome = await reader<unknown>("/api/v1/model-profiles");
if (outcome.state !== "ok") return fromReadFailure(outcome);
const profiles = validateModelProfiles(outcome.data);
@@ -105,71 +102,46 @@ export async function planModelProbe(
);
if (matches.length !== 1)
return failure("not_found", "model_profile_not_found");
const profile = matches[0];
if (!Number.isSafeInteger(profile.revision) || profile.revision < 1) {
const revision = matches[0].revision;
if (!Number.isSafeInteger(revision) || revision < 1) {
return failure("invalid", "model_profile_revision_invalid");
}
const plan = withDigest({
schema_version: 1 as const,
kind: "model_profile_probe" as const,
operation_id: randomUUID(),
profile_id: profileId,
expected_profile_revision: profile.revision,
});
return { schema_version: 1, product: PRODUCT, state: "preview", plan };
}
export async function executeModelProbe(
value: unknown,
reader: CoreReader,
operations: CoreOperations,
): Promise<ModelProbeFlowDocument> {
const plan = validatePlan(value);
if (!plan) return failure("invalid", "model_probe_plan_invalid");
// A 404 or a legacy/malformed body is not support. Some old Core versions
// ignore the operation body and start an untracked external model request.
const support = await reader<unknown>(operationPath(plan.operation_id));
const support = await reader<unknown>(operationPath(operationId));
if (support.state !== "ok") {
return failure(
"disabled",
"model_probe_operations_unsupported",
plan.operation_id,
operationId,
);
}
const supportReceipt = validateReceipt(support.data, plan.operation_id);
const supportReceipt = validateReceipt(support.data, operationId);
if (supportReceipt?.state !== "not_found") {
return failure(
"disabled",
"model_probe_operations_unsupported",
plan.operation_id,
operationId,
);
}
const outcome = await operations.post<unknown>(
`/api/v1/model-profiles/${plan.profile_id}/probe`,
{
operation_id: plan.operation_id,
expected_profile_revision: plan.expected_profile_revision,
},
const posted = await operations.post<unknown>(
`/api/v1/model-profiles/${profileId}/probe`,
{ operation_id: operationId, expected_profile_revision: revision },
);
if (outcome.state === "rejected") {
return failure("rejected", outcome.code, plan.operation_id);
if (posted.state === "rejected") {
return failure("rejected", posted.code, operationId);
}
if (outcome.state === "committed") {
return receiptDocument(
outcome.data,
plan.operation_id,
plan.profile_id,
plan.expected_profile_revision,
);
if (posted.state === "committed") {
return receiptDocument(posted.data, operationId, profileId, revision);
}
const reconciled = await readModelProbeReceipt(reader, plan.operation_id);
const reconciled = await readModelProbeReceipt(reader, operationId);
return reconciled.state === "not_found"
? failure(
"unknown",
"model_probe_receipt_not_found_after_attempt",
plan.operation_id,
operationId,
)
: reconciled;
}
@@ -190,32 +162,24 @@ export async function readModelProbeReceipt(
return receiptDocument(outcome.data, operationId);
}
export function inspectModelProbePlan(value: unknown): ModelProbePlan | null {
return validatePlan(value);
}
export function modelProbeExitCode(document: ModelProbeFlowDocument): number {
return ["preview", "completed"].includes(document.state) ? 0 : 1;
// `pending` is a submitted probe still running: the submit succeeded.
return ["completed", "pending"].includes(document.state) ? 0 : 1;
}
export function renderModelProbeText(document: ModelProbeFlowDocument): string {
if (document.state === "preview") {
return [
"预览:模型能力探测",
`模型配置:${document.plan.profile_id}`,
`预期版本:${document.plan.expected_profile_revision}`,
`操作 ID:${document.plan.operation_id}`,
`确认摘要:${document.plan.digest}`,
"未发起模型请求。请保存 --json 输出,再用 models probe --plan-file <path> --confirm。",
"",
].join("\n");
}
return [
const lines = [
`探测状态:${document.state}`,
`操作 ID:${document.operation_id ?? "无"}`,
"code" in document ? `代码:${document.code}` : "已返回有界操作回执。",
"",
].join("\n");
];
if (document.state === "pending" && document.operation_id) {
lines.push(
`探测在后台进行;用 memhub models receipt ${document.operation_id} 查看结果。`,
);
}
lines.push("");
return lines.join("\n");
}
function receiptDocument(
@@ -249,31 +213,6 @@ function receiptDocument(
};
}
function validatePlan(value: unknown): ModelProbePlan | null {
const plan = record(value);
if (
plan?.schema_version !== 1 ||
plan.kind !== "model_profile_probe" ||
typeof plan.operation_id !== "string" ||
!UUID.test(plan.operation_id) ||
typeof plan.profile_id !== "string" ||
!UUID.test(plan.profile_id) ||
!positiveInteger(plan.expected_profile_revision) ||
typeof plan.digest !== "string" ||
!DIGEST.test(plan.digest)
) {
return null;
}
const expected = withDigest({
schema_version: 1 as const,
kind: "model_profile_probe" as const,
operation_id: plan.operation_id,
profile_id: plan.profile_id,
expected_profile_revision: plan.expected_profile_revision,
});
return expected.digest === plan.digest ? expected : null;
}
function validateReceipt(
value: unknown,
operationId: string,
@@ -409,27 +348,6 @@ function failure(
};
}
function withDigest<T extends object>(body: T): T & { digest: string } {
return {
...body,
digest: createHash("sha256").update(canonical(body)).digest("hex"),
};
}
function canonical(value: unknown): string {
if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`;
if (value && typeof value === "object") {
return `{${Object.keys(value)
.sort()
.map(
(key) =>
`${JSON.stringify(key)}:${canonical((value as Record<string, unknown>)[key])}`,
)
.join(",")}}`;
}
return JSON.stringify(value);
}
function record(value: unknown): Record<string, unknown> | null {
return value !== null && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
+2 -2
View File
@@ -41,7 +41,7 @@ export type ModelReportDocument =
| {
schema_version: 1;
product: string;
state: "denied" | "unavailable" | "not_found" | "invalid";
state: "denied" | "unavailable" | "not_found" | "invalid" | "rejected";
code: string;
};
@@ -136,7 +136,7 @@ function project(profile: ModelProfile): SafeModelProfile {
}
function failure(
state: "denied" | "unavailable" | "not_found" | "invalid",
state: "denied" | "unavailable" | "not_found" | "invalid" | "rejected",
code: string,
): ModelReportDocument {
return { schema_version: 1, product: PRODUCT, state, code };
+15 -42
View File
@@ -126,29 +126,12 @@ describe("models activate", () => {
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
});
it("previews without contacting Core", async () => {
const result = await capturedRun(["models", "activate", PROFILE, "--json"]);
expect(result.code).toBe(0);
const document = JSON.parse(result.stdout);
expect(document.state).toBe("preview");
expect(document.plan.kind).toBe("model_profile_activate");
// Activation has no CAS body, so the plan must not pretend to carry one.
expect(document.plan).not.toHaveProperty("expected_revision");
expect(transport.post).not.toHaveBeenCalled();
});
it("posts the activate endpoint with --confirm and reports the profile", async () => {
it("posts the activate endpoint directly and reports the profile", async () => {
transport.post.mockResolvedValue({
status: 200,
data: profileResponse(),
});
const result = await capturedRun([
"models",
"activate",
PROFILE,
"--confirm",
"--json",
]);
const result = await capturedRun(["models", "activate", PROFILE, "--json"]);
expect(result.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
`/api/v1/model-profiles/${PROFILE}/activate`,
@@ -196,31 +179,22 @@ describe("models activate", () => {
});
});
describe("models edit", () => {
it("previews a patch from a config file without writing", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "om-model-edit-"));
const config = path.join(directory, "patch.json");
await writeFile(
config,
JSON.stringify({ expected_revision: 4, name: "Renamed" }),
{ mode: 0o600 },
);
const result = await capturedRun([
"models",
"edit",
PROFILE,
"--config-file",
config,
"--json",
]);
describe("models remove", () => {
it("previews without --yes and changes nothing", async () => {
const result = await capturedRun(["models", "remove", PROFILE, "--json"]);
expect(result.code).toBe(0);
const document = JSON.parse(result.stdout);
expect(document.state).toBe("preview");
expect(document.plan.patch).toEqual({ name: "Renamed" });
expect(transport.patch).not.toHaveBeenCalled();
expect(JSON.parse(result.stdout)).toMatchObject({
command: "models remove",
state: "preview",
target: PROFILE,
applied: false,
});
expect(runtime.inspectRuntimeStatus).not.toHaveBeenCalled();
});
});
it("sends the exact patch body with the CAS revision on confirm", async () => {
describe("models edit", () => {
it("sends the exact patch body with the CAS revision directly", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "om-model-edit-"));
const config = path.join(directory, "patch.json");
await writeFile(
@@ -242,7 +216,6 @@ describe("models edit", () => {
PROFILE,
"--config-file",
config,
"--confirm",
"--json",
]);
expect(result.code).toBe(0);
+34 -190
View File
@@ -3,14 +3,12 @@
* activate (`POST /model-profiles/{id}/activate`) and edit
* (`PATCH /model-profiles/{id}`).
*
* Core owns the semantics. These flows reuse the same two-phase convention as
* `models configure`: plan without `--confirm` (or produce a plan file), then
* apply exactly that plan. The edit body mirrors the Web console's
* `ModelProfilePatch`, including `expected_revision` as the compare-and-swap
* guard Core requires.
* Core owns the semantics. Both run directly: activation is reversible (activate
* another profile) and an edit is guarded by `expected_revision`, the
* compare-and-swap Core requires, mirroring the Web console's
* `ModelProfilePatch`.
*/
import { createHash, randomUUID } from "node:crypto";
import type { ModelProfile } from "@one-memory/contracts";
import {
buildModelProfileUpdateRequest,
@@ -22,29 +20,7 @@ import { PRODUCT } from "./version.js";
const PROFILE_ID = /^[A-Za-z0-9][\w.-]{0,127}$/;
export type ModelWritePlan =
| {
schema_version: 1;
kind: "model_profile_activate";
profile_id: string;
digest: string;
}
| {
schema_version: 1;
kind: "model_profile_edit";
profile_id: string;
expected_revision: number;
patch: Record<string, unknown>;
digest: string;
};
export type ModelWriteDocument =
| {
schema_version: 1;
product: string;
state: "preview";
plan: ModelWritePlan;
}
| {
schema_version: 1;
product: string;
@@ -60,38 +36,6 @@ export type ModelWriteDocument =
code: string;
};
function canonical(value: unknown): string {
if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`;
if (value !== null && typeof value === "object") {
return `{${Object.entries(value as Record<string, unknown>)
.sort(([left], [right]) => left.localeCompare(right))
.map(([key, item]) => `${JSON.stringify(key)}:${canonical(item)}`)
.join(",")}}`;
}
if (value === undefined) return "null";
return JSON.stringify(value);
}
function digestOf(value: unknown): string {
return createHash("sha256").update(canonical(value)).digest("hex");
}
function preview(
plan:
| Omit<
Extract<ModelWritePlan, { kind: "model_profile_activate" }>,
"digest"
>
| Omit<Extract<ModelWritePlan, { kind: "model_profile_edit" }>, "digest">,
): ModelWriteDocument {
return {
schema_version: 1,
product: PRODUCT,
state: "preview",
plan: { ...plan, digest: digestOf(plan) } as ModelWritePlan,
};
}
function invalid(code: string): ModelWriteDocument {
return { schema_version: 1, product: PRODUCT, state: "invalid", code };
}
@@ -122,87 +66,45 @@ function projectProfile(profile: ModelProfile): Record<string, unknown> {
};
}
// --- activate -------------------------------------------------------------
/**
* Activation has no request body, so the plan carries only the target. There
* is deliberately no `expected_revision`: Core would never read one, and
* recording a check it does not perform would be a fake guard.
*/
export function planModelProfileActivate(input: {
profileId: string;
}): ModelWriteDocument {
return PROFILE_ID.test(input.profileId)
? preview({
schema_version: 1,
kind: "model_profile_activate",
profile_id: input.profileId,
})
: invalid("model_profile_activate_input_invalid");
}
export async function executeModelProfileActivate(
plan: Extract<ModelWritePlan, { kind: "model_profile_activate" }>,
/** Make `profileId` the active profile. Activation has no request body, so
* there is no revision for Core to check. */
export async function activateModelProfile(
profileId: string,
operations: CoreOperations,
): Promise<ModelWriteDocument> {
if (!PROFILE_ID.test(profileId)) {
return invalid("model_profile_activate_input_invalid");
}
const result = await operations.post<unknown>(
`/api/v1/model-profiles/${encodeURIComponent(plan.profile_id)}/activate`,
`/api/v1/model-profiles/${encodeURIComponent(profileId)}/activate`,
{},
);
return modelProfileResult(plan.profile_id, result);
return modelProfileResult(profileId, result);
}
// --- edit -----------------------------------------------------------------
export function planModelProfileEdit(input: {
profileId: string;
patch: unknown;
}): ModelWriteDocument {
// The shared builder owns patch validation (14-key allow-list,
// expected_revision CAS, field bounds, api_key/clear_api_key exclusivity).
// A rejected patch is `invalid`, never a plan the caller could confirm.
let request: ReturnType<typeof buildModelProfileUpdateRequest>;
try {
request = buildModelProfileUpdateRequest(input.profileId, input.patch);
} catch (error) {
if (error instanceof ModelProfileInputError) {
return invalid("model_profile_edit_input_invalid");
}
throw error;
}
const patch = request.body as Record<string, unknown>;
if (!PROFILE_ID.test(input.profileId)) {
/** Apply a bounded patch (carrying `expected_revision`) to one profile; the
* key, when given, comes from a controlled file and never from the patch. */
export async function editModelProfile(
profileId: string,
patch: unknown,
apiKey: string | undefined,
operations: CorePatchOperations,
): Promise<ModelWriteDocument> {
if (
!PROFILE_ID.test(profileId) ||
patch === null ||
typeof patch !== "object" ||
Array.isArray(patch) ||
"api_key" in patch
) {
return invalid("model_profile_edit_input_invalid");
}
return preview({
schema_version: 1,
kind: "model_profile_edit",
profile_id: input.profileId,
expected_revision: patch.expected_revision as number,
// `api_key` is read from a controlled file at apply time, exactly like
// `models configure`; the plan never carries the credential. The revision
// is a top-level field, so it is stripped from the nested patch rather
// than being duplicated in both places.
patch: Object.fromEntries(
Object.entries(patch).filter(
([key]) => key !== "api_key" && key !== "expected_revision",
),
),
});
}
export async function executeModelProfileEdit(
plan: Extract<ModelWritePlan, { kind: "model_profile_edit" }>,
operations: CorePatchOperations,
apiKey: string | undefined,
): Promise<ModelWriteDocument> {
// Re-validate through the shared builder at apply time too, so a hand-edited
// plan cannot smuggle an out-of-bounds patch past the write.
// The shared builder owns patch validation (allow-list, expected_revision
// CAS, field bounds, api_key/clear_api_key exclusivity).
let request: ReturnType<typeof buildModelProfileUpdateRequest>;
try {
request = buildModelProfileUpdateRequest(plan.profile_id, {
...plan.patch,
expected_revision: plan.expected_revision,
request = buildModelProfileUpdateRequest(profileId, {
...(patch as Record<string, unknown>),
...(apiKey === undefined ? {} : { api_key: apiKey }),
});
} catch (error) {
@@ -212,7 +114,7 @@ export async function executeModelProfileEdit(
throw error;
}
const result = await operations.patch<unknown>(request.path, request.body);
return modelProfileResult(plan.profile_id, result);
return modelProfileResult(profileId, result);
}
function modelProfileResult(
@@ -254,69 +156,11 @@ function modelProfileResult(
return committed(operationId, profile);
}
export function validateModelWritePlan(value: unknown): ModelWritePlan | null {
if (value === null || typeof value !== "object" || Array.isArray(value)) {
return null;
}
const candidate = value as Record<string, unknown>;
if (candidate.schema_version !== 1 || typeof candidate.kind !== "string") {
return null;
}
const { digest, ...withoutDigest } = candidate;
if (typeof digest !== "string" || digest !== digestOf(withoutDigest)) {
return null;
}
switch (candidate.kind) {
case "model_profile_activate":
return typeof candidate.profile_id === "string" &&
PROFILE_ID.test(candidate.profile_id) &&
Object.keys(candidate).length === 3
? (candidate as unknown as ModelWritePlan)
: null;
case "model_profile_edit": {
if (
typeof candidate.profile_id !== "string" ||
!PROFILE_ID.test(candidate.profile_id) ||
!Number.isSafeInteger(candidate.expected_revision)
) {
return null;
}
// The shared builder is the single judge of a patch body; the plan must
// not carry `api_key` (that is read from a controlled file at apply).
let built: ReturnType<typeof buildModelProfileUpdateRequest>;
try {
built = buildModelProfileUpdateRequest(candidate.profile_id, {
...(candidate.patch as Record<string, unknown>),
expected_revision: candidate.expected_revision,
});
} catch {
return null;
}
if ("api_key" in built.body) return null;
return Object.keys(candidate).length === 6
? (candidate as unknown as ModelWritePlan)
: null;
}
default:
return null;
}
}
export function modelWriteExitCode(document: ModelWriteDocument): number {
return ["preview", "committed"].includes(document.state) ? 0 : 1;
return document.state === "committed" ? 0 : 1;
}
export function renderModelWriteText(document: ModelWriteDocument): string {
if (document.state === "preview") {
const plan = document.plan;
return [
`预览:${plan.kind}`,
`目标配置:${plan.profile_id}`,
`确认摘要:${plan.digest}`,
"未执行任何写操作。请把 --json 输出保存为 plan 文件,再用对应命令加 --confirm。",
"",
].join("\n");
}
if (document.state === "committed") {
return `已保存模型配置:${document.receipt.profile_id}\nrevision=${document.receipt.revision}\n`;
}
-57
View File
@@ -1,57 +0,0 @@
import { constants } from "node:fs";
import { type FileHandle, open } from "node:fs/promises";
import { readBoundedJson } from "./bounded-file.js";
export type PlanAttempt = {
schema_version: 1;
operation_id: string;
plan_digest: string;
};
/**
* Durable client fence written before a mutating request. Once present, the
* same plan may only query its operation receipt; it is never blindly posted
* again after an unknown result.
*/
export async function claimPlanAttempt(
planFile: string,
attempt: PlanAttempt,
): Promise<"first" | "prior" | "invalid"> {
const journal = `${planFile}.one-memory-attempt`;
let handle: FileHandle;
try {
handle = await open(
journal,
constants.O_WRONLY |
constants.O_CREAT |
constants.O_EXCL |
(constants.O_NOFOLLOW ?? 0),
0o600,
);
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "EEXIST") return "invalid";
const current = await readBoundedJson(journal, 1024);
return sameAttempt(current, attempt) ? "prior" : "invalid";
}
try {
await handle.writeFile(`${JSON.stringify(attempt)}\n`, "utf8");
await handle.sync();
return "first";
} catch {
return "invalid";
} finally {
await handle.close().catch(() => undefined);
}
}
function sameAttempt(value: unknown, expected: PlanAttempt): boolean {
return (
value !== null &&
typeof value === "object" &&
!Array.isArray(value) &&
(value as Record<string, unknown>).schema_version === 1 &&
(value as Record<string, unknown>).operation_id === expected.operation_id &&
(value as Record<string, unknown>).plan_digest === expected.plan_digest &&
Object.keys(value).length === 3
);
}
+6 -10
View File
@@ -104,7 +104,7 @@ describe("package matrix harness", () => {
});
});
it("fails closed on every destructive UI-write command without a Runtime", {
it("fails closed on every write command without a Runtime", {
timeout: 120_000,
}, async () => {
const runtimeDist = await mkdtemp(
@@ -140,7 +140,7 @@ describe("package matrix harness", () => {
expect(document.state).toBe("verified");
const results = document.commands.stopped_destructive.results;
// Every destructive command fails closed while Runtime is stopped.
expect(Object.keys(results)).toHaveLength(7);
expect(Object.keys(results)).toHaveLength(8);
for (const [name, result] of Object.entries(results)) {
expect(result, `${name} did not fail closed`).toEqual({
exit: 3,
@@ -164,7 +164,7 @@ describe("package matrix harness", () => {
const shim = [
"#!/usr/bin/env node",
"const a=process.argv.slice(2);",
"if(['calls','notes','graph','models','memories'].includes(a[0])){",
"if(['graph-width','models','memories','sources'].includes(a[0])){",
' process.stdout.write(\'{"schema_version":1,"product":"memhub","state":"committed","operation_id":"x","receipt":{}}\\n\');',
" process.exit(0);",
"}",
@@ -190,7 +190,7 @@ describe("package matrix harness", () => {
expect(failure?.state).toBe("failed");
expect(
failure?.failures.filter((item) => item.includes("did not fail closed")),
).toHaveLength(7);
).toHaveLength(8);
});
it("can require an expected bundle SHA and reports bundle metadata", {
@@ -291,9 +291,7 @@ describe("package matrix harness", () => {
"sync",
"memories",
"profile",
"calls",
"notes",
"graph",
"graph-width",
"models",
"host",
"doctor",
@@ -311,14 +309,12 @@ describe("package matrix harness", () => {
"uninstall",
"delete-data",
"recall",
"setup",
"scan",
"progress",
"jobs",
"path",
"scan-roots",
"adapter",
"write",
"task",
"serve",
];
const catalog = {
@@ -145,13 +145,15 @@ describe("memhub short-command contract", () => {
[command.usage, command.alternative_usage ?? ""].join("\n"),
).join("\n");
for (const shortForm of [
"memhub calls clear",
"memhub notes add",
"memhub notes process",
"memhub memories add",
"memhub memories add-status",
"memhub models setup",
"memhub models add",
"memhub models test",
"memhub models activate",
"memhub models edit",
"memhub graph width",
"memhub graph-width",
"memhub sources connect",
"memhub write submit",
"memhub write status",
"memhub memories update",
@@ -159,6 +161,10 @@ describe("memhub short-command contract", () => {
]) {
expect(flattened).toContain(shortForm);
}
// No "note" concept and no plan files on the CLI surface.
for (const removed of ["memhub notes", "memhub calls", "--plan-file"]) {
expect(flattened).not.toContain(removed);
}
expect(COMMANDS.some((command) => command.name === "permission")).toBe(
false,
);
@@ -211,7 +217,7 @@ describe("memhub short-command contract", () => {
// but an earlier catalog text still claimed they were "尚不可用". A user
// reading `help path` would not try a command that works.
it("does not call the wired path/adapter commands unavailable", () => {
for (const name of ["path", "adapter"]) {
for (const name of ["scan-roots", "adapter"]) {
const command = COMMANDS.find((entry) => entry.name === name);
expect(command).toBeDefined();
const text = [
@@ -277,7 +283,7 @@ describe("memhub short-command contract", () => {
}
});
it("keeps plan-file apply previews free of diagnostic side effects", {
it("keeps destructive previews free of diagnostic side effects", {
timeout: 20_000,
}, async () => {
expect(
@@ -291,9 +297,10 @@ describe("memhub short-command contract", () => {
);
try {
for (const [index, argv] of [
["sources", "import", "--plan-file", path.join(root, "plan.json")],
["models", "configure", "--plan-file", path.join(root, "plan.json")],
["memories", "apply", "--plan-file", path.join(root, "plan.json")],
["memories", "delete", "m-1"],
["models", "remove", "44444444-4444-4444-8444-444444444444"],
["backup", "restore", "--source", path.join(root, "backup.sqlite3")],
["graph-width", "--decision", "rebuild_with_new"],
].entries()) {
const dataDir = path.join(root, `data-${index}`);
await import("node:fs/promises").then(({ mkdir }) =>
@@ -305,8 +312,8 @@ describe("memhub short-command contract", () => {
stdio: "pipe",
});
} catch {
// The missing plan or stopped Runtime is expected; the guard is
// that even the failure path did not add a diagnostic file.
// A failure path is acceptable; the guard is that no preview adds
// a diagnostic file.
}
expect(await readdir(dataDir)).toEqual([]);
}
+122 -264
View File
@@ -1,122 +1,56 @@
// A6: the rules behind the six product commands.
// The rules behind the commands that attach to the Runtime directly.
//
// The parsers are where the contract actually lives — "needs --yes to change
// anything" and "no write without evidence" are decisions, not formatting — so
// they are tested directly rather than only through the CLI.
// The parsers are where the contract actually lives — "only irreversible
// operations ask for --yes" and "unknown flags are rejected" are decisions,
// not formatting — so they are tested directly rather than only through the
// CLI.
import { describe, expect, it } from "vitest";
import { parseOpenAIMessages } from "./core-client.js";
import {
asPlanDocument,
describe as describePlan,
parseProductCommand,
parseBackupCommand,
parseScanCommand,
parseScanRootsCommand,
parseTaskCommand,
parseWriteCommand,
restoreConfirmationMatches,
} from "./product-flows.js";
describe("mutating commands are two-phase", () => {
it("plans without --yes", () => {
const parsed = parseProductCommand("scan", ["--root", "/tmp/a"]);
expect(parsed?.confirmed).toBe(false);
});
it("confirms with --yes", () => {
const parsed = parseProductCommand("scan", ["--root", "/tmp/a", "--yes"]);
expect(parsed?.confirmed).toBe(true);
});
it("says what it would do", () => {
const parsed = parseProductCommand("scan", ["--root", "/tmp/a"]);
// `?.plan!` would let a null parse pass silently; assert the parse first.
if (parsed === null) {
throw new Error("scan with a root must parse");
}
expect(describePlan(parsed.plan)).toContain("scan: /tmp/a");
});
it("marks a plan as not applied", () => {
// Assert the parse itself: `?.` let a null result satisfy this test for as
// long as `backup create` was unparseable.
const parsed = parseProductCommand("backup", [
"create",
"--target",
"/tmp/b",
]);
if (parsed === null) {
throw new Error("backup create must parse");
}
expect(parsed.plan.verb).toBe("create");
expect(asPlanDocument(parsed.plan).applied).toBe(false);
});
it("parses the verb for every command that takes one", () => {
// `path add` / `path remove` / `backup create` were rejected outright
// before the verb was skipped in parsing.
describe("scan-roots", () => {
it("reads the root, source and recursive flag off add", () => {
expect(
parseProductCommand("path", ["add", "--root", "/tmp/x"])?.plan.action,
).toBe("add scan root");
expect(
parseProductCommand("path", ["remove", "--root", "/tmp/x"])?.plan.action,
).toBe("remove scan root");
expect(
parseProductCommand("backup", ["restore", "--source", "/tmp/b"])?.plan
.action,
).toBe("restore backup");
parseScanRootsCommand([
"add",
"--root",
"/tmp/x",
"--source",
"codex",
"--no-recursive",
]),
).toEqual({
kind: "add",
root: "/tmp/x",
source: "codex",
recursive: false,
});
expect(parseScanRootsCommand(["add", "--root", "/tmp/x"])).toEqual({
kind: "add",
root: "/tmp/x",
recursive: true,
});
});
it("defaults a scan to all registered roots", () => {
const parsed = parseProductCommand("scan", []);
expect(parsed?.plan.target).toBe("all registered roots");
});
});
describe("unknown flags are rejected", () => {
it("rejects a typo'd flag rather than ignoring it", () => {
// A silently ignored --yes would turn a real apply into a plan, and the
// user would believe the change had been made.
expect(parseProductCommand("scan", ["--yesx"])).toBeNull();
expect(parseProductCommand("scan", ["--root"])).toBeNull();
expect(parseProductCommand("scan", ["--root", "--yes"])).toBeNull();
});
it("rejects an unknown verb for backup", () => {
it("distinguishes remove from add and refuses add-only flags on remove", () => {
expect(parseScanRootsCommand(["remove", "--root", "/tmp/x"])).toEqual({
kind: "remove",
root: "/tmp/x",
});
expect(
parseProductCommand("backup", ["make", "--target", "/tmp/b"]),
parseScanRootsCommand(["remove", "--root", "/tmp/x", "--no-recursive"]),
).toBeNull();
});
// `path` and `scan` share one parser. Before this guard, an unknown verb was
// silently treated as `path add` or a default `scan`, so `--yes` executed a
// different operation than the one the caller named.
it("rejects an unknown verb for path without turning it into add", () => {
expect(
parseProductCommand("path", ["frobnicate", "--root", "/tmp/x", "--yes"]),
).toBeNull();
});
it("rejects an unknown verb for scan without turning it into all", () => {
expect(
parseProductCommand("scan", ["frobnicate", "--all", "--yes"]),
).toBeNull();
expect(parseProductCommand("scan", ["frobnicate", "--yes"])).toBeNull();
});
it("still accepts the documented verbless scan", () => {
expect(parseProductCommand("scan", ["--all", "--yes"])?.plan.all).toBe(
true,
);
expect(
parseProductCommand("scan", ["--source", "codex"])?.plan.sourceId,
).toBe("codex");
});
it("rejects flags that the named verb does not consume", () => {
// These were accepted and then silently dropped. An operator who typed
// `path remove --source codex` would believe the source filter applied.
expect(
parseProductCommand("path", [
parseScanRootsCommand([
"remove",
"--root",
"/tmp/x",
@@ -124,137 +58,103 @@ describe("unknown flags are rejected", () => {
"codex",
]),
).toBeNull();
});
it("is direct, but keeps accepting a legacy --yes", () => {
expect(parseScanRootsCommand(["add", "--root", "/tmp/x", "--yes"])).toEqual(
{ kind: "add", root: "/tmp/x", recursive: true },
);
});
it("rejects unknown verbs, typos and repeated flags", () => {
expect(parseScanRootsCommand(["create", "--root", "/tmp/x"])).toBeNull();
expect(parseScanRootsCommand(["add", "--rot", "/tmp/x"])).toBeNull();
expect(parseScanRootsCommand(["add"])).toBeNull();
expect(
parseProductCommand("path", [
"remove",
"--root",
"/tmp/x",
"--no-recursive",
]),
parseScanRootsCommand(["add", "--root", "/a", "--root", "/b"]),
).toBeNull();
expect(
parseProductCommand("path", ["add", "--root", "/tmp/x", "--full"]),
parseScanRootsCommand(["add", "--root", "/a", "--yes", "--yes"]),
).toBeNull();
});
});
describe("scan", () => {
it("needs a source: there is no endpoint that scans a bare path", () => {
expect(parseScanCommand(["--source", "codex"])).toEqual({
kind: "source",
sourceId: "codex",
});
expect(parseScanCommand([])).toBeNull();
expect(parseScanCommand(["--root", "/tmp/a"])).toBeNull();
});
it("no longer accepts --full, which was never sent to Core", () => {
expect(parseScanCommand(["--source", "codex", "--full"])).toBeNull();
});
it("keeps --all as a hidden alias and a legacy --yes", () => {
expect(parseScanCommand(["--all", "--yes"])).toEqual({ kind: "all" });
expect(parseScanCommand(["--all", "--source", "codex"])).toBeNull();
expect(parseScanCommand(["--source", "codex", "--yes"])).toEqual({
kind: "source",
sourceId: "codex",
});
});
it("rejects a verb, a bad source id and repeated flags", () => {
expect(parseScanCommand(["all"])).toBeNull();
expect(parseScanCommand(["--source", "Bad Id"])).toBeNull();
expect(
parseScanCommand(["--source", "codex", "--source", "cursor"]),
).toBeNull();
expect(
parseProductCommand("backup", [
"create",
"--target",
"/tmp/b",
"--source",
"/tmp/s",
]),
parseScanCommand(["--source", "codex", "--json", "--json"]),
).toBeNull();
});
});
describe("backup", () => {
it("creates directly and requires a target", () => {
expect(parseBackupCommand(["create", "--target", "/tmp/b"])).toEqual({
verb: "create",
target: "/tmp/b",
});
expect(parseBackupCommand(["create"])).toBeNull();
expect(
parseProductCommand("backup", [
"restore",
"--source",
"/tmp/s",
"--target",
"/tmp/t",
]),
parseBackupCommand(["create", "--target", "/tmp/b", "--source", "/x"]),
).toBeNull();
});
// A later value silently won before this guard. For `scan --source` that
// means one extra shell/make variable can change which source is scanned
// while the operator believes both were applied.
it("rejects repeated value-bearing flags instead of letting the last one win", () => {
it("restores only with --yes and requires a source", () => {
expect(parseBackupCommand(["restore", "--source", "/tmp/b"])).toEqual({
verb: "restore",
source: "/tmp/b",
confirmed: false,
});
expect(
parseProductCommand("scan", ["--source", "codex", "--source", "cursor"]),
).toBeNull();
expect(
parseProductCommand("path", [
"add",
"--root",
"/tmp/a",
"--root",
"/tmp/b",
]),
).toBeNull();
expect(
parseProductCommand("backup", [
"create",
"--target",
"/tmp/a",
"--target",
"/tmp/b",
]),
).toBeNull();
parseBackupCommand(["restore", "--source", "/tmp/b", "--yes"]),
).toMatchObject({ confirmed: true });
expect(parseBackupCommand(["restore", "--yes"])).toBeNull();
});
it("rejects repeated boolean flags instead of treating them as idempotent", () => {
expect(parseProductCommand("scan", ["--all", "--all", "--yes"])).toBeNull();
expect(
parseProductCommand("scan", ["--full", "--full", "--yes"]),
).toBeNull();
expect(
parseProductCommand("path", [
"add",
"--root",
"/tmp/a",
"--no-recursive",
"--no-recursive",
]),
).toBeNull();
expect(
parseProductCommand("backup", [
"create",
"--target",
"/tmp/a",
"--yes",
"--yes",
]),
).toBeNull();
});
it("rejects a repeated global --json spelling", () => {
expect(
parseProductCommand("scan", ["--root", "/tmp/a", "--json", "--json"]),
).toBeNull();
expect(
parseProductCommand("backup", [
"create",
"--target",
"/tmp/a",
"--json",
"--json",
]),
).toBeNull();
});
it("requires a source when restoring", () => {
expect(
parseProductCommand("backup", ["restore", "--target", "/tmp/b"]),
).toBeNull();
});
it("parses the explicit restore confirmation path", () => {
const parsed = parseProductCommand("backup", [
it("accepts the legacy --confirm <path> and checks it lexically", () => {
const parsed = parseBackupCommand([
"restore",
"--source",
"/tmp/backups/a.sqlite3",
"/tmp/b",
"--yes",
"--confirm",
"/tmp/backups/a.sqlite3",
"/tmp/b",
]);
expect(parsed).not.toBeNull();
expect(parsed?.confirmPath).toBe("/tmp/backups/a.sqlite3");
expect(
restoreConfirmationMatches("/tmp/backups/a.sqlite3", parsed?.confirmPath),
).toBe(true);
expect(
restoreConfirmationMatches("/tmp/backups/a.sqlite3", undefined),
).toBe(false);
expect(
restoreConfirmationMatches(
"/tmp/backups/a.sqlite3",
"/tmp/backups/../backups/a.sqlite3",
),
).toBe(false);
expect(parsed).toMatchObject({ confirmPath: "/tmp/b" });
expect(restoreConfirmationMatches("/tmp/b", "/tmp/b")).toBe(true);
expect(restoreConfirmationMatches("/tmp/b", undefined)).toBe(true);
expect(restoreConfirmationMatches("/tmp/b", "/tmp/./b")).toBe(false);
});
it("requires a target when creating a backup", () => {
expect(parseProductCommand("backup", ["create"])).toBeNull();
it("rejects an unknown verb", () => {
expect(parseBackupCommand(["delete", "--target", "/tmp/b"])).toBeNull();
});
});
@@ -305,11 +205,19 @@ describe("write", () => {
).toBeNull();
});
it("parses status under write", () => {
it("parses status under write, positional or --id", () => {
expect(parseWriteCommand(["status", "--id", "request-1"])).toEqual({
kind: "status",
id: "request-1",
});
expect(parseWriteCommand(["status", "request-1"])).toEqual({
kind: "status",
id: "request-1",
});
});
it("rejects a repeated --session instead of letting the last one win", () => {
expect(parseWriteCommand([...base, "--session", "s2"])).toBeNull();
});
it("marks --live as priority live", () => {
@@ -366,53 +274,3 @@ describe("task", () => {
expect(parseTaskCommand(["--id", "req-1", "--json", "--json"])).toBeNull();
});
});
describe("path add and remove carry what the endpoint needs", () => {
it("reads the source and recursive flag off the path add", () => {
const parsed = parseProductCommand("path", [
"add",
"--root",
"/tmp/scanme",
"--source",
"codex",
"--no-recursive",
]);
// `?.plan!` let a null parse pass silently, so assert the parse itself.
if (parsed === null) {
throw new Error("path add must parse");
}
expect(parsed.plan.action).toBe("add scan root");
expect(parsed.plan.target).toBe("/tmp/scanme");
expect(parsed.plan.detail).toContain("source=codex");
expect(parsed.plan.detail).toContain("recursive=false");
});
it("distinguishes remove from add", () => {
const parsed = parseProductCommand("path", [
"remove",
"--root",
"/tmp/scanme",
]);
if (parsed === null) {
throw new Error("path remove must parse");
}
// The action is the discriminator at apply time: add and remove share a
// `--root`, so a mis-set action would register the directory it meant to delete.
expect(parsed.plan.action).toBe("remove scan root");
});
it("does not silently drop --yes", () => {
const parsed = parseProductCommand("path", [
"add",
"--root",
"/tmp/a",
"--yes",
]);
expect(parsed?.confirmed).toBe(true);
// Without this, a typo'd `--yes` would fall through to a plan and the user
// would believe the change had been made.
const without = parseProductCommand("path", ["add", "--root", "/tmp/a"]);
expect(without?.confirmed).toBe(false);
});
});
+153 -272
View File
@@ -1,258 +1,163 @@
// A6: the product commands Runtime exposes on the CLI.
// Parsers for the commands that attach to the Runtime directly:
// `scan-roots` (alias `path`), `scan`, `backup`, `write`.
//
// These are the six commands the objective names: path, adapter, scan, write,
// task, backup. Two rules run through all of them:
// Rules shared by all of them:
//
// - **Mutating commands are two-phase.** Without `--yes` they print a plan and
// change nothing. This is the convention the existing install/service
// commands already follow, and it is why `scan` and `backup` default to a
// plan rather than quietly doing work.
// - **Output is machine-readable.** `--json` emits one document on stdout so an
// agent can consume it; the human form is text. Never both, so a caller that
// parses stdout is not surprised by prose.
// - **Only irreversible operations ask for `--yes`.** `backup restore`
// overwrites the database, so without `--yes` it previews and changes
// nothing. Registering a scan root, scanning a source, creating a backup and
// submitting a write are direct. A legacy `--yes` on a direct command is
// accepted and ignored so older scripts keep working.
// - **Unknown or repeated flags are rejected**, never ignored: a dropped typo
// would silently change what the command does.
// - **Output is machine-readable.** `--json` emits one document on stdout.
//
// The commands talk to the Runtime's HTTP API, not to the database and not to
// HY. That is the A6 rule: CLI and Electron are clients of the same Runtime.
// the SDK: CLI and desktop are clients of the same Runtime.
/** What a mutating command would do, before it is allowed to do it. */
export interface Plan {
readonly action: string;
readonly target: string;
readonly detail?: string;
/** Which form of the command this plan is for, when one name has several.
*
* `backup` has `create` and `restore`, and applying the wrong one is not a
* harmless mistake — so the verb travels with the plan the user confirmed.
*/
readonly verb?: string;
/** The source the Runtime endpoint addresses, when the plan names one.
*
* Carried on the plan rather than re-parsed at apply time so the plan the
* user confirmed and the request actually sent cannot disagree.
*/
readonly sourceId?: string;
/** `scan --all`: every source Runtime actually found, not a named one. */
readonly all?: boolean;
export type ScanRootsCommand =
| {
readonly kind: "add";
readonly root: string;
readonly source?: string;
readonly recursive: boolean;
}
| { readonly kind: "remove"; readonly root: string };
export type ScanCommand =
| { readonly kind: "source"; readonly sourceId: string }
/** Hidden alias of `sources connect --all-detected --import`. */
| { readonly kind: "all" };
export type BackupCommand =
| { readonly verb: "create"; readonly target: string }
| {
readonly verb: "restore";
readonly source: string;
readonly confirmed: boolean;
/** Legacy `--confirm <path>`: accepted, and must repeat `--source`. */
readonly confirmPath?: string;
};
interface ParsedFlags {
readonly values: Map<string, string>;
readonly switches: Set<string>;
}
export function planAddRoot(
root: string,
source: string | undefined,
recursive: boolean,
): Plan {
return {
action: "add scan root",
target: root,
detail: `source=${source ?? "auto"} recursive=${recursive}`,
};
}
export function planRemoveRoot(root: string): Plan {
return { action: "remove scan root", target: root };
}
export function planScan(root: string | undefined, full: boolean): Plan {
return {
action: "scan",
target: root ?? "all registered roots",
detail: `full=${full}`,
};
}
export function planScanSource(sourceId: string, full: boolean): Plan {
return {
action: "scan",
target: `source ${sourceId}`,
detail: `full=${full}`,
sourceId,
};
}
export function planScanAll(full: boolean): Plan {
return {
action: "scan",
target: "all detected sources",
detail: `full=${full}`,
all: true,
};
}
export function planBackup(target: string): Plan {
return { action: "create backup", target, verb: "create" };
}
export function planRestore(source: string): Plan {
return { action: "restore backup", target: source, verb: "restore" };
}
export interface ProductCommand {
readonly plan: Plan;
readonly confirmed: boolean;
/** For destructive verbs with a named artifact, the exact value repeated by
* the user. `backup restore` requires this to match `--source`. */
readonly confirmPath?: string;
}
/** Parse a mutating product command.
*
* Unknown flags are rejected rather than ignored: a typo'd `--yes` that is
* silently dropped would turn a real apply into a plan, and the user would
* believe the change had been made.
*/
export function parseProductCommand(
name: string,
/** Read `--flag value` pairs and bare switches; null on anything unknown,
* repeated or missing its value. */
function parseFlags(
args: readonly string[],
): ProductCommand | null {
let root: string | undefined;
let source: string | undefined;
let target: string | undefined;
let recursive = true;
let full = false;
let all = false;
let confirmed = false;
let confirmPath: string | undefined;
const seen = new Set<string>();
const seenValues = new Set<string>();
const flags = new Set([
"--root",
"--source",
"--target",
"--confirm",
"--no-recursive",
"--full",
"--all",
"--yes",
"--json",
]);
// The first token is the verb for commands that have one (`path add`,
// `backup create`). It used to fall into the flag loop and be rejected as an
// unknown flag, which made every one of these commands permanently
// unparseable — `path add --root X` printed "参数无效" even though `help`
// documented it. The tests missed this because they asserted
// `parsed?.plan!` and null satisfies `?.` + `!`.
const verb = args[0];
const start = verb !== undefined && !verb.startsWith("--") ? 1 : 0;
for (let index = start; index < args.length; index += 1) {
valueFlags: readonly string[],
switchFlags: readonly string[],
): ParsedFlags | null {
const values = new Map<string, string>();
const switches = new Set<string>();
for (let index = 0; index < args.length; index += 1) {
const arg = args[index];
if (!flags.has(arg)) return null;
// A repeated flag is a malformed argv, not an idempotent spelling. The
// other command parsers already apply this rule to --json as well.
if (seen.has(arg)) return null;
seen.add(arg);
if (arg === "--yes") {
confirmed = true;
continue;
}
if (arg === "--confirm") {
const value = args[index + 1];
if (value === undefined || value.startsWith("--")) return null;
if (confirmPath !== undefined) return null;
confirmPath = value;
index += 1;
continue;
}
if (arg === "--json") continue;
if (arg === "--no-recursive") {
recursive = false;
continue;
}
if (arg === "--full") {
full = true;
continue;
}
if (arg === "--all") {
all = true;
if (switchFlags.includes(arg)) {
if (switches.has(arg)) return null;
switches.add(arg);
continue;
}
if (!valueFlags.includes(arg) || values.has(arg)) return null;
const value = args[index + 1];
if (value === undefined || value.startsWith("--")) return null;
if (seenValues.has(arg)) return null;
seenValues.add(arg);
values.set(arg, value);
index += 1;
if (arg === "--root") root = value;
else if (arg === "--source") source = value;
else target = value;
}
return { values, switches };
}
if (name === "path") {
if (confirmPath !== undefined) return null;
if (target !== undefined) return null;
if (root === undefined) return null;
if (source === undefined && args.includes("--source")) return null;
if (seen.has("--full") || seen.has("--all")) return null;
if (args[0] !== "add" && args[0] !== "remove") return null;
if (
args[0] === "remove" &&
(seen.has("--source") || seen.has("--no-recursive"))
) {
return null;
}
const SOURCE_ID = /^[a-z0-9][a-z0-9-]{0,127}$/;
/** `scan-roots add --root <path> [--source <id>] [--no-recursive]` and
* `scan-roots remove --root <path>` (`list` is answered separately). */
export function parseScanRootsCommand(
args: readonly string[],
): ScanRootsCommand | null {
const verb = args[0];
if (verb === "add") {
const flags = parseFlags(
args.slice(1),
["--root", "--source"],
["--no-recursive", "--yes", "--json"],
);
const root = flags?.values.get("--root");
if (!flags || root === undefined) return null;
const source = flags.values.get("--source");
if (source !== undefined && !SOURCE_ID.test(source)) return null;
return {
plan:
args[0] === "add"
? planAddRoot(root, source, recursive)
: planRemoveRoot(root),
confirmed,
kind: "add",
root,
...(source === undefined ? {} : { source }),
recursive: !flags.switches.has("--no-recursive"),
};
}
if (name === "scan") {
if (confirmPath !== undefined) return null;
if (target !== undefined) return null;
if (seen.has("--no-recursive")) return null;
if (args[0] !== undefined && !args[0].startsWith("--")) return null;
// `--all` means "every source Runtime actually found", so naming one too
// would make the command mean two different things.
if (all && source !== undefined) return null;
if (all) return { plan: planScanAll(full), confirmed };
// A source is the thing the Runtime scan endpoint addresses. A bare root is
// still accepted for planning (it means "all registered roots") but cannot
// be applied, because there is no endpoint that scans a raw path — the CLI
// must not pretend it can.
if (source === undefined) return { plan: planScan(root, full), confirmed };
return { plan: planScanSource(source, full), confirmed };
if (verb === "remove") {
const flags = parseFlags(args.slice(1), ["--root"], ["--yes", "--json"]);
const root = flags?.values.get("--root");
return flags && root !== undefined ? { kind: "remove", root } : null;
}
if (name === "backup") {
const verb = args[0];
if (verb !== "create" && verb !== "restore") return null;
if (seen.has("--full") || seen.has("--all") || seen.has("--no-recursive")) {
return null;
}
if (verb === "restore") {
if (source === undefined) return null;
if (target !== undefined) return null;
return {
plan: planRestore(source),
confirmed,
...(confirmPath === undefined ? {} : { confirmPath }),
};
}
if (confirmPath !== undefined) return null;
if (source !== undefined) return null;
if (target === undefined) return null;
return null;
}
/** `scan --source <id>`. A scan names a source: the Runtime has no endpoint
* that scans a bare path, so a missing `--source` is a usage error rather
* than a preview of something that can never run. */
export function parseScanCommand(args: readonly string[]): ScanCommand | null {
const flags = parseFlags(args, ["--source"], ["--all", "--yes", "--json"]);
if (!flags) return null;
const source = flags.values.get("--source");
if (flags.switches.has("--all")) {
return source === undefined ? { kind: "all" } : null;
}
return source !== undefined && SOURCE_ID.test(source)
? { kind: "source", sourceId: source }
: null;
}
/** `backup create --target <path>` and
* `backup restore --source <path> [--yes]`. */
export function parseBackupCommand(
args: readonly string[],
): BackupCommand | null {
const verb = args[0];
if (verb === "create") {
const flags = parseFlags(args.slice(1), ["--target"], ["--yes", "--json"]);
const target = flags?.values.get("--target");
return flags && target !== undefined ? { verb: "create", target } : null;
}
if (verb === "restore") {
const flags = parseFlags(
args.slice(1),
["--source", "--confirm"],
["--yes", "--json"],
);
const source = flags?.values.get("--source");
if (!flags || source === undefined) return null;
const confirmPath = flags.values.get("--confirm");
return {
plan: planBackup(target),
confirmed,
verb: "restore",
source,
confirmed: flags.switches.has("--yes"),
...(confirmPath === undefined ? {} : { confirmPath }),
};
}
return null;
}
/** Whether `--confirm <path>` names the same restore source.
/** Whether a legacy `--confirm <path>` names the same restore source.
*
* The comparison is lexical on purpose: the CLI must not resolve symlinks or
* touch the filesystem before Core has validated the managed source. A
* different string is a different confirmation, so a typo or a path escape
* can never be applied as a side effect of `--yes`.
* different string is a different confirmation.
*/
export function restoreConfirmationMatches(
source: string,
confirmPath: string | undefined,
): boolean {
return confirmPath === source;
return confirmPath === undefined || confirmPath === source;
}
export interface WriteSubmitCommand {
@@ -272,41 +177,33 @@ export interface WriteStatusCommand {
export type WriteCommand = WriteSubmitCommand | WriteStatusCommand;
/** Parse SDK-native `write submit|status`. */
/** Parse `write submit ...` and `write status <request-id>` (or `--id`). */
export function parseWriteCommand(
args: readonly string[],
): WriteCommand | null {
if (args.filter((item) => item === "--json").length > 1) return null;
const meaningful = args.filter((item) => item !== "--json");
if (meaningful[0] === "status") {
const status = parseTaskCommand(meaningful.slice(1));
const rest = meaningful.slice(1);
if (rest.length === 1 && !rest[0].startsWith("--")) {
return { kind: "status", id: rest[0] };
}
const status = parseTaskCommand(rest);
return status === null ? null : { kind: "status", id: status.id };
}
// `write --session ...` without `submit` stays accepted for older scripts.
const input = meaningful[0] === "submit" ? meaningful.slice(1) : meaningful;
let session: string | undefined;
let source = "memhub-cli";
let messagesFile: string | undefined;
let requestId: string | undefined;
let memoryAt: string | undefined;
let priority: "live" | "normal" | undefined;
for (let index = 0; index < input.length; index += 1) {
const arg = input[index];
if (arg === "--live") {
priority = "live";
continue;
}
const value = input[index + 1];
if (value === undefined || value.startsWith("--")) return null;
index += 1;
if (arg === "--session") session = value;
else if (arg === "--source") source = value;
else if (arg === "--messages-file") messagesFile = value;
else if (arg === "--request-id") requestId = value;
else if (arg === "--memory-at") memoryAt = value;
else return null;
}
const flags = parseFlags(
input,
["--session", "--source", "--messages-file", "--request-id", "--memory-at"],
["--live"],
);
if (!flags) return null;
const session = flags.values.get("--session");
const source = flags.values.get("--source") ?? "memhub-cli";
const messagesFile = flags.values.get("--messages-file");
const requestId = flags.values.get("--request-id");
const memoryAt = flags.values.get("--memory-at");
if (session === undefined || messagesFile === undefined || !source)
return null;
if (
@@ -323,39 +220,23 @@ export function parseWriteCommand(
...(requestId === undefined ? {} : { request_id: requestId }),
...(memoryAt === undefined ? {} : { memory_at: memoryAt }),
};
return priority === undefined ? command : { ...command, priority };
return flags.switches.has("--live")
? { ...command, priority: "live" }
: command;
}
export interface TaskCommand {
readonly id: string;
}
/** `--id <request-id>`: the legacy `task` spelling of `write status`. */
export function parseTaskCommand(args: readonly string[]): TaskCommand | null {
if (args.filter((item) => item === "--json").length > 1) return null;
let id: string | undefined;
for (let index = 0; index < args.length; index += 1) {
const arg = args[index];
if (arg === "--json") continue;
if (arg !== "--id") return null;
const value = args[index + 1];
if (value === undefined || value.startsWith("--")) return null;
index += 1;
id = value;
}
const flags = parseFlags(
args.filter((item) => item !== "--json"),
["--id"],
[],
);
const id = flags?.values.get("--id");
return id === undefined ? null : { id };
}
/** The human form. Kept separate from the JSON form on purpose. */
export function describe(plan: Plan): string {
const line = `${plan.action}: ${plan.target}`;
return plan.detail ? `${line} (${plan.detail})` : line;
}
/** The machine form: one document, nothing else on stdout. */
export function asPlanDocument(plan: Plan): {
schema_version: 1;
plan: Plan;
applied: boolean;
} {
return { schema_version: 1, plan, applied: false };
}
+69 -4
View File
@@ -697,6 +697,66 @@ describe("CLI-21 memory reads", () => {
expect(report.memories[1].content.length).toBeLessThan(400);
});
it("lists working and coding memories with their scene and kind", async () => {
transport.get.mockResolvedValue({
status: 200,
data: {
total: 2,
offset: 0,
limit: 20,
totals: { chat: 1, working: 1, coding: 0 },
window_id: null,
window_reset: false,
memories: [
memory("m1", "对话记忆"),
{
...memory("e1", "修复测试\n固定随机种子"),
layer: "experience",
agent_id: "",
owner: null,
source_raw_ids: [],
scene: "working",
kind: "experience",
variant: "working",
task: "修复测试",
solution: "固定随机种子",
created_at: 1_790_000_000,
updated_at: 1_790_000_100,
},
],
},
});
const streams = captureStreams();
try {
expect(await run(["memories", "list", "--json"], {})).toBe(0);
} finally {
streams.restore();
}
const report = JSON.parse(streams.stdout());
expect(report.page).toMatchObject({ state: "ok", total: 2, returned: 2 });
expect(report.memories[0]).toMatchObject({
memory_id: "m1",
scene: "chat",
kind: "chat",
variant: null,
});
expect(report.memories[1]).toMatchObject({
memory_id: "e1",
scene: "working",
kind: "experience",
variant: "working",
layer: "experience",
});
const text = captureStreams();
try {
expect(await run(["memories", "list"], {})).toBe(0);
} finally {
text.restore();
}
expect(text.stdout()).toContain("- e1 [working] active/experience");
});
it("passes the requested filters through as query parameters", async () => {
await run(
[
@@ -818,8 +878,11 @@ describe("CLI-21 memory reads", () => {
layer: "L2_FACT",
status: "active",
});
expect(report.evidence).toEqual([]);
expect(report.history).toEqual([]);
// Schema 2, like add/update/delete; the v1-era always-empty
// `evidence`/`history` fields are gone.
expect(report.schema_version).toBe(2);
expect(report).not.toHaveProperty("evidence");
expect(report).not.toHaveProperty("history");
});
// Core registers /api/v2/memories/graph before /{memory_id}, and that static
@@ -1077,14 +1140,16 @@ describe("backup create target refusal is rejected, not usage_error", () => {
let code: number;
try {
code = await run(
["backup", "create", "--target", "/tmp/outside.db", "--yes", "--json"],
["backup", "create", "--target", "/tmp/outside.db", "--json"],
{},
);
} finally {
streams.restore();
}
expect(code!).toBe(1);
const document = JSON.parse(streams.stderr());
// A result, on stdout like every --json document.
expect(streams.stderr()).toBe("");
const document = JSON.parse(streams.stdout());
expect(document.status).not.toBe("usage_error");
expect(document).toMatchObject({
command: "backup",
@@ -341,7 +341,11 @@ function validated<T>(
):
| { state: "ok"; data: T }
| { state: Exclude<ReadState["state"], "ok">; code: string } {
if (outcome.state !== "ok") return outcome;
if (outcome.state !== "ok") {
// A Core refusal of the request itself reads as an invalid answer here.
const state = outcome.state === "rejected" ? "invalid" : outcome.state;
return { state, code: outcome.code };
}
const data = validate(outcome.data);
return data === null
? { state: "invalid", code: "response_invalid" }
+4
View File
@@ -102,6 +102,10 @@ export function uiSessionFailure(code: string): { code: string } {
export function uiExitCode(result: UiResult): number {
if (result.state === "opened") return 0;
// A stopped Runtime is the same exit 3 every other command reports.
if (result.state === "blocked" && result.code === "runtime_not_running") {
return 3;
}
if (result.state === "blocked") return 4;
return 1;
}
-479
View File
@@ -1,479 +0,0 @@
import { mkdtemp, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
}));
vi.mock("axios", () => ({
default: { create: vi.fn(() => transport), get: vi.fn() },
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
const EVENT = "11111111-1111-4111-8111-111111111111";
const UPLOAD = "22222222-2222-4222-8222-222222222222";
const NOTE = "33333333-3333-4333-8333-333333333333";
beforeEach(() => {
vi.clearAllMocks();
// clearAllMocks keeps prior mock implementations, so a mockResolvedValue
// from an earlier test would leak into the next one and fake its response.
transport.get.mockReset();
transport.post.mockReset();
transport.put.mockReset();
transport.patch.mockReset();
transport.delete.mockReset();
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue({
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
});
});
describe("calls clear", () => {
it("reports runtime_stopped (exit 3) without attempting a write", async () => {
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "stopped",
code: "runtime_not_running",
});
const result = await capturedRun([
"calls",
"clear",
"--all",
"--confirm",
"--json",
]);
expect(result.code).toBe(3);
expect(JSON.parse(result.stdout).code).toBe("runtime_not_running");
expect(transport.post).not.toHaveBeenCalled();
});
it("previews without contacting Core", async () => {
const result = await capturedRun([
"calls",
"clear",
"--event",
EVENT,
"--json",
]);
expect(result.code).toBe(0);
const document = JSON.parse(result.stdout);
expect(document.state).toBe("preview");
expect(document.plan.kind).toBe("call_events_clear");
expect(transport.post).not.toHaveBeenCalled();
expect(transport.delete).not.toHaveBeenCalled();
});
it("sends the exact Core body with --confirm", async () => {
transport.post.mockResolvedValue({ status: 200, data: { removed: 1 } });
const result = await capturedRun([
"calls",
"clear",
"--event",
EVENT,
"--confirm",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/call-events/clear",
{ event_ids: [EVENT], user_confirmed: true },
undefined,
);
expect(JSON.parse(result.stdout).state).toBe("committed");
});
it("clears all without naming events", async () => {
transport.post.mockResolvedValue({ status: 200, data: { removed: 4 } });
const result = await capturedRun([
"calls",
"clear",
"--all",
"--confirm",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/call-events/clear-all",
{ user_confirmed: true },
undefined,
);
});
it("rejects a malformed event id without contacting Core", async () => {
const result = await capturedRun([
"calls",
"clear",
"--event",
"not-a-uuid",
"--json",
]);
expect(result.code).toBe(2);
expect(transport.post).not.toHaveBeenCalled();
});
it("never reports committed for a malformed success body", async () => {
transport.post.mockResolvedValue({ status: 200, data: { wrong: true } });
const result = await capturedRun([
"calls",
"clear",
"--event",
EVENT,
"--confirm",
"--json",
]);
expect(result.code).toBe(1);
const document = JSON.parse(result.stdout);
expect(document.state).toBe("unknown");
expect(document.code).toBe("ui_write_receipt_invalid");
});
});
describe.skip("retired import-file", () => {
it("never uploads without --confirm", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "om-import-"));
const file = path.join(directory, "export.json");
await writeFile(file, '{"conversations":[]}', { mode: 0o600 });
const result = await capturedRun([
"import-file",
"upload",
"--file",
file,
"--json",
]);
expect(result.code).toBe(0);
const document = JSON.parse(result.stdout);
expect(document.state).toBe("preview");
expect(document.plan.operation_id).toBeDefined();
expect(transport.post).not.toHaveBeenCalled();
});
it("uploads the file with a deterministic operation id on confirm", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "om-import-"));
const file = path.join(directory, "export.json");
await writeFile(file, '{"conversations":[]}', { mode: 0o600 });
transport.post.mockResolvedValue({
status: 201,
data: {
schema_version: 1,
operation_id: "ignored-here",
state: "completed",
upload_id: UPLOAD,
upload_state: "staged",
},
});
const preview = await capturedRun([
"import-file",
"upload",
"--file",
file,
"--json",
]);
const plan = JSON.parse(preview.stdout).plan;
transport.post.mockResolvedValue({
status: 201,
data: {
schema_version: 1,
operation_id: plan.operation_id,
state: "completed",
upload_id: UPLOAD,
upload_state: "staged",
},
});
const result = await capturedRun([
"import-file",
"upload",
"--file",
file,
"--confirm",
"--json",
]);
expect(result.code).toBe(0);
const url = transport.post.mock.calls.at(-1)?.[0] as string;
expect(url).toContain("/api/v1/import-files?");
expect(url).toContain(`operation_id=${plan.operation_id}`);
expect(url).toContain("filename=export.json");
expect(JSON.parse(result.stdout).state).toBe("committed");
});
it("commits and discards only with explicit confirmation", async () => {
const preview = await capturedRun([
"import-file",
"commit",
UPLOAD,
"--json",
]);
expect(preview.code).toBe(0);
expect(JSON.parse(preview.stdout).state).toBe("preview");
expect(transport.post).not.toHaveBeenCalled();
transport.post.mockResolvedValue({
status: 200,
data: {
upload_id: UPLOAD,
filename: "x.json",
vendor: "generic",
source_id: "generic-export",
messages: 0,
conversations: 0,
human_messages: 0,
agent_messages: 0,
skipped_records: 0,
ignored_files: 0,
checksum: "a".repeat(64),
created_at: "T",
committed: true,
},
});
const committed = await capturedRun([
"import-file",
"commit",
UPLOAD,
"--confirm",
"--json",
]);
expect(committed.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
`/api/v1/import-files/${UPLOAD}/commit`,
{ confirmed: true },
undefined,
);
transport.delete.mockResolvedValue({ status: 204, data: "" });
const discarded = await capturedRun([
"import-file",
"discard",
UPLOAD,
"--confirm",
"--json",
]);
expect(discarded.code).toBe(0);
expect(JSON.parse(discarded.stdout).state).toBe("committed");
expect(transport.delete).toHaveBeenCalledWith(
`/api/v1/import-files/${UPLOAD}`,
undefined,
);
});
it("maps a Core 404 to a stable rejected code, not a fake success", async () => {
transport.post.mockResolvedValue({
status: 404,
data: { detail: "import_file_not_found" },
});
const result = await capturedRun([
"import-file",
"commit",
UPLOAD,
"--confirm",
"--json",
]);
expect(result.code).toBe(1);
const document = JSON.parse(result.stdout);
expect(document.state).toBe("rejected");
expect(document.code).toBe("import_file_not_found");
});
});
describe("notes / extraction / graph / manual extract", () => {
it("adds a note from a bounded file and re-verifies content on apply", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "om-note-"));
const content = path.join(directory, "note.txt");
await writeFile(content, "记住这个决定", { mode: 0o600 });
const preview = await capturedRun([
"notes",
"add",
"--content-file",
content,
"--json",
]);
expect(preview.code).toBe(0);
const planFile = path.join(directory, "plan.json");
await writeFile(planFile, preview.stdout, { mode: 0o600 });
transport.post.mockResolvedValue({
status: 201,
// A realistic Core `MemoryNoteReceipt`: the shared parser requires
// `scope` and `memory_ids` too, so a minimal `{evidence_id,state}` stub
// is a malformed body, not a success.
data: {
evidence_id: NOTE,
text: "记住这个决定",
scope: "all-apps",
created_at: "2026-09-19T00:00:00Z",
state: "pending",
task_id: null,
memory_ids: [],
error: null,
},
});
const applied = await capturedRun([
"notes",
"apply",
"--plan-file",
planFile,
"--content-file",
content,
"--confirm",
"--json",
]);
expect(applied.code).toBe(0);
const body = transport.post.mock.calls.at(-1)?.[1] as Record<
string,
unknown
>;
expect(body.text).toBe("记住这个决定");
expect(body.scope).toBe("all-apps");
expect(JSON.parse(applied.stdout).state).toBe("committed");
});
it.skip("plans then applies the extraction window", async () => {
const preview = await capturedRun([
"extraction",
"set",
"--window",
"6m",
"--json",
]);
expect(preview.code).toBe(0);
expect(JSON.parse(preview.stdout).state).toBe("preview");
expect(transport.put).not.toHaveBeenCalled();
transport.put.mockResolvedValue({
status: 200,
data: { window_preset: "6m" },
});
const applied = await capturedRun([
"extraction",
"set",
"--window",
"6m",
"--confirm",
"--json",
]);
expect(applied.code).toBe(0);
expect(transport.put).toHaveBeenCalledWith(
"/api/v1/extraction-settings",
{ window_preset: "6m" },
undefined,
);
});
it("does not call graph width without --confirm", async () => {
const preview = await capturedRun([
"graph",
"width",
"--decision",
"rebuild_with_new",
"--json",
]);
expect(preview.code).toBe(0);
expect(transport.post).not.toHaveBeenCalled();
transport.post.mockResolvedValue({
status: 200,
data: { archived_path: "/tmp/old", exported_path: null, mismatch: null },
});
const applied = await capturedRun([
"graph",
"width",
"--decision",
"rebuild_with_new",
"--confirm",
"--json",
]);
expect(applied.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/graph-width",
{ decision: "rebuild_with_new" },
undefined,
);
});
it.skip("plans manual extraction and applies it only on confirm", async () => {
const preview = await capturedRun([
"processing",
"extract",
"--evidence",
NOTE,
"--json",
]);
expect(preview.code).toBe(0);
expect(JSON.parse(preview.stdout).state).toBe("preview");
expect(transport.post).not.toHaveBeenCalled();
transport.post.mockResolvedValue({
status: 200,
data: { promoted: 1, batches_planned: 2, batches_current: 1 },
});
const applied = await capturedRun([
"processing",
"extract",
"--evidence",
NOTE,
"--confirm",
"--json",
]);
expect(applied.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
"/api/v1/memory-processing/manual-extract",
{ evidence_ids: [NOTE] },
undefined,
);
});
});
async function capturedRun(arguments_: string[]) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return {
code: await run(arguments_, {
ONE_MEMORY_DATA_DIR: "/synthetic/data",
}),
stdout,
stderr,
};
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
@@ -1,501 +0,0 @@
/**
* The remaining Web Console mutating operations the CLI did not expose:
* 端内新增记忆 (memory-notes), extraction window, graph width, and manual
* extraction.
*
* Core owns every rule; this module only validates the input it is about to
* send, performs one explicit two-phase write, and normalises the receipt so
* the CLI never reports a write as committed without evidence.
*/
import { createHash, randomUUID } from "node:crypto";
import {
buildMemoryNoteSaveRequest,
MemoryNoteInputError,
parseExtractionSettingsReceipt,
parseGraphWidthDecisionReceipt,
parseManualExtractReceipt,
parseMemoryNoteReceipt,
} from "@one-memory/operations";
import type { CoreOperations, CorePutOperations } from "./core-operations.js";
import { PRODUCT } from "./version.js";
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const DIGEST = /^[0-9a-f]{64}$/;
const NOTE_MAX_CHARACTERS = 12_000;
const EXTRACTION_PRESETS = ["3m", "6m", "1y", "all"] as const;
const GRAPH_DECISIONS = [
"keep_existing",
"rebuild_with_new",
"export_first",
] as const;
const NOTE_STATES = [
"waiting_for_model",
"waiting_for_permission",
"pending",
"running",
"failed",
"ready",
"no_memories",
] as const;
export type NoteState = (typeof NOTE_STATES)[number];
export type ExtendedWritePlan =
| {
schema_version: 1;
kind: "memory_note_add";
request_id: string;
content_sha256: string;
content_characters: number;
digest: string;
}
| {
schema_version: 1;
kind: "memory_note_process";
evidence_id: string;
digest: string;
}
| {
schema_version: 1;
kind: "extraction_window_set";
window_preset: (typeof EXTRACTION_PRESETS)[number];
digest: string;
}
| {
schema_version: 1;
kind: "graph_width_decide";
decision: (typeof GRAPH_DECISIONS)[number];
digest: string;
}
| {
schema_version: 1;
kind: "manual_extract";
evidence_ids: string[];
digest: string;
};
export type ExtendedWriteOutcome =
| {
schema_version: 1;
product: string;
state: "preview";
plan: ExtendedWritePlan;
}
| {
schema_version: 1;
product: string;
state: "committed" | "recovered";
operation_id: string;
receipt: Record<string, unknown>;
}
| {
schema_version: 1;
product: string;
state: "rejected" | "unknown" | "invalid";
operation_id?: string;
code: string;
};
function canonical(value: unknown): string {
if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`;
if (value !== null && typeof value === "object") {
return `{${Object.entries(value as Record<string, unknown>)
.sort(([left], [right]) => left.localeCompare(right))
.map(([key, item]) => `${JSON.stringify(key)}:${canonical(item)}`)
.join(",")}}`;
}
if (value === undefined) return "null";
return JSON.stringify(value);
}
function digestOf(value: unknown): string {
return createHash("sha256").update(canonical(value)).digest("hex");
}
type Draft =
| Omit<Extract<ExtendedWritePlan, { kind: "memory_note_add" }>, "digest">
| Omit<Extract<ExtendedWritePlan, { kind: "memory_note_process" }>, "digest">
| Omit<
Extract<ExtendedWritePlan, { kind: "extraction_window_set" }>,
"digest"
>
| Omit<Extract<ExtendedWritePlan, { kind: "graph_width_decide" }>, "digest">
| Omit<Extract<ExtendedWritePlan, { kind: "manual_extract" }>, "digest">;
function preview(plan: Draft): ExtendedWriteOutcome {
return {
schema_version: 1,
product: PRODUCT,
state: "preview",
plan: { ...plan, digest: digestOf(plan) } as ExtendedWritePlan,
};
}
function invalid(code: string): ExtendedWriteOutcome {
return { schema_version: 1, product: PRODUCT, state: "invalid", code };
}
function committed(
operationId: string,
receipt: Record<string, unknown>,
state: "committed" | "recovered" = "committed",
): ExtendedWriteOutcome {
return {
schema_version: 1,
product: PRODUCT,
state,
operation_id: operationId,
receipt,
};
}
function rejected(
operationId: string | undefined,
code: string,
): ExtendedWriteOutcome {
return {
schema_version: 1,
product: PRODUCT,
state: "rejected",
...(operationId === undefined ? {} : { operation_id: operationId }),
code,
};
}
function unknown(
operationId: string | undefined,
code: string,
): ExtendedWriteOutcome {
return {
schema_version: 1,
product: PRODUCT,
state: "unknown",
...(operationId === undefined ? {} : { operation_id: operationId }),
code,
};
}
// --- memory-notes ---------------------------------------------------------
export function planMemoryNoteAdd(text: string): ExtendedWriteOutcome {
// The plan keeps only the note's identity, like `memories plan add`: the
//正文 lives in a bounded file and is re-read and re-hashed at apply time, so
// a plan file never carries the user's private text.
if (!validNoteText(text)) return invalid("memory_note_text_invalid");
const requestId = randomUUID();
return preview({
schema_version: 1,
kind: "memory_note_add",
request_id: requestId,
content_sha256: createHash("sha256").update(text, "utf8").digest("hex"),
content_characters: [...text].length,
});
}
export function validNoteText(text: string): boolean {
return (
text.length > 0 &&
[...text].length <= NOTE_MAX_CHARACTERS &&
text.trim().length > 0 &&
![...text].some((character) => {
const code = character.charCodeAt(0);
return code <= 8 || (code >= 11 && code <= 31) || code === 127;
})
);
}
export function planMemoryNoteProcess(
evidenceId: string,
): ExtendedWriteOutcome {
return UUID.test(evidenceId)
? preview({
schema_version: 1,
kind: "memory_note_process",
evidence_id: evidenceId,
})
: invalid("memory_note_id_invalid");
}
function noteReceipt(
operationId: string,
data: { evidence_id: string; state: NoteState },
): ExtendedWriteOutcome {
return committed(operationId, { ...data });
}
export async function executeMemoryNoteAdd(
plan: Extract<ExtendedWritePlan, { kind: "memory_note_add" }>,
operations: CoreOperations,
text: string,
): Promise<ExtendedWriteOutcome> {
if (
!validNoteText(text) ||
createHash("sha256").update(text, "utf8").digest("hex") !==
plan.content_sha256 ||
[...text].length !== plan.content_characters
) {
return invalid("memory_note_content_mismatch");
}
// The shared builder owns the save body (request_id/text/scope=all-apps), so
// the CLI and the Web console post the same three fields. Its own text check
// already ran above, so a throw here means an unusable plan.
let request: ReturnType<typeof buildMemoryNoteSaveRequest>;
try {
request = buildMemoryNoteSaveRequest({
request_id: plan.request_id,
text,
scope: "all-apps",
} as Parameters<typeof buildMemoryNoteSaveRequest>[0]);
} catch (error) {
if (error instanceof MemoryNoteInputError) {
return invalid("memory_note_content_mismatch");
}
throw error;
}
const result = await operations.post<unknown>(request.path, request.body);
if (result.state === "rejected") {
return rejected(plan.request_id, result.code);
}
if (result.state === "unknown") {
return unknown(plan.request_id, result.code);
}
const receipt = asNoteReceipt(result.data);
return receipt === null
? unknown(plan.request_id, "ui_write_receipt_invalid")
: noteReceipt(plan.request_id, receipt);
}
export async function executeMemoryNoteProcess(
plan: Extract<ExtendedWritePlan, { kind: "memory_note_process" }>,
operations: CoreOperations,
): Promise<ExtendedWriteOutcome> {
const result = await operations.post<unknown>(
`/api/v1/memory-notes/${plan.evidence_id}/process`,
{},
);
if (result.state === "rejected") {
return rejected(plan.evidence_id, result.code);
}
if (result.state === "unknown") {
return unknown(plan.evidence_id, result.code);
}
const receipt = asNoteReceipt(result.data);
return receipt === null
? unknown(plan.evidence_id, "ui_write_receipt_invalid")
: noteReceipt(plan.evidence_id, receipt);
}
function asNoteReceipt(
value: unknown,
): { evidence_id: string; state: NoteState } | null {
// The shared parser is the single judge of a note receipt; a malformed body
// returns null and the caller reports `unknown`, never a saved note.
const receipt = parseMemoryNoteReceipt(value);
return receipt === null
? null
: { evidence_id: receipt.evidence_id, state: receipt.state as NoteState };
}
// --- extraction window / graph width / manual extract ---------------------
export function planExtractionWindowSet(
windowPreset: string,
): ExtendedWriteOutcome {
return (EXTRACTION_PRESETS as readonly string[]).includes(windowPreset)
? preview({
schema_version: 1,
kind: "extraction_window_set",
window_preset: windowPreset as (typeof EXTRACTION_PRESETS)[number],
})
: invalid("extraction_window_preset_invalid");
}
export function planGraphWidthDecide(decision: string): ExtendedWriteOutcome {
return (GRAPH_DECISIONS as readonly string[]).includes(decision)
? preview({
schema_version: 1,
kind: "graph_width_decide",
decision: decision as (typeof GRAPH_DECISIONS)[number],
})
: invalid("graph_width_decision_invalid");
}
export function planManualExtract(evidenceIds: string[]): ExtendedWriteOutcome {
const ids = [...new Set(evidenceIds)];
return ids.length === evidenceIds.length && ids.every((id) => UUID.test(id))
? preview({
schema_version: 1,
kind: "manual_extract",
evidence_ids: ids,
})
: invalid("manual_extract_evidence_ids_invalid");
}
export async function executeExtractionWindowSet(
plan: Extract<ExtendedWritePlan, { kind: "extraction_window_set" }>,
operations: CorePutOperations,
): Promise<ExtendedWriteOutcome> {
const result = await operations.put<unknown>("/api/v1/extraction-settings", {
window_preset: plan.window_preset,
});
if (result.state === "rejected") {
return rejected(plan.window_preset, result.code);
}
if (result.state === "unknown") {
return unknown(plan.window_preset, result.code);
}
// The shared parser also enforces that the receipt echoes the requested
// preset, so a response belonging to another write cannot pass.
const parsed = parseExtractionSettingsReceipt(
result.data,
plan.window_preset,
);
if (parsed === null) {
return unknown(plan.window_preset, "ui_write_receipt_invalid");
}
return committed(plan.window_preset, { window_preset: plan.window_preset });
}
export async function executeGraphWidthDecide(
plan: Extract<ExtendedWritePlan, { kind: "graph_width_decide" }>,
operations: CoreOperations,
): Promise<ExtendedWriteOutcome> {
const result = await operations.post<unknown>("/api/v1/graph-width", {
decision: plan.decision,
});
if (result.state === "rejected") {
return rejected(plan.decision, result.code);
}
if (result.state === "unknown") {
return unknown(plan.decision, result.code);
}
const receipt = parseGraphWidthDecisionReceipt(result.data);
if (receipt === null) {
return unknown(plan.decision, "ui_write_receipt_invalid");
}
return committed(plan.decision, {
archived_path: receipt.archived_path ?? null,
exported_path: receipt.exported_path ?? null,
mismatch: receipt.mismatch ?? null,
});
}
export async function executeManualExtract(
plan: Extract<ExtendedWritePlan, { kind: "manual_extract" }>,
operations: CoreOperations,
): Promise<ExtendedWriteOutcome> {
const result = await operations.post<unknown>(
"/api/v1/memory-processing/manual-extract",
{ evidence_ids: plan.evidence_ids },
);
if (result.state === "rejected") {
return rejected("manual_extract", result.code);
}
if (result.state === "unknown") {
return unknown("manual_extract", result.code);
}
const receipt = parseManualExtractReceipt(result.data);
if (receipt === null) {
return unknown("manual_extract", "ui_write_receipt_invalid");
}
return committed("manual_extract", {
promoted: receipt.promoted,
batches_planned: receipt.batches_planned,
batches_current: receipt.batches_current,
});
}
export function validateExtendedWritePlan(
value: unknown,
): ExtendedWritePlan | null {
if (value === null || typeof value !== "object" || Array.isArray(value)) {
return null;
}
const candidate = value as Record<string, unknown>;
if (candidate.schema_version !== 1 || typeof candidate.kind !== "string") {
return null;
}
const { digest, ...withoutDigest } = candidate;
if (typeof digest !== "string" || digest !== digestOf(withoutDigest)) {
return null;
}
switch (candidate.kind) {
case "memory_note_add":
return typeof candidate.request_id === "string" &&
UUID.test(candidate.request_id) &&
typeof candidate.content_sha256 === "string" &&
DIGEST.test(candidate.content_sha256) &&
Number.isSafeInteger(candidate.content_characters) &&
(candidate.content_characters as number) >= 1 &&
(candidate.content_characters as number) <= NOTE_MAX_CHARACTERS &&
Object.keys(candidate).length === 6
? (candidate as unknown as ExtendedWritePlan)
: null;
case "memory_note_process":
return typeof candidate.evidence_id === "string" &&
UUID.test(candidate.evidence_id) &&
Object.keys(candidate).length === 4
? (candidate as unknown as ExtendedWritePlan)
: null;
case "extraction_window_set":
return (EXTRACTION_PRESETS as readonly string[]).includes(
String(candidate.window_preset),
) && Object.keys(candidate).length === 4
? (candidate as unknown as ExtendedWritePlan)
: null;
case "graph_width_decide":
return (GRAPH_DECISIONS as readonly string[]).includes(
String(candidate.decision),
) && Object.keys(candidate).length === 4
? (candidate as unknown as ExtendedWritePlan)
: null;
case "manual_extract":
return Array.isArray(candidate.evidence_ids) &&
candidate.evidence_ids.every(
(id) => typeof id === "string" && UUID.test(id),
) &&
Object.keys(candidate).length === 4
? (candidate as unknown as ExtendedWritePlan)
: null;
default:
return null;
}
}
export function extendedWriteExitCode(document: ExtendedWriteOutcome): number {
return ["preview", "committed", "recovered"].includes(document.state) ? 0 : 1;
}
export function renderExtendedWriteText(
document: ExtendedWriteOutcome,
): string {
if (document.state === "preview") {
const lines = ["以下操作尚未执行;加 --confirm 后才会写入:"];
const plan = document.plan;
switch (plan.kind) {
case "memory_note_add":
lines.push(`新增端内记忆:${plan.content_characters} 字符`);
break;
case "memory_note_process":
lines.push(`处理端内记忆:${plan.evidence_id}`);
break;
case "extraction_window_set":
lines.push(`设置提取窗口:${plan.window_preset}`);
break;
case "graph_width_decide":
lines.push(`图谱宽度决定:${plan.decision}`);
break;
case "manual_extract":
lines.push(`手动提取:${plan.evidence_ids.length} 条证据`);
break;
}
lines.push(`plan_digest:${plan.digest}`);
return `${lines.join("\n")}\n`;
}
if (document.state === "committed" || document.state === "recovered") {
return `${document.state === "recovered" ? "已从回执恢复" : "已提交"}:${document.operation_id}\n${JSON.stringify(document.receipt)}\n`;
}
const code = "code" in document ? document.code : "ui_write_state_invalid";
return `操作未完成:${document.state}(${code})\n`;
}
-144
View File
@@ -1,144 +0,0 @@
import { describe, expect, it, vi } from "vitest";
import type {
CoreDeleteOperations,
CoreOperations,
} from "./core-operations.js";
import {
executeCallEventsClear,
executeCallEventsClearAll,
executeImportFileCommit,
executeImportFileDiscard,
planCallEventsClear,
planCallEventsClearAll,
planImportFileCommit,
planImportFileDiscard,
planImportFileUpload,
validateUiWritePlan,
} from "./ui-write-flows.js";
const EVENT = "11111111-1111-4111-8111-111111111111";
const UPLOAD = "22222222-2222-4222-8222-222222222222";
function planOf(value: ReturnType<typeof planCallEventsClear>) {
if (value.state !== "preview") throw new Error("expected preview");
return value.plan;
}
describe("UI write plans", () => {
it("permits only valid call-event ids and rejects duplicates", () => {
expect(planCallEventsClear([EVENT]).state).toBe("preview");
expect(planCallEventsClear([]).state).toBe("invalid");
expect(planCallEventsClear([EVENT, EVENT]).state).toBe("invalid");
});
it("plans all remaining UI write operations", () => {
expect(planCallEventsClearAll().state).toBe("preview");
expect(planImportFileCommit(UPLOAD).state).toBe("preview");
expect(planImportFileDiscard(UPLOAD).state).toBe("preview");
expect(
planImportFileUpload({
path: "/tmp/export.json",
bytes: 42,
sha256: "a".repeat(64),
inputKind: "raw_conversation",
}).state,
).toBe("preview");
});
it("rejects a tampered plan digest", async () => {
const plan = planOf(planCallEventsClear([EVENT]));
expect(validateUiWritePlan(plan)).toEqual(plan);
expect(validateUiWritePlan({ ...plan, event_ids: [UPLOAD] })).toBeNull();
});
});
describe("UI write execution", () => {
it("posts clear with Core's confirmation field", async () => {
const post = vi.fn(async () => ({
state: "committed" as const,
data: { removed: 1 },
}));
const plan = planOf(planCallEventsClear([EVENT]));
if (plan.kind !== "call_events_clear") throw new Error("kind");
const result = await executeCallEventsClear(plan, {
post,
} as unknown as CoreOperations);
expect(result.state).toBe("committed");
expect(post).toHaveBeenCalledWith("/api/v1/call-events/clear", {
event_ids: [EVENT],
user_confirmed: true,
});
});
it("posts clear-all with Core's confirmation field", async () => {
const post = vi.fn(async () => ({
state: "committed" as const,
data: { removed: 2 },
}));
const plan = planOf(planCallEventsClearAll());
if (plan.kind !== "call_events_clear_all") throw new Error("kind");
await executeCallEventsClearAll(plan, {
post,
} as unknown as CoreOperations);
expect(post).toHaveBeenCalledWith("/api/v1/call-events/clear-all", {
user_confirmed: true,
});
});
it("uses POST commit and DELETE discard endpoints", async () => {
const post = vi.fn(async () => ({
state: "committed" as const,
data: {
upload_id: UPLOAD,
filename: "x.json",
vendor: "generic",
source_id: "generic-export",
messages: 0,
conversations: 0,
human_messages: 0,
agent_messages: 0,
skipped_records: 0,
ignored_files: 0,
checksum: "a".repeat(64),
created_at: "T",
committed: true,
},
}));
const del = vi.fn(async <T>() => ({
state: "committed" as const,
data: {} as T,
}));
const operations = {
post,
delete: del,
} as unknown as CoreOperations & CoreDeleteOperations;
const commit = planOf(planImportFileCommit(UPLOAD));
if (commit.kind !== "import_file_commit") throw new Error("kind");
await executeImportFileCommit(commit, operations);
expect(post).toHaveBeenCalledWith(`/api/v1/import-files/${UPLOAD}/commit`, {
confirmed: true,
});
const discard = planOf(planImportFileDiscard(UPLOAD));
if (discard.kind !== "import_file_discard") throw new Error("kind");
expect((await executeImportFileDiscard(discard, operations)).state).toBe(
"committed",
);
expect(del).toHaveBeenCalledWith(`/api/v1/import-files/${UPLOAD}`);
});
it("treats a malformed success receipt as unknown", async () => {
const plan = planOf(planCallEventsClear([EVENT]));
if (plan.kind !== "call_events_clear") throw new Error("kind");
const result = await executeCallEventsClear(plan, {
post: vi.fn(async () => ({
state: "committed" as const,
data: { wrong: true },
})),
} as unknown as CoreOperations);
expect(result).toMatchObject({
state: "unknown",
code: "ui_write_receipt_invalid",
});
});
});
-555
View File
@@ -1,555 +0,0 @@
/**
* The Web Console's remaining mutating operations, exposed to the CLI without
* duplicating Core semantics. Each operation is planned first; callers apply
* the exact plan they reviewed.
*/
import { createHash } from "node:crypto";
import { constants } from "node:fs";
import { type FileHandle, open } from "node:fs/promises";
import type {
CoreDeleteOperations,
CoreOperations,
CoreUploadOperations,
} from "./core-operations.js";
import { PRODUCT } from "./version.js";
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const INPUT_KINDS = ["raw_conversation", "memory_content"] as const;
const MAX_UPLOAD_BYTES = 128 * 1024 * 1024;
export type UiWritePlan =
| {
schema_version: 1;
kind: "call_events_clear";
event_ids: string[];
digest: string;
}
| {
schema_version: 1;
kind: "call_events_clear_all";
digest: string;
}
| {
schema_version: 1;
kind: "import_file_commit";
upload_id: string;
digest: string;
}
| {
schema_version: 1;
kind: "import_file_discard";
upload_id: string;
digest: string;
}
| {
schema_version: 1;
kind: "import_file_upload";
operation_id: string;
path: string;
filename: string;
input_kind: (typeof INPUT_KINDS)[number];
bytes: number;
sha256: string;
digest: string;
};
export type UiWriteOutcome =
| {
schema_version: 1;
product: string;
state: "preview";
plan: UiWritePlan;
}
| {
schema_version: 1;
product: string;
state: "committed" | "recovered";
operation_id: string;
receipt: Record<string, unknown>;
}
| {
schema_version: 1;
product: string;
state: "rejected" | "unknown" | "invalid";
operation_id?: string;
code: string;
receipt?: Record<string, unknown>;
};
function digestOf(value: unknown): string {
return createHash("sha256").update(canonical(value)).digest("hex");
}
function canonical(value: unknown): string {
if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`;
if (value !== null && typeof value === "object") {
return `{${Object.entries(value as Record<string, unknown>)
.sort(([left], [right]) => left.localeCompare(right))
.map(([key, item]) => `${JSON.stringify(key)}:${canonical(item)}`)
.join(",")}}`;
}
if (value === undefined) return "null";
return JSON.stringify(value);
}
export type UiWritePlanDraft =
| Omit<Extract<UiWritePlan, { kind: "call_events_clear" }>, "digest">
| Omit<Extract<UiWritePlan, { kind: "call_events_clear_all" }>, "digest">
| Omit<Extract<UiWritePlan, { kind: "import_file_commit" }>, "digest">
| Omit<Extract<UiWritePlan, { kind: "import_file_discard" }>, "digest">
| Omit<Extract<UiWritePlan, { kind: "import_file_upload" }>, "digest">;
function preview(plan: UiWritePlanDraft): UiWriteOutcome {
return {
schema_version: 1,
product: PRODUCT,
state: "preview",
plan: { ...plan, digest: digestOf(plan) } as UiWritePlan,
};
}
function invalid(code: string): UiWriteOutcome {
return { schema_version: 1, product: PRODUCT, state: "invalid", code };
}
function committed(
operationId: string,
receipt: Record<string, unknown>,
state: "committed" | "recovered" = "committed",
): UiWriteOutcome {
return {
schema_version: 1,
product: PRODUCT,
state,
operation_id: operationId,
receipt,
};
}
export function planCallEventsClear(
eventIds: readonly string[],
): UiWriteOutcome {
if (
eventIds.length < 1 ||
eventIds.length > 100 ||
new Set(eventIds).size !== eventIds.length ||
eventIds.some((id) => !UUID.test(id))
) {
return invalid("call_event_ids_invalid");
}
return preview({
schema_version: 1,
kind: "call_events_clear",
event_ids: [...eventIds],
});
}
export function planCallEventsClearAll(): UiWriteOutcome {
return preview({ schema_version: 1, kind: "call_events_clear_all" });
}
export function planImportFileCommit(uploadId: string): UiWriteOutcome {
return UUID.test(uploadId)
? preview({
schema_version: 1,
kind: "import_file_commit",
upload_id: uploadId,
})
: invalid("import_file_id_invalid");
}
export function planImportFileDiscard(uploadId: string): UiWriteOutcome {
return UUID.test(uploadId)
? preview({
schema_version: 1,
kind: "import_file_discard",
upload_id: uploadId,
})
: invalid("import_file_id_invalid");
}
export function planImportFileUpload(input: {
path: string;
bytes: number;
sha256: string;
inputKind: "raw_conversation" | "memory_content";
}): UiWriteOutcome {
const filename = input.path.split(/[/\\]/).pop() ?? "";
if (
input.path.length === 0 ||
!Number.isSafeInteger(input.bytes) ||
input.bytes < 0 ||
input.bytes > MAX_UPLOAD_BYTES ||
!/^[0-9a-f]{64}$/.test(input.sha256) ||
!INPUT_KINDS.includes(input.inputKind) ||
filename.length === 0 ||
[...filename].length > 200 ||
!/\.(?:zip|json|md|txt)$/i.test(filename)
) {
return invalid("import_file_upload_invalid");
}
return preview({
schema_version: 1,
kind: "import_file_upload",
operation_id: fileUploadOperationId(
filename,
input.inputKind,
input.sha256,
),
path: input.path,
filename,
input_kind: input.inputKind,
bytes: input.bytes,
sha256: input.sha256,
});
}
/**
* Read one regular, non-symlinked export without buffering past the 128 MiB
* limit. Returns the bytes plus the checksum the plan records, so apply can
* prove the file it uploads is the one the caller reviewed.
*/
export async function readImportFileUpload(
filePath: string,
inputKind: "raw_conversation" | "memory_content",
): Promise<{ plan: UiWritePlan; bytes: Buffer } | null> {
let handle: FileHandle;
try {
handle = await open(
filePath,
constants.O_RDONLY |
(constants.O_NONBLOCK ?? 0) |
(constants.O_NOFOLLOW ?? 0),
);
} catch {
return null;
}
try {
const stats = await handle.stat();
if (!stats.isFile() || stats.nlink !== 1 || stats.size > MAX_UPLOAD_BYTES) {
return null;
}
const buffer = Buffer.alloc(Number(stats.size));
let offset = 0;
while (offset < buffer.length) {
const chunk = await handle.read(
buffer,
offset,
buffer.length - offset,
offset,
);
if (chunk.bytesRead === 0) break;
offset += chunk.bytesRead;
}
if (offset !== buffer.length) return null;
const checksum = createHash("sha256").update(buffer).digest("hex");
const result = planImportFileUpload({
path: filePath,
bytes: buffer.byteLength,
sha256: checksum,
inputKind,
});
if (result.state !== "preview") return null;
return { plan: result.plan, bytes: buffer };
} catch {
return null;
} finally {
await handle.close().catch(() => undefined);
}
}
/**
* Core keys upload idempotency by `operation_id` and rejects a replay whose
* fingerprint differs. Deriving it from the file's own identity means a retry
* after a dropped response resumes the same staging operation instead of
* creating a second upload; a genuinely different payload maps to a different
* operation and cannot overwrite the first.
*/
function fileUploadOperationId(
filename: string,
inputKind: (typeof INPUT_KINDS)[number],
checksum: string,
): string {
const hex = createHash("sha256")
.update(JSON.stringify([filename, inputKind, checksum]))
.digest("hex");
return [
hex.slice(0, 8),
hex.slice(8, 12),
`4${hex.slice(13, 16)}`,
`${(8 + (parseInt(hex[16]!, 16) % 4)).toString(16)}${hex.slice(17, 20)}`,
hex.slice(20, 32),
].join("-");
}
export function validateUiWritePlan(value: unknown): UiWritePlan | null {
if (value === null || typeof value !== "object" || Array.isArray(value)) {
return null;
}
const candidate = value as Record<string, unknown>;
if (candidate.schema_version !== 1 || typeof candidate.kind !== "string") {
return null;
}
const { digest, ...withoutDigest } = candidate;
if (typeof digest !== "string" || digest !== digestOf(withoutDigest)) {
return null;
}
switch (candidate.kind) {
case "call_events_clear":
return Array.isArray(candidate.event_ids) &&
candidate.event_ids.length >= 1 &&
candidate.event_ids.length <= 100 &&
new Set(candidate.event_ids).size === candidate.event_ids.length &&
candidate.event_ids.every(
(id) => typeof id === "string" && UUID.test(id),
)
? (candidate as unknown as UiWritePlan)
: null;
case "call_events_clear_all":
return Object.keys(candidate).length === 3
? (candidate as unknown as UiWritePlan)
: null;
case "import_file_commit":
case "import_file_discard":
return typeof candidate.upload_id === "string" &&
UUID.test(candidate.upload_id) &&
Object.keys(candidate).length === 4
? (candidate as unknown as UiWritePlan)
: null;
case "import_file_upload":
return typeof candidate.operation_id === "string" &&
UUID.test(candidate.operation_id) &&
typeof candidate.path === "string" &&
typeof candidate.filename === "string" &&
candidate.filename === candidate.path.split(/[/\\]/).pop() &&
/^[0-9a-f]{64}$/.test(String(candidate.sha256)) &&
INPUT_KINDS.includes(
candidate.input_kind as (typeof INPUT_KINDS)[number],
) &&
Number.isSafeInteger(candidate.bytes) &&
(candidate.bytes as number) >= 0 &&
(candidate.bytes as number) <= 128 * 1024 * 1024
? (candidate as unknown as UiWritePlan)
: null;
default:
return null;
}
}
export function uiWriteExitCode(document: UiWriteOutcome): number {
return ["preview", "committed", "recovered"].includes(document.state) ? 0 : 1;
}
export function renderUiWriteText(document: UiWriteOutcome): string {
if (document.state === "preview") {
return `以下操作尚未执行;加 --confirm 后才会写入:\n${document.plan.kind}\nplan_digest:${document.plan.digest}\n`;
}
if (document.state === "committed" || document.state === "recovered") {
return `已提交:${document.operation_id}\n${JSON.stringify(document.receipt)}\n`;
}
const code = "code" in document ? document.code : "ui_write_state_invalid";
return `操作未完成:${document.state}(${code})\n`;
}
export async function executeCallEventsClear(
plan: Extract<UiWritePlan, { kind: "call_events_clear" }>,
operations: CoreOperations,
): Promise<UiWriteOutcome> {
const result = await operations.post<unknown>("/api/v1/call-events/clear", {
event_ids: plan.event_ids,
user_confirmed: true,
});
return outcome(result, plan.event_ids.join(","), ["removed"]);
}
export async function executeCallEventsClearAll(
plan: Extract<UiWritePlan, { kind: "call_events_clear_all" }>,
operations: CoreOperations,
): Promise<UiWriteOutcome> {
const result = await operations.post<unknown>(
"/api/v1/call-events/clear-all",
{ user_confirmed: true },
);
return outcome(result, "all", ["removed"]);
}
export async function executeImportFileCommit(
plan: Extract<UiWritePlan, { kind: "import_file_commit" }>,
operations: CoreOperations,
): Promise<UiWriteOutcome> {
const result = await operations.post<unknown>(
`/api/v1/import-files/${plan.upload_id}/commit`,
{ confirmed: true },
);
return outcome(result, plan.upload_id, [
"upload_id",
"filename",
"vendor",
"source_id",
"messages",
"conversations",
"human_messages",
"agent_messages",
"skipped_records",
"ignored_files",
"checksum",
"created_at",
"committed",
]);
}
export async function executeImportFileDiscard(
plan: Extract<UiWritePlan, { kind: "import_file_discard" }>,
operations: CoreDeleteOperations,
): Promise<UiWriteOutcome> {
const result = await operations.delete<unknown>(
`/api/v1/import-files/${plan.upload_id}`,
);
// Core answers 204 with no body. A successful delete is a committed
// operation even though there is no receipt to validate; requiring a body
// here would report every real discard as `ui_write_receipt_invalid`.
return outcome(result, plan.upload_id, null);
}
/** Upload one local file through the same recoverable Core staging flow the
* Web console uses. The operation id is part of the plan, so a retry after a
* dropped response resumes the same staging record instead of duplicating it.
*/
export async function executeImportFileUpload(
plan: Extract<UiWritePlan, { kind: "import_file_upload" }>,
operations: CoreUploadOperations,
): Promise<UiWriteOutcome> {
// Re-read the file at apply time and re-derive its identity. A plan that
// names a path whose content changed after review is refused rather than
// uploading something the caller never saw.
const reread = await readImportFileUpload(plan.path, plan.input_kind);
if (reread === null) {
return invalid("import_file_upload_unreadable");
}
const bytes = reread.bytes;
const checksum = createHash("sha256").update(bytes).digest("hex");
if (
bytes.byteLength !== plan.bytes ||
checksum !== plan.sha256 ||
reread.plan.digest !== plan.digest
) {
return invalid("import_file_upload_content_mismatch");
}
const result = await operations.uploadImportFile<unknown>({
bytes,
filename: plan.filename,
inputKind: plan.input_kind,
operationId: plan.operation_id,
});
if (result.state === "rejected") {
return {
schema_version: 1,
product: PRODUCT,
state: "rejected",
operation_id: plan.operation_id,
code: result.code,
};
}
if (result.state === "unknown") {
return {
schema_version: 1,
product: PRODUCT,
state: "unknown",
operation_id: plan.operation_id,
code: result.code,
};
}
const receipt =
result.data !== null &&
typeof result.data === "object" &&
!Array.isArray(result.data)
? (result.data as Record<string, unknown>)
: null;
if (
receipt === null ||
receipt.operation_id !== plan.operation_id ||
typeof receipt.state !== "string"
) {
return {
schema_version: 1,
product: PRODUCT,
state: "unknown",
operation_id: plan.operation_id,
code: "ui_write_receipt_invalid",
};
}
// Core reports the staged upload as `completed`; `pending`/`not_found` here
// means the operation record did not settle, so the result is not known.
if (receipt.state !== "completed") {
return {
schema_version: 1,
product: PRODUCT,
state: "unknown",
operation_id: plan.operation_id,
code: `import_upload_${receipt.state}`,
};
}
const uploadId =
typeof receipt.upload_id === "string" ? receipt.upload_id : undefined;
if (uploadId === undefined || !UUID.test(uploadId)) {
return {
schema_version: 1,
product: PRODUCT,
state: "unknown",
operation_id: plan.operation_id,
code: "ui_write_receipt_invalid",
};
}
return committed(plan.operation_id, {
operation_id: plan.operation_id,
upload_id: uploadId,
upload_state: receipt.upload_state ?? null,
});
}
function outcome(
result:
| Awaited<ReturnType<CoreOperations["post"]>>
| Awaited<ReturnType<CoreDeleteOperations["delete"]>>,
operationId: string,
required: readonly string[] | null,
): UiWriteOutcome {
if (result.state === "rejected") {
return {
schema_version: 1,
product: PRODUCT,
state: "rejected",
operation_id: operationId,
code: result.code,
};
}
if (result.state === "unknown") {
return {
schema_version: 1,
product: PRODUCT,
state: "unknown",
operation_id: operationId,
code: result.code,
};
}
if (required === null) {
return committed(operationId, {});
}
const data =
result.data !== null &&
typeof result.data === "object" &&
!Array.isArray(result.data)
? (result.data as Record<string, unknown>)
: null;
if (data === null || required.some((key) => !Object.hasOwn(data, key))) {
return {
schema_version: 1,
product: PRODUCT,
state: "unknown",
operation_id: operationId,
code: "ui_write_receipt_invalid",
};
}
return committed(operationId, data);
}
+15 -1
View File
@@ -13,7 +13,16 @@ const GLOBAL_OPTIONS: readonly CommandOption[] = [
summary: "显示帮助;可与 --json 组合输出机器可读目录。",
},
{ flag: "--version, -V", summary: "显示 CLI 版本。" },
{ flag: "--json", summary: "以 JSON 输出到 stdout,供脚本与 Agent 消费。" },
{
flag: "--json",
summary:
"以 JSON 输出到 stdout,供脚本与 Agent 消费;结果与失败都在 stdout,只有参数错误写 stderr。",
},
{
flag: "--data-dir <path>",
summary:
"本次使用的 MemHub 数据目录(绝对路径);默认 ONE_MEMORY_DATA_DIR > 安装回执 > ~/.one-memory。",
},
];
export function renderGeneralHelp(): string {
@@ -33,6 +42,9 @@ export function renderGeneralHelp(): string {
(exit) => ` ${exit.code} ${pad(exit.name, 17)}${exit.summary}`,
),
"",
"确认约定:只有不可逆操作(memories delete、models remove、delete-data、backup restore、graph-width --decision rebuild_with_new)需要 --yes;不加时只说明将要做什么,不做任何改动,exit 0。delete-data 与 backup restore 还要用 --confirm <path> 再写一遍目标路径。install/upgrade/uninstall/service-install/service-uninstall 不加 --yes 时只出计划。其它写命令直接执行。",
"异步操作返回 ID,用对应命令查进度:sources receipt、models receipt、memories add-status、write status。",
"目前仅支持 macOS 与 Linux 连接本机 Runtime。",
"stdout 只输出数据,进度与诊断写入 stderr。",
`运行 ${PRODUCT} help <command> 查看单个命令的参数。`,
"",
@@ -55,6 +67,8 @@ export function renderCommandHelp(command: CommandDescriptor): string {
);
}
lines.push(
"Global options: --json、--data-dir <path>(见 memhub help)。",
"",
"Exit codes:",
...EXIT_CODES.filter((exit) => command.exit_codes.includes(exit.code)).map(
(exit) => ` ${exit.code} ${pad(exit.name, 17)}${exit.summary}`,
+1 -1
View File
@@ -12,7 +12,7 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.5",
"hy-memory==2.0.8",
# chromadb arrives through the HY SDK; locked here so an unrelated upgrade
# cannot move the SDK's store. Built-in embedding consumes numpy directly.
"chromadb==1.5.9",
+446 -105
View File
@@ -47,8 +47,8 @@ from one_memory.background_work import shutdown_bulk_pool, warm_bulk_pool
from one_memory.config import Settings, load_settings
from one_memory.context_routes import sdk_context_router
from one_memory.conversation_processing import (
_conversation_chunks,
memory_task_timeout_seconds,
turn_batches,
)
from one_memory.database_access import (
MAIN_DATABASE_REPORT_SECONDS,
@@ -74,7 +74,11 @@ from one_memory.exception_reporting import (
report_exception,
watched_task,
)
from one_memory.external_memory_access import external_access, require_external_endpoint
from one_memory.external_memory_access import (
ExternalMemoryAccess,
external_access,
require_external_endpoint,
)
from one_memory.extraction_settings import ExtractionSettings, ExtractionSettingsStore
from one_memory.file_imports import FileImportStore, file_import_router
from one_memory.hy_graph_admin import GraphWidthDecision, GraphWidthMismatch
@@ -87,20 +91,26 @@ from one_memory.hy_runtime_config import (
set_diagnostic_log_level,
)
from one_memory.hy_sdk_runtime import (
AmbiguousMemoryIdError,
SdkClientHandle,
builtin_retrieval_snapshot,
close_all_shared_runtimes,
hold_stores_offline,
release_stores,
sdk_relevance_floor,
)
from one_memory.hy_sdk_types import (
HySdkError,
HySdkMemoryPage,
HySdkMemoryRecord,
HySdkRawHit,
HySdkRawPage,
HySdkRawRecord,
HySdkReadError,
HySdkUserProfile,
LibraryHit,
library_keyword_hits,
parse_sdk_raw_search,
parse_sdk_search,
)
from one_memory.imports.access import (
@@ -152,16 +162,17 @@ from one_memory.imports.sources.registry import (
)
from one_memory.imports.storage import EvidenceStore
from one_memory.imports.supervisor import ImportSupervisor
from one_memory.library_search_window import LibrarySearchWindows, window_key
from one_memory.library_upgrade import LibraryUpgradeError, prepare_library
from one_memory.loop_watchdog import LoopWatchdog
from one_memory.memory_additions import (
LegacyMemoryAdditionReader,
MemoryAdditionStore,
memory_add_router,
)
from one_memory.memory_governance import (
MemoryGovernanceStore,
)
from one_memory.memory_notes import (
LegacyMemoryNoteReader,
MemoryNoteStore,
memory_note_router,
)
from one_memory.memory_qa import MemoryAnswerError
from one_memory.memory_queue import (
ExtractionWindowPreset,
@@ -214,6 +225,7 @@ from one_memory.sdk_memory_qa import (
SdkMemoryAnswerer,
)
from one_memory.sdk_memory_runtime import SdkMemoryRuntime
from one_memory.sdk_migration_log import error_chain_text, error_frames
from one_memory.sdk_write_processor import SdkWriteCoordinator
from one_memory.secrets import (
KeyringSecretStore,
@@ -396,10 +408,20 @@ class ConnectSourcesResponse(BaseModel):
request_id: str | None = None
SdkMemoryKind = Literal["chat", "experience", "fact"]
SdkMemoryVariant = Literal["working", "coding"]
class SdkMemoryUpdateRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
# Chat: the memory text. Experience: the solution. Fact: the statement.
content: str = Field(min_length=1, max_length=200_000)
# Optional namespace, as listed on the memory (`kind`, `variant`).
kind: SdkMemoryKind | None = None
variant: SdkMemoryVariant | None = None
# Experiences only: also replace the task.
task: str | None = Field(default=None, min_length=1, max_length=200_000)
class SdkUserProfileResponse(BaseModel):
@@ -691,7 +713,8 @@ def create_app(
raise SdkMigrationRequired()
if app.state.memory_authority is MemoryAuthority.SDK:
# The metadata repair after a cutover reads every bundle: it is
# recovery work, so it runs after readiness (`recover_after_ready`).
# recovery work, so it runs after readiness (`recover_after_ready`)
# while the SDK's stores are held offline (see there).
app.state.memory_governance_store = RetiredGovernanceStore(database_path)
app.state.evidence_store = RetiredEvidenceStore(database_path)
else:
@@ -1007,34 +1030,31 @@ def create_app(
app.state.memory_write_intake = UnifiedWriteIntake(
memory_write_queue,
source_writes_permitted=lambda source_id: (
source_id == "one-memory-manual" or _source_writes_on(app, source_id)
),
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
)
app.state.start_memory_backfill = lambda: _start_backfill(app)
async def enqueue_stable_evidence(record: EvidenceRecord) -> None:
priority = "live" if live_sync_arrival.get() else "normal"
live = live_sync_arrival.get()
conversation = await asyncio.to_thread(
app.state.evidence_store.list_conversation, record.evidence_id
)
for chunk in _conversation_chunks(conversation, max_turns=12, completed_only=True):
if (
record not in chunk
or not any(item.role == "user" for item in chunk)
or not any(item.role == "assistant" for item in chunk)
):
continue
# Legacy authority. The turn rule groups the whole conversation
# from its start, so a batch's records and key never change:
# history (end of file) writes it all, live writes complete
# batches and leaves the rest to catch-up (`_run_backfill`).
batches = turn_batches(conversation, flush=not live).batches
for batch in _batches_settled_by(batches, record):
await asyncio.to_thread(
app.state.memory_write_intake.submit_records,
chunk,
priority=priority,
batch,
priority="live" if live else "normal",
)
app.state.memory_note_store = MemoryNoteStore(
app.state.memory_addition_store = MemoryAdditionStore(
app.state.evidence_store,
(
LegacyMemoryNoteReader(
LegacyMemoryAdditionReader(
app.state.memory_task_store,
app.state.memory_governance_store,
)
@@ -1170,10 +1190,14 @@ def create_app(
except Exception as error:
failed.append(name)
app.state.recovery_failed_steps = tuple(failed)
# Where it failed, as `file:line:function` frames of
# the exception chain, never its message: a bare
# `error_type=KeyError` named no location.
logger.warning(
"core_recovery_step_failed step=%s error_type=%s",
"core_recovery_step_failed step=%s error_type=%s chain=%s",
name,
type(error).__name__,
error_chain_text(error),
)
diagnostic_event(
level="warn",
@@ -1181,20 +1205,42 @@ def create_app(
event="core.recovery_step_failed",
outcome="failed",
error_type=type(error).__name__,
safe_stack=error_frames(error) or None,
duration_ms=elapsed_ms(step_started),
details={"phase": name},
)
async def repair_sdk_metadata() -> None:
# `sdk-migrate` repairs right after its cutover and writes
# the receipt, so this normally finds it and returns. It
# stays for homes whose repair failed before that (no
# receipt). It opens its own Chroma client on the SDK's
# vector store, and Chroma's shared System is not safe
# beside another client: a first library request that
# opened the SDK client meanwhile lost it (KeyError).
# The stores were held offline before readiness, so any
# SDK client waits for this step.
from one_memory.hy_sdk_migration import (
repair_sdk_metadata_from_bundle,
)
repair = await asyncio.to_thread(
repair_sdk_metadata_from_bundle,
data_dir=runtime_settings.data_dir,
database_path=database_path,
if not app.state.sdk_stores_held:
logger.warning("sdk_metadata_repair_deferred reason=sdk_client_open")
return
# The hold is this step's from here; shutdown leaves it.
app.state.sdk_stores_held = False
work = asyncio.ensure_future(
asyncio.to_thread(
repair_sdk_metadata_from_bundle,
data_dir=runtime_settings.data_dir,
database_path=database_path,
)
)
# Released when the thread ends, not when this coroutine
# is cancelled: a cancelled await leaves the thread
# writing Chroma.
work.add_done_callback(lambda _work: release_stores(runtime_settings.data_dir))
repair = await asyncio.shield(work)
if repair.vector_rows or repair.coding_rows:
logger.info(
"sdk_metadata_repair_completed mapped=%d vector=%d coding=%d",
@@ -1318,7 +1364,12 @@ def create_app(
)
app.state.recovery_task = None
app.state.sdk_stores_held = False
try:
if app.state.memory_authority is MemoryAuthority.SDK:
# Before the first request can open an SDK client; the
# metadata repair step releases it.
app.state.sdk_stores_held = hold_stores_offline(runtime_settings.data_dir)
diagnostic_event(
level="info",
component="core.lifecycle",
@@ -1375,6 +1426,10 @@ def create_app(
if app.state.live_sync_watch is not None:
await app.state.live_sync_watch.shutdown()
await app.state.live_sync_supervisor.shutdown()
if app.state.sdk_stores_held:
# Recovery never reached the metadata repair step.
release_stores(runtime_settings.data_dir)
app.state.sdk_stores_held = False
await app.state.import_supervisor.shutdown()
app.state.memory_write_drain.stop()
if app.state.memory_write_drain_task is not None:
@@ -1441,7 +1496,7 @@ def create_app(
Depends(local_api_guard(runtime_settings, valid_ui_session=ui_session_valid)),
Depends(require_external_endpoint),
]
app.include_router(memory_note_router(protected))
app.include_router(memory_add_router(protected))
app.include_router(file_import_router(protected))
app.include_router(sdk_context_router(protected))
@@ -2805,11 +2860,24 @@ def create_app(
dependencies=protected,
)
async def sdk_home_summary() -> SdkHomeSummary:
"""The home card: the memory page's count and its newest memory.
The count is `memory_stats` total (chat + working + coding
experiences, facts not counted), the same set the memory page lists
with scene="all". Without a scene `list_memories` counts chat only,
which showed 55 on the home card against 94 on the memory page.
"""
try:
client = await _sdk_client_handle(app).client()
stats = await asyncio.to_thread(
client.memory_stats,
user_id="local",
statuses=["active"],
)
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
scene="all",
statuses=["active"],
limit=1,
offset=0,
@@ -2819,8 +2887,9 @@ def create_app(
_log_skipped_sdk_memories(skipped)
except Exception as error:
raise _sdk_http_error(error, "hy_memory_list_failed") from None
total = stats.get("total") if isinstance(stats, dict) else None
return SdkHomeSummary(
memory_count=page.total,
memory_count=total if isinstance(total, int) and total >= 0 else page.total,
latest=page.memories[0] if page.memories else None,
observed_at=datetime.now(UTC),
)
@@ -2835,8 +2904,24 @@ def create_app(
source: str | None = Query(default=None, min_length=1),
query: str | None = Query(default=None, max_length=512),
offset: int = Query(default=0, ge=0),
limit: int = Query(default=100, ge=1, le=500),
limit: int = Query(default=LIBRARY_PAGE_SIZE, ge=1, le=500),
window_id: str | None = Query(default=None, min_length=1, max_length=64),
scene: Literal["all", "chat", "working", "coding"] = Query(default="all"),
) -> HySdkMemoryPage:
"""One page of memories across chat, working and coding.
Without `query`: `list_memories(scene="all")`, newest update first;
`total` is the SDK's count across the three scenes. `scene` narrows
the page to one scene: chat lists the chat store, working/coding list
that scene's experiences (the SDK lists experiences only, all active).
With `query`: keyword search over one SDK relevance window (see
`_library_search_page`). The first request (no `window_id`) searches
and returns a `window_id`; later pages pass it back with their offset
and are cut from the same matches, so pages never overlap or skip. A
`window_id` that expired or was searched with other filters searches
again and answers page 1 with `window_reset: true`.
"""
try:
client = await _sdk_client_handle(app).client()
search_query = (query or "").strip()
@@ -2846,59 +2931,26 @@ def create_app(
status_code=422,
detail="sdk_search_supports_active_status_only",
)
# SDK search has no count-only contract and, on the local store,
# no keyword channel. Ask for one bounded relevance window with
# no score floor, then keep the rows that contain the words, so
# a word in the library is found and a word in none of it finds
# nothing. The whole library is never listed in Core.
# A larger window made one search take long enough to freeze
# the page; 20 is the bounded window the library searches.
fetch = 20
search_raw = await asyncio.to_thread(
client.search,
search_query,
user_ids=["local"],
agent_ids=[],
include_sources=[source] if source else None,
limit=fetch,
min_score=0.0,
schema_min_score=0.0,
schema_limit=fetch,
# The library lists stored memories; proactive intentions
# are a recall-time lane the list never shows.
intention_limit=0,
scene="normal",
reader=HY_MEMORY_READER,
)
hit_ids, window = library_keyword_hits(search_raw, search_query)
rows = await _get_sdk_rows(client, list(hit_ids[offset : offset + limit]))
memories: list[HySdkMemoryRecord] = []
skipped = 0
for row in rows:
if row is None:
continue
try:
record = HySdkMemoryRecord.from_sdk(row)
except HySdkReadError:
skipped += 1
continue
if record.status == "active":
memories.append(record)
_log_skipped_sdk_memories(skipped)
return HySdkMemoryPage(
memories=tuple(memories),
total=len(hit_ids),
limit=limit,
return await _library_search_page(
app,
client,
query=search_query,
source=source,
statuses=list(statuses),
offset=offset,
search_window=fetch if window >= fetch else None,
limit=limit,
window_id=window_id,
scene=scene,
)
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
scene=scene,
statuses=list(statuses),
include_sources=[source] if source else None,
offset=offset,
limit=limit,
order="desc",
)
page, skipped = HySdkMemoryPage.from_sdk_skipping_invalid(
raw, limit=limit, offset=offset
@@ -2928,11 +2980,13 @@ def create_app(
"""
try:
client = await _sdk_client_handle(app).client()
# The map draws the page the list shows, so it reads the same
# source filter and window rather than the newest memories overall.
# The map visualizes the active source scope, independently of the
# left list's 20-row pagination window.
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
scene="all",
statuses=["active"],
include_sources=[source] if source else None,
limit=limit,
offset=offset,
@@ -3066,9 +3120,18 @@ def create_app(
response_model=HySdkMemoryRecord,
dependencies=protected,
)
async def get_sdk_memory(memory_id: str) -> HySdkMemoryRecord:
async def get_sdk_memory(
memory_id: str,
kind: SdkMemoryKind | None = None,
variant: SdkMemoryVariant | None = None,
) -> HySdkMemoryRecord:
_check_memory_namespace(kind, variant)
try:
memory = await _get_sdk_memory(_sdk_client_handle(app), memory_id)
memory = await _get_sdk_memory(
_sdk_client_handle(app), memory_id, kind=kind, variant=variant
)
except AmbiguousMemoryIdError:
raise HTTPException(status_code=409, detail="memory_id_ambiguous") from None
except Exception as error:
raise _sdk_http_error(error, "hy_memory_get_failed") from None
if memory is None:
@@ -3084,13 +3147,40 @@ def create_app(
memory_id: str,
request: SdkMemoryUpdateRequest,
) -> HySdkMemoryRecord:
_check_memory_namespace(request.kind, request.variant)
if request.task is not None and request.kind in {"chat", "fact"}:
raise HTTPException(status_code=422, detail="memory_task_requires_experience")
namespace = _namespace_kwargs(request.kind, request.variant)
try:
client = await app.state.sdk_write_coordinator.client()
result = await asyncio.to_thread(client.update, memory_id, request.content)
if not isinstance(result, dict) or result.get("success") is not True:
raise HySdkError("hy_memory_update_rejected")
raw = await asyncio.to_thread(client.get, memory_id)
async with app.state.sdk_write_coordinator.use_client() as client:
update_kwargs: dict[str, object] = dict(namespace)
if request.task is not None:
update_kwargs["task"] = request.task
result = await asyncio.to_thread(
client.update, memory_id, request.content, **update_kwargs
)
if not isinstance(result, dict):
raise HySdkError("hy_memory_update_rejected")
if result.get("success") is not True:
if "not found" in str(result.get("error_message") or ""):
raise HTTPException(status_code=404, detail="memory_not_found")
raise HySdkError("hy_memory_update_rejected")
# The SDK names the namespace it wrote; read that one back.
if result.get("kind") in {"experience", "fact"}:
namespace = {"kind": result["kind"]}
if result.get("variant") in {"working", "coding"}:
namespace["variant"] = result["variant"]
raw = await asyncio.to_thread(client.get, memory_id, **namespace)
memory = None if raw is None else HySdkMemoryRecord.from_sdk(raw)
except HTTPException:
raise
except AmbiguousMemoryIdError:
raise HTTPException(status_code=409, detail="memory_id_ambiguous") from None
except ValueError as error:
if request.task is None:
raise _sdk_http_error(error, "hy_memory_update_failed") from None
# The SDK refuses `task` for anything but an experience.
raise HTTPException(status_code=422, detail="memory_task_requires_experience") from None
except Exception as error:
raise _sdk_http_error(error, "hy_memory_update_failed") from None
if memory is None:
@@ -3102,10 +3192,17 @@ def create_app(
status_code=204,
dependencies=protected,
)
async def delete_sdk_memory(memory_id: str) -> Response:
async def delete_sdk_memory(
memory_id: str,
kind: SdkMemoryKind | None = None,
variant: SdkMemoryVariant | None = None,
) -> Response:
_check_memory_namespace(kind, variant)
try:
client = await app.state.sdk_write_coordinator.client()
result = await asyncio.to_thread(client.delete, memory_id)
async with app.state.sdk_write_coordinator.use_client() as client:
result = await asyncio.to_thread(
client.delete, memory_id, **_namespace_kwargs(kind, variant)
)
if not isinstance(result, dict):
raise HySdkError("hy_memory_delete_rejected")
error_code = result.get("error_code")
@@ -3113,6 +3210,8 @@ def create_app(
raise HySdkError(str(error_code or "hy_memory_delete_incomplete"))
if result.get("success") is not True and result.get("deleted_count") != 0:
raise HySdkError("hy_memory_delete_rejected")
except AmbiguousMemoryIdError:
raise HTTPException(status_code=409, detail="memory_id_ambiguous") from None
except Exception as error:
raise _sdk_http_error(error, "hy_memory_delete_failed") from None
return Response(status_code=204)
@@ -3239,6 +3338,7 @@ def create_app(
http_request: Request,
) -> RecallReceipt:
from one_memory.call_event_recorder import record_legacy_result
from one_memory.recall_fallback import recall_with_fallback
started = time.monotonic()
await _require_external_recall_scope(
@@ -3246,7 +3346,16 @@ def create_app(
)
target = _recall_target(app)
access = external_access(http_request.headers)
receipt = await target.recall_async(request)
receipt = await recall_with_fallback(
request,
model_ready=lambda: _recall_model_ready(app),
recall_memories=target.recall_async,
has_memories=lambda: _library_has_memories(app),
raw_sources=lambda: _recall_raw_sources(app, request, access, started),
search_raw=lambda query, limit, sources: _search_sdk_raw(
app, query, limit=limit, sources=sources
),
)
event_store = getattr(app.state, "call_event_store", None)
if event_store is not None and request.query != "__om_probe__":
try:
@@ -3335,9 +3444,9 @@ def create_app(
"""Retry unknown writes only behind the user's explicit confirmation."""
del request
reconciled = await _sdk_memory_runtime(app).reconcile_unknown_outputs()
from one_memory.memory_server.durable_queue import DurableWriteQueue
queue = DurableWriteQueue(app.state.memory_write_queue_path)
# The startup queue knows the authority. A fresh legacy-mode queue
# would recreate the job-evidence table the SDK cutover dropped.
queue = app.state.memory_write_queue
unknown = queue.unknown_jobs(limit=2048)
if app.state.memory_authority is MemoryAuthority.LEGACY:
for job in unknown:
@@ -4273,14 +4382,11 @@ async def _run_backfill(app: FastAPI) -> None:
_evidence_store(app).list_conversation,
evidence_id,
)
for chunk in _conversation_chunks(
conversation,
max_turns=12,
completed_only=True,
):
if not any(item.evidence_id in evidence_ids for item in chunk):
# Catch-up is history: the last turn and a short batch are written.
for batch in turn_batches(conversation, flush=True).batches:
if not any(item.evidence_id in evidence_ids for item in batch):
continue
await asyncio.to_thread(intake.submit_records, chunk, priority="normal")
await asyncio.to_thread(intake.submit_records, batch, priority="normal")
cursor = evidence_ids[-1]
except asyncio.CancelledError:
raise
@@ -4299,6 +4405,22 @@ async def _run_backfill(app: FastAPI) -> None:
logger.exception("memory catch-up failed, no further memories will be made")
def _batches_settled_by(
batches: tuple[tuple[EvidenceRecord, ...], ...],
record: EvidenceRecord,
) -> list[tuple[EvidenceRecord, ...]]:
"""The batches one arriving record can have made writable.
Its own batch, and the last two: a user message completes the turn
before it, which can close that turn's batch and, at the 8000-character
cut, the batch before it. Submitting is idempotent, so an older batch
named again is only a hit.
"""
chosen = [batch for batch in batches if record in batch]
chosen.extend(batch for batch in batches[-2:] if batch not in chosen)
return chosen
def _sdk_memory_runtime(app: FastAPI) -> SdkMemoryRuntime:
return cast(
SdkMemoryRuntime,
@@ -4306,6 +4428,100 @@ def _sdk_memory_runtime(app: FastAPI) -> SdkMemoryRuntime:
)
async def _recall_model_ready(app: FastAPI) -> bool:
"""A generation model is configured and usable, so memories can exist."""
try:
profile = await asyncio.to_thread(_model_profile_store(app).active, _secret_store(app))
except Exception:
return False
return profile is not None and bool(profile.capabilities.model_available)
async def _library_has_memories(app: FastAPI) -> bool:
"""Any active memory at all. Asked only after an empty memory search."""
try:
client = await _sdk_client_handle(app).client()
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
statuses=["active"],
limit=1,
offset=0,
order="desc",
)
page, _ = HySdkMemoryPage.from_sdk_skipping_invalid(raw, limit=1, offset=0)
except Exception:
# Unknown is not "empty": keep the memory answer rather than guess.
return True
return page.total > 0
async def _recall_raw_sources(
app: FastAPI,
request: RecallRequest,
access: ExternalMemoryAccess | None,
started: float,
) -> frozenset[str] | None:
"""The raw sources this caller may read: the `/context` raw-lane rule.
Recall's own grant does not cover original conversations. A host reads
them only with `om_conversations` and its granted, unrevoked sources; a
trusted local caller reads every source.
"""
if access is None:
return None
from one_memory.context_access import resolve_context_authority
from one_memory.context_models import ContextSearchRequest
query = ContextSearchRequest(
query=request.query,
kinds=("raw",),
limit=request.limit,
character_budget=request.character_budget,
timeout_ms=request.timeout_ms,
)
try:
authority = await asyncio.to_thread(
resolve_context_authority,
query,
access,
_evidence_store(app).database_path,
deadline=started + request.timeout_ms / 1_000,
)
except (HTTPException, TimeoutError):
return frozenset()
if not authority.raw_allowed:
return frozenset()
return frozenset(authority.raw_sources or ())
async def _search_sdk_raw(
app: FastAPI,
query: str,
*,
limit: int,
sources: frozenset[str] | None,
) -> tuple[HySdkRawHit, ...]:
"""The imported conversations nearest the query: the SDK raw layer."""
client = await _sdk_client_handle(app).client()
raw = await asyncio.to_thread(
client.search,
query,
user_ids=["local"],
agent_ids=[],
include_sources=sorted(sources) if sources else None,
limit=limit,
min_score=0.0,
schema_min_score=0.0,
schema_limit=0,
intention_limit=0,
scene="chat",
reader=HY_MEMORY_READER,
layers=["l1_raw"],
)
return parse_sdk_raw_search(raw)
def _sdk_client_handle(app: FastAPI) -> SdkClientHandle:
handle = cast(SdkClientHandle | None, app.state.sdk_client_handle)
if handle is None:
@@ -4328,25 +4544,143 @@ def _log_skipped_sdk_memories(count: int) -> None:
_SDK_GET_CONCURRENCY = 8
async def _get_sdk_rows(client: Any, memory_ids: list[str]) -> list[object]:
async def _get_sdk_rows(client: Any, hits: list[LibraryHit]) -> list[object]:
gate = asyncio.Semaphore(_SDK_GET_CONCURRENCY)
async def one(memory_id: str) -> object:
async def one(hit: LibraryHit) -> object:
async with gate:
return await asyncio.to_thread(client.get, memory_id)
# Name the namespace the search found the row in, so an id that
# also exists elsewhere cannot make the read ambiguous.
return await asyncio.to_thread(
client.get, hit.memory_id, **_namespace_kwargs(hit.kind, hit.variant)
)
return list(await asyncio.gather(*(one(memory_id) for memory_id in memory_ids)))
return list(await asyncio.gather(*(one(hit) for hit in hits)))
async def _get_sdk_memory(
handle: SdkClientHandle,
memory_id: str,
*,
kind: str | None = None,
variant: str | None = None,
) -> HySdkMemoryRecord | None:
client = await handle.client()
raw = await asyncio.to_thread(client.get, memory_id)
raw = await asyncio.to_thread(client.get, memory_id, **_namespace_kwargs(kind, variant))
return None if raw is None else HySdkMemoryRecord.from_sdk(raw)
def _namespace_kwargs(kind: str | None, variant: str | None) -> dict[str, str]:
"""The SDK's kind/variant arguments, only the ones the caller gave."""
namespace: dict[str, str] = {}
if kind is not None:
namespace["kind"] = kind
if variant is not None:
namespace["variant"] = variant
return namespace
def _check_memory_namespace(kind: str | None, variant: str | None) -> None:
if kind == "chat" and variant is not None:
raise HTTPException(status_code=422, detail="memory_variant_requires_productivity_kind")
# Keyword search reads one SDK relevance window this large, once per window.
LIBRARY_SEARCH_WINDOW = 200
# The memory panel's page size; the API still accepts 1..500 for the CLI.
LIBRARY_PAGE_SIZE = 20
def _library_search_windows(app: FastAPI) -> LibrarySearchWindows:
windows = getattr(app.state, "library_search_windows", None)
if windows is None:
windows = LibrarySearchWindows()
app.state.library_search_windows = windows
return cast(LibrarySearchWindows, windows)
async def _library_search_page(
app: FastAPI,
client: Any,
*,
query: str,
source: str | None,
statuses: list[str],
offset: int,
limit: int,
window_id: str | None,
scene: str = "all",
) -> HySdkMemoryPage:
"""One page of a keyword search, cut from a cached result window.
SDK search has no count-only contract and, on the local store, no keyword
channel. The search asks once for a bounded relevance window across chat,
working and coding (no reranking, no score floor), keeps the rows that
contain every word, and caches those matches under a `window_id`. Every
page of that search is a slice of the same matches, read fresh from the
SDK so edits and deletes show. The whole library is never listed in Core.
"""
windows = _library_search_windows(app)
key = window_key(query, source, statuses, scene)
window = windows.get(window_id, key) if window_id else None
window_reset = window_id is not None and window is None
if window is None:
fetch = LIBRARY_SEARCH_WINDOW
search_raw = await asyncio.to_thread(
client.search,
query,
scene="all",
rerank=False,
user_ids=["local"],
agent_ids=[],
include_sources=[source] if source else None,
limit=fetch,
min_score=0.0,
working_fact_min_score=0.0,
working_experience_min_score=0.0,
schema_min_score=0.0,
schema_limit=fetch,
# The library lists stored memories; proactive intentions are a
# recall-time lane the list never shows.
intention_limit=0,
reader=HY_MEMORY_READER,
)
hits, held = library_keyword_hits(search_raw, query)
if scene != "all":
# One scene: keep that scene's matches from the same fused window.
hits = tuple(
hit
for hit in hits
if (hit.kind == "chat" if scene == "chat" else hit.variant == scene)
)
window = windows.put(key, hits, fetch if held >= fetch else None)
if window_reset:
offset = 0
rows = await _get_sdk_rows(client, list(window.hits[offset : offset + limit]))
memories: list[HySdkMemoryRecord] = []
skipped = 0
for row in rows:
if row is None:
continue
try:
record = HySdkMemoryRecord.from_sdk(row)
except HySdkReadError:
skipped += 1
continue
if record.status == "active":
memories.append(record)
_log_skipped_sdk_memories(skipped)
return HySdkMemoryPage(
memories=tuple(memories),
total=len(window.hits),
limit=limit,
offset=offset,
search_window=window.search_window,
window_id=window.window_id,
window_reset=window_reset,
)
async def _search_sdk_memories(
handle: SdkClientHandle,
query: str,
@@ -4382,10 +4716,17 @@ def _evidence_store(app: FastAPI) -> EvidenceStore:
def _source_writes_on(app: FastAPI, source_id: str) -> bool:
"""Whether this source's conversations may become memories."""
"""Whether this source's conversations may become memories.
A hand-added memory is allowed unless its grant was explicitly revoked;
a missing grant row is allowed. The SDK add path writes no grant row, so
requiring one left every SDK-authority manual add unclaimable: Pending
forever with attempts=0 while later jobs ran. Intake, claim and execute
all ask this one question.
"""
if source_id == "one-memory-manual":
receipt = _grant_store(app).receipt(user_id="local", source_id=source_id)
return receipt is not None and receipt.revoked_at is None
return receipt is None or receipt.revoked_at is None
return _source_processing_enabled(app, source_id)
@@ -187,8 +187,9 @@ def record_legacy_result(
host_session_id=None,
injection_elapsed_ms=None,
elapsed_ms=elapsed_ms,
raw_count=0,
memory_count=len(receipt.hits),
# A raw-fallback recall returned conversation excerpts, not memories.
raw_count=len(receipt.hits) if receipt.mode == "raw" else 0,
memory_count=0 if receipt.mode == "raw" else len(receipt.hits),
references=(),
)
)
+47 -2
View File
@@ -314,6 +314,10 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
details={
"runtime_build": runtime_build,
"confirmed": confirmed,
# The Runtime the upgrade replaced, as the orchestrator read it
# from that Runtime's install manifest; never guessed.
"source_version": _migration_source("ONE_MEMORY_MIGRATION_SOURCE_VERSION"),
"source_build": _migration_source("ONE_MEMORY_MIGRATION_SOURCE_BUILD"),
**authority_snapshot(database_path),
},
)
@@ -386,6 +390,7 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
stream.write(f"{step_names.get(step, step)}\n")
stream.flush()
restore: dict[str, object] = {}
try:
with _migration_lock(settings.data_dir):
result = run_startup_migration(
@@ -394,6 +399,7 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
sdk_version=HY_MEMORY_VERSION,
namespace=HY_MEMORY_NAMESPACE,
progress=progress,
restore_report=restore,
)
except Exception as error:
from one_memory.sdk_migration_verify import MemoryReadbackFailed
@@ -402,6 +408,10 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
failed: dict[str, object] = {"schema_version": 1, "state": "failed", "code": code}
if isinstance(error, MemoryReadbackFailed):
failed["readback"] = error.report.as_document()
# The migration's own code stays the result; the restore is secondary.
failed["data_restored"] = restore.get("state") == "restored"
if restore.get("state") == "failed":
failed["restore_code"] = restore.get("code")
write_migration_log(
settings.data_dir,
writer="core-sdk-migrate",
@@ -417,10 +427,13 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
"code_is_fallback": code == "sdk_migration_preflight_failed"
and not _names_code(error, code),
**({"readback": failed["readback"]} if "readback" in failed else {}),
"restore": restore.get("state", "not_attempted"),
**({"restore_code": restore["code"]} if "code" in restore else {}),
**authority_snapshot(database_path),
},
)
_write_restore(stream, as_json, failed, f"迁移未完成:{code}")
restored_text = ";数据已恢复到升级前的状态" if failed["data_restored"] else ""
_write_restore(stream, as_json, failed, f"迁移未完成:{code}{restored_text}")
return 1
document = result.as_document()
write_migration_log(
@@ -446,6 +459,14 @@ def sdk_migrate(options: Sequence[str], stream: TextIO) -> int:
return 0
_SAFE_SOURCE = re.compile(r"^[0-9A-Za-z][0-9A-Za-z.+\-]{0,63}$")
def _migration_source(name: str) -> str:
value = os.environ.get(name, "").strip()
return value if _SAFE_SOURCE.fullmatch(value) else "unknown"
@contextlib.contextmanager
def _migration_lock(data_dir: Path) -> Iterator[None]:
"""One `sdk-migrate` per home at a time.
@@ -545,8 +566,18 @@ def _sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
as_json = "--json" in options
settings = load_settings()
database_path = settings.data_dir / "one-memory.sqlite3"
if not database_path.is_file():
from one_memory.sdk_migration_restore import restore_pending
if restore_pending(settings.data_dir):
# A restore was killed between its renames; the database may be
# elsewhere. Only `sdk-migrate` may finish it.
document: dict[str, object] = {
"schema_version": 1,
"authority": "legacy",
"state": "required",
}
elif not database_path.is_file():
document = {
"schema_version": 1,
"authority": "sdk",
"state": "not_required",
@@ -619,8 +650,22 @@ def _refuse_unmigrated_home(data_dir: Path) -> None:
"""
from one_memory.sdk_data_plane_migration import MemoryAuthority, SdkDataPlaneMigrationStore
from one_memory.sdk_migration_log import authority_snapshot, write_migration_log
from one_memory.sdk_migration_restore import restore_pending
database_path = data_dir / "one-memory.sqlite3"
if restore_pending(data_dir):
# A killed restore may have moved the database aside: serving now
# would create an empty home over the user's data.
write_migration_log(
data_dir,
writer="core-startup",
event="core.refused_legacy",
outcome="refused",
code="sdk_migration_required",
details={"stage": "serve_entry", "restore_pending": True},
)
print("sdk_migration_required", file=sys.stderr, flush=True)
raise SystemExit(SDK_MIGRATION_REQUIRED_EXIT)
if not database_path.is_file():
return
try:
@@ -4,9 +4,10 @@ from __future__ import annotations
import os
import re
from collections.abc import Sequence
from dataclasses import dataclass
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord
from one_memory.memory_tasks import segment_evidence_record
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
MEMORY_TASK_TIMEOUT_MARGIN_SECONDS = 60.0
MEMORY_TASK_TIMEOUT_MIN_SECONDS = 180.0
@@ -29,10 +30,13 @@ _HOST_ENVELOPE = re.compile(
r")",
re.IGNORECASE | re.DOTALL,
)
_WHOLE_TURN_CHARACTERS = 60_000
_WINDOW_BYTE_CEILING = 2_000_000
_HISTORY_MERGE_TURNS = 8
_HISTORY_MERGE_CHARACTERS = 8_000
#: The write rule (`turn_batches`): every 5 complete turns, or fewer when
#: the next turn would take the batch past 8000 HY-read characters.
TURN_BATCH_TURNS = 5
TURN_BATCH_CHARACTERS = 8_000
#: A live conversation's last turn counts as complete, and a short batch is
#: written, after this long without a new message in that conversation.
TURN_IDLE_FLUSH_SECONDS = 600
def memory_task_timeout_seconds(model_timeout_seconds: float) -> float:
@@ -57,180 +61,83 @@ def _coding_tool_result_max_bytes() -> int:
_HY_TOOL_CHARACTERS = _coding_tool_result_max_bytes()
def _conversation_chunks(
records: list[EvidenceRecord],
*,
max_characters: int | None = None,
max_turns: int | None = None,
completed_only: bool = False,
) -> list[tuple[EvidenceRecord, ...]]:
"""Build bounded, complete user/assistant/tool windows."""
@dataclass(frozen=True)
class TurnBatches:
"""What a conversation's records become: batches to write, records to keep."""
#: Each batch is whole turns; every batch holds at least one user message.
batches: tuple[tuple[EvidenceRecord, ...], ...]
#: Not written yet: the open last turn, complete turns short of a batch,
#: or records with no user message before them. Fed back in, in order,
#: with the conversation's next records.
carry: tuple[EvidenceRecord, ...]
def conversation_turns(records: Sequence[EvidenceRecord]) -> list[tuple[EvidenceRecord, ...]]:
"""One user message and every record after it up to the next user message.
Records before the first user message are not a turn of their own: they
join the first turn (an assistant proposal and the user turn adopting it
reach HY together). With no user message at all the result is one group
that does not open with a user, which `turn_batches` never writes.
"""
turns: list[list[EvidenceRecord]] = []
for record in records:
if not _is_hy_turn_record(record):
continue
pending_proposal = bool(turns) and not _opens_turn(turns[-1][0])
if not turns or (_opens_turn(record) and not pending_proposal):
# Until the first user message, the group is leading records only.
leading = bool(turns) and not any(_opens_turn(item) for item in turns[-1])
if not turns or (_opens_turn(record) and not leading):
turns.append([record])
else:
turns[-1].append(record)
if completed_only and turns and _trailing_user_turn_is_open(turns[-1]):
turns.pop()
whole = _whole_turns() and max_characters is None
if whole:
size = _hy_read_length
max_characters = _WHOLE_TURN_CHARACTERS
max_turns = max_turns or 1_000_000
else:
size = lambda record: len(record.verbatim) # noqa: E731
max_characters = max_characters or 12_000
max_turns = max_turns or 24
chunks: list[tuple[EvidenceRecord, ...]] = []
for turn in turns:
original_question = turn[0] if _opens_turn(turn[0]) else None
expanded: list[EvidenceRecord] = []
for record in turn:
reserve = (
size(original_question)
if original_question is not None
and record is not original_question
and size(original_question) < max_characters
else 0
)
segment_limit = max(1, max_characters - reserve)
if size(record) <= segment_limit:
expanded.append(record)
continue
spans = _semantic_segment_spans(record.verbatim, segment_limit)
expanded.extend(
segment_evidence_record(record, index=index, start=start, end=end)
if len(spans) > 1
else record
for index, (start, end) in enumerate(spans)
)
question = expanded[0] if expanded and _opens_turn(expanded[0]) else None
current: list[EvidenceRecord] = []
for record in expanded:
if current and (
len(current) >= max_turns
or sum(size(item) for item in current) + size(record) > max_characters
or sum(_window_bytes(item) for item in current) + _window_bytes(record)
> _WINDOW_BYTE_CEILING
):
chunks.append(tuple(current))
current = (
[question]
if question is not None
and max_turns > 1
and size(question) + size(record) <= max_characters
else []
)
current.append(record)
chunks.append(tuple(current))
return _merge_tool_only_chunks(chunks)
return [tuple(turn) for turn in turns]
def _semantic_segment_spans(text: str, limit: int) -> list[tuple[int, int]]:
if len(text) <= limit:
return [(0, len(text))]
spans: list[tuple[int, int]] = []
start = 0
separators = ("\n\n", "\n", "。", "!", "?", ". ", "! ", "? ")
while start < len(text):
hard_end = min(len(text), start + limit)
if hard_end == len(text):
end = hard_end
else:
floor = start + max(1, limit // 2)
candidates = [text.rfind(separator, floor, hard_end) for separator in separators]
boundary = max(candidates)
if boundary < floor:
end = hard_end
else:
matched = next(
separator for separator in separators if text.startswith(separator, boundary)
)
end = boundary + len(matched)
spans.append((start, end))
start = end
return spans
def turn_batches(records: Sequence[EvidenceRecord], *, flush: bool) -> TurnBatches:
"""Group a conversation's records into write batches by the turn rule.
def _merge_tool_only_chunks(
chunks: list[tuple[EvidenceRecord, ...]],
) -> list[tuple[EvidenceRecord, ...]]:
if not any(chunk and all(record.role == "tool" for record in chunk) for chunk in chunks):
return chunks
merged: list[tuple[EvidenceRecord, ...]] = []
leading_tools: list[EvidenceRecord] = []
for chunk in chunks:
if not chunk:
continue
if all(record.role == "tool" for record in chunk):
if merged:
merged[-1] = (*merged[-1], *chunk)
else:
leading_tools.extend(chunk)
continue
if leading_tools:
chunk = (*leading_tools, *chunk)
leading_tools = []
merged.append(chunk)
return [] if leading_tools else merged
def _merge_historical_chunks(
chunks: list[tuple[EvidenceRecord, ...]],
*,
max_turns: int = _HISTORY_MERGE_TURNS,
max_characters: int = _HISTORY_MERGE_CHARACTERS,
) -> list[tuple[EvidenceRecord, ...]]:
if not chunks:
return []
merged: list[tuple[EvidenceRecord, ...]] = []
current: list[EvidenceRecord] = []
current_turns = 0
- A turn is complete once the next user message follows it; `flush`
completes the last one too (history at end of file, or 10 idle minutes).
- Every 5 complete turns are one batch. Before 5, a turn that would take
the batch past 8000 characters starts the next batch; one turn over
8000 is a batch by itself and is never split.
- Characters are what HY reads: user/assistant text in full, a tool
record up to `_HY_TOOL_CHARACTERS`.
- Fewer than 5 complete turns within 8000 are written only on `flush`.
- No batch is written without a user message; records with none before
them are carried, and dropped on `flush`.
"""
turns = conversation_turns(records)
if turns and not any(_opens_turn(record) for record in turns[0]):
# Records with no user message yet: nothing to write until one comes.
return TurnBatches(batches=(), carry=() if flush else turns[0])
complete = turns if flush else turns[:-1]
open_tail = () if flush or not turns else turns[-1]
batches: list[tuple[EvidenceRecord, ...]] = []
current: list[tuple[EvidenceRecord, ...]] = []
current_characters = 0
for chunk in chunks:
chunk_characters = sum(len(record.verbatim) for record in chunk)
if current and (
current_turns >= max_turns or current_characters + chunk_characters > max_characters
):
merged.append(tuple(current))
current = []
current_turns = 0
current_characters = 0
current.extend(chunk)
current_turns += 1
current_characters += chunk_characters
if current:
merged.append(tuple(current))
return merged
def _chunk_input_ref(chunk: tuple[EvidenceRecord, ...]) -> str:
user_records = tuple(item for item in chunk if item.role == "user")
return user_records[-1].evidence_id if user_records else chunk[-1].evidence_id
for turn in complete:
characters = sum(_hy_read_length(record) for record in turn)
if current and current_characters + characters > TURN_BATCH_CHARACTERS:
batches.append(tuple(record for item in current for record in item))
current, current_characters = [], 0
current.append(turn)
current_characters += characters
if len(current) >= TURN_BATCH_TURNS or current_characters > TURN_BATCH_CHARACTERS:
batches.append(tuple(record for item in current for record in item))
current, current_characters = [], 0
if current and flush:
batches.append(tuple(record for item in current for record in item))
current = []
carry = tuple(record for item in current for record in item) + tuple(open_tail)
return TurnBatches(batches=tuple(batches), carry=carry)
def _opens_turn(record: EvidenceRecord) -> bool:
return record.role == "user"
def _trailing_user_turn_is_open(turn: list[EvidenceRecord]) -> bool:
if len(turn) != 1:
return False
opener = turn[0]
return (
_opens_turn(opener)
and opener.content_kind is EvidenceContentKind.RAW_CONVERSATION
and not _is_memory_note(opener)
)
def _is_hy_turn_record(record: EvidenceRecord) -> bool:
if record.role == "tool":
return record.content_kind is EvidenceContentKind.RAW_CONVERSATION and bool(
@@ -254,23 +161,6 @@ def _is_hy_input_turn(record: EvidenceRecord) -> bool:
return _HOST_ENVELOPE.search(source[:2_000]) is None
def _is_memory_note(record: EvidenceRecord) -> bool:
meta = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
return (
record.source_id == "one-memory-manual"
and meta is not None
and meta.raw_meta.get("input_mode") == "direct_memory_note"
)
def _window_bytes(record: EvidenceRecord) -> int:
return len(record.verbatim.encode("utf-8"))
def _whole_turns() -> bool:
return os.environ.get("ONE_MEMORY_SPLIT_TURNS", "").strip() != "1"
def _hy_read_length(record: EvidenceRecord) -> int:
if record.role == "tool":
return min(len(record.verbatim), _HY_TOOL_CHARACTERS)
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.5"
HY_MEMORY_VERSION = "2.0.8"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -109,11 +109,30 @@ def apply_diagnostic_log_level(level: DiagnosticLogLevel) -> None:
if any(name == root or name.startswith(f"{root}.") for root in _NOISY_LIBRARY_LOGGERS):
logging.getLogger(name).setLevel(noisy_level)
# HY configures its text log once, on the first client, at the level of
# that moment. Its file handler must follow later changes too, or a live
# switch to debug never reaches hy_memory.log.
for handler in logging.getLogger("hy_memory").handlers:
if isinstance(handler, logging.FileHandler):
handler.setLevel(selected)
os.environ["MEMORY_LOG_LEVEL"] = logging.getLevelName(selected)
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "true" if level == "debug" else "false"
_apply_hy_pipeline_policy()
os.environ["MEMORY_LOG_LLM_CONTENT"] = "true" if level == "debug" else "false"
def _apply_hy_pipeline_policy() -> None:
"""Keep HY's per-step pipeline log and trace off at every level.
Nothing in MemHub reads them: Core's own diagnostics carry the step
timing, and the JSONL/SQLite copies only grew on disk. With both off
the SDK skips the per-step bookkeeping altogether.
"""
os.environ["MEMORY_PIPELINE_LOG_ENABLED"] = "false"
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "false"
def hy_memory_version() -> str:
"""Return the installed SDK version, failing if the pinned dist is absent."""
@@ -131,7 +150,9 @@ def configure_hy_runtime_environment(data_dir: Path) -> None:
os.environ["MEMORY_LOG_DIR"] = str(log_dir)
level = diagnostic_log_level(data_dir)
os.environ["MEMORY_LOG_LEVEL"] = level.upper()
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "true" if level == "debug" else "false"
_apply_hy_pipeline_policy()
# Still meaningful with the pipeline log off: it decides whether HY's
# DEBUG lines in hy_memory.log carry full model prompts and responses.
os.environ["MEMORY_LOG_LLM_CONTENT"] = "true" if level == "debug" else "false"
os.environ["MEMORY_MEMORY_OPERATIONS_ENABLED"] = "false"
os.environ["ANONYMIZED_TELEMETRY"] = "False"
@@ -82,7 +82,10 @@ def repair_sdk_metadata_from_bundle(
database_path: Path,
now: datetime | None = None,
) -> MetadataRepairResult:
"""Restore metadata lost by historical cutovers from their verified bundles."""
"""Restore metadata lost by historical cutovers from their verified bundles.
Offline only: no SDK client may be open on this home (see `_repair_vector`).
"""
migration_id = _sdk_migration_id(database_path)
if migration_id is None:
return MetadataRepairResult(None, 0, 0, 0)
@@ -326,15 +329,14 @@ def _repair_vector(
directory = data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / "vector"
if not (directory / "chroma.sqlite3").is_file():
return 0
import chromadb
from chromadb.config import Settings
from one_memory.sdk_vector_reattach import open_offline_chroma
client = chromadb.PersistentClient(
path=str(directory),
settings=Settings(anonymized_telemetry=False),
)
# The SDK client is not open yet (startup has not created its handle), so
# this is the only writer; release the store before the SDK takes it.
client = open_offline_chroma(directory)
# This must be the only Chroma client on the directory: Chroma shares one
# System per path, and this client's close() drops it from under any
# other (KeyError on its next lookup). `sdk-migrate` runs it after the
# cutover with Runtime stopped; Core's startup fallback runs it while
# `hold_stores_offline` keeps every SDK client waiting.
try:
repaired = 0
ids = tuple(mapping)
@@ -424,6 +426,7 @@ def _repair_coding(
path = root / name
if not path.is_file():
continue
_ensure_coding_schema(path)
with closing(sqlite3.connect(path)) as database:
tables = {
str(row[0])
@@ -435,10 +438,14 @@ def _repair_coding(
):
if table not in tables:
continue
# _ensure_coding_schema added it; an SDK without the column
# (before 2.0.7) leaves the time to the vector repair alone.
has_memory_at = "memory_at" in _columns(database, table)
rows = database.execute(
f"SELECT {identifier},source,created_at FROM {table}"
f"SELECT {identifier},source,created_at,"
f"{'memory_at' if has_memory_at else 'NULL'} FROM {table}"
).fetchall()
for memory_id, source, created_at in rows:
for memory_id, source, created_at, memory_at in rows:
desired = mapping.get(str(memory_id))
if desired is None:
continue
@@ -447,6 +454,13 @@ def _repair_coding(
if desired.source and source in {None, "", "default"}:
updates.append("source=?")
values.append(desired.source)
if (
has_memory_at
and desired.memory_at is not None
and memory_at != desired.memory_at
):
updates.append("memory_at=?")
values.append(desired.memory_at)
if desired.created_at is not None:
desired_created_at = (
datetime.fromtimestamp(desired.created_at, tz=UTC)
@@ -468,6 +482,32 @@ def _repair_coding(
return repaired
def _ensure_coding_schema(path: Path) -> None:
"""Bring a Coding/Working store to the installed SDK's schema.
A migrated home's stores were written by an older SDK and may lack columns
the repair writes (``memory_at`` since hy-memory 2.0.7). The repair runs
before any SDK client opens them, so the SDK's own schema step is run here
rather than repeating its DDL. ``CodingMemoryStore`` has no public
schema-only entry point: ``initialize()`` also opens its Chroma
collections, which must stay closed during the offline repair. So only its
SQLite step runs, on a store object that carries nothing but that path.
"""
from hy_memory.coding.store import ( # type: ignore[import-untyped]
CodingMemoryStore,
)
# Relies on the pinned SDK's internals (hy-memory==2.0.8 is pinned exactly).
store = CodingMemoryStore.__new__(CodingMemoryStore)
store._db_path = str(path)
store._conn = None
try:
store._init_sqlite_sync()
finally:
if store._conn is not None:
store._conn.close()
def _batches[T](values: list[T] | tuple[T, ...]) -> Iterator[tuple[T, ...]]:
for start in range(0, len(values), _BATCH):
yield tuple(values[start : start + _BATCH])
+110 -14
View File
@@ -17,6 +17,9 @@ from pathlib import Path
from typing import Any, ClassVar
from hy_memory import ( # type: ignore[import-untyped]
AmbiguousMemoryIdError as AmbiguousMemoryIdError, # re-export for api.py
)
from hy_memory import (
HyMemoryClient,
MemoryConfig,
SharedRuntime,
@@ -33,7 +36,12 @@ from one_memory.hy_graph_admin import (
)
from one_memory.hy_sdk_types import HySdkRuntimeError
from one_memory.local_embedding import LocalEmbedder
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind
from one_memory.log_file import confine_hy_sdk_logging
from one_memory.model_profiles import (
ModelProfileSnapshot,
ProviderKind,
disable_thinking_params,
)
_RUNTIME_LOCK = threading.RLock()
BUILTIN_RETRIEVAL_PROFILE_ID = "builtin-retrieval"
@@ -49,6 +57,26 @@ class _RuntimeEntry:
_RUNTIMES: dict[Path, _RuntimeEntry] = {}
# Homes whose SDK-owned local stores are held offline for maintenance, and
# the client constructions in flight per home. Chroma keeps one process-wide
# System per path with no lock around create, lookup and release, so a
# second client that opens or closes beside the SDK's pulls the System out
# from under it (KeyError on the next lookup). Maintenance therefore runs
# only while no SDK client exists for that home, and new clients wait.
_STORE_GATE = threading.Condition()
_OFFLINE_STORES: set[Path] = set()
_OPENING: dict[Path, int] = {}
# One client constructs at a time per home, its first-time store setup
# included. The constructor opens Chroma's `PersistentClient`, and two first
# opens on one directory race over that same unlocked System cache. On a
# clean install the write drain's client and a UI read client started in the
# same second; neither reached `VectorStore ready`, and the first write job
# was lost as unknown. Taken inside `acquire_client`, after the offline gate
# and never while `_STORE_GATE` is held, so it cannot deadlock with
# `hold_stores_offline`, which refuses while `_OPENING` counts a client.
_CONSTRUCTING: dict[Path, threading.Lock] = {}
class LocalOnnxEmbeddingService:
"""Public ``EmbeddingService`` implementation backed by MemHub's ONNX model."""
@@ -241,14 +269,22 @@ def build_sdk_config(
embedding_provider, embedding_endpoint, embedding_model, dimensions
)
llm: dict[str, Any] = {
"provider": "openai",
"model": profile.model,
"api_key": generation_key,
"base_url": generation_endpoint,
}
# The SDK sends `extra_body` on every chat call, chat and coding/working
# alike. Without it DeepSeek spent the whole agent_max_tokens on thinking
# and answered finish_reason=length with no content.
thinking_off = disable_thinking_params(profile)
if thinking_off:
llm["extra_body"] = thinking_off
return MemoryConfig.from_dict(
{
"llm": {
"provider": "openai",
"model": profile.model,
"api_key": generation_key,
"base_url": generation_endpoint,
},
"llm": llm,
"embedder": {
"provider": "openai",
"model": (
@@ -292,7 +328,60 @@ def acquire_client(
data_dir: Path,
http_transport_factory: Callable[[], Any] | None = None,
) -> ClientLease:
"""Construct a public SDK client and return its explicit runtime lease."""
"""Construct a public SDK client and return its explicit runtime lease.
Waits while `hold_stores_offline` holds this home's stores.
"""
key = data_dir.resolve()
with _STORE_GATE:
while key in _OFFLINE_STORES:
_STORE_GATE.wait()
_OPENING[key] = _OPENING.get(key, 0) + 1
try:
return _acquire_client(
profile,
api_key=api_key,
data_dir=data_dir,
http_transport_factory=http_transport_factory,
)
finally:
with _STORE_GATE:
_OPENING[key] -= 1
if not _OPENING[key]:
del _OPENING[key]
_STORE_GATE.notify_all()
def hold_stores_offline(data_dir: Path) -> bool:
"""Hold this home's SDK stores for offline maintenance; False if a client is open.
While held, `acquire_client` for the home waits. Every SDK client in Core
is built there, so nothing reads or writes Chroma beside the maintenance.
`release_stores` ends the hold; it is idempotent.
"""
key = data_dir.resolve()
with _STORE_GATE:
with _RUNTIME_LOCK:
open_runtime = key in _RUNTIMES
if key in _OFFLINE_STORES or _OPENING.get(key) or open_runtime:
return False
_OFFLINE_STORES.add(key)
return True
def release_stores(data_dir: Path) -> None:
with _STORE_GATE:
_OFFLINE_STORES.discard(data_dir.resolve())
_STORE_GATE.notify_all()
def _acquire_client(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None,
) -> ClientLease:
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir)
config.http_transport_factory = http_transport_factory
mismatch = inspect_graph_width_mismatch(config)
@@ -303,16 +392,23 @@ def acquire_client(
embedding_service: LocalOnnxEmbeddingService | None = None
if (profile.embedding_provider_kind or profile.provider_kind) is ProviderKind.LOCAL_ONNX:
embedding_service = LocalOnnxEmbeddingService()
with _STORE_GATE:
constructing = _CONSTRUCTING.setdefault(key, threading.Lock())
try:
client = HyMemoryClient(
config=config,
mode="ultra" if graph_store_enabled() else "pro",
runtime=runtime,
embedding_service=embedding_service,
)
with constructing:
client = HyMemoryClient(
config=config,
mode="ultra" if graph_store_enabled() else "pro",
runtime=runtime,
embedding_service=embedding_service,
)
except BaseException:
_release_runtime(key)
raise
finally:
# The constructor's first step configures HY's text log (once per
# process), so by here it exists whether or not construction succeeded.
confine_hy_sdk_logging()
return ClientLease(client, key)
+296 -21
View File
@@ -3,6 +3,8 @@
from __future__ import annotations
from collections.abc import Mapping
from datetime import UTC, datetime
from typing import Literal, NamedTuple, cast
from pydantic import BaseModel, ConfigDict, Field
@@ -29,7 +31,21 @@ class HySdkReadError(HySdkError):
"""A public SDK read operation failed or returned an invalid shape."""
MemoryScene = Literal["chat", "working", "coding"]
MemoryKindName = Literal["chat", "experience", "fact"]
MemoryVariantName = Literal["working", "coding"]
class HySdkMemoryRecord(BaseModel):
"""One memory of any scene, in the unified shape the library lists.
Chat memories keep every field they always had. Working/coding memories
(experiences and facts) carry ``scene``/``kind``/``variant`` so a client
can open, edit and delete them again without guessing the namespace; an
experience also keeps ``task`` and ``solution`` apart, and ``content`` is
the readable ``task\nsolution`` text the SDK lists.
"""
model_config = ConfigDict(frozen=True, extra="forbid")
memory_id: str
@@ -48,17 +64,36 @@ class HySdkMemoryRecord(BaseModel):
source: str | None = None
memory_at: int | None = None
gmt_created: int | None = None
scene: MemoryScene = "chat"
kind: MemoryKindName = "chat"
# The productivity namespace ("working" | "coding"); None for chat. Pass it
# back with `kind` on get/update/delete.
variant: MemoryVariantName | None = None
task: str | None = None
solution: str | None = None
# Epoch seconds. `updated_at` falls back to `created_at`.
created_at: int | None = None
updated_at: int | None = None
@classmethod
def from_sdk(cls, raw: object) -> HySdkMemoryRecord:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_record_shape_unknown")
if raw.get("kind") in {"experience", "fact"}:
return cls._from_productivity(raw)
if any(
not isinstance(raw.get(key), str) or not str(raw[key]).strip()
for key in ("memory_id", "content")
):
raise HySdkReadError("hy_memory_record_shape_unknown")
try:
gmt_created = (
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
)
created_at = _epoch(raw.get("created_at"))
if created_at is None:
created_at = gmt_created
updated_at = _epoch(raw.get("updated_at"))
return cls(
memory_id=str(raw["memory_id"]),
content=str(raw["content"]),
@@ -77,9 +112,73 @@ class HySdkMemoryRecord(BaseModel):
),
source=str(raw["source"]) if isinstance(raw.get("source"), str) else None,
memory_at=raw.get("memory_at") if isinstance(raw.get("memory_at"), int) else None,
gmt_created=(
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
gmt_created=gmt_created,
created_at=created_at,
updated_at=updated_at if updated_at is not None else created_at,
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_record_shape_unknown") from error
@classmethod
def _from_productivity(cls, raw: Mapping[str, object]) -> HySdkMemoryRecord:
"""An experience or fact from list/search(scene="all") or get().
get() names the namespace in ``variant`` (its ``scene`` is the legacy
"productivity"); list/search name it in ``scene``.
"""
kind = str(raw["kind"])
variant = next(
(
value
for value in (raw.get("variant"), raw.get("scene"))
if value in {"working", "coding"}
),
None,
)
id_key = "experience_id" if kind == "experience" else "fact_id"
memory_id = raw.get("memory_id") or raw.get(id_key)
if variant is None or not isinstance(memory_id, str) or not memory_id.strip():
raise HySdkReadError("hy_memory_record_shape_unknown")
task: str | None = None
solution: str | None = None
if kind == "experience":
task = str(raw.get("task") or "").strip()
solution = str(raw.get("solution") or "").strip()
content = "\n".join(part for part in (task, solution) if part)
else:
statement = raw.get("statement")
content = str(statement if isinstance(statement, str) else raw.get("content") or "")
if not content.strip():
raise HySdkReadError("hy_memory_record_shape_unknown")
status = str(raw.get("status") or "active")
memory_at = raw.get("memory_at")
created_at = _epoch(raw.get("created_at"))
updated_at = _epoch(raw.get("updated_at"))
try:
return cls(
memory_id=memory_id,
content=content,
layer=kind,
status=status,
is_latest=status == "active",
tags=_string_tuple(raw.get("tags")),
user_id=str(raw.get("user_id") or ""),
agent_id=str(raw.get("agent_id") or ""),
session_id=(
str(raw["session_id"]) if isinstance(raw.get("session_id"), str) else None
),
source=str(raw["source"]) if isinstance(raw.get("source"), str) else None,
# Conversation time; hy-memory 2.0.7 stores it for experiences
# and facts. Older stores leave it unknown.
memory_at=memory_at if isinstance(memory_at, int) else None,
gmt_created=created_at,
scene=cast(MemoryScene, variant),
kind=cast(MemoryKindName, kind),
variant=cast(MemoryVariantName, variant),
task=task,
solution=solution,
created_at=created_at,
updated_at=updated_at if updated_at is not None else created_at,
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_record_shape_unknown") from error
@@ -95,6 +194,15 @@ class HySdkMemoryPage(BaseModel):
# Set on a search whose relevance window was full: matches were looked for
# only among this many most relevant memories, so `total` may undercount.
search_window: int | None = None
# Keyword search only: the server-side result window this page was cut
# from. Pass it back with the next offset to page the same results.
window_id: str | None = None
# True when the window_id the client sent had expired or did not match the
# query: the search ran again and this is page 1 of the new window.
window_reset: bool = False
# Empty-query listing only: how many memories each scene holds under the
# same filters (`total` is their sum).
totals: dict[str, int] | None = None
@classmethod
def from_sdk(cls, raw: object, *, limit: int, offset: int) -> HySdkMemoryPage:
@@ -123,6 +231,12 @@ class HySdkMemoryPage(BaseModel):
) -> tuple[HySdkMemoryPage, int]:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_list_shape_unknown")
if raw.get("scene") == "all":
return cls._parse_all_scenes(raw, limit=limit, offset=offset, skip_invalid=skip_invalid)
if raw.get("scene") == "productivity":
return cls._parse_experiences(
raw, limit=limit, offset=offset, skip_invalid=skip_invalid
)
rows: list[object] = []
total = 0
for bucket_name, rows_name in (("vdb", "memories"), ("graph", "nodes")):
@@ -163,6 +277,107 @@ class HySdkMemoryPage(BaseModel):
)
return page, skipped
@classmethod
def _parse_all_scenes(
cls,
raw: Mapping[str, object],
*,
limit: int,
offset: int,
skip_invalid: bool,
) -> tuple[HySdkMemoryPage, int]:
"""``list_memories(scene="all")``: chat, working and coding, newest first."""
rows = raw.get("memories")
total = raw.get("total")
if (
not isinstance(rows, list)
or not isinstance(total, int)
or isinstance(total, bool)
or total < 0
):
raise HySdkReadError("hy_memory_list_shape_unknown")
totals_raw = raw.get("totals")
totals = (
{
str(scene): count
for scene, count in totals_raw.items()
if isinstance(count, int) and not isinstance(count, bool) and count >= 0
}
if isinstance(totals_raw, Mapping)
else None
)
memories: list[HySdkMemoryRecord] = []
seen: set[tuple[str, str, str | None]] = set()
skipped = 0
for row in rows:
try:
record = HySdkMemoryRecord.from_sdk(row)
except HySdkReadError:
if not skip_invalid:
raise
skipped += 1
continue
key = (record.memory_id, record.kind, record.variant)
if key in seen:
continue
seen.add(key)
memories.append(record)
return (
cls(
memories=tuple(memories),
total=total,
limit=limit,
offset=offset,
totals=totals,
),
skipped,
)
@classmethod
def _parse_experiences(
cls,
raw: Mapping[str, object],
*,
limit: int,
offset: int,
skip_invalid: bool,
) -> tuple[HySdkMemoryPage, int]:
"""``list_memories(scene="working"|"coding")``: that scene's experiences.
The SDK names the scene once, in ``variant``; each row is a bare
experience, so the kind and variant are added before parsing.
"""
rows = raw.get("experiences")
total = raw.get("total")
variant = raw.get("variant")
if (
not isinstance(rows, list)
or not isinstance(total, int)
or isinstance(total, bool)
or total < 0
or variant not in {"working", "coding"}
):
raise HySdkReadError("hy_memory_list_shape_unknown")
memories: list[HySdkMemoryRecord] = []
skipped = 0
for row in rows:
try:
if not isinstance(row, Mapping):
raise HySdkReadError("hy_memory_record_shape_unknown")
record = HySdkMemoryRecord.from_sdk(
{**row, "kind": "experience", "variant": variant}
)
except HySdkReadError:
if not skip_invalid:
raise
skipped += 1
continue
memories.append(record)
return (
cls(memories=tuple(memories), total=total, limit=limit, offset=offset),
skipped,
)
class HySdkRawRecord(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
@@ -327,38 +542,79 @@ def parse_sdk_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit
return tuple(found)
def library_keyword_hits(raw: object, query: str) -> tuple[tuple[str, ...], int]:
"""Ids in an SDK search window whose text contains every query term.
class LibraryHit(NamedTuple):
"""One keyword match in a library search window, with its namespace."""
memory_id: str
kind: MemoryKindName
variant: MemoryVariantName | None
def library_keyword_hits(raw: object, query: str) -> tuple[tuple[LibraryHit, ...], int]:
"""Memories in an SDK search window whose text contains every query term.
The local SDK store has no keyword channel (Chroma's keyword search is
empty), so search is pure vector similarity: a word in the library can
score below the floor, and a word in none of it still scores above. The
library search asks the SDK for its most relevant window and keeps only the
rows that actually contain the words, most relevant first. Returns those ids
and how many distinct rows the window held.
rows that actually contain the words, most relevant first. Returns those
hits and how many distinct rows the window held.
A ``search(scene="all")`` window mixes chat memories with working/coding
experiences and facts; their text is the unified ``content`` (an
experience's task and solution, a fact's statement). The fused order is
kept as the SDK ranked it.
"""
rows = _reply_rows(raw, "hy_memory_search_shape_unknown")
terms = query.casefold().split()
seen: set[str] = set()
matched: list[str] = []
for item in sorted(rows, key=_score, reverse=True):
memory_id = item.get("memory_id") or item.get("id")
if not isinstance(memory_id, str) or not memory_id or memory_id in seen:
if isinstance(raw, Mapping) and raw.get("scene") == "all":
rows = raw.get("memories")
if not isinstance(rows, list) or any(not isinstance(row, Mapping) for row in rows):
raise HySdkReadError("hy_memory_search_shape_unknown")
ordered: list[Mapping[str, object]] = list(rows)
else:
ordered = sorted(
_reply_rows(raw, "hy_memory_search_shape_unknown"), key=_score, reverse=True
)
seen: set[LibraryHit] = set()
matched: list[LibraryHit] = []
for item in ordered:
hit = _library_hit(item)
if hit is None or hit in seen:
continue
seen.add(memory_id)
content = item.get("content")
seen.add(hit)
tags = item.get("tags")
text = " ".join(
[
content if isinstance(content, str) else "",
*(tag for tag in (tags if isinstance(tags, list) else ()) if isinstance(tag, str)),
]
).casefold()
parts = [item.get(key) for key in ("content", "task", "solution", "statement")] + list(
tags if isinstance(tags, list) else ()
)
text = " ".join(part for part in parts if isinstance(part, str)).casefold()
if all(term in text for term in terms):
matched.append(memory_id)
matched.append(hit)
return tuple(matched), len(seen)
def _library_hit(item: Mapping[str, object]) -> LibraryHit | None:
kind = item.get("kind")
if kind in {"experience", "fact"}:
variant = item.get("variant") if item.get("variant") in {"working", "coding"} else None
variant = variant or (
item.get("scene") if item.get("scene") in {"working", "coding"} else None
)
memory_id = item.get("memory_id") or item.get(
"experience_id" if kind == "experience" else "fact_id"
)
if variant is None or not isinstance(memory_id, str) or not memory_id:
return None
return LibraryHit(
memory_id,
"experience" if kind == "experience" else "fact",
"coding" if variant == "coding" else "working",
)
memory_id = item.get("memory_id") or item.get("id")
if not isinstance(memory_id, str) or not memory_id:
return None
return LibraryHit(memory_id, "chat", None)
def parse_sdk_raw_search(raw: object) -> tuple[HySdkRawHit, ...]:
return tuple(
HySdkRawHit.from_sdk_search(row)
@@ -366,6 +622,25 @@ def parse_sdk_raw_search(raw: object) -> tuple[HySdkRawHit, ...]:
)
def _epoch(value: object) -> int | None:
"""Epoch seconds from an int or an ISO-8601 string (naive means UTC)."""
if isinstance(value, bool):
return None
if isinstance(value, int):
return value
if isinstance(value, float):
return int(value)
if isinstance(value, str) and value.strip():
try:
parsed = datetime.fromisoformat(value.strip())
except ValueError:
return None
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=UTC)
return int(parsed.timestamp())
return None
def _string_tuple(value: object) -> tuple[str, ...]:
if not isinstance(value, list):
return ()
@@ -662,11 +662,15 @@ class ImportRunner:
and self.sdk_queue is not None
and event.cursor.read_to_end
):
# History ends at this end of file, so its last turn
# and a short batch are written. Live polling keeps
# them for the next increment or the idle flush.
self.sdk_queue.finalize_sdk_file(
user_id=current.user_id,
source_id=current.source_id,
file_key=event.cursor.file_key,
connection=connection,
flush=not self._live_import,
)
continue
if self.on_stable_evidence is not None and event.cursor.read_to_end:
@@ -0,0 +1,113 @@
"""Short-lived result windows for the memory library's keyword search.
A library search asks the SDK once for a bounded relevance window and keeps
the keyword matches. Paging must cut every page from that same list: running
the search again per page can rank differently and duplicate or skip memories
between pages. The matches are kept here under an opaque ``window_id`` that
the client passes back with the next offset.
A window is bound to the filters it was searched with (query, source,
statuses, scene). It expires after ``ttl_seconds`` without being read, and the
store keeps at most ``max_windows`` windows, dropping the least recently read
first.
Only memory identities are kept, never content: each page is read fresh from
the SDK, so an edit or a delete shows on the next page read.
"""
from __future__ import annotations
import secrets
import threading
import time
from collections import OrderedDict
from collections.abc import Callable
from dataclasses import dataclass, field
from one_memory.hy_sdk_types import LibraryHit
DEFAULT_WINDOW_TTL_SECONDS = 300.0
DEFAULT_MAX_WINDOWS = 32
WindowKey = tuple[str, str | None, tuple[str, ...], str]
def window_key(
query: str, source: str | None, statuses: list[str], scene: str = "all"
) -> WindowKey:
return (query, source, tuple(sorted(set(statuses))), scene)
@dataclass(frozen=True)
class SearchWindow:
window_id: str
key: WindowKey
hits: tuple[LibraryHit, ...]
# Set when the SDK window was full, so matches outside it were not looked
# for and `total` may undercount.
search_window: int | None
@dataclass
class _Entry:
window: SearchWindow
last_read: float = field(default=0.0)
class LibrarySearchWindows:
def __init__(
self,
*,
ttl_seconds: float = DEFAULT_WINDOW_TTL_SECONDS,
max_windows: int = DEFAULT_MAX_WINDOWS,
clock: Callable[[], float] = time.monotonic,
) -> None:
self._ttl = ttl_seconds
self._max = max(1, max_windows)
self._clock = clock
self._entries: OrderedDict[str, _Entry] = OrderedDict()
self._lock = threading.Lock()
def get(self, window_id: str, key: WindowKey) -> SearchWindow | None:
"""The window, if it is still live and was searched with `key`."""
with self._lock:
now = self._clock()
self._expire(now)
entry = self._entries.get(window_id)
if entry is None or entry.window.key != key:
return None
entry.last_read = now
self._entries.move_to_end(window_id)
return entry.window
def put(
self,
key: WindowKey,
hits: tuple[LibraryHit, ...],
search_window: int | None,
) -> SearchWindow:
window = SearchWindow(
window_id=secrets.token_urlsafe(12),
key=key,
hits=hits,
search_window=search_window,
)
with self._lock:
now = self._clock()
self._expire(now)
self._entries[window.window_id] = _Entry(window=window, last_read=now)
while len(self._entries) > self._max:
self._entries.popitem(last=False)
return window
def __len__(self) -> int:
with self._lock:
return len(self._entries)
def _expire(self, now: float) -> None:
stale = [
window_id
for window_id, entry in self._entries.items()
if now - entry.last_read >= self._ttl
]
for window_id in stale:
del self._entries[window_id]
+17 -15
View File
@@ -105,10 +105,17 @@ def capture_stdio(source_id: str, stdin: TextIO, output: TextIO) -> int:
def sdk_migration_capture_fenced(data_dir: Path) -> bool:
"""Ignore stale fences; only a live Desktop migration pauses hook writes."""
"""Ignore stale fences; a live migration or a failed one pauses hook writes.
A failed upgrade migration leaves the fence `frozen` (no pid): the home was
put back as it was before the upgrade and takes no writes until the next
migration attempt, which replaces the fence and lifts it on success.
"""
marker = data_dir / SDK_MIGRATION_CAPTURE_FENCE
try:
document = json.loads(marker.read_text(encoding="utf-8"))
if isinstance(document, dict) and document.get("frozen") is True:
return True
pid = int(document["pid"])
if pid < 1:
raise ValueError("invalid migration fence pid")
@@ -249,24 +256,19 @@ def record_capture_rejection(
def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed: bool) -> None:
"""Add the record to its conversation's live assembling row.
The turn rule decides what is written (`turn_batches`): a turn is
complete when the next user message arrives, batches are 5 turns or
8000 characters, and Core writes the rest after 10 quiet minutes.
"""
if not changed or not _sync_open(store.database_path, record.source_id):
return
from one_memory.conversation_processing import _conversation_chunks
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
intake = UnifiedWriteIntake(DurableWriteQueue(store.database_path, evidence=store))
for chunk in _conversation_chunks(
store.list_conversation(record.evidence_id),
max_turns=12,
completed_only=True,
):
if (
record in chunk
and any(item.role == "user" for item in chunk)
and any(item.role == "assistant" for item in chunk)
):
intake.submit_records(chunk, priority="live")
DurableWriteQueue(store.database_path, evidence=store).assemble_live_records(
(record,), file_key=f"hook:{record.conversation_id}"
)
def _conversation(payload: dict[str, object]) -> str | None:
+55 -2
View File
@@ -20,8 +20,11 @@ def start_logging_to_file(data_dir: Path) -> Path:
nowhere. That includes the loop watchdog: the one report that names what
stalled the app was unreadable in exactly the situation it exists for.
The active diagnostic collection level controls Core, HY SDK, and provider
client records. Uvicorn keeps its request log on its own non-propagating
The active diagnostic collection level controls Core and provider client
records. The HY SDK keeps its own text log, hy_memory.log under the SDK's
log directory, and stops propagating here once its first client is built;
`confine_hy_sdk_logging` then forwards only its warnings and errors to
this file. Uvicorn keeps its request log on its own non-propagating
loggers.
"""
destination = log_path(data_dir)
@@ -49,3 +52,53 @@ def start_logging_to_file(data_dir: Path) -> Path:
if root.level > logging.INFO or root.level == logging.NOTSET:
root.setLevel(logging.INFO)
return destination
HY_SDK_LOGGER = "hy_memory"
class _CoreLogForwarder(logging.Handler):
"""Hands HY warnings to Core's own core.log handler, rotation and format."""
def __init__(self, target: logging.Handler) -> None:
super().__init__(level=logging.WARNING)
self.target = target
def emit(self, record: logging.LogRecord) -> None:
self.target.handle(record)
def confine_hy_sdk_logging() -> None:
"""Keep HY's text log in hy_memory.log, with only its problems in core.log.
HY's first client sets up hy_memory.log plus a stdout and a stderr stream
on the "hy_memory" logger, and stops it propagating to the root. Under
launchd those streams are service.log and service-error.log, so every SDK
line was written twice, and none reached core.log. Drop the streams and
forward WARNING and above to core.log, which the Debug panel reads; the
SDK keeps conversation text out of those levels.
Idempotent. Does nothing until HY has configured its logger, because a
propagating logger already reaches core.log through the root.
"""
sdk = logging.getLogger(HY_SDK_LOGGER)
if sdk.propagate:
return
for handler in list(sdk.handlers):
if isinstance(handler, logging.StreamHandler) and not isinstance(
handler, logging.FileHandler
):
sdk.removeHandler(handler)
if any(isinstance(handler, _CoreLogForwarder) for handler in sdk.handlers):
return
core = next(
(
handler
for handler in logging.getLogger().handlers
if isinstance(handler, RotatingFileHandler)
and Path(handler.baseFilename).name == "core.log"
),
None,
)
if core is not None:
sdk.addHandler(_CoreLogForwarder(core))
@@ -1,5 +1,13 @@
"""Add one memory directly: `POST /api/v2/memories` and its status read.
The public API has no "notes": a hand-added memory is filed as manual Evidence
and turned into memories by the durable write queue. Storage keeps its
pre-rename identifiers (`memory_note_requests`, source `one-memory-manual`,
`input_mode=direct_memory_note`, conversation `note:<request_id>`) because they
are persisted and feed evidence ids; renaming them would change stored data.
"""
import asyncio
import json
import sqlite3
from collections.abc import Sequence
from datetime import UTC, datetime
@@ -28,7 +36,7 @@ from one_memory.memory_tasks import MemoryTaskStatus, MemoryTaskStore
from one_memory.secrets import SecretStoreUnavailableError
SOURCE_ID = "one-memory-manual"
MemoryNoteState = Literal[
MemoryAddState = Literal[
"waiting_for_model",
"waiting_for_permission",
"pending",
@@ -39,7 +47,7 @@ MemoryNoteState = Literal[
]
class MemoryNoteInput(BaseModel):
class MemoryAddInput(BaseModel):
model_config = ConfigDict(extra="forbid")
request_id: UUID
@@ -50,25 +58,25 @@ class MemoryNoteInput(BaseModel):
@classmethod
def meaningful_text(cls, value: str) -> str:
if not value.strip():
raise ValueError("memory_note_empty")
raise ValueError("memory_add_empty")
return value
class MemoryNoteReceipt(BaseModel):
class MemoryAddReceipt(BaseModel):
model_config = ConfigDict(extra="forbid")
evidence_id: str
text: str
scope: Literal["all-apps"] = "all-apps"
created_at: datetime
state: MemoryNoteState
state: MemoryAddState
task_id: str | None = None
memory_ids: list[str] = Field(default_factory=list)
error: str | None = None
class LegacyMemoryNoteReader:
"""Read pre-cutover note receipts without retaining any write runtime."""
class LegacyMemoryAdditionReader:
"""Read pre-cutover addition receipts without retaining any write runtime."""
def __init__(
self,
@@ -79,7 +87,7 @@ class LegacyMemoryNoteReader:
self.governance = governance
class MemoryNoteStore:
class MemoryAdditionStore:
def __init__(
self,
evidence: EvidenceStore,
@@ -92,44 +100,52 @@ class MemoryNoteStore:
self.evidence = evidence
self.processor = processor
# D52: the production composition root passes a UnifiedWriteIntake so a
# manual note reaches the durable queue like every other entry. It stays
# manual addition reaches the durable queue like every other entry. It stays
# optional so isolated tests can keep constructing the store directly.
self.intake = intake
self.sdk_authority = sdk_authority
self.write_queue = write_queue
def save(self, note: MemoryNoteInput) -> EvidenceRecord:
def save(self, addition: MemoryAddInput) -> EvidenceRecord:
now = datetime.now(UTC)
value = EvidenceInput(
source_id=SOURCE_ID,
conversation_id=f"note:{note.request_id}",
message_id=str(note.request_id),
conversation_id=f"note:{addition.request_id}",
message_id=str(addition.request_id),
role="user",
written_by=WrittenBy.PERSON,
created_at=now,
verbatim=note.text,
verbatim=addition.text,
origin_meta={
EvidenceOrigin.MANUAL_IMPORT: OriginMetadata(
raw_meta={
"input_mode": "direct_memory_note",
"recall_scope": note.scope,
"recall_scope": addition.scope,
}
)
},
)
if self.sdk_authority:
if self.intake is None or self.write_queue is None:
raise RuntimeError("sdk_memory_note_intake_unavailable")
raise RuntimeError("sdk_memory_add_intake_unavailable")
record = materialize_write_input(value, observed_at=now)
with main_database(self.write_queue.database_path).read() as database:
# No grant row is written here: a missing row means allowed,
# only an explicit revocation refuses (same as the claim gate).
grant = database.execute(
"SELECT revoked_at FROM source_grants WHERE user_id='local' AND source_id=?",
(SOURCE_ID,),
).fetchone()
existing = database.execute(
"SELECT request_id FROM memory_note_requests WHERE note_id=?",
(record.evidence_id,),
).fetchone()
if grant is not None and grant[0] is not None:
raise ValueError("memory_add_source_revoked")
if existing is not None:
current = self.require(record.evidence_id)
if current.verbatim != record.verbatim:
raise ValueError("memory_note_request_conflict")
raise ValueError("memory_add_conflict")
return current
receipt = self.intake.submit_records(
(record,),
@@ -159,13 +175,13 @@ class MemoryNoteStore:
(SOURCE_ID,),
).fetchone()
if grant and grant[0] is not None:
raise ValueError("memory_note_source_revoked")
raise ValueError("memory_add_source_revoked")
existing = db.execute(
"SELECT verbatim FROM evidence WHERE evidence_id=?", (value.evidence_id,)
).fetchone()
if existing:
if existing[0] != note.text:
raise ValueError("memory_note_request_conflict")
if existing[0] != addition.text:
raise ValueError("memory_add_conflict")
else:
self.evidence.put(value, connection=db)
db.execute(
@@ -218,10 +234,10 @@ class MemoryNoteStore:
(SOURCE_ID,),
).fetchone()
if grant is None or grant[0] is not None:
raise ValueError("memory_note_source_revoked")
raise ValueError("memory_add_source_revoked")
return record
def receipt(self, evidence_id: str) -> MemoryNoteReceipt:
def receipt(self, evidence_id: str) -> MemoryAddReceipt:
record = self.require(evidence_id)
if self.sdk_authority:
assert self.write_queue is not None
@@ -232,7 +248,7 @@ class MemoryNoteStore:
).fetchone()[0]
durable = self.write_queue.get(str(request_id))
durable_state = durable.state if durable is not None else "failed"
mapped: MemoryNoteState = {
mapped: MemoryAddState = {
"pending": "pending",
"running": "running",
"waiting_for_model": "waiting_for_model",
@@ -241,7 +257,7 @@ class MemoryNoteStore:
"ready" if durable is not None and durable.sdk_memory_ids else "no_memories"
),
}.get(durable_state, "failed")
return MemoryNoteReceipt(
return MemoryAddReceipt(
evidence_id=evidence_id,
text=sanitize_model_input(record.verbatim),
created_at=record.created_at or record.imported_at,
@@ -250,7 +266,7 @@ class MemoryNoteStore:
error=durable.detail if durable is not None else "write_missing",
)
if self.processor is None:
raise RuntimeError("legacy_memory_note_processor_unavailable")
raise RuntimeError("legacy_memory_add_processor_unavailable")
task = self.processor.tasks.task_for_evidence(evidence_id)
memory_ids = []
if task:
@@ -258,7 +274,7 @@ class MemoryNoteStore:
memory = self.processor.governance.get(output.memory_id, user_id="local")
if memory and memory.status is MemoryStatus.CONFIRMED:
memory_ids.append(memory.memory_id)
state: MemoryNoteState
state: MemoryAddState
if task is None or task.status == MemoryTaskStatus.WAITING_FOR_MODEL:
state = "waiting_for_model"
elif task.status == MemoryTaskStatus.WAITING_FOR_PERMISSION:
@@ -274,7 +290,7 @@ class MemoryNoteStore:
else:
state = "failed"
error = task.last_error_code if task else None
return MemoryNoteReceipt(
return MemoryAddReceipt(
evidence_id=record.evidence_id,
text=sanitize_model_input(record.verbatim),
created_at=record.created_at or record.imported_at,
@@ -284,33 +300,7 @@ class MemoryNoteStore:
error=error,
)
def list(self) -> list[MemoryNoteReceipt]:
if self.sdk_authority:
assert self.write_queue is not None
with main_database(self.write_queue.database_path).read() as database:
ids = [
str(row[0])
for row in database.execute(
"SELECT note_id FROM memory_note_requests ORDER BY created_at DESC LIMIT 50"
)
]
return [self.receipt(key) for key in ids]
with main_database(self.evidence.database_path).read() as db:
ids = [
row[0]
for row in db.execute(
"SELECT e.evidence_id,o.raw_meta_json FROM evidence e "
"JOIN source_grants g USING(user_id,source_id) "
"JOIN evidence_origins o USING(evidence_id) WHERE e.user_id='local' "
"AND e.source_id=? AND g.revoked_at IS NULL AND o.origin='manual_import' "
"ORDER BY e.imported_at DESC,e.evidence_id LIMIT 50",
(SOURCE_ID,),
)
if json.loads(row[1]).get("input_mode") == "direct_memory_note"
]
return [self.receipt(key) for key in ids]
def receipt_for_request(self, request_id: UUID) -> MemoryNoteReceipt:
def receipt_for_request(self, request_id: UUID) -> MemoryAddReceipt:
identity = str(request_id)
if self.sdk_authority:
probe = EvidenceInput(
@@ -338,107 +328,65 @@ class MemoryNoteStore:
return self.receipt(str(row[0]))
def memory_note_router(dependencies: Sequence[Depends]) -> APIRouter:
router = APIRouter(
prefix="/api/v1/memory-notes", tags=["memory-notes"], dependencies=dependencies
)
def memory_add_router(dependencies: Sequence[Depends]) -> APIRouter:
"""`POST /api/v2/memories` and `GET /api/v2/memories/additions/{request_id}`.
def store(request: Request) -> MemoryNoteStore:
return cast(MemoryNoteStore, request.app.state.memory_note_store)
Both paths are outside the single-segment `/api/v2/memories/{memory_id}`
routes; external (enveloped) callers are refused by
`require_external_endpoint` because neither is a broker read route.
"""
router = APIRouter(prefix="/api/v2/memories", tags=["memories"], dependencies=dependencies)
async def schedule(
notes: MemoryNoteStore,
record: EvidenceRecord,
*,
readmit: bool = False,
) -> MemoryNoteReceipt:
def store(request: Request) -> MemoryAdditionStore:
return cast(MemoryAdditionStore, request.app.state.memory_addition_store)
async def schedule(additions: MemoryAdditionStore, record: EvidenceRecord) -> None:
try:
if notes.sdk_authority:
return await asyncio.to_thread(notes.receipt, record.evidence_id)
if notes.processor is None:
raise RuntimeError("legacy_memory_note_processor_unavailable")
if notes.intake is None:
if additions.sdk_authority:
return
if additions.processor is None:
raise RuntimeError("legacy_memory_add_processor_unavailable")
if additions.intake is None:
# Isolated tests construct the store without a queue.
await notes.processor.enqueue_conversation_off_thread(record)
await additions.processor.enqueue_conversation_off_thread(record)
else:
# Production: submit to the durable queue first. The drain runs
# the same supervisor path, so create/process semantics are
# unchanged; replaying the same request returns the original
# `entry:<evidence_id>` job instead of a second HY call.
if readmit:
# `/{evidence_id}/process` is the explicit retry signal.
# It also revives pre-upgrade permission waiters.
await asyncio.to_thread(notes.intake.queue.readmit_parked)
submitted = await asyncio.to_thread(
notes.intake.submit_evidence,
# Production: submit to the durable queue. Replaying the same
# request returns the original `entry:<evidence_id>` job
# instead of a second HY call.
await asyncio.to_thread(
additions.intake.submit_evidence,
record,
first_round=1,
last_round=1,
priority="live",
)
if readmit:
await asyncio.to_thread(
notes.intake.readmit_cancelled_evidence,
record,
job_id=submitted.record.job_id,
first_round=1,
last_round=1,
priority="live",
)
except SecretStoreUnavailableError:
saved = await asyncio.to_thread(notes.receipt, record.evidence_id)
return saved.model_copy(
update={"state": "waiting_for_model", "error": "secure_storage_unavailable"}
)
return await asyncio.to_thread(notes.receipt, record.evidence_id)
# The receipt already reports the saved text; the queue retries
# once secure storage is back.
return
@router.post("", response_model=MemoryNoteReceipt, status_code=201)
async def create(
note: MemoryNoteInput,
@router.post("", response_model=MemoryAddReceipt, status_code=201)
async def add_memory(
addition: MemoryAddInput,
request: Request,
background: BackgroundTasks,
) -> MemoryNoteReceipt:
notes = store(request)
) -> MemoryAddReceipt:
additions = store(request)
try:
record = await asyncio.to_thread(notes.save, note)
record = await asyncio.to_thread(additions.save, addition)
except ValueError as error:
raise HTTPException(409, str(error)) from error
saved = await asyncio.to_thread(notes.receipt, record.evidence_id)
background.add_task(schedule, notes, record)
saved = await asyncio.to_thread(additions.receipt, record.evidence_id)
background.add_task(schedule, additions, record)
return saved
@router.get("", response_model=list[MemoryNoteReceipt])
def listing(request: Request) -> list[MemoryNoteReceipt]:
return store(request).list()
@router.get("/operations/{request_id}", response_model=MemoryNoteReceipt)
def operation(request_id: UUID, request: Request) -> MemoryNoteReceipt:
@router.get("/additions/{request_id}", response_model=MemoryAddReceipt)
def memory_add_status(request_id: UUID, request: Request) -> MemoryAddReceipt:
try:
return store(request).receipt_for_request(request_id)
except KeyError as error:
raise HTTPException(404, "memory_note_operation_not_found") from error
@router.get("/{evidence_id}", response_model=MemoryNoteReceipt)
def get(evidence_id: str, request: Request) -> MemoryNoteReceipt:
try:
return store(request).receipt(evidence_id)
except KeyError as error:
raise HTTPException(404, "memory_note_not_found") from error
raise HTTPException(404, "memory_add_not_found") from error
except ValueError as error:
raise HTTPException(409, str(error)) from error
@router.post("/{evidence_id}/process", response_model=MemoryNoteReceipt)
async def process(evidence_id: str, request: Request) -> MemoryNoteReceipt:
notes = store(request)
try:
record = await asyncio.to_thread(notes.require, evidence_id)
except KeyError as error:
raise HTTPException(404, "memory_note_not_found") from error
except ValueError as error:
raise HTTPException(409, str(error)) from error
# This route is the explicit "the condition that parked it may have
# changed" signal (the person just granted consent or fixed the model),
# which is when `readmit_parked` is meant to run.
return await schedule(notes, record, readmit=True)
return router
@@ -24,10 +24,11 @@ from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
from typing import Any, Literal
from uuid import uuid4
from one_memory.background_work import in_the_background
from one_memory.conversation_processing import TURN_IDLE_FLUSH_SECONDS, turn_batches
from one_memory.database_access import main_database
from one_memory.diagnostic_log import diagnostic_event
from one_memory.imports.models import EvidenceOrigin, EvidenceRecord
@@ -51,6 +52,7 @@ from one_memory.memory_tasks import (
from one_memory.memory_tasks import (
POLICY_STOPPED_CODES as _TASK_POLICY_STOPPED_CODES,
)
from one_memory.sdk_migration_log import error_chain_text
from one_memory.service_levels import ServiceLevel, service_level_context
logger = logging.getLogger(__name__)
@@ -240,14 +242,16 @@ PERMISSION_REQUIRED_CODES = frozenset(
)
#: Model-side codes the SDK write coordinator raises before it can lease a
#: client (no active profile, missing credential, secure storage down). They prove
#: nothing was sent, so the write parks instead of being lost. Transport codes
#: that can arrive after a request was in flight stay `unknown`.
#: client (no active profile, missing credential, secure storage down, or the
#: client failed to construct). They prove nothing was sent, so the write
#: parks instead of being lost. Transport codes that can arrive after a
#: request was in flight stay `unknown`.
MODEL_NOT_READY_CODES = frozenset(
{
"model_profile_not_active",
"model_secret_unavailable",
"secure_storage_unavailable",
"hy_memory_client_unavailable",
}
)
@@ -291,6 +295,23 @@ _VALID_JOB_STATES = frozenset(
#: A row in one of these states can never be moved back to a live one.
_TERMINAL_WRITE_STATES = frozenset({COMPLETED, FAILED, DEAD, CANCELLED})
#: The legacy task rows a write job can be linked to. The SDK cutover drops
#: them (`target_schema.LEGACY_TABLES_IN_DROP_ORDER`, inputs before tasks), so
#: the job-only paths must not assume any of them.
_LEGACY_TASK_LINK_TABLES = ("memory_tasks", "memory_task_inputs", "memory_write_job_evidence")
def _legacy_task_links_present(connection: sqlite3.Connection) -> bool:
present = {
str(row[0])
for row in connection.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name IN (?,?,?)",
_LEGACY_TASK_LINK_TABLES,
)
}
return present == set(_LEGACY_TASK_LINK_TABLES)
#: Columns a current schema declares but an older `memory_write_jobs` may lack.
#: `CREATE TABLE IF NOT EXISTS` is a no-op for an existing table, so without
#: this an upgraded install would keep an older queue that every claim/lease
@@ -316,6 +337,12 @@ LEGACY_MEMORY_WRITE_JOB_COLUMNS = (
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)
EVENT_TIME_WINDOW = timedelta(minutes=10)
#: Sources whose jobs are a person adding a memory by hand ("添加记忆"), not
#: history. History is digested strictly old to new through the event-time
#: window; a note is stamped "now", so inside that window it waited behind the
#: whole unprocessed history. These jobs are claimed ahead of the window,
#: still one at a time per source and in order within their session.
DIRECT_WRITE_SOURCES = frozenset({"one-memory-manual"})
@dataclass(frozen=True)
@@ -888,16 +915,30 @@ class DurableWriteQueue:
priority: str,
connection: sqlite3.Connection,
) -> None:
"""Append fresh-install input to an assembling queue row.
"""Append input to its conversation's assembling queue row.
The row is not claimable until the scanner commits EOF and finalizes
it. No Evidence table is involved.
The row is not claimable. `finalize_sdk_file` turns what is complete
into write jobs (`turn_batches`) and keeps the rest here for the next
increment; `flush_idle_assemblies` writes a live row's rest once the
conversation has been quiet. `updated_at` moves only when a record is
new or revised, so a re-read does not keep a quiet conversation open.
No Evidence table is involved.
"""
assembly_key = (
f"assembly:{record.user_id}:{record.source_id}:"
f"{hashlib.sha256(file_key.encode()).hexdigest()}:"
f"{hashlib.sha256(record.conversation_id.encode()).hexdigest()}"
)
session_key = internal_session_id_for(record.conversation_id, record.source_id)
written = connection.execute(
"SELECT 1 FROM memory_write_jobs AS job, json_each(job.evidence_ids_json) AS ref "
"WHERE job.session_key=? AND job.state!='assembling' AND ref.value=? LIMIT 1",
(session_key, record.evidence_id),
).fetchone()
if written is not None:
# A re-read of a record a job already holds: staging it again
# would regroup it with later turns into a second job.
return
row = connection.execute(
"SELECT job_id,payload_json,priority FROM memory_write_jobs "
"WHERE idempotency_key=? AND state='assembling'",
@@ -908,8 +949,20 @@ class DurableWriteQueue:
payload = json.loads(str(row["payload_json"]))
records = list(payload.get("input_records") or [])
dumped = record.model_dump(mode="json")
if not any(item.get("evidence_id") == record.evidence_id for item in records):
index = next(
(
position
for position, item in enumerate(records)
if item.get("evidence_id") == record.evidence_id
),
None,
)
changed = index is None or records[index] != dumped
if index is None:
records.append(dumped)
elif changed:
# A revision replaces the carried copy in place.
records[index] = dumped
now = datetime.now(UTC).isoformat()
payload_json = json.dumps(
{
@@ -920,15 +973,13 @@ class DurableWriteQueue:
sort_keys=True,
)
if row is not None:
merged_priority = "live" if "live" in (row["priority"], priority) else "normal"
if not changed and merged_priority == row["priority"]:
return
connection.execute(
"UPDATE memory_write_jobs SET payload_json=?,priority=?,updated_at=? "
"WHERE job_id=?",
(
payload_json,
"live" if row["priority"] == "live" or priority == "live" else "normal",
now,
str(row["job_id"]),
),
"UPDATE memory_write_jobs SET payload_json=?,priority=?,"
"updated_at=CASE WHEN ? THEN ? ELSE updated_at END WHERE job_id=?",
(payload_json, merged_priority, int(changed), now, str(row["job_id"])),
)
return
queue_seq = int(
@@ -946,7 +997,7 @@ class DurableWriteQueue:
(
str(uuid4()),
queue_seq,
internal_session_id_for(record.conversation_id, record.source_id),
session_key,
record.source_id,
assembly_key,
assembly_key,
@@ -976,92 +1027,215 @@ class DurableWriteQueue:
source_id: str,
file_key: str,
connection: sqlite3.Connection,
flush: bool = True,
) -> tuple[str, ...]:
"""Write what one file's assembling rows hold that the turn rule completes.
`flush` (history at end of file) completes the last turn and writes a
short batch; without it (live polling) those stay in the row.
"""
rows = connection.execute(
"SELECT * FROM memory_write_jobs WHERE state='assembling' "
"AND source_key=? ORDER BY queue_seq",
(source_id,),
).fetchall()
finalized = []
finalized: list[str] = []
for row in rows:
payload = json.loads(str(row["payload_json"]))
if payload.get("file_key") != file_key:
continue
records = tuple(
EvidenceRecord.model_validate(item)
for item in payload.get("input_records") or []
if item.get("user_id") == user_id
finalized.extend(
self._write_assembled_turns(connection, row, user_id=user_id, flush=flush)
)
if not records:
connection.execute(
"DELETE FROM memory_write_jobs WHERE job_id=?",
(str(row["job_id"]),),
return tuple(finalized)
def assemble_live_records(
self,
records: tuple[EvidenceRecord, ...],
*,
file_key: str,
) -> tuple[str, ...]:
"""Stage live records and write what is complete, in one transaction.
For live capture outside the import runner (host hooks): the same
assembling rows, turn rule and idle flush as live file polling.
"""
if not records:
return ()
first = records[0]
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
for record in records:
self.stage_sdk_input(
record, file_key=file_key, priority="live", connection=connection
)
continue
first = records[0]
session_id = internal_session_id_for(
first.conversation_id,
first.source_id,
)
material = ":".join(f"{item.evidence_id}:{item.content_hash}" for item in records)
request_id = f"turn:v2:{hashlib.sha256(material.encode()).hexdigest()}"
existing = connection.execute(
"SELECT job_id FROM memory_write_jobs WHERE idempotency_key=? AND job_id!=?",
(request_id, str(row["job_id"])),
).fetchone()
if existing is not None:
connection.execute(
"DELETE FROM memory_write_jobs WHERE job_id=?",
(str(row["job_id"]),),
)
finalized.append(str(existing["job_id"]))
continue
request = WriteRequest(
internal_session_id=session_id,
return self.finalize_sdk_file(
user_id=first.user_id,
source_id=first.source_id,
batch_identity=BatchIdentity(
internal_session_id=session_id,
first_round=1,
last_round=1,
),
evidence_ids=[item.evidence_id for item in records],
input_records=records,
idempotency_key=request_id,
priority="live" if str(row["priority"]) == "live" else "normal",
event_at=max(
(item.created_at for item in records if item.created_at),
default=first.imported_at,
),
time_quality="exact" if any(item.created_at for item in records) else "approximate",
file_key=file_key,
connection=connection,
flush=False,
)
def flush_idle_assemblies(self, *, now: datetime | None = None) -> tuple[str, ...]:
"""Write the rest of every live conversation quiet for 10 minutes.
Its last turn counts as complete and a batch short of 5 turns is
written. History rows are left alone: their end of file flushes them.
"""
current = now or datetime.now(UTC)
cutoff = (current - timedelta(seconds=TURN_IDLE_FLUSH_SECONDS)).isoformat()
query = (
"SELECT * FROM memory_write_jobs WHERE state='assembling' AND priority='live' "
"AND updated_at<=? ORDER BY queue_seq"
)
# A read first: most ticks find nothing and must not take the writer.
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
if connection.execute(query, (cutoff,)).fetchone() is None:
return ()
finalized: list[str] = []
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
for row in connection.execute(query, (cutoff,)).fetchall():
finalized.extend(self._write_assembled_turns(connection, row, flush=True))
return tuple(finalized)
def _write_assembled_turns(
self,
connection: sqlite3.Connection,
row: sqlite3.Row,
*,
flush: bool,
user_id: str | None = None,
) -> list[str]:
"""Turn one assembling row's complete batches into jobs; keep the carry."""
payload = json.loads(str(row["payload_json"]))
records = tuple(
EvidenceRecord.model_validate(item)
for item in payload.get("input_records") or []
if user_id is None or item.get("user_id") == user_id
)
grouped = turn_batches(records, flush=flush)
priority: Literal["live", "normal"] = "live" if str(row["priority"]) == "live" else "normal"
job_ids = [
self._insert_turn_batch(connection, batch, priority=priority)
for batch in grouped.batches
]
if not grouped.carry:
connection.execute(
"UPDATE memory_write_jobs SET session_key=?,batch_id=?,"
"idempotency_key=?,payload_json=?,evidence_ids_json=?,state='pending',"
"event_at=?,time_quality=?,updated_at=? WHERE job_id=?",
"DELETE FROM memory_write_jobs WHERE job_id=?", (str(row["job_id"]),)
)
elif grouped.batches or len(grouped.carry) != len(records):
# `updated_at` stays: the idle clock counts new messages only.
connection.execute(
"UPDATE memory_write_jobs SET payload_json=? WHERE job_id=?",
(
session_id,
request.batch_identity.batch_id,
request_id,
json.dumps(
request.model_dump(
mode="json",
exclude={"event_at", "time_quality"},
),
{
**payload,
"input_records": [
record.model_dump(mode="json") for record in grouped.carry
],
},
sort_keys=True,
),
json.dumps(list(request.evidence_ids)),
(
request.event_at.isoformat()
if request.event_at
else datetime.now(UTC).isoformat()
),
request.time_quality,
datetime.now(UTC).isoformat(),
str(row["job_id"]),
),
)
finalized.append(str(row["job_id"]))
return tuple(finalized)
return job_ids
def _insert_turn_batch(
self,
connection: sqlite3.Connection,
records: tuple[EvidenceRecord, ...],
*,
priority: Literal["live", "normal"],
) -> str:
"""One pending job for one batch of whole turns, keyed by its content.
Same records, same key: a re-read returns the job already written.
"""
first = records[0]
session_id = internal_session_id_for(first.conversation_id, first.source_id)
material = ":".join(f"{item.evidence_id}:{item.content_hash}" for item in records)
request_id = f"turn:v2:{hashlib.sha256(material.encode()).hexdigest()}"
existing = connection.execute(
"SELECT job_id FROM memory_write_jobs WHERE idempotency_key=?",
(request_id,),
).fetchone()
if existing is not None:
return str(existing["job_id"])
request = WriteRequest(
internal_session_id=session_id,
source_id=first.source_id,
batch_identity=BatchIdentity(
internal_session_id=session_id,
first_round=1,
last_round=1,
),
evidence_ids=[item.evidence_id for item in records],
input_records=records,
idempotency_key=request_id,
priority=priority,
)
# The batch's time is its first user message's, as for every other job.
event_at, time_quality = self._event_time_for_request(request)
now = datetime.now(UTC)
job_id = str(uuid4())
queue_seq = int(
connection.execute(
"SELECT COALESCE(MAX(queue_seq),0)+1 FROM memory_write_jobs"
).fetchone()[0]
)
event = event_at or now
if event.tzinfo is None:
event = event.replace(tzinfo=UTC)
connection.execute(
"INSERT INTO memory_write_jobs ("
"job_id,queue_seq,session_key,source_key,batch_id,idempotency_key,"
"payload_json,evidence_ids_json,state,priority,attempts,max_attempts,"
"lease_owner,lease_expires_at,hy_invoked,last_error,event_at,time_quality,"
"created_at,updated_at,sdk_raw_ids_json,sdk_memory_ids_json"
") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
(
job_id,
queue_seq,
session_id,
first.source_id,
request.batch_identity.batch_id,
request_id,
json.dumps(
request.model_dump(mode="json", exclude={"event_at", "time_quality"}),
sort_keys=True,
),
json.dumps(list(request.evidence_ids)),
PENDING,
priority,
0,
3,
None,
None,
0,
None,
event.astimezone(UTC).isoformat(),
time_quality,
now.isoformat(),
now.isoformat(),
"[]",
"[]",
),
)
if not self.sdk_authority:
# Legacy catch-up reads coverage from these references.
connection.executemany(
"INSERT INTO memory_write_job_evidence (job_id, evidence_id) VALUES (?,?)",
((job_id, evidence_id) for evidence_id in request.evidence_ids),
)
return job_id
def _freeze_request_inputs(self, request: WriteRequest) -> WriteRequest:
if request.input_records or self.evidence is None:
@@ -1828,21 +2002,24 @@ class DurableWriteQueue:
or str(row["last_error"]) in NON_RETRYABLE_POST_WRITE_CODES
):
return 0
connection.execute(
"""
UPDATE memory_tasks
SET status='pending', runner_token=NULL, attempts=0,
last_error_code='model_content_manual_retry_requested', updated_at=?
WHERE status='unknown' AND task_id IN (
SELECT inputs.task_id
FROM memory_task_inputs AS inputs, json_each(inputs.refs_json) AS ref
JOIN memory_write_job_evidence AS linked
ON linked.evidence_id=json_extract(ref.value, '$.evidence_id')
WHERE linked.job_id=?
# After the SDK cutover the legacy task tables are gone and the
# job is the only row to reset.
if not self.sdk_authority and _legacy_task_links_present(connection):
connection.execute(
"""
UPDATE memory_tasks
SET status='pending', runner_token=NULL, attempts=0,
last_error_code='model_content_manual_retry_requested', updated_at=?
WHERE status='unknown' AND task_id IN (
SELECT inputs.task_id
FROM memory_task_inputs AS inputs, json_each(inputs.refs_json) AS ref
JOIN memory_write_job_evidence AS linked
ON linked.evidence_id=json_extract(ref.value, '$.evidence_id')
WHERE linked.job_id=?
)
""",
(now, job_id),
)
""",
(now, job_id),
)
changed = connection.execute(
"""
UPDATE memory_write_jobs
@@ -1944,9 +2121,7 @@ class DurableWriteQueue:
f"WHERE state=? AND job_id IN ({placeholders})",
(PENDING, now, UNKNOWN, *job_ids),
)
if connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_tasks'"
).fetchone():
if _legacy_task_links_present(connection):
connection.execute(
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
f"last_error_code='model_content_reconciled_absent_retry', updated_at=? "
@@ -2004,9 +2179,7 @@ class DurableWriteQueue:
f"WHERE state=? AND attempts=1 AND job_id IN ({placeholders})",
(PENDING, now, UNKNOWN, *job_ids),
)
if connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_tasks'"
).fetchone():
if _legacy_task_links_present(connection):
connection.execute(
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
f"last_error_code='model_content_unknown_auto_retry', updated_at=? "
@@ -2023,6 +2196,14 @@ class DurableWriteQueue:
def resolve_unknown_from_linked_output(self, job_id: str) -> bool:
"""Close an unknown durable row when local governance output proves success."""
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
if not (
_legacy_task_links_present(connection)
and connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_task_outputs'"
).fetchone()
):
# No legacy governance output survives the SDK cutover.
return False
row = connection.execute(
"""
SELECT 1
@@ -2405,9 +2586,22 @@ async def execute_job(
if code == "model_profile_not_active":
logger.debug("durable_write_job_parked job_id=%s error_code=%s", job_id, code)
elif state in PARKED_STATES:
logger.info("durable_write_job_parked job_id=%s error_code=%s", job_id, code)
logger.info(
"durable_write_job_parked job_id=%s error_code=%s chain=%s",
job_id,
code,
error_chain_text(error),
)
else:
logger.error("durable_write_job_failed job_id=%s error_code=%s", job_id, code)
# Where it failed, as `file:line:function` frames of the exception
# chain, never its message: `error_code=AttributeError` alone
# named no location.
logger.error(
"durable_write_job_failed job_id=%s error_code=%s chain=%s",
job_id,
code,
error_chain_text(error),
)
return
await in_the_background(
queue.settle,
@@ -2494,6 +2688,9 @@ class DurableWriteDrain:
#: a model outage clears on its own, and a queue that never goes idle
#: must still re-admit the write once the wait has elapsed.
self._last_model_readmit: datetime | None = None
#: A live conversation quiet for 10 minutes has its last turn and short
#: batch written (`flush_idle_assemblies`); checked on the same timer.
self._last_assembly_flush: datetime | None = None
#: Idle polls do not need to hit the single writer every 50ms. A
#: 1m Evidence table makes a miss expensive enough that the tail
#: call-event write can be behind it; back off only while idle and
@@ -2601,6 +2798,39 @@ class DurableWriteDrain:
placeholders = ",".join("?" for _ in enabled_sources)
source_clause = f"AND candidate.source_key IN ({placeholders}) "
source_params = tuple(sorted(enabled_sources))
direct_sources = ",".join("?" for _ in DIRECT_WRITE_SOURCES)
# Same eligibility as the window claim below (free lease, no
# running job of its source, nothing earlier in its session), but
# not bound to the window: a note must not wait for history.
direct_rows = connection.execute(
"SELECT candidate.job_id,candidate.source_key "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ f"AND candidate.source_key IN ({direct_sources}) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.queue_seq",
(
PENDING,
now.isoformat(),
*source_params,
*sorted(DIRECT_WRITE_SOURCES),
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchall()
window = connection.execute(
"SELECT window_anchor,window_end FROM memory_write_schedule_state WHERE singleton=1"
).fetchone()
@@ -2654,6 +2884,7 @@ class DurableWriteDrain:
if active is None:
connection.execute("DELETE FROM memory_write_schedule_state WHERE singleton=1")
window = None
anchor: Any = None
if window is None:
anchor = connection.execute(
"SELECT candidate.event_at "
@@ -2682,8 +2913,10 @@ class DurableWriteDrain:
*_FIFO_BLOCKING_STATES,
),
).fetchone()
if anchor is None:
if anchor is None and not direct_rows:
return ()
has_window = window is not None or anchor is not None
if window is None and anchor is not None:
anchor_at = datetime.fromisoformat(str(anchor[0]))
if anchor_at.tzinfo is None:
anchor_at = anchor_at.replace(tzinfo=UTC)
@@ -2695,39 +2928,41 @@ class DurableWriteDrain:
"VALUES (1,?,?,?)",
(anchor_text, end_text, now.isoformat()),
)
rows = connection.execute(
"SELECT candidate.job_id,candidate.source_key "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ "AND candidate.event_at>=? AND candidate.event_at<? "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.event_at,candidate.queue_seq",
(
PENDING,
now.isoformat(),
*source_params,
anchor_text,
end_text,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchall()
rows: list[Any] = []
if has_window:
rows = connection.execute(
"SELECT candidate.job_id,candidate.source_key "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ "AND candidate.event_at>=? AND candidate.event_at<? "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.event_at,candidate.queue_seq",
(
PENDING,
now.isoformat(),
*source_params,
anchor_text,
end_text,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchall()
selected: list[str] = []
selected_sources: set[str] = set()
for row in rows:
for row in (*direct_rows, *rows):
source_key = str(row["source_key"])
if source_key in selected_sources:
continue
@@ -2988,6 +3223,15 @@ class DurableWriteDrain:
self.queue.readmit_model_waiters,
operation_name="readmit_model_waiters",
)
if (
self._last_assembly_flush is None
or tick_started - self._last_assembly_flush >= self._reap_interval
):
self._last_assembly_flush = tick_started
await self._run_idle_maintenance(
self.queue.flush_idle_assemblies,
operation_name="flush_idle_assemblies",
)
if self.reconcile_unknown is not None and (
self._last_unknown_reconcile is None
or tick_started - self._last_unknown_reconcile
@@ -98,23 +98,6 @@ class UnifiedWriteIntake:
enqueued=durable.enqueued,
)
def readmit_cancelled_evidence(
self,
record: EvidenceRecord,
*,
job_id: str,
first_round: int,
last_round: int,
priority: WritePriority = "normal",
) -> int:
request = self._evidence_request(
record,
first_round=first_round,
last_round=last_round,
priority=priority,
)
return self.queue.readmit_cancelled_permission_confirmed(job_id, request)
@staticmethod
def _evidence_request(
record: EvidenceRecord,
@@ -1136,20 +1136,31 @@ class ModelProfileStore:
)
def disable_thinking_params(
profile: ModelProfile | ModelProfileSnapshot,
) -> dict[str, object]:
"""Provider-specific thinking-off params, empty unless the profile opts in.
The one mapping for every request Core builds itself and for the SDK's
`llm.extra_body`, so a model that thinks on one path thinks on none.
"""
if not profile.disable_thinking:
return {}
if profile.provider_kind is ProviderKind.OPENAI_COMPATIBLE:
if urlparse(profile.base_url or "").hostname == "api.deepseek.com":
return {"thinking": {"type": "disabled"}}
return {"reasoning_effort": "none"}
if profile.provider_kind is ProviderKind.OLLAMA:
return {"think": False}
return {}
def attach_disable_thinking(
profile: ModelProfile | ModelProfileSnapshot,
payload: dict[str, object],
) -> dict[str, object]:
"""Attach provider-specific thinking-off params when the profile opts in."""
if not profile.disable_thinking:
return payload
if profile.provider_kind is ProviderKind.OPENAI_COMPATIBLE:
if urlparse(profile.base_url or "").hostname == "api.deepseek.com":
payload["thinking"] = {"type": "disabled"}
else:
payload["reasoning_effort"] = "none"
elif profile.provider_kind is ProviderKind.OLLAMA:
payload["think"] = False
payload.update(disable_thinking_params(profile))
return payload
+10 -1
View File
@@ -81,7 +81,11 @@ class RecallHit(BaseModel):
score: float = Field(ge=0, le=1)
#: HY already rendered memory time/evolution/capsule context into content.
#: Hosts must not append their own governance timestamp in this case.
content_format: Literal["plain", "hy-sdk"] = "plain"
#: "raw" is an excerpt of an imported conversation, not a memory.
content_format: Literal["plain", "hy-sdk", "raw"] = "plain"
#: Raw excerpts only: the app and conversation the excerpt came from.
source: str | None = None
session_id: str | None = None
class RecallReceipt(BaseModel):
@@ -103,6 +107,11 @@ class RecallReceipt(BaseModel):
#: whether the answer is that nothing was found or that reading was not
#: allowed.
excluded_sources: tuple[str, ...] = ()
#: "raw" when recall searched the imported conversations instead of
#: memories, because no generation model is configured or no memory
#: exists yet (`fallback_reason`). Raw hits are excerpts, not memories.
mode: Literal["memory", "raw"] = "memory"
fallback_reason: Literal["model_unavailable", "no_memories"] | None = None
class RecallTarget(Protocol):
@@ -0,0 +1,149 @@
"""Recall that degrades to the original conversations instead of answering empty.
MemHub_recall searches the memories the SDK distilled from conversations. That
needs a configured generation model to have produced any, and a library that
holds some. Without either, the memory search can only come back empty, so
recall searches the SDK's raw layer (the imported conversations themselves)
instead: plain retrieval over what was imported, the same lane
`/api/v2/context/search` reads, under the same raw-source authority.
"""
from __future__ import annotations
import asyncio
import time
from collections.abc import Awaitable, Callable
from datetime import UTC, datetime
from typing import Literal
from one_memory.hy_sdk_types import HySdkRawHit
from one_memory.recall import (
PROBE_QUERY,
RecallHit,
RecallReceipt,
RecallRequest,
RecallState,
TruncationBudget,
)
FallbackReason = Literal["model_unavailable", "no_memories"]
#: Raw sources this caller may read, or None for every source (a trusted local
#: caller). An empty set means the caller may read no raw conversation.
RawSources = frozenset[str] | None
async def recall_with_fallback(
request: RecallRequest,
*,
model_ready: Callable[[], Awaitable[bool]],
recall_memories: Callable[[RecallRequest], Awaitable[RecallReceipt]],
has_memories: Callable[[], Awaitable[bool]],
raw_sources: Callable[[], Awaitable[RawSources]],
search_raw: Callable[[str, int, RawSources], Awaitable[tuple[HySdkRawHit, ...]]],
) -> RecallReceipt:
"""Memory recall when a model and memories exist, otherwise the raw layer.
The library check runs only after an empty memory search: with memories
present and nothing relevant, the answer stays an honest empty memory
result rather than a different kind of hit.
"""
if request.query == PROBE_QUERY:
return await recall_memories(request)
started = time.monotonic()
if await model_ready():
receipt = await recall_memories(request)
if receipt.hits or await has_memories():
return receipt
reason: FallbackReason = "no_memories"
else:
reason = "model_unavailable"
remaining = request.timeout_ms / 1_000 - (time.monotonic() - started)
return await _raw_recall(
request,
reason=reason,
remaining_seconds=remaining,
raw_sources=raw_sources,
search_raw=search_raw,
)
async def _raw_recall(
request: RecallRequest,
*,
reason: FallbackReason,
remaining_seconds: float,
raw_sources: Callable[[], Awaitable[RawSources]],
search_raw: Callable[[str, int, RawSources], Awaitable[tuple[HySdkRawHit, ...]]],
) -> RecallReceipt:
def receipt(
hits: tuple[RecallHit, ...] = (),
*,
used: int = 0,
truncated_by: TruncationBudget | None = None,
unexamined: int = 0,
) -> RecallReceipt:
return RecallReceipt(
state=RecallState.GROUNDED if hits else RecallState.EMPTY,
query=request.query,
scope=request.scope,
hits=hits,
used_characters=used,
truncated=truncated_by is not None,
truncated_by=truncated_by,
unexamined_count=unexamined,
mode="raw",
fallback_reason=reason,
)
sources = await raw_sources()
if sources is not None and not sources:
# The caller may read no original conversation: an empty answer, not
# a wider grant than recall had.
return receipt()
if remaining_seconds <= 0:
return receipt(truncated_by=TruncationBudget.TIMEOUT)
try:
found = await asyncio.wait_for(
search_raw(request.query, request.limit, sources),
timeout=remaining_seconds,
)
except TimeoutError:
return receipt(truncated_by=TruncationBudget.TIMEOUT)
hits: list[RecallHit] = []
used = 0
truncated_by: TruncationBudget | None = None
for index, hit in enumerate(found):
if sources is not None and hit.source not in sources:
continue
if len(hits) >= request.limit:
truncated_by = TruncationBudget.COUNT
return receipt(
tuple(hits),
used=used,
truncated_by=truncated_by,
unexamined=len(found) - index,
)
if used + len(hit.content) > request.character_budget:
truncated_by = TruncationBudget.CHARACTER
continue
hits.append(
RecallHit(
memory_id=hit.raw_id,
content=hit.content,
kind="raw",
scopes=(),
evidence_ids=(),
updated_at=(
datetime.fromtimestamp(hit.memory_at, tz=UTC)
if hit.memory_at is not None
else datetime(1970, 1, 1, tzinfo=UTC)
),
score=hit.score,
content_format="raw",
source=hit.source,
session_id=hit.session_id,
)
)
used += len(hit.content)
return receipt(tuple(hits), used=used, truncated_by=truncated_by)
@@ -5,7 +5,8 @@ from __future__ import annotations
import asyncio
import logging
import re
from contextlib import suppress
from collections.abc import AsyncIterator
from contextlib import AsyncExitStack, asynccontextmanager, suppress
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
@@ -164,44 +165,45 @@ class SdkMemoryRuntime:
if str(error) == "model_profile_not_active":
return HyGraphSnapshot(consulted=False)
raise
try:
client = await self.coordinator.client()
except Exception as error:
raise HySdkError("hy_memory_graph_read_failed") from error
return await asyncio.to_thread(
read_graph_snapshot,
client,
"local",
"*",
limit,
)
async with AsyncExitStack() as stack:
try:
client = await stack.enter_async_context(self.coordinator.use_client())
except Exception as error:
raise HySdkError("hy_memory_graph_read_failed") from error
return await asyncio.to_thread(
read_graph_snapshot,
client,
"local",
"*",
limit,
)
async def digest(self) -> dict[str, object]:
client = await self.coordinator.client()
try:
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
limit=1_000,
offset=0,
)
agent_ids = _agent_ids(raw)
digests = [
{
"agent_id": agent_id,
"schemas_recorded": 0,
**dict(
await asyncio.to_thread(
client.evolve,
user_id="local",
agent_id=agent_id,
)
),
}
for agent_id in agent_ids
]
except Exception as error:
raise HySdkError("hy_memory_digest_failed") from error
async with self.coordinator.use_client() as client:
try:
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
limit=1_000,
offset=0,
)
agent_ids = _agent_ids(raw)
digests = [
{
"agent_id": agent_id,
"schemas_recorded": 0,
**dict(
await asyncio.to_thread(
client.evolve,
user_id="local",
agent_id=agent_id,
)
),
}
for agent_id in agent_ids
]
except Exception as error:
raise HySdkError("hy_memory_digest_failed") from error
return {
"success": all(result.get("success") is not False for result in digests),
"scope_count": len(digests),
@@ -240,36 +242,39 @@ class SdkMemoryRuntime:
decision=decision,
)
async def _mutation_client(self) -> Any:
@asynccontextmanager
async def _mutation_client(self) -> AsyncIterator[Any]:
"""The write client, with no-model and keychain refusals as SDK codes.
Callers record these against the edit or deletion instead of failing
the request that already committed it.
"""
try:
return await self.coordinator.client()
except RuntimeError as error:
code = str(error)
if not _STABLE_CODE.fullmatch(code):
code = "hy_memory_client_unavailable"
raise HySdkError(code) from error
async with AsyncExitStack() as stack:
try:
client = await stack.enter_async_context(self.coordinator.use_client())
except RuntimeError as error:
code = str(error)
if not _STABLE_CODE.fullmatch(code):
code = "hy_memory_client_unavailable"
raise HySdkError(code) from error
yield client
async def update_hy_memory(self, memory_id: str, content: str) -> dict[str, object]:
client = await self._mutation_client()
try:
result = await asyncio.to_thread(client.update, memory_id, content)
except Exception as error:
raise HySdkError("hy_memory_update_failed") from error
async with self._mutation_client() as client:
try:
result = await asyncio.to_thread(client.update, memory_id, content)
except Exception as error:
raise HySdkError("hy_memory_update_failed") from error
if not isinstance(result, dict) or result.get("success") is not True:
raise HySdkError("hy_memory_update_rejected")
return result
async def finish_deletion(self, memory_id: str) -> None:
client = await self._mutation_client()
try:
result = await asyncio.to_thread(client.delete, memory_id)
except Exception as error:
raise HySdkError("hy_memory_delete_failed") from error
async with self._mutation_client() as client:
try:
result = await asyncio.to_thread(client.delete, memory_id)
except Exception as error:
raise HySdkError("hy_memory_delete_failed") from error
if not isinstance(result, dict):
raise HySdkError("hy_memory_delete_rejected")
error_code = result.get("error_code")
@@ -125,6 +125,31 @@ def describe_error(error: BaseException) -> dict[str, object]:
return {"chain": chain}
_PY_FRAME = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_.\-]*\.py:\d{1,7}:[A-Za-z_][A-Za-z0-9_]*$")
def error_frames(error: BaseException, *, limit: int = 16) -> tuple[str, ...]:
"""The raised error's innermost `file.py:line:function` frames, for a stack field.
Library frames are kept (a KeyError raised inside Chroma is located by
them); `<module>`-style names are dropped because stack fields refuse them.
"""
chain = describe_error(error)["chain"]
frames = chain[0].get("frames", ()) if chain else () # type: ignore[index,union-attr]
return tuple(frame for frame in frames if _PY_FRAME.fullmatch(frame))[-limit:]
def error_chain_text(error: BaseException) -> str:
"""`describe_error` as one log token: `Type@file:line:function>...;Cause@...`.
For a plain text log (core.log). Types and frames only, never a message.
"""
return ";".join(
f"{entry['type']}@{'>'.join(entry.get('frames', ()))}" # type: ignore[arg-type]
for entry in describe_error(error)["chain"] # type: ignore[union-attr]
)
def _build() -> dict[str, object]:
from one_memory import __version__
@@ -0,0 +1,504 @@
"""Put a home back exactly as it was when a failed `sdk-migrate` started.
A migration that fails before SDK authority is committed has already changed
the home: schema migrations on `one-memory.sqlite3`, legacy work moved into the
durable queue, renamed Chroma collections under `hy-memory/governed-v2`, ledger
rows. The fixed release that follows must start from the original data, not
from that half-way state, so the runner restores the verified full-home bundle
it took before the first change (`restore point`).
The restore is staged and then swapped in with renames:
1. the bundle is verified (every file hashed against its manifest);
2. its database, config and SDK root are copied next to their live places and
verified again;
3. a journal lists every rename, then the live items (database with its
`-wal`/`-shm`, `config.json`, `hy-memory/governed-v2`) are renamed into
`backups/sdk-migration-failed/` and the staged copies are renamed in.
A rename either happened or did not, and each step is idempotent, so a process
killed during step 3 is finished by the next run (`finish_interrupted_restore`)
before anything else reads the home. Core refuses to serve and the status
probe reports `required` while a journal exists.
After SDK authority is committed nothing is restored: the home already belongs
to the SDK and the legacy tables may be gone.
"""
from __future__ import annotations
import errno
import hashlib
import json
import os
import shutil
import sys
import time
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from one_memory.database_access import close_main_databases
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
from one_memory.sdk_data_plane_migration import DataPlaneMigrationError
RESTORE_POINT = ".sdk-migration-restore-point.json"
RESTORE_STAGING = ".sdk-migration-restore"
JOURNAL = "journal.json"
FAILED_STATE = "sdk-migration-failed"
RESTORED_MARK = "restored.json"
DATABASE = "one-memory.sqlite3"
CONFIG = "config.json"
_DATABASE_SIDECARS = ("-wal", "-shm", "-journal")
_WRITER = "core-sdk-migrate"
class MigrationRestoreFailed(DataPlaneMigrationError):
"""The pre-migration data could not be put back; the home stays blocked."""
@dataclass(frozen=True)
class RestoreResult:
bundle_id: str
files: int
bytes: int
duration_ms: float
def as_document(self) -> dict[str, object]:
return {
"bundle_id": self.bundle_id,
"files": self.files,
"bytes": self.bytes,
"duration_ms": self.duration_ms,
}
def _home(data_dir: Path) -> Path:
return data_dir.resolve()
def _staging(home: Path) -> Path:
return home / RESTORE_STAGING
def _journal(home: Path) -> Path:
return _staging(home) / JOURNAL
def _sdk_staging(home: Path) -> Path:
return home / "hy-memory" / f".{HY_MEMORY_NAMESPACE}.restore-staging"
def failed_state_path(data_dir: Path) -> Path:
return _home(data_dir) / "backups" / FAILED_STATE
def _write_json(path: Path, document: dict[str, object]) -> None:
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
temporary.write_text(json.dumps(document, sort_keys=True) + "\n", encoding="utf-8")
os.chmod(temporary, 0o600)
_fsync(temporary)
temporary.replace(path)
_fsync_directory(path.parent)
def _fsync(path: Path) -> None:
descriptor = os.open(path, os.O_RDONLY)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)
def _fsync_directory(path: Path) -> None:
# Windows cannot open a directory; its renames are durable on return.
try:
_fsync(path)
except OSError:
return
def _log(
data_dir: Path,
event: str,
outcome: str,
*,
migration_id: str | None,
code: str | None = None,
error: BaseException | None = None,
details: dict[str, object] | None = None,
) -> None:
from one_memory.sdk_migration_log import write_migration_log
write_migration_log(
data_dir,
writer=_WRITER,
event=event,
outcome=outcome,
migration_id=migration_id,
code=code,
error=error,
details=details,
)
# --- restore point -----------------------------------------------------------
def record_restore_point(data_dir: Path, *, migration_id: str, bundle_id: str) -> None:
"""Name the verified bundle that holds this home as the migration found it.
Written before the first change, so a run killed part-way is rolled back
by the next one instead of resumed.
"""
_write_json(
_home(data_dir) / RESTORE_POINT,
{
"schema_version": 1,
"migration_id": migration_id,
"bundle_id": bundle_id,
"created_at": datetime.now(UTC).isoformat(),
},
)
def read_restore_point(data_dir: Path) -> dict[str, str] | None:
try:
document = json.loads((_home(data_dir) / RESTORE_POINT).read_text(encoding="utf-8"))
except FileNotFoundError:
return None
except (OSError, ValueError) as error:
raise MigrationRestoreFailed("sdk_migration_restore_point_invalid") from error
if (
not isinstance(document, dict)
or document.get("schema_version") != 1
or not isinstance(document.get("migration_id"), str)
or not isinstance(document.get("bundle_id"), str)
):
raise MigrationRestoreFailed("sdk_migration_restore_point_invalid")
return {"migration_id": document["migration_id"], "bundle_id": document["bundle_id"]}
def clear_restore_point(data_dir: Path) -> None:
(_home(data_dir) / RESTORE_POINT).unlink(missing_ok=True)
def restore_pending(data_dir: Path) -> bool:
"""A restore was interrupted between its renames; nothing may read the home."""
return _journal(_home(data_dir)).is_file()
# --- restore -----------------------------------------------------------------
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def _release_handles() -> None:
"""Close what this process holds on the home; Windows cannot rename open files."""
close_main_databases()
chromadb = sys.modules.get("chromadb")
if chromadb is None:
return
try:
from chromadb.api.shared_system_client import SharedSystemClient
SharedSystemClient.clear_system_cache()
except Exception: # noqa: BLE001 - best effort; a held file fails the rename instead
return
def _verify_copy(copy: Path, expected: list[dict[str, object]], prefix: str) -> None:
"""The staged tree holds exactly the bundle's files under `prefix`."""
wanted = {
str(item["path"])[len(prefix) :]: item
for item in expected
if str(item["path"]).startswith(prefix)
}
present = (
{path.relative_to(copy).as_posix() for path in copy.rglob("*") if path.is_file()}
if copy.is_dir()
else set()
)
if present != set(wanted):
raise MigrationRestoreFailed("sdk_migration_restore_verify_failed")
for relative, item in wanted.items():
path = copy / relative
if path.stat().st_size != int(str(item["bytes"])) or _sha256(path) != item["sha256"]:
raise MigrationRestoreFailed("sdk_migration_restore_verify_failed")
def _plan(home: Path, *, config_staged: bool) -> list[tuple[str, str]]:
"""Every rename of the swap, as paths relative to the home, in order."""
failed = Path("backups") / FAILED_STATE
staging = Path(RESTORE_STAGING)
sdk_live = Path("hy-memory") / HY_MEMORY_NAMESPACE
steps: list[tuple[Path, Path]] = [
(Path(f"{DATABASE}{suffix}"), failed / "management" / f"{DATABASE}{suffix}")
for suffix in _DATABASE_SIDECARS
]
steps += [
(Path(DATABASE), failed / "management" / DATABASE),
(Path(CONFIG), failed / "management" / CONFIG),
(sdk_live, failed / "sdk" / HY_MEMORY_NAMESPACE),
(staging / "management" / DATABASE, Path(DATABASE)),
(_sdk_staging(home).relative_to(home), sdk_live),
]
if config_staged:
steps.append((staging / "management" / CONFIG, Path(CONFIG)))
return [(source.as_posix(), target.as_posix()) for source, target in steps]
def _apply(
home: Path,
steps: list[list[str]] | list[tuple[str, str]],
done: list[tuple[Path, Path]],
) -> None:
"""Run each rename once; a step whose source is already gone is skipped.
`done` collects the renames this call made, so a caller can undo them.
"""
for source_name, target_name in steps:
source, target = home / source_name, home / target_name
source_exists = source.exists() or source.is_symlink()
target_exists = target.exists() or target.is_symlink()
if not source_exists:
continue
if target_exists:
raise MigrationRestoreFailed("sdk_migration_restore_conflict")
target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
source.replace(target)
done.append((source, target))
for directory in (home, home / "hy-memory"):
_fsync_directory(directory)
def _finish(home: Path, bundle_id: str) -> None:
# The restore point goes first: a kill after it leaves only the journal,
# whose steps are all done, and the next run just tidies up.
clear_restore_point(home)
shutil.rmtree(_sdk_staging(home), ignore_errors=True)
_journal(home).unlink(missing_ok=True)
shutil.rmtree(_staging(home), ignore_errors=True)
bundle = home / "backups" / f"{bundle_id}.sdk-bundle"
try:
# A later verified bundle supersedes this one (`prune_restored_bundles`).
_write_json(
bundle / RESTORED_MARK,
{"schema_version": 1, "restored_at": datetime.now(UTC).isoformat()},
)
except OSError:
return
def restore_pre_migration_state(
data_dir: Path,
*,
bundle_id: str,
migration_id: str | None,
reason: str,
) -> RestoreResult:
"""Put the restore-point bundle back in place of the half-migrated home.
The caller holds the migration lock with Runtime stopped and host capture
fenced. Raises `MigrationRestoreFailed`; before the swap the home is left as
the failure left it, and the restore point stays for the next run.
"""
home = _home(data_dir)
started = time.monotonic()
_log(
home,
"restore.started",
"started",
migration_id=migration_id,
details={"bundle_id": bundle_id, "reason": reason},
)
try:
result = _restore(home, bundle_id=bundle_id, started=started)
except BaseException as error:
code = getattr(error, "code", None)
if not (isinstance(code, str) and code.startswith("sdk_migration_restore")):
code = (
"sdk_migration_restore_disk_space_insufficient"
if isinstance(error, OSError) and error.errno in {errno.ENOSPC, errno.EDQUOT}
else "sdk_migration_restore_failed"
)
_log(
home,
"restore.failed",
"failed",
migration_id=migration_id,
code=code,
error=error,
details={
"bundle_id": bundle_id,
"reason": reason,
"duration_ms": (time.monotonic() - started) * 1000,
"swap_pending": restore_pending(home),
},
)
if not isinstance(error, Exception) or (
isinstance(error, MigrationRestoreFailed) and error.code == code
):
raise
raise MigrationRestoreFailed(code) from error
_log(
home,
"restore.finished",
"succeeded",
migration_id=migration_id,
details={**result.as_document(), "reason": reason},
)
return result
def _restore(home: Path, *, bundle_id: str, started: float) -> RestoreResult:
from one_memory.sdk_migration_bundle import inspect_sdk_migration_bundle
if restore_pending(home):
raise MigrationRestoreFailed("sdk_migration_restore_pending")
manifest = inspect_sdk_migration_bundle(data_dir=home, bundle_id=bundle_id)
bundle = home / "backups" / f"{bundle_id}.sdk-bundle"
listed = manifest.get("files")
files = [item for item in listed if isinstance(item, dict)] if isinstance(listed, list) else []
total = sum(int(str(item["bytes"])) for item in files)
if shutil.disk_usage(home).free < total:
raise MigrationRestoreFailed("sdk_migration_restore_disk_space_insufficient")
_release_handles()
staging = _staging(home)
sdk_staging = _sdk_staging(home)
shutil.rmtree(staging, ignore_errors=True)
shutil.rmtree(sdk_staging, ignore_errors=True)
(staging / "management").mkdir(parents=True, mode=0o700)
try:
shutil.copy2(bundle / "management" / DATABASE, staging / "management" / DATABASE)
config_staged = manifest.get("config") is not None
if config_staged:
shutil.copy2(bundle / "management" / CONFIG, staging / "management" / CONFIG)
_verify_copy(staging / "management", files, "management/")
sdk_source = bundle / "sdk" / HY_MEMORY_NAMESPACE
if not sdk_source.is_dir() or sdk_source.is_symlink():
raise MigrationRestoreFailed("sdk_migration_restore_bundle_incomplete")
sdk_staging.parent.mkdir(parents=True, exist_ok=True)
shutil.copytree(sdk_source, sdk_staging, symlinks=True)
_verify_copy(sdk_staging, files, f"sdk/{HY_MEMORY_NAMESPACE}/")
for path in (staging / "management").iterdir():
_fsync(path)
except BaseException:
shutil.rmtree(staging, ignore_errors=True)
shutil.rmtree(sdk_staging, ignore_errors=True)
raise
failed = failed_state_path(home)
# One failed state is kept for diagnosis; the newest replaces the last.
shutil.rmtree(failed, ignore_errors=True)
if failed.exists():
raise MigrationRestoreFailed("sdk_migration_restore_failed_state_busy")
steps = _plan(home, config_staged=config_staged)
_write_json(
_journal(home),
{"schema_version": 1, "bundle_id": bundle_id, "steps": [list(step) for step in steps]},
)
done: list[tuple[Path, Path]] = []
try:
_apply(home, steps, done)
except BaseException as error:
if not isinstance(error, Exception):
# Interrupted (a signal tearing the process down): leave the
# journal; the next run finishes the swap forward.
raise
try:
for source, target in reversed(done):
target.replace(source)
except OSError:
# The journal stays: the next run finishes the swap forward.
raise MigrationRestoreFailed("sdk_migration_restore_swap_interrupted") from error
_journal(home).unlink(missing_ok=True)
shutil.rmtree(staging, ignore_errors=True)
shutil.rmtree(sdk_staging, ignore_errors=True)
raise
_finish(home, bundle_id)
return RestoreResult(
bundle_id=bundle_id,
files=len(files),
bytes=total,
duration_ms=(time.monotonic() - started) * 1000,
)
def finish_interrupted_restore(data_dir: Path, *, migration_id: str | None = None) -> bool:
"""Complete a swap a killed process left between its renames. True if it did."""
home = _home(data_dir)
journal = _journal(home)
if not journal.is_file():
return False
started = time.monotonic()
try:
document = json.loads(journal.read_text(encoding="utf-8"))
steps = document["steps"]
bundle_id = str(document["bundle_id"])
if not isinstance(steps, list) or not all(
isinstance(step, list) and len(step) == 2 for step in steps
):
raise ValueError("journal steps invalid")
_release_handles()
_apply(home, steps, [])
_finish(home, bundle_id)
except Exception as error:
code = (
error.code
if isinstance(error, MigrationRestoreFailed)
else "sdk_migration_restore_resume_failed"
)
_log(
home,
"restore.failed",
"failed",
migration_id=migration_id,
code=code,
error=error,
details={"reason": "interrupted_swap", "swap_pending": True},
)
if isinstance(error, MigrationRestoreFailed):
raise
raise MigrationRestoreFailed(code) from error
_log(
home,
"restore.finished",
"succeeded",
migration_id=migration_id,
details={
"bundle_id": bundle_id,
"reason": "interrupted_swap",
"duration_ms": (time.monotonic() - started) * 1000,
},
)
return True
def prune_restored_bundles(data_dir: Path, *, keep: set[str]) -> int:
"""Drop bundles already restored, once a newer verified bundle exists.
Each holds the same pre-migration home a newer bundle now holds (capture
stayed fenced since), so it is only a second copy of it.
"""
backups = _home(data_dir) / "backups"
removed = 0
for mark in backups.glob(f"*.sdk-bundle/{RESTORED_MARK}"):
bundle = mark.parent
if bundle.name.removesuffix(".sdk-bundle") in keep:
continue
shutil.rmtree(bundle, ignore_errors=True)
removed += 0 if bundle.exists() else 1
return removed
def discard_failed_state(data_dir: Path) -> None:
"""The failed half-migrated copy is only useful until a migration succeeds."""
shutil.rmtree(failed_state_path(data_dir), ignore_errors=True)
@@ -18,6 +18,7 @@ import time
from collections.abc import Callable, Iterator
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
@@ -34,6 +35,8 @@ from one_memory.memory_server.durable_queue import (
COMPLETED,
FAILED,
REAPED_PAYLOAD_MARKER,
RUNNING,
UNKNOWN,
DurableWriteQueue,
)
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
@@ -316,11 +319,45 @@ def move_pending_dispatches(database_path: Path) -> int:
return len(conversations)
#: Why a pre-cutover write job that may have reached HY was closed.
ABANDONED_AT_CUTOVER = "abandoned_at_cutover"
def settle_legacy_unknown_write_jobs(database_path: Path) -> int:
"""Close write jobs the legacy pipeline left possibly-sent. Safe to repeat.
They ran under the legacy pipeline, so the SDK's `get_write_outcome` knows
none of them and the unknown reconcile can never settle them: a migrated
1.0.4 home showed 1000+ unknown forever. They are not retried, since they
may already have written and the SDK has no idempotency key. `cancelled`,
not `failed`: the upgrade ended them, nothing broke, and a `failed` count
would put the home in the failed state for good. `hy_invoked` is kept, so
no readmission path can pick one up again.
"""
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
if "memory_write_jobs" not in _tables(database):
return 0
now = datetime.now(UTC).isoformat()
with main_database(database_path).write(autocommit=True) as database:
# A `running` row already marked invoked would be reclaimed to
# `unknown` on the next open, so it is the same case.
return int(
database.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,"
"lease_expires_at=NULL,last_error=?,updated_at=? "
"WHERE state=? OR (state=? AND hy_invoked=1)",
(CANCELLED, ABANDONED_AT_CUTOVER, now, UNKNOWN, RUNNING),
).rowcount
)
def abandoned_legacy_work(database_path: Path) -> dict[str, int]:
"""Count legacy work that ends with the cutover instead of blocking it.
A task that may have reached the SDK is not replayed: the SDK already holds
whatever it committed. Pending user deletions are not carried over.
Settled write jobs are counted by their code, so a repeated preparation
reports the same number.
"""
abandoned: dict[str, int] = {}
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
@@ -331,6 +368,11 @@ def abandoned_legacy_work(database_path: Path) -> dict[str, int]:
"memory_tasks",
"status='unknown' OR (status='running' AND hy_invocation_count > 0)",
),
(
"unknown_write_jobs",
"memory_write_jobs",
f"state='{CANCELLED}' AND last_error='{ABANDONED_AT_CUTOVER}'",
),
("deletions", "memory_deletions", "status IN ('pending','running')"),
)
for name, table, where in checks:
@@ -353,6 +395,10 @@ def prepare_legacy_work_for_cutover(
running, so a failure names it.
"""
trace = {} if trace is None else trace
# First: a possibly-sent job never runs again, so its input is not
# frozen, and input released since must not block the cutover.
trace["sub_step"] = "unknown_write_jobs"
settle_legacy_unknown_write_jobs(database_path)
trace["sub_step"] = "write_queue"
frozen, moved = prepare_legacy_write_queue_for_cutover(database_path)
trace["sub_step"] = "pending_dispatches"
@@ -483,30 +529,50 @@ def _resumable_identity(database_path: Path) -> tuple[str, str]:
return migration_id, f"{migration_id}-pre-cutover"
@contextmanager
def capture_fence(data_dir: Path) -> Iterator[None]:
"""Host capture hooks return silently while this process is alive."""
def _write_capture_fence(data_dir: Path, document: dict[str, object]) -> None:
marker = data_dir / CAPTURE_FENCE
temporary = marker.with_name(f"{marker.name}.{os.getpid()}.tmp")
temporary.write_text(
json.dumps({"schema_version": 1, "pid": os.getpid()}) + "\n",
encoding="utf-8",
)
temporary.write_text(json.dumps(document) + "\n", encoding="utf-8")
os.chmod(temporary, 0o600)
temporary.replace(marker)
@contextmanager
def capture_fence(data_dir: Path) -> Iterator[None]:
"""Host capture hooks return silently while this process is alive.
A failed run leaves the fence frozen instead of lifting it: the home was
put back (or is half-migrated) and must not take new writes until the next
migration attempt replaces the fence with its own and succeeds.
"""
marker = data_dir / CAPTURE_FENCE
_write_capture_fence(data_dir, {"schema_version": 1, "pid": os.getpid()})
try:
yield
finally:
except BaseException:
try:
_write_capture_fence(
data_dir, {"schema_version": 1, "frozen": True, "reason": "sdk_migration_failed"}
)
except OSError:
marker.unlink(missing_ok=True)
raise
else:
marker.unlink(missing_ok=True)
temporary.unlink(missing_ok=True)
finally:
marker.with_name(f"{marker.name}.{os.getpid()}.tmp").unlink(missing_ok=True)
def _ensure_bundle(*, data_dir: Path, bundle_id: str, runtime_build: str) -> dict[str, object]:
def _ensure_bundle(
*, data_dir: Path, bundle_id: str, runtime_build: str
) -> tuple[dict[str, object], bool]:
"""The verified bundle, and whether this call created it."""
from one_memory.sdk_migration_bundle import (
MigrationBundleRejected,
create_sdk_migration_bundle,
)
created = True
try:
create_sdk_migration_bundle(
data_dir=data_dir,
@@ -517,7 +583,67 @@ def _ensure_bundle(*, data_dir: Path, bundle_id: str, runtime_build: str) -> dic
except MigrationBundleRejected as error:
if error.code != "sdk_migration_bundle_exists":
raise
return inspect_sdk_migration_bundle(data_dir=data_dir, bundle_id=bundle_id)
created = False
return inspect_sdk_migration_bundle(data_dir=data_dir, bundle_id=bundle_id), created
def _restore_point_bundle(
*, data_dir: Path, migration_id: str, bundle_id: str, created: bool, runtime_build: str
) -> str:
"""The bundle that holds this home exactly as this run found it.
The cutover bundle qualifies when this run created it. One left by an
earlier run (an older build's blocked ledger row) may predate later writes,
so restoring it could drop them; a fresh restore-point bundle is taken.
"""
if created:
return bundle_id
from one_memory.sdk_migration_bundle import create_sdk_migration_bundle
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%S%fZ")
restore_id = f"{migration_id}-restore-point-{stamp}"
create_sdk_migration_bundle(
data_dir=data_dir,
bundle_id=restore_id,
runtime_build=runtime_build,
runtime_stopped=True,
)
inspect_sdk_migration_bundle(data_dir=data_dir, bundle_id=restore_id)
return restore_id
def _recover_interrupted_attempt(data_dir: Path, database_path: Path, log: _StepLog) -> None:
"""Roll back what a killed earlier run left, before anything reads the home."""
from one_memory.sdk_migration_restore import (
MigrationRestoreFailed,
clear_restore_point,
finish_interrupted_restore,
read_restore_point,
restore_pre_migration_state,
)
finish_interrupted_restore(data_dir)
point = read_restore_point(data_dir)
if point is None:
return
with log.step("preflight.interrupted_attempt") as step:
try:
authority = SdkDataPlaneMigrationStore(database_path).authority()
except Exception as error:
# Never restore over a home that may already be on SDK authority.
raise MigrationRestoreFailed("sdk_migration_restore_authority_unknown") from error
step["authority"] = authority.value
if authority is MemoryAuthority.SDK:
clear_restore_point(data_dir)
step["restored"] = False
return
restore_pre_migration_state(
data_dir,
bundle_id=point["bundle_id"],
migration_id=point["migration_id"],
reason="interrupted_attempt",
)
step["restored"] = True
def _vacuum(database_path: Path) -> None:
@@ -537,14 +663,28 @@ def run_startup_migration(
namespace: str,
progress: Callable[[str], None] = lambda _step: None,
open_client: Callable[[ModelProfileSnapshot], AbstractContextManager[Any]] | None = None,
restore_report: dict[str, object] | None = None,
) -> StartupMigrationResult:
"""Bring one stopped home to SDK authority. Safe to repeat after any failure.
The caller must stop Runtime and keep its service from restarting.
`open_client` yields a public SDK client configured as the Runtime will be;
tests replace it.
A failure before SDK authority is committed puts the home back exactly as
this run found it (`sdk_migration_restore`), so the next run, possibly of a
fixed release, starts from the original data. The original exception is
re-raised either way; `restore_report` learns what the restore did
(`state`: `restored`, `failed` with its own `code`, or `skipped`).
"""
from one_memory.sdk_migration_restore import read_restore_point, restore_pending
database_path = data_dir / "one-memory.sqlite3"
log = _StepLog(data_dir)
# Before anything reads the home: a run killed part-way is rolled back.
if restore_pending(data_dir) or read_restore_point(data_dir) is not None:
with capture_fence(data_dir):
_recover_interrupted_attempt(data_dir, database_path, log)
if not database_path.is_file():
return StartupMigrationResult(state="not_required")
@@ -553,7 +693,6 @@ def run_startup_migration(
return open_client(profile)
return runtime_sdk_client(database_path, data_dir, profile)
log = _StepLog(data_dir)
store = SdkDataPlaneMigrationStore(database_path)
with log.step("preflight.authority_read") as step:
authority, authority_migration = store.authority_state()
@@ -613,6 +752,12 @@ def run_startup_migration(
from one_memory.database_access import close_main_databases
from one_memory.memory_server.migration_runner import SqliteMigrationStore, migrate
from one_memory.memory_server.migrations import MIGRATIONS
from one_memory.sdk_migration_restore import (
clear_restore_point,
discard_failed_state,
prune_restored_bundles,
record_restore_point,
)
with log.step("preflight.resumable_identity") as step:
migration_id, bundle_id = _resumable_identity(database_path)
@@ -630,66 +775,106 @@ def run_startup_migration(
raise DataPlaneMigrationError("sdk_migration_model_profile_required")
log.check("preflight.model_profile_check", None)
with capture_fence(data_dir):
progress("bundle")
with log.step("bundle.ensure") as step:
manifest = _ensure_bundle(
data_dir=data_dir,
bundle_id=bundle_id,
runtime_build=runtime_build,
)
step["bundle_id"] = bundle_id
progress("prepare")
with log.step("prepare.schema_migrate") as step:
asyncio.run(migrate(SqliteMigrationStore(database_path), MIGRATIONS))
close_main_databases()
step["schema_version"] = MIGRATIONS[-1].version
# Builds through 19 left a staged SDK copy beside the original store.
# Settling it before the bundle means the bundle holds the original
# store, and a restore never brings the staged copy back.
with log.step("prepare.recover_abandoned_staging"):
recover_abandoned_staging(
data_dir=data_dir,
database_path=database_path,
namespace=namespace,
)
with log.step("prepare.ledger_start"):
store.start(
migration_id=migration_id,
source_runtime_build=str(manifest.get("runtime_build") or ""),
target_sdk_version=sdk_version,
source_schema_version=MIGRATIONS[-1].version,
namespace_fingerprint=hashlib.sha256(namespace.encode()).hexdigest(),
backend_fingerprint=hashlib.sha256(b"chroma:kuzu:sqlite").hexdigest(),
backup_bundle_id=bundle_id,
progress("bundle")
with log.step("bundle.ensure") as step:
manifest, created = _ensure_bundle(
data_dir=data_dir,
bundle_id=bundle_id,
runtime_build=runtime_build,
)
with log.step("prepare.legacy_work") as step:
preparation = prepare_legacy_work_for_cutover(database_path, step)
step.update(preparation.as_document())
reattached = ()
readback = None
if profile is not None:
progress("vectors")
with log.step("vectors.reattach") as step:
reattached = reattach_legacy_vectors(data_dir, profile)
step["collections"] = len(reattached)
if not legacy.empty:
assert profile is not None
progress("verify")
with log.step("verify.readback") as step:
with client_for(profile) as client:
readback = verify_readback(client, legacy)
step.update(readback.as_document())
if not readback.passed:
store.set_phase(migration_id, phase="readback_failed", state="blocked")
raise MemoryReadbackFailed(readback)
with log.step("prepare.mark_prepared"):
store.set_phase(migration_id, phase=PREPARED_PHASE, state="ready")
progress("cutover")
with log.step("cutover.commit"):
commit_sdk_cutover(
step["bundle_id"] = bundle_id
step["created"] = created
with log.step("bundle.restore_point") as step:
restore_bundle = _restore_point_bundle(
data_dir=data_dir,
migration_id=migration_id,
bundle_id=bundle_id,
created=created,
runtime_build=runtime_build,
)
record_restore_point(data_dir, migration_id=migration_id, bundle_id=restore_bundle)
step["bundle_id"] = restore_bundle
step["pruned"] = prune_restored_bundles(data_dir, keep={bundle_id, restore_bundle})
# From here until SDK authority is committed, a failure puts the
# restore point back. Only `cutover.commit` can fail after its own
# commit; the database decides then.
cutover_started = False
try:
progress("prepare")
with log.step("prepare.schema_migrate") as step:
asyncio.run(migrate(SqliteMigrationStore(database_path), MIGRATIONS))
close_main_databases()
step["schema_version"] = MIGRATIONS[-1].version
with log.step("prepare.ledger_start"):
store.start(
migration_id=migration_id,
source_runtime_build=str(manifest.get("runtime_build") or ""),
target_sdk_version=sdk_version,
source_schema_version=MIGRATIONS[-1].version,
namespace_fingerprint=hashlib.sha256(namespace.encode()).hexdigest(),
backend_fingerprint=hashlib.sha256(b"chroma:kuzu:sqlite").hexdigest(),
backup_bundle_id=bundle_id,
)
with log.step("prepare.legacy_work") as step:
preparation = prepare_legacy_work_for_cutover(database_path, step)
step.update(preparation.as_document())
reattached = ()
readback = None
if profile is not None:
progress("vectors")
with log.step("vectors.reattach") as step:
reattached = reattach_legacy_vectors(data_dir, profile)
step["collections"] = len(reattached)
if not legacy.empty:
assert profile is not None
progress("verify")
with log.step("verify.readback") as step:
with client_for(profile) as client:
readback = verify_readback(client, legacy)
step.update(readback.as_document())
if not readback.passed:
store.set_phase(migration_id, phase="readback_failed", state="blocked")
raise MemoryReadbackFailed(readback)
with log.step("prepare.mark_prepared"):
store.set_phase(migration_id, phase=PREPARED_PHASE, state="ready")
progress("cutover")
cutover_started = True
with log.step("cutover.commit"):
commit_sdk_cutover(
data_dir=data_dir,
database_path=database_path,
migration_id=migration_id,
bundle_id=bundle_id,
sdk_version=sdk_version,
)
except Exception:
_restore_after_failure(
data_dir=data_dir,
database_path=database_path,
migration_id=migration_id,
bundle_id=bundle_id,
sdk_version=sdk_version,
bundle_id=restore_bundle,
cutover_started=cutover_started,
report=restore_report,
)
raise
# SDK authority is committed: from here on the home is never restored.
clear_restore_point(data_dir)
if restore_bundle != bundle_id:
shutil.rmtree(
data_dir.resolve() / "backups" / f"{restore_bundle}.sdk-bundle",
ignore_errors=True,
)
discard_failed_state(data_dir)
_repair_metadata_after_cutover(data_dir, database_path, log)
progress("cleanup")
with log.step("cleanup.drop_legacy_tables") as step:
dropped = resume_legacy_cleanup(data_dir=data_dir, database_path=database_path)
@@ -706,3 +891,71 @@ def run_startup_migration(
reattached=reattached,
readback=readback,
)
def _repair_metadata_after_cutover(data_dir: Path, database_path: Path, log: _StepLog) -> None:
"""Restore lost metadata here, where Runtime is stopped and no Chroma client is open.
Writes the repair receipt, so Core's startup fallback finds it and does
nothing. Authority is already committed: a failure is logged and left to
that fallback instead of failing the migration.
"""
from one_memory.hy_sdk_migration import repair_sdk_metadata_from_bundle
try:
with log.step("cutover.metadata_repair") as step:
repair = repair_sdk_metadata_from_bundle(data_dir=data_dir, database_path=database_path)
step["mapped"] = repair.mapped
step["vector_rows"] = repair.vector_rows
step["coding_rows"] = repair.coding_rows
except Exception:
return
def _restore_after_failure(
*,
data_dir: Path,
database_path: Path,
migration_id: str,
bundle_id: str,
cutover_started: bool,
report: dict[str, object] | None,
) -> None:
"""Put the restore point back after a pre-cutover failure. Never raises.
The caller re-raises the migration's own error: the restore is secondary,
so its outcome only goes to the log and to `report`.
"""
from one_memory.sdk_migration_log import write_migration_log
from one_memory.sdk_migration_restore import restore_pre_migration_state
report = {} if report is None else report
if cutover_started:
try:
committed = SdkDataPlaneMigrationStore(database_path).authority() is MemoryAuthority.SDK
except Exception: # noqa: BLE001 - never restore over an unknown authority
committed = True
if committed:
write_migration_log(
data_dir,
writer="core-sdk-migrate",
event="restore.skipped",
outcome="succeeded",
migration_id=migration_id,
details={"reason": "authority_committed_or_unknown"},
)
report["state"] = "skipped"
return
try:
restore_pre_migration_state(
data_dir,
bundle_id=bundle_id,
migration_id=migration_id,
reason="migration_failed",
)
except Exception as error: # noqa: BLE001 - the migration's own failure is raised
code = getattr(error, "code", None)
report["state"] = "failed"
report["code"] = code if isinstance(code, str) else "sdk_migration_restore_failed"
return
report["state"] = "restored"
Loaded 100 of 282 files, more files were not shown because too many files have changed in this diff. Show more