Compare commits

...
Author SHA1 Message Date
alvinfeiandClaude Opus 5.5 3742b3e453 feat(memhub): configurable SDK retrieval and extraction features
MemHub hard-codes none of the SDK's optional features any more. A person
can turn them on from the CLI or the API; with nothing set the SDK config
is byte for byte what it was, so the App behaves as before.

- Features (one global `memory_features` document in config.json, beside
  `memory_mode`; strict booleans, unknown keys refused, absent = SDK
  default):
  - graph_rerank: graph_store.graph_rerank_enabled, usage A, ultra only.
  - same_series_timeline: recall.same_series_timeline_enabled, needs
    graph_rerank.
  - episode_recall: recall.episode_recall_enabled, usage B, pro or ultra.
  - session_episodes: recall.session_episode_enabled, the episode write
    side.
  - episode_summary: recall.episode_summary_enabled, one LLM call per
    closed episode.
  - intention: llm.enable_intention, on by default.
  - few_shot: llm.few_shot_enabled.
- How they reach the SDK: build_sdk_config sets the MemoryConfig fields
  per client. Every exposed flag is a config field read at call time, never
  os.environ; the SDK's own env var still sets the default of an unset
  feature. A feature never touches the vector store, embedder or store
  paths, so collection identity is unchanged.
- A change rebuilds the handles like a mode change: write handles are keyed
  by (profile, revision, mode, features) and retired with the #218 use
  count; the read handle (ActiveProfileReadHandle) is keyed by (profile,
  mode, features) and builds a new client on the next read, closing the old
  one once no request uses it.
- API: GET/PUT /api/v1/settings/memory-features. Each feature reports its
  effective value, configured value, default, whether it is active in the
  current mode (e.g. graph_rerank in pro: requires_ultra), cost, SDK field,
  env var and a description. PUT: a value sets, null resets, an absent key
  is kept, anything else is 422.
- Status: pipeline.features (the non-default ones) in
  /api/v1/memory-processing, memhub status --json and progress --json.
- CLI: memhub settings features [set <name>=<bool> ... [--yes] |
  reset [<name> ...]]; set previews without --yes.
- Contract: docs/contracts/memory-features-v1.md. It has the inventory,
  what is left out and why, and zh/en/es copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 19:54:44 +08:00
feitianxiang 98bc52b9f4 Merge pull request #253 from Tencent-Hunyuan/feat/memhub-mode-evolve
feat(memhub): memory mode (lite/pro/ultra), evolve and nightly schedule (B, stacked on #252)
2026-11-01 19:53:15 +08:00
feitianxiang 30fea9394f Merge pull request #252 from Tencent-Hunyuan/feat/memhub-sdk-surface
feat(memhub): SDK search params, memory layers, raw, v2 source aliases, profile versions (A)
2026-11-01 19:53:02 +08:00
feitianxiang e966e913b6 Merge pull request #258 from Tencent-Hunyuan/release/hy-memory-2.0.10
release: hy-memory 2.0.10
2026-11-01 19:32:15 +08:00
alvinfeiandClaude Opus 5.5 ba6248b90b docs: capitalise the 2.0.10 changelog entry
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 19:26:34 +08:00
alvinfeiandClaude Opus 5.5 6cad02bc82 release: hy-memory 2.0.10
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 19:26:07 +08:00
feitianxiang fdc57822b5 Merge pull request #254 from Tencent-Hunyuan/feat/sdk-trimmed-raw
feat(sdk): store raw conversations already trimmed; history concatenates without re-trimming
2026-11-01 19:24:43 +08:00
alvinfeiandClaude Opus 5.5 4e1748f65a feat(memhub): memory mode (lite/pro/ultra), evolve and its nightly schedule
- Mode: a saved setting (config.json memory_mode, default pro);
  ONE_MEMORY_GRAPH_STORE still forces ultra for development.
  - API: GET/PUT /api/v1/settings/memory-mode, which reports the switch's
    effects and the graph backfill.
  - CLI: memhub settings mode [lite|pro|ultra] [--yes], which prints the plan
    first.
- The mode drives every SDK client.
  - acquire_client passes it to HyMemoryClient. The read handle rebuilds on a
    change.
  - Write handles are keyed by (profile, revision, mode) and retired with the
    #218 use count, so a batch in flight finishes in the old mode.
  - Jobs read the mode when they run: a job completed in lite is never
    reprocessed; pending jobs take the new mode.
  - Mode never touches the vector collection identity.
- lite needs no generation model: the built-in embedder writes raw, and the
  claim gate admits lite without an active profile. lite never calls
  add_extracted (a note is one raw user turn through add). Recall serves the
  raw layer.
- → ultra: build_memory_graph once per known user, in the background, with a
  status (graph_build).
- Raw retention removed in every mode: MemHub deletes no raw (owner
  decision); the retention loop, its startup step and its test are gone.
- P1-3 / R7: POST /api/v2/evolve answers 202 with a run receipt;
  GET /api/v2/evolve[/{run_id}].
  - Ultra only: 409 memory_mode_not_ultra otherwise.
  - One run per user. Never beside that user's write: it waits for the write
    in flight, and holds that user's jobs (released pre-SDK) until it ends.
  - Loops while should_continue, up to 20 rounds.
  - Per-user last_run_at, last_result and last_error in evolve-state.json
    (counts and codes only).
  - CLI: memhub evolve [--user-id] [--agent-id] [--max-batches] [--wait].
- Nightly scheduler: 03:00 local by default, ultra only, with catch-up when
  the last run is older than the slot or 24h.
  - Users come from Core's records (local, grant holders, write-queue users)
    that the SDK holds memories for.
  - GET/PUT /api/v1/settings/evolve-schedule.
- R10: pipeline {mode, configured_mode, forced_by_env, evolve, last_evolve_at,
  graph_build} in /api/v1/memory-processing, memhub status --json and
  memhub progress.
- Contract for the App: docs/contracts/memory-mode-evolve-v1.md, including
  what SDK ultra additionally enables and zh/en/es copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 19:09:19 +08:00
alvinfeiandClaude Opus 5.5 b20331aa39 Merge main into feat/sdk-trimmed-raw
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 18:56:23 +08:00
feitianxiang 104fbcce09 Merge pull request #256 from Tencent-Hunyuan/feat/sdk-reembed
feat(sdk): re-embed vector stores and the graph into a new embedding space
2026-11-01 18:54:40 +08:00
alvinfeiandClaude Opus 5.5 302fd30af1 feat(sdk): re-embed vector stores and the graph into a new embedding space
Add hy_memory.reembed for switching the embedding model while writes are
paused. Source and target are named explicitly; a space's identity is the
model name plus the width.

- vector_space_names, plan_reembed, reembed_vector_space (resumable,
  cancellable at batch boundaries, widths validated before any upsert, target
  wins on existing ids, source never written), verify_reembed,
  drop_vector_space.
- scan_vector_spaces and diff_spaces for MemHub's startup reconcile of stray
  spaces.
- reembed_graph builds a new Kuzu DB at the target width, copies every node
  and edge, recomputes Memory/beh/Topic/HierarchyTopic embeddings and rebuilds
  the HNSW index; the source graph is opened read-only. verify_graph_reembed
  and drop_graph.

EmbedService.embed_batch now chunks to embedder.batch_size (default 64),
honours Retry-After on 429, takes a per-call max_retries, and rejects vectors
whose width differs from embedding_dims.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 18:33:52 +08:00
alvinfeiandClaude Opus 5.5 8d0a069950 feat(memhub): expose SDK search, layer, raw and profile-version surface
- P2-1 / R8: POST /api/v2/context/search and /api/v1/recall pass the public
  client.search knobs through, validated: min_score, schema_min_score,
  working_fact_min_score, working_experience_min_score, schema_limit,
  intention_limit, rerank, reader (legacy|hybrid), scene
  (chat|working|coding|all), search_mode (default|agentic),
  include_sources, exclude_sources, created_after, session_ids, agent_ids,
  workspace_id, branch (needs workspace_id), layers. The v2 ask routes take
  the floors and filters. Absent, every route makes the call it made before
  (relevance floor 0.35, scene normal, hybrid reader). A caller's floor is
  also enforced on what Core returns, per lane (the fused scene=all score is
  a rank, so the lane score is used): min_score=0.9 answers only >= 0.9.
  user_ids, embed_override, query_embedding and request_id are not exposed.
- P2-2 / R9: memory-library takes layers (or layer), statuses,
  exclude_sources, session_ids and agent_ids, repeated or comma separated;
  a layer filter lists the chat store and refuses scene=working|coding.
  New GET /api/v2/memory-library/layers counts memories per SDK layer.
  The memory graph takes layers, session_ids and exclude_sources.
- P2-4: /api/v2/raw lists layers=["l1_raw"] with statuses active+shadow (pro
  shadows a raw once extracted), filters by source, session_id, statuses
  and order, cuts after/before in Core, and returns each turn's messages
  with roles, a sha256 content_hash, status and created.
- P2-3: GET /api/v2/sources, POST /api/v2/sources/{id}/sync,
  POST /api/v2/import-jobs, GET /api/v2/import-jobs[/{id}] reuse the v1
  handlers on one user's lane (Hermes only for another user; another
  user's job answers 404).
- Profile: GET /api/v2/user-profile/versions (get_user_basic_info) returns
  the L0 version chains; memhub profile --versions prints them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 18:28:54 +08:00
feitianxiang cdb44658cf Merge pull request #255 from Tencent-Hunyuan/fix/upgrade-migration-issues
fix(memhub): upgrade migration without a model profile, tolerant readback, grant triggers after cutover, reads use the active embedding
2026-11-01 18:26:00 +08:00
alvinfeiandClaude Opus 5.5 2ab04291a9 fix(memhub): the embedding endpoint no longer follows the generation base_url
A remote embedding with no `embedding_base_url` resolved to the generation
`base_url` at runtime (`build_sdk_config`, `vector_collection_bases`, the
probe). The collection name hashes that endpoint, so a proxy or region edit
of the generation endpoint moved writes to a new, empty collection, and
providers without embeddings were sent embedding requests.

- `resolve_embedding_endpoint` no longer inherits; creating or updating a
  remote (non-local_onnx) embedding without an endpoint is refused with
  422 `embedding_base_url_required`. The runtime and probe read only
  `embedding_base_url`. The API key stays shared.
- Opening the profile store pins, once and idempotently, every stored
  remote embedding left without an endpoint to the one it resolved to
  (the generation base_url, or the provider default), so the collection
  name is byte-identical. `read_active_profile` applies the same rule for
  `sdk-migrate`, which may run before Core has opened the store.
- Web: settings write the prefilled endpoint instead of relying on the
  fallback; onboarding writes an empty custom field as the endpoint it
  means. CLI headless configs now get the 422 code.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 18:11:20 +08:00
alvinfeiandClaude Opus 5.5 3772a69c6b feat(sdk): store raw conversations already trimmed; history concatenates without re-trimming
Raw is kept forever (MemHub dropped its retention; lite stores nothing else),
so trim it once when it is written instead of on every history read.

- hy_memory/utils/raw_trim.py: one idempotent rule per speaker label.
  user/system kept; assistant and other speakers cut to 500 chars + "...";
  tool head/tail with coding.tool_result_max_bytes (default 2048).
- writer: L1_RAW (lite/pro/ultra) is flattened from trimmed messages and
  flagged custom.raw_trimmed; request.messages (extractor input) untouched.
- _get_recent_history concatenates flagged raws as stored; legacy unflagged
  raws are trimmed with the same rule on read.
- client: memory_history ADD record uses the same trimmed messages.
- coding preproc: tool/assistant truncation made idempotent so re-trimming
  coding_trace history no longer rewrites omitted-count markers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 18:02:40 +08:00
alvinfeiandClaude Opus 5.5 80d3b55723 fix(memhub): reads use the active profile's embedding space
The read handle was always built from `builtin_retrieval_snapshot()`, while
writes follow the active profile, so a custom embedding (2560 dims, say)
wrote `memhub_<sha>_2560` and every read opened the empty built-in 512-dim
collection.

`_sdk_client_handle` now returns an `ActiveProfileReadHandle`. It opens a
`retrieval_view` of the active profile: the profile's embedding fields with
the built-in generation placeholder, so a read never needs the generation
model; a remote OpenAI-compatible embedding gets the profile's shared key.
With no active profile it stays on the built-in space.

A profile change builds a new client and retires the old one, closed once no
task still uses it, as `SdkWriteCoordinator` does (#218). A use is the task
that asked for the client, so every caller (library, search, recall and its
raw fallback, graph, profile, raw, home summary, ask, context, multi-user
routes) keeps its shape; each raw-retention pass runs in its own task.

Refs #242

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 17:50:21 +08:00
alvinfeiandClaude Opus 5.5 b5ec0861f0 fix(memhub): migrate without a model profile and check readback against the pre-migration store
#234: with no active model profile the migration aborted in preflight with
`sdk_migration_model_profile_required`. No profile means no generation
model, not no embedding: the vectors are the built-in model's, the space the
Runtime reads with. The runner, and `sdk-migration-status`, now use
`builtin_retrieval_snapshot()` for reattach, the collection bases and the
readback client; `runtime_sdk_client` resolves no secret for a key-less
profile. When legacy memories have vectors only under another embedding
space (a profile since deleted), the preflight stops with
`sdk_migration_vector_profile_missing` before anything changes.

#229/#240: the readback required every confirmed governed chat id to read
back, but a 1.0.4 library can hold confirmed rows whose SDK node was merged
or deleted before the upgrade (658 confirmed, 657 readable, upgrade failed).
The ids in the vector store are now read before reattach; chat is held to
the confirmed ids present there, and the rest is only counted as
`chat_missing_before_migration`. A memory present before the migration but
unreadable after it still fails, and if none of the confirmed chats is in
the store none is excused.

Fixes #234, Fixes #229, Fixes #240

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 17:15:51 +08:00
alvinfeiandClaude Opus 5.5 47e938da1b fix(memhub): drop grant triggers that read a table the cutover dropped
The two `source_grants` triggers of the visibility cache select from
`governed_memory_evidence`, which the SDK cutover drops. SQLite resolves a
trigger body only when it fires, so every grant insert or update on a
migrated home failed with "no such table": revoke, connect and the
multi-user Hermes connect all answered 503.

`install_triggers` now installs a trigger only when every table its body
names exists, and drops an old copy otherwise. Startup on SDK authority
(which does not install the cache) and the cutover cleanup drop the stale
copies an earlier build left behind.

Fixes #233

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 16:20:34 +08:00
feitianxiang f5ea0af75d Merge pull request #251 from Tencent-Hunyuan/fix/hermes-injected-content
feat(memhub): land #250 on main — multi-user Runtime, hy-memory 2.0.9
2026-11-01 14:39:43 +08:00
alvinfeiandClaude Opus 5.5 1c2f48579f Merge main into fix/hermes-injected-content to land #250
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 14:33:00 +08:00
feitianxiang d29615abeb Merge pull request #250 from Tencent-Hunyuan/feat/memhub-multi-user
feat(memhub): multi-user Runtime — user_id on memory APIs and CLI, per-user Hermes profiles, hy-memory 2.0.9
2026-11-01 14:25:05 +08:00
feitianxiang 0965519e16 Merge pull request #249 from Tencent-Hunyuan/fix/hermes-injected-content
fix(memhub): drop Hermes gateway notices and unwrap relayed messages on import; no false 'damaged log' warning
2026-11-01 14:24:05 +08:00
alvinfeiandClaude Opus 5.5 7e9f0d98d8 fix(memhub): another user's memory id answers like a missing one
DELETE of another user's id answered 404 while a missing id answered 204,
so comparing the two told that the id existed. It now answers 204 and
deletes nothing; GET and PUT answer the same 404 as a missing id, and
receipt and addition reads already matched. A test compares every by-id
route's response for another user's id with a nonexistent id's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 10:23:10 +08:00
alvinfeiandClaude Opus 5.5 0ff4538f36 feat(memhub-cli): --user-id for memory commands, connect a Hermes profile
- Global --user-id <id> (default local) for memories list/show/add/
  add-status/update/delete, recall, profile and write submit/status; sent
  only when it is not the default, so default requests are unchanged.
- memhub sources connect hermes --user-id <id> --root <path> [--import]
  grants one Hermes profile to its own user; refused for other sources.
- memhub sources list shows the user of every grant (from
  GET /api/v1/source-grants, skipped on an older Core).
- Help catalog, contract tests and docs/cli/EXPERIENCE.md updated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 10:12:31 +08:00
alvinfeiandClaude Opus 5.5 a9709788d8 feat(memhub): multi-user Core — user_id on memory APIs, per-user Hermes grants
One Runtime now serves many user_ids. A user is the root that is read,
never a session or platform field; the App never sends user_id and keeps
reading and writing `local`, and only Hermes may hold another user's grant.

- R1: every memory read/write route takes an optional user_id (query for
  GET/DELETE, body for POST/PUT), default `local`, pattern
  ^[A-Za-z0-9._-]{1,64}$ (no `:`).
- R2: user_id reaches the SDK: Evidence, durable payload and client.add
  carry it; reads filter by user. get/update/delete/history and
  get_write_outcome take no user in hy-memory 2.0.9, so Core reads the
  memory first and answers 404 for another user's id; memory-write keys
  are namespaced per user and addition receipts are keyed by the user's
  Evidence id.
- R3: grants, cursors, jobs, live sync and the claim gate work per
  (user_id, source_id) through lane keys (`hermes` for local,
  `hermes@wife` otherwise). POST /api/v1/sources/connect takes user_id and
  root (Hermes only, 422 otherwise); revoke takes ?user_id; new
  GET /api/v1/source-grants lists every user's grants. Another user's
  grant syncs while it stands; the App's per-source switch stays local's.
- R4: a Hermes root scan never enters profiles/** (discovery, history and
  live sync share the same walk).
- R5: session keys, assembling rows, FIFO source keys and request ids are
  per user, so two users' turns never share a batch.
- R6: memory routes refuse undeclared query parameters and body fields
  with 422.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 10:12:31 +08:00
alvinfeiandClaude Opus 5.5 c1e5916b51 build(memhub): pin hy-memory 2.0.9
Core pins and locks the published hy-memory 2.0.9 from PyPI; the
version constant, README, architecture notes, SBOM, notices and their
tests follow. CHANGELOG records 2.0.9 as published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 09:12:21 +08:00
alvinfeiandClaude Opus 5.5 97de5480bd Merge github/main (hy-memory 2.0.9 release) into feat/memhub-multi-user
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 09:11:25 +08:00
feitianxiang 97d9309ca8 Merge pull request #248 from Tencent-Hunyuan/feat/sdk-2.0.9
fix(sdk): placeholder values never overwrite the basic profile (2.0.9)
2026-11-01 08:26:49 +08:00
alvinfeiandClaude Opus 5.5 b1b3e0a291 fix(sdk): drop the user-turn filter for basic_info
Owner decision on #248: the writer no longer filters basic_info against
the user turns. The placeholder guard (with healing on read) and the
prompt hard rules stay; the guard alone keeps an assistant quoting
"name is 用户" out of the profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 08:18:32 +08:00
alvinfeiandClaude Opus 5.5 53507c7abe fix(memhub): only call non-JSON diagnostic lines damaged in memhub logs
After an upgrade the diagnostic files still hold an older build's records.
They are valid JSON that the reader leaves out, but were counted as damaged,
so `memhub logs` warned "跳过 N 行损坏的诊断记录" and a support export
failed with diagnostic_export_invalid_lines. Count them apart as
`unrecognized_lines` (reported only in the JSON empty-state record) and
keep the warning for lines that are not JSON.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 05:10:19 +08:00
alvinfeiandClaude Opus 5.5 5a1d252787 fix(memhub): drop Hermes gateway notices and unwrap relayed messages on import
Hermes stores its gateway's own posts in `user` rows: finished background
process notices (often with the gateway's "not written by a human" note),
voice placeholders that only name an audio file, and "Reply with exactly:
PONG" probes. Relayed Telegram messages arrive wrapped in out-of-band
markers and a gateway-origin JSON block. All of it reached extraction as
the person's words.

The Hermes adapter now cleans `user` rows (state.db and session
transcripts) before evidence is built, so the content hash and dedup see
the cleaned text:
- skipped with new reason codes `injected_machine_notice`,
  `voice_placeholder` and `probe` (media-only rows as `no_text`), counted
  in `import.parse.issue_summary` like other parse skips;
- out-of-band markers, the gateway-origin block and the interrupted
  response label are removed, keeping only the person's message.

Assistant and tool rows are unchanged. Tests use real rows with the
Telegram ids and home paths replaced by fake values.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 05:10:19 +08:00
alvinfeiandClaude Opus 5.5 8895fea775 release: hy-memory 2.0.9
Placeholder values no longer overwrite the user's basic profile (L0). Not
yet published to PyPI.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 04:56:01 +08:00
alvinfeiandClaude Opus 5.5 78882f3300 fix(sdk): placeholder values never overwrite the basic profile
A weak extraction model filled basic_info with "用户", "The user" or "N/A"
when the user stated nothing, and each placeholder superseded the real L0
value. upsert_basic_profile now drops placeholder values and writes no new
version when nothing is left; profile assembly skips stored placeholders.
The writer keeps a basic_info field only when its value appears in a user
message of the request. The zh and en extraction prompts add a hard rule
and a small-talk counter-example.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 04:56:01 +08:00
feitianxiang 988667cd87 Merge pull request #224 from Tencent-Hunyuan/chore/build-36-docs
chore(release): bump MemHub build number to 36 for v1.0.5
2026-10-28 22:07:22 +08:00
alvinfeiandClaude Opus 5.5 72968f831f chore(release): bump MemHub build number to 36
The macOS v1.0.5 artifacts were built from release/memhub-build-36-main
(dbba403): main's code at a621262 with build 36. Main moves to the same
build so the Linux Runtimes built from the v1.0.5 tag match them. The
release docs name build 36.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 22:02:27 +08:00
feitianxiang af25f51f9e Merge pull request #223 from Tencent-Hunyuan/docs/release-1.0.5
docs: prepare MemHub v1.0.5 (news, build 35, macOS arm64 only)
2026-10-28 21:59:59 +08:00
alvinfeiandClaude Opus 5.5 b3f13e130f docs: MemHub 1.0.5 news, build 35, macOS arm64 only
The README news line announces v1.0.5 briefly; details live in the release
notes. The release facts said build 14 and promised a macOS Intel build that
this version does not ship. CHANGELOG's Unreleased section becomes MemHub
1.0.5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 21:58:28 +08:00
kristinhe1111 a621262fac Merge pull request #222 from Tencent-Hunyuan/fix/memory-detail-structure-and-stability
fix(memhub): stabilize memory detail navigation
2026-10-28 20:44:13 +08:00
kristinhe e027635d3f Merge remote-tracking branch 'origin/main' into fix/memory-detail-structure-and-stability
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	memhub/apps/web/src/MemoryMap.test.tsx
2026-10-28 20:36:06 +08:00
feitianxiang 4cb33b7108 Merge pull request #217 from Tencent-Hunyuan/fix/memory-source-filter-sync
fix(memhub): 同步记忆筛选范围并优化首次启动状态
2026-10-28 20:32:24 +08:00
alvinfeiandClaude Opus 5.5 b50c9f33eb fix(memhub): give the map its own graph-loading state
The map hid the previous scope's graph by passing stale into the library
view, which also blanked the list status, pagination and empty states.
A separate graphLoading flag now hides the old graph and shows
"正在读取记忆地图…" while stale keeps its original meaning and text.
The client-side revoked-source graph filter is dropped to match Core.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:24:39 +08:00
alvinfeiandClaude Opus 5.5 545b7d38bf fix(memhub): keep revoked-source memories on the memory map
Memories already distilled from a revoked source stay visible, so the
graph endpoint no longer checks revocation before or after the SDK read,
no longer filters memories by revoked source and reports the SDK total
again. The map and the list now agree. The scene="all" and
statuses=["active"] scope stays.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:24:39 +08:00
feitianxiang 5f78d67363 Merge pull request #221 from Tencent-Hunyuan/fix/linux-lifecycle-queue-check
fix(memhub): Linux lifecycle checks the write queue, not recall without a model
2026-10-28 20:17:45 +08:00
alvinfei 33d7713d4e Merge remote-tracking branch 'github/main' into fix/memory-source-filter-sync 2026-10-28 20:14:52 +08:00
kristinheandCursor 68a4b546f9 fix(memhub): stabilize memory detail navigation
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 20:11:12 +08:00
kristinhe cbcc50f35e Merge remote-tracking branch 'origin/main' into fix/memory-source-filter-sync
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	memhub/apps/web/src/MemorySpace.tsx
2026-10-28 20:09:35 +08:00
alvinfeiandClaude Opus 5.5 528d2c9fd7 fix(memhub): Linux lifecycle checks the write queue, not recall without a model
The lifecycle verification imported a conversation on a runner with no
generation model and then expected recall to find it. Under SDK authority
nothing reaches the SDK, not even its raw layer, until a model is active,
so the check could never pass and blocked the Linux release archive.

It now asserts what the runner can prove: the imported and appended turns
reach Core's durable write queue, and survive an idempotent upgrade.
docs/TODO-raw-recall-without-model.md records the product gap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:05:44 +08:00
kristinhe1111 0eb9a2844e Merge pull request #216 from Tencent-Hunyuan/fix/onboarding-language-menu-clipping
fix(memhub): keep option menus inside viewport
2026-10-28 20:01:37 +08:00
feitianxiang 8e56bf7a37 Merge pull request #218 from Tencent-Hunyuan/fix/sdk-client-race-and-unknown-jobs
fix(memhub): serialize SDK client startup, log failure frames, settle pre-upgrade unknown writes
2026-10-28 19:49:24 +08:00
alvinfeiandClaude Opus 5.5 352fffb3da fix(memhub): serialize SDK client startup, log failure frames, settle pre-upgrade unknown writes
- hy_sdk_runtime: construct one HyMemoryClient at a time per home. Two
  first-time Chroma opens on a fresh vector dir race Chroma's unlocked
  System cache; on a clean install the drain's and the UI's clients both
  failed and the first write job was lost as unknown.
- sdk_write_processor: a client that fails to construct raises
  hy_memory_client_unavailable before HY is invoked, which parks the job
  (waiting_for_model) instead of leaving it unknown.
- sdk_write_coordinator: a retired profile's handle closes only after its
  last in-flight use (defensive; a profile revision change used to close
  the client under a running write).
- durable_queue: durable_write_job_failed/parked log the exception chain as
  file:line:function frames, never the message.
- sdk_migration_runner: the cutover settles pre-cutover unknown (and
  invoked running) write jobs as cancelled/abandoned_at_cutover and counts
  them as abandoned.unknown_write_jobs; never retried.
- durable_queue/api: retry-unknown and the reconcile helpers no longer
  assume the dropped legacy task tables; the endpoint uses the startup
  queue so it does not recreate memory_write_job_evidence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:39:13 +08:00
kristinheandCursor 5103492762 fix(memhub): harden memory overview recovery
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 19:26:01 +08:00
kristinheandCursor 082aeee1f2 fix(memhub): align graph scope and startup states
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 19:16:48 +08:00
kristinheandCursor 201901d35e fix(memhub): keep option menus inside viewport
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-28 19:04:11 +08:00
feitianxiang 42c0ca1d14 Merge pull request #215 from Tencent-Hunyuan/feat/memhub-sdk-logs-quiet
feat(memhub): HY pipeline log off, SDK text logs only in hy_memory.log (hy-memory 2.0.8)
2026-10-28 15:46:40 +08:00
alvinfeiandClaude Opus 5.5 0ca2e56ccc chore(release): bump MemHub build number to 35
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:38:05 +08:00
alvinfeiandClaude Opus 5.5 f328cdbf8e docs: record hy-memory 2.0.8 as published
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:38:05 +08:00
alvinfeiandClaude Opus 5.5 3f57a3c978 build(memhub): lock hy-memory 2.0.8 from PyPI
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:37:51 +08:00
alvinfeiandClaude Opus 5.5 19a7af0e71 feat(memhub): remove the HY pipeline section from the Debug panel
MemHub no longer writes the HY pipeline JSONL, so the Desktop diagnostics
reader stops scanning hy-memory/governed-v2/logs and the renderer drops the
"hy-memory" service and "pipeline" source. A test covers leftover pipeline
files and hy_memory.log staying out of the panel.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:36:50 +08:00
alvinfeiandClaude Opus 5.5 5791c0d1e7 feat(memhub): HY pipeline log off, SDK text logs only in hy_memory.log
- MEMORY_PIPELINE_LOG_ENABLED and MEMORY_PIPELINE_TRACE_ENABLED are false
  at every diagnostic level, both at startup and on a live level change.
  MEMORY_LOG_LLM_CONTENT keeps its debug-only setting: it still decides
  whether HY's DEBUG lines in hy_memory.log carry full prompts/responses.
- After each HyMemoryClient construction (success or failure), Core drops
  the SDK's stdout/stderr StreamHandlers, which under launchd duplicated
  every SDK line into service.log/service-error.log, and forwards SDK
  WARNING+ to Core's core.log handler so the Debug panel still shows SDK
  problems.
- A live level change also sets the level of HY's hy_memory.log handler,
  which the SDK otherwise fixes at the level of the first client.
- log_file.py no longer claims core.log covers the HY SDK at every level.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:36:50 +08:00
alvinfeiandClaude Opus 5.5 bf92105210 build(memhub): pin hy-memory 2.0.8
2.0.8 adds MEMORY_PIPELINE_LOG_ENABLED and keeps conversation text out of
INFO logs. apps/core/uv.lock is not updated: 2.0.8 is not on PyPI yet, so
the lock is regenerated (uv lock --directory apps/core) once it is
published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:36:50 +08:00
feitianxiang 3387247842 Merge pull request #214 from Tencent-Hunyuan/feat/sdk-pipeline-log-switch
feat(sdk): switch to turn the pipeline log off, keep conversation text out of INFO logs (2.0.8)
2026-10-28 15:34:41 +08:00
alvinfeiandClaude Opus 5.5 791285f4c9 fix(sdk): keep model output out of the reconcile parse-failure warning
The final RECONCILE JSON parse failure warning embedded the raw response
whenever MEMORY_LOG_LLM_CONTENT was on. It now reports the length; the raw
output is already logged per attempt at DEBUG.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:06:09 +08:00
alvinfeiandClaude Opus 5.5 dc762df78b feat(sdk): switch to turn the pipeline log off, keep conversation text out of INFO logs (2.0.8)
MEMORY_PIPELINE_LOG_ENABLED (default true) turns off the pipeline JSONL
log. With the DB trace also off, store_pipeline_log becomes a no-op that
does no timing, redaction, background work or database write. The init line
states the real on/off state.

Lines that carried conversation or memory text at INFO/WARNING now log ids,
counts and lengths; the text goes to DEBUG: [add] input, search queries and
rewrites, coding tasks and fact statements/subjects, reconcile op detail,
extracted info, embedding inputs and unparsable model output.

Bump hy-memory to 2.0.8 (not yet published).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:03:21 +08:00
feitianxiang 8277a72aea Merge pull request #206 from rahul05ranjan/fix/runtime-stale-lease-boot-clean
fix(runtime): 清理早于本次开机的服务锁,异常关机后不再崩溃循环
2026-10-28 14:42:46 +08:00
feitianxiang 5eed0f81fb Merge pull request #200 from u7-u7/fix/literal-special-token-counting
fix(tokens): count literal special-token spellings as ordinary text
2026-10-28 14:36:16 +08:00
feitianxiang 59ca5afca6 Merge pull request #199 from u7-u7/fix/guarded-thread-context
fix(async): preserve context in guarded thread calls
2026-10-28 14:36:11 +08:00
feitianxiang 2648012ad5 Merge pull request #203 from u7-u7/fix/migration-log-write-order
fix(memhub): preserve migration log write order
2026-10-28 14:36:05 +08:00
alvinfeiandClaude Opus 5.5 63a80092c2 fix(runtime): identify stale leases by boot id and publish them atomically
Comparing the lock directory's mtime with Date.now() - os.uptime() breaks
when the wall clock jumps forward after a lease is taken (no RTC, VM
resume, NTP or manual correction): a live lease looks older than the boot
and is cleared, so a second Runtime can open the same data dir.

owner.json now records the kernel boot identity (kern.bootsessionuuid on
macOS, /proc/sys/kernel/random/boot_id on Linux, null elsewhere), read
once per process. removeStaleLease clears a lease whose boot_id differs
from the current one, which still fixes the recycled-PID crash loop of
#184 without consulting the clock; when either id is unknown the PID
probe decides as before. bootTime, BOOT_SLACK_MS and defaultBootTime are
gone; the boot id source is injectable via RuntimeServiceOptions.bootId.

The lease is published in one step: owner.json is written into a
runtime-service.lock.tmp-<uuid> staging dir that is then renamed onto the
lock, so a power cut can no longer leave a lock without an owner. Because
rename() replaces an existing empty directory, an existing lock of any
kind is reported as taken first. releaseLease also removes the lock with
a single rename, so this version never shows an owner-less lock; staging
and cleared-lease debris is swept once the lease is held.

An owner-less lock can therefore only come from an older version. It is
cleared when its mtime predates the kernel-reported boot time
(kern.boottime, /proc/stat btime) and otherwise keeps failing as
runtime_service_lock_invalid, as before.

Refs #184

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 14:33:33 +08:00
feitianxiang 6ba49bf112 Merge pull request #209 from Tencent-Hunyuan/fix/memory-map-node-navigation
fix(memhub): open source memories from map nodes
2026-10-28 14:10:28 +08:00
feitianxiang cf410aa9cf Merge pull request #210 from Tencent-Hunyuan/fix/harness-refresh-on-upgrade
fix(memhub): refresh harness skills and MCP entry after an upgrade
2026-10-28 14:08:01 +08:00
alvinfeiandClaude Opus 5.5 30c0b1d5ff fix(memhub): refresh harness skills and MCP entry after an upgrade
After an upgrade, hosts kept the call-rule skill and MCP entry an older
release wrote, so models were still told to call MemHub_ask and the other
removed tools, and WorkBuddy kept listing them from its tool-list cache.

- Stamp every managed MCP entry with ONE_MEMORY_MCP_TOOLSET, a tool
  contract version kept beside the tool list (mcp-toolset.ts). WorkBuddy
  fingerprints the stdio transport, env included, so a new stamp makes it
  restart the server and drop the cached list. An entry without the
  current stamp reads as stale in skill drift; verify still passes it.
- HostPermissions.refreshInstalledHosts runs the existing repoint and
  stale-sync planners for every host this data dir installed, leaves
  user-edited skills alone, emits allowlisted host_refresh.* diagnostics
  (host ids and counts only), and records the build it completed in
  host-permissions/host-refresh.json so it runs once per build.
- The Runtime calls it as it starts (covering App and `memhub upgrade`),
  and Desktop calls it after an embedded Runtime is up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:58:21 +08:00
feitianxiang 1086f4d582 Merge pull request #208 from Tencent-Hunyuan/feat/memhub-experience-memory-at
fix(memhub): experiences show their conversation time (hy-memory 2.0.7)
2026-10-28 11:50:41 +08:00
alvinfeiandClaude Opus 5.5 32539e7a82 chore(release): bump MemHub build number to 34
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:44:42 +08:00
alvinfeiandClaude Opus 5.5 cfae020a31 test(memhub): install docs name hy-memory 2.0.7
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:44:42 +08:00
alvinfeiandClaude Opus 5.5 5d51513ec8 docs: record hy-memory 2.0.7 as published
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:37:30 +08:00
alvinfeiandClaude Opus 5.5 9db5b49ccb build(memhub): lock hy-memory 2.0.7 from PyPI
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:37:23 +08:00
alvinfeiandClaude Opus 5.5 cb7df2a022 fix(memhub): metadata repair adds the SDK's memory_at column itself
The repair wrote memory_at only when the coding store already had the
column, which depended on the cutover readback having opened the store
through a 2.0.7 client first. On a migrated 1.0.4 home it now brings each
coding.sqlite3 / coding_cv.sqlite3 to the installed SDK's schema before
repairing, by running CodingMemoryStore's own SQLite schema step (its
initialize() would also open Chroma, which must stay closed offline). The
SDK's DDL is not repeated here.

The test now starts from a 2.0.6-shaped store without the column and
expects the bundle's conversation time in it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:37:13 +08:00
alvinfeiandClaude Opus 5.5 812d1c3846 build(memhub): pin hy-memory 2.0.7
2.0.7 stores and returns the conversation time of working/coding
experiences. apps/core/uv.lock is not updated: 2.0.7 is not on PyPI yet,
so the lock is regenerated (uv lock --directory apps/core) once it is
published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:36:56 +08:00
alvinfeiandClaude Opus 5.5 c8b660651e fix(memhub): experiences show their conversation time
After the SDK upgrade 124 working/coding experiences read 时间未知: the SDK
never stored their conversation time. hy-memory 2.0.7 stores memory_at for
experiences and facts; Core now maps it from list, get and search, so the
memory library, detail, map list and home summary carry it.

An experience stored before 2.0.7 has none. The list labels its write time
写入于 and the detail 写入时间, instead of 时间未知 or calling it an import.

Metadata repair also writes the conversation time recovered from the
migration bundle into coding_memory_meta / working_fact when the store has
the memory_at column. The repair version is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:36:56 +08:00
feitianxiang bdd2d9cf84 Merge pull request #207 from Tencent-Hunyuan/feat/sdk-coding-memory-at
feat(sdk): coding and working experiences keep their conversation time (2.0.7)
2026-10-28 11:33:29 +08:00
alvinfeiandClaude Opus 5.5 fc2e51a8cc release: hy-memory 2.0.7
Coding and working experiences and facts keep their conversation time
(memory_at) and return it from list, get and search. Not yet published to
PyPI.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:09:54 +08:00
alvinfeiandClaude Opus 5.5 d4fca0639e feat(sdk): coding and working experiences keep their conversation time
client.add passed memory_at only to the chat path; the productivity writer
dropped it and coding_memory_meta / working_fact had no column for it, so
every experience had only its write time and MemHub showed it as 时间未知.

Both tables gain a nullable memory_at column (epoch seconds), added to
existing coding/working databases on open. The writer stamps it on drafts
and new facts, defaulting to the request time as chat does; an UPDATE keeps
the later of the stored and incoming times. list_memories (coding, working,
all), get and search return memory_at in chat's format.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:09:54 +08:00
Rahul Ranjanandrahul05ranjan a17230dadb fix(runtime): clear leases that predate the current boot
A forced power-off leaves runtime-service.lock behind with no chance to
release it (#184). After the reboot the liveness probe cannot tell the
dead owner from the process that reused its PID, so launchd's KeepAlive
restarts the wrapper into runtime_service_already_running without end;
and a power cut inside the mkdir -> privateWrite window leaves the lock
without owner.json at all, which fails as runtime_service_lock_invalid
forever.

A lease created before the current boot can never have a living owner:
every process from that boot is gone. removeStaleLease now compares the
lock directory's mtime with the boot time (os.uptime(), injectable via
bootTime for tests) and clears such leases before it ever trusts owner
or PID. A two-minute slack absorbs forward clock steps taken since
boot, so a lease taken right after boot is not mistaken for stale.
Same-boot leases keep the old behavior exactly: live owner still
refused, dead owner still reclaimed, corrupt same-boot lease still
refused.

Fixes #184

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
2026-10-28 01:00:05 +05:30
u7-u7 d76bfa6474 fix(memhub): preserve migration log write order 2026-10-28 00:49:54 +08:00
u7-u7 e9698a293c fix(tokens): count literal special-token spellings as ordinary text 2026-10-28 00:28:14 +08:00
u7-u7 6718ba8bdf fix(async): preserve context in guarded thread calls 2026-10-27 10:41:13 +08:00
221 changed files with 20284 additions and 1452 deletions

No files matched your search

+1
View File
@@ -40,6 +40,7 @@ MEMORY_EMBEDDING_DIMS=1024
# MEMORY_EMBEDDER_TIMEOUT=15
# MEMORY_EMBEDDER_MAX_RETRIES=2
# MEMORY_EMBEDDER_RETRY_DELAY=0.3
# MEMORY_EMBEDDER_BATCH_SIZE=64
# MEMORY_EMBEDDER_EXTRA_HEADERS={}
# MEMORY_EMBEDDER_EXTRA_BODY={}
+154
View File
@@ -4,6 +4,99 @@ Hy Memory and MemHub are versioned and released independently.
## Unreleased
## Hy Memory 2.0.10 — 2026-11-01
- New `hy_memory.reembed` module for switching the embedding
model. Writers must be paused for the whole run. Nothing is derived from
the provider or endpoint: source and target are named explicitly, and the
identity of a space is the model name plus the width (`space_identity`).
Only Chroma and Kuzu are supported.
- `vector_space_names(config, base_name=None)` returns every physical
collection a client opens for the six vector roles: memory, entities,
coding keys and working facts, each in the working and coding variants.
- `reembed_vector_space(source, target, embedding=...)` pages every raw
source collection, covering all users and all layers including L1_RAW. It
embeds the text the write path embedded (`reembed_text`: the canonical
state's `state_retrieval_text`, a schema's `retrieval_text`, otherwise the
document). It then writes each record into the target with the same id,
document and metadata. Each batch's widths are checked before anything is
written. A cancel stops at a batch boundary (`ReembedCancelled`). A rerun
skips the ids the target already holds, so it resumes, and a merge never
overwrites the target. The source is never written.
- `plan_reembed`, `verify_reembed` and `drop_vector_space` complete the
vector side. `verify_reembed` checks ids, metadata, per-user counts and
the width, and runs a self-recall sample. `drop_vector_space` refuses
collections that a live client in this process holds open.
- `scan_vector_spaces(persist_directory)` is read-only. It lists every
space on disk with its roles, counts, stored width, per-user counts and
the models recorded in metadata. `diff_spaces(source, target)` returns the
ids that the source holds and the target lacks, whatever their status.
- `reembed_graph(source_path, target_path, embedding=..., target_dims=...)`
builds a new Kuzu graph at the new width. It copies every node and edge
and recomputes `Memory.embedding`, `Memory.beh_embedding`, and the Topic
and HierarchyTopic embeddings, then rebuilds the HNSW index. The source
graph is opened read-only and kept for rollback. A run that fails or is
cancelled leaves no target behind, and a rerun starts over.
`verify_graph_reembed` and `drop_graph` complete the graph side.
- `EmbedService.embed_batch` now splits large batches into
requests of `embedder.batch_size` texts each (`MEMORY_EMBEDDER_BATCH_SIZE`,
default 64); before, it sent them all in one request. On a 429 it waits for
the `Retry-After` time plus jitter, capped at 60 s. It takes a per-call
`max_retries`. A returned vector whose width differs from the configured
`embedding_dims` now raises `EmbeddingDimensionMismatchError` and is not
retried.
- Raw conversations are stored already trimmed. MemHub now keeps every raw
forever and lite mode stores nothing else, so tool output and long pasted
logs no longer stay in full. One rule (`hy_memory/utils/raw_trim.py`) is
applied to each message when the raw is written:
- `user` and `system` turns are kept in full;
- `assistant` turns and other speakers (`[John]:`) longer than 500
characters, after stripping, are cut to the first 500 plus `...`. This is
the limit history injection used before;
- `tool` turns keep their head and tail, and the middle becomes
`[...N bytes omitted by truncate...]`, with the coding limit
`coding.tool_result_max_bytes` (`MEMORY_CODING_TOOL_RESULT_MAX_BYTES`,
default 2048).
The rule is idempotent. It covers chat `L1_RAW` in lite, pro and ultra (lite
keeps tool turns, because it has no coding route) and the `memory_history`
ADD record. The coding/working `coding_trace` was already trimmed when it
was written. Its tool and assistant truncations are now idempotent, so
reading the history back no longer rewrites the omitted-count markers. The
text embedded for `L1_RAW` is the trimmed text. New raws carry
`custom.raw_trimmed=true`.
- Extractor history ("last k messages") now joins the stored raws directly
and does not trim them again. Raws written before this change have no
`raw_trimmed` flag, so they are still trimmed with the same rule when they
are read. The extractor still receives the current batch in full (tool
turns are still removed on the chat route).
## MemHub 1.0.5 — 2026-10-28
Build 36, on `hy-memory==2.0.8`. Linux x64/arm64 Runtime and macOS arm64
App and Runtime. Upgrading from 1.0.4 migrates the existing library to SDK
authority on first launch.
- MemHub turns off the HY pipeline log and trace at every diagnostic level
(hy-memory 2.0.8 `MEMORY_PIPELINE_LOG_ENABLED=false`,
`MEMORY_PIPELINE_TRACE_ENABLED=false`), so the per-step JSONL files under
`hy-memory/governed-v2/logs/` and the `pipeline_logs` rows stop growing.
The Debug panel no longer shows the HY pipeline section. Existing files are
left in place.
- HY SDK text logs are written once, to `hy_memory.log`. They are no longer
copied to `service.log` and `service-error.log` through the SDK's
stdout/stderr streams. SDK warnings and errors also go to `core.log`, so the
Debug panel still shows them. A live switch of the diagnostic level now also
changes the level of `hy_memory.log`.
- MemHub shows when working and coding experiences were talked about. After
the SDK upgrade every one of them read "时间未知" (time unknown) because the
SDK never stored their conversation time; hy-memory 2.0.7 stores it and
Core now passes it through the memory list, detail, search and home
summary. An experience stored before 2.0.7 shows its write time, labelled
"写入于" / "写入时间", instead of "时间未知". Metadata repair also writes
the conversation time it recovers from the migration bundle into the SDK's
coding/working stores when they already have the column.
- The SDK migration no longer fails its first run on a home with legacy
vectors: reattaching them left a Chroma instance open with its own settings,
and the readback that followed could not open the same store
@@ -68,6 +161,67 @@ Hy Memory and MemHub are versioned and released independently.
`hy_node_permissions` proof table. Evidence, governed memories, model
profiles, SDK stores, and source/host grants are preserved.
## Hy Memory 2.0.9 — 2026-11-01
- Published
[`hy-memory==2.0.9`](https://pypi.org/project/hy-memory/2.0.9/).
- Placeholder values no longer overwrite the user's basic profile (L0). A
small extraction model filled `basic_info` with `{"name": "用户"}`,
`The user` or `N/A` when the user had said nothing about themselves, and
each placeholder superseded the real value, so the L0 chain read
费天翔 → Fei → 用户 → The user. Two defences:
- `upsert_basic_profile` drops placeholder values: empty, `用户`,
`the user`, `user`, `n/a`, `na`, `none`, `null`, `unknown`, `未提供`,
`未知`, `不详`, `-`, `?` and the field's own name, compared
case-insensitively with whitespace and punctuation stripped. When nothing
is left it writes no new version and no new L0 head. Placeholders already
stored in the chain no longer override earlier real values when the
profile is assembled, so `get_user_profile` of an affected user shows the
last real value again without any migration. This also stops an
assistant quoting an old memory ("The user's name is 用户.") from being
written back.
- The extraction prompts (zh and en) state the rule: only what the user
said in a `user` turn, never the field name or a placeholder, `{}`
otherwise, with a small-talk counter-example.
## Hy Memory 2.0.8 — 2026-10-28
- Published
[`hy-memory==2.0.8`](https://pypi.org/project/hy-memory/2.0.8/).
- New `MEMORY_PIPELINE_LOG_ENABLED` (default `true`) turns the pipeline JSONL
log (`{MEMORY_LOG_DIR}/<subdir>/<date>.log`) off. With `false` no JSONL
file or directory is written. When `MEMORY_PIPELINE_TRACE_ENABLED` is also
`false`, `store_pipeline_log` becomes a no-op that returns `True` at once:
no timing, redaction, background task or database write. The init log line
now states whether the JSONL log is on or off.
- Conversation and memory text no longer appears in INFO or WARNING logs. The
`[add]` input, search queries and rewrites, coding experience tasks, fact
statements and subjects, reconcile op details, extracted info, embedding
inputs and unparsable model output moved to DEBUG. INFO lines keep ids,
counts and character lengths; parse-failure warnings remain but report only
the length of the output, e.g. `failed to parse output (812 chars)`. The
`[embed]` INFO line now carries `request.input_chars` instead of
`request.input`. The `JSON_PARSE_FAILED` error message from extraction no
longer embeds the model output either.
## Hy Memory 2.0.7 — 2026-10-28
- Published
[`hy-memory==2.0.7`](https://pypi.org/project/hy-memory/2.0.7/).
- Coding and working experiences and facts now keep their conversation time.
`add(..., memory_at=...)` reached only the chat path before; the
productivity path dropped it, so every experience had only its write time
and MemHub showed them as "时间未知" (time unknown). `coding_memory_meta`
and `working_fact` gain a nullable `memory_at` column (epoch seconds) in
both the working and coding stores; existing databases get it on open, and
their old rows stay `None`. Without `memory_at` a write uses the time the
request arrived, as chat does. When an experience is updated by a later
conversation it keeps the later of the two times.
- `list_memories` (`scene="coding"`, `"working"` and `"all"`), `get` and
`search` return `memory_at` for experiences and facts in the same field and
format as chat (epoch seconds, `None` when unknown). Experience search
results now also carry `created_at`.
## Hy Memory 2.0.6 — 2026-10-27
- Published
+1
View File
@@ -33,6 +33,7 @@ English | [中文](README.zh.md)
## Latest News
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** quantized **FP8** (**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**) and **GGUF** (**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**) versions of **Hy-Memory-30B-A3B** on Hugging Face!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **Released [MemHub v1.0.5](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.5)** with Chinese, English and Spanish interfaces, one-time pairing codes for more custom Harnesses, and smoother browsing, search, startup and recovery, on **`hy-memory==2.0.8`**. See the release notes for details.
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **Released [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** and **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**, with a signed macOS desktop app and verified macOS/Linux Runtimes!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.4`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.4/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** on Hugging Face along with its [technical report](https://arxiv.org/abs/2610.22196) on attributed reinforcement learning for long-term memory writing!
+1
View File
@@ -33,6 +33,7 @@
## 最新动态
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **Hy-Memory-30B-A3B** 的 **FP8**(**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**)与 **GGUF**(**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**)量化版本模型!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **正式发布 [MemHub v1.0.5](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.5)**:支持中文、英文和西班牙语界面,可通过一次性配对码连接更多自定义 Harness,并优化了记忆浏览、搜索、启动和异常恢复;底层使用 **`hy-memory==2.0.8`**。详见发布说明。
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **正式发布 [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** 与 **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**,提供已签名的 macOS 桌面应用及经过验证的 macOS/Linux Runtime!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.4`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.4/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** 长期记忆写入模型及[技术报告](https://arxiv.org/abs/2610.22196)!
+2
View File
@@ -48,6 +48,7 @@
| `MEMORY_EMBEDDER_TIMEOUT` | `15` | 单次超时,秒 |
| `MEMORY_EMBEDDER_MAX_RETRIES` | `2` | 最大重试次数 |
| `MEMORY_EMBEDDER_RETRY_DELAY` | `0.3` | 重试间隔,秒 |
| `MEMORY_EMBEDDER_BATCH_SIZE` | `64` | `embed_batch` 单次请求最多发送的文本条数,超出按此切块 |
| `MEMORY_EMBEDDER_EXTRA_HEADERS` | — | JSON object,附加 HTTP headers |
| `MEMORY_EMBEDDER_EXTRA_BODY` | — | JSON object,附加请求 body |
@@ -191,6 +192,7 @@ SDK 不设置 Rerank Search 并发上限。开启后模型在 `HyMemoryClient`
| `MEMORY_LOG_FORMAT` | `text` | `text` / `json` |
| `MEMORY_LOG_DIR` | — | 统一日志根目录 |
| `MEMORY_TRACE_LOG_DIR` | — | JSONL 日志目录的兼容旧名;优先使用 `MEMORY_LOG_DIR` |
| `MEMORY_PIPELINE_LOG_ENABLED` | `true` | 是否写 pipeline JSONL 日志(`{MEMORY_LOG_DIR}/<subdir>/<date>.log`);`false` 完全不写 |
| `MEMORY_PIPELINE_TRACE_ENABLED` | `true` | 是否写入 cache 的 pipeline trace 表 |
| `MEMORY_MEMORY_OPERATIONS_ENABLED` | `false` | 是否写入 memory operation 审计记录 |
| `MEMORY_HISTORY_ENABLE` | `true` | 是否启用历史记录 |
+53
View File
@@ -55,6 +55,33 @@ except Exception:
from .client import HyMemoryClient
from .inspector import MemoryInspector
from .runtime import SharedRuntime
from .reembed import (
EmbeddingSpaceIdentity,
GraphReembedPlan,
GraphReembedProgress,
GraphReembedReport,
GraphVerification,
ReembedCancelled,
ReembedPlan,
ReembedProgress,
ReembedReport,
ReembedVerification,
ScannedSpace,
SpaceDiff,
VectorSpace,
diff_spaces,
drop_graph,
drop_vector_space,
plan_reembed,
reembed_graph,
reembed_text,
reembed_vector_space,
scan_vector_spaces,
space_identity,
vector_space_names,
verify_graph_reembed,
verify_reembed,
)
from .snapshot import (
DataPlaneRestore,
DataPlaneSnapshot,
@@ -129,6 +156,32 @@ __all__ = [
"DataPlaneSnapshotRejected",
"restore_local_data_plane",
"snapshot_local_data_plane",
# 换 embedding 模型:把向量库和图重新 embed 进新的 embedding 空间
"EmbeddingSpaceIdentity",
"GraphReembedPlan",
"GraphReembedProgress",
"GraphReembedReport",
"GraphVerification",
"ReembedCancelled",
"ReembedPlan",
"ReembedProgress",
"ReembedReport",
"ReembedVerification",
"ScannedSpace",
"SpaceDiff",
"VectorSpace",
"diff_spaces",
"drop_graph",
"drop_vector_space",
"plan_reembed",
"reembed_graph",
"reembed_text",
"reembed_vector_space",
"scan_vector_spaces",
"space_identity",
"vector_space_names",
"verify_graph_reembed",
"verify_reembed",
# 配置
"MemoryConfig",
# 公共扩展协议 / 精确读取类型
+37 -3
View File
@@ -682,6 +682,23 @@ Output:
```
John's occupation and location are HIS attributes, so they go into `memory` with "John" as the subject. They must NEVER land in `basic_info.occupation` / `basic_info.location` — `basic_info` is exclusively the user's own profile. Only `name` (Tim, from the user message's `name` field) belongs there.
## Example 11 — Counter-example: small talk has no basic_info
New Messages:
[{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello! The user's name is The user. How can I help?"}]
Memory Date: 2025-09-03
Output:
```json
{
"memory": [],
"intentions": [],
"basic_info": {}
}
```
The user stated nothing about themselves. The assistant's line is not a user turn, and "The user" is a placeholder, not a name. Output `{}` — never `{"name": "The user"}` or `{"name": "user"}`.
---
"""
@@ -749,6 +766,14 @@ Rules:
fields (e.g. name, age, location, occupation, employer) **explicitly stated**
in the NEW user messages. Never infer, never invent. If no field is explicitly
stated, output `{{}}`.
* HARD RULE: fill a field ONLY if the user stated it in a `user` turn.
Anything said only by the assistant (including the assistant quoting an
earlier memory such as "The user's name is X") does NOT count. If nothing
qualifies, output `"basic_info": {{}}`.
* NEVER write the field name itself or a placeholder as a value ("user",
"The user", "用户", "name", "N/A", "unknown", "none", "-"). Omit the field.
* Negative example: small talk such as `[user]: Hi` / `[assistant]: How can
I help?`, or `[user]: Reply with exactly: PONG` → `"basic_info": {{}}`.
* Exception for `name`: the `name` field on the user's own message, or a leading
line "The user's name is X" / "用户名叫X", is an authoritative source for
`name` — even though the user never said it out loud.
@@ -817,6 +842,14 @@ Rules:
fields (e.g. name, age, location, occupation, employer) **explicitly stated**
in the NEW user messages. Never infer, never invent. If no field is explicitly
stated, output `{{}}`.
* HARD RULE: fill a field ONLY if the user stated it in a `user` turn.
Anything said only by the assistant (including the assistant quoting an
earlier memory such as "The user's name is X") does NOT count. If nothing
qualifies, output `"basic_info": {{}}`.
* NEVER write the field name itself or a placeholder as a value ("user",
"The user", "用户", "name", "N/A", "unknown", "none", "-"). Omit the field.
* Negative example: small talk such as `[user]: Hi` / `[assistant]: How can
I help?`, or `[user]: Reply with exactly: PONG` → `"basic_info": {{}}`.
* Exception for `name`: the `name` field on the user's own message, or a leading
line "The user's name is X" / "用户名叫X", is an authoritative source for
`name` — even though the user never said it out loud.
@@ -1177,7 +1210,7 @@ class Extractor:
if not tc_id:
logger.error(
f"[extractor] tool_call_id is empty! "
f"raw tc: {json.dumps(tc_raw, ensure_ascii=False, default=str)[:500]}"
f"raw_keys={list(tc_raw.keys())}"
)
try:
t_res = await tool_registry.dispatch(
@@ -1311,11 +1344,12 @@ class Extractor:
if parsed is None:
if not all_tool_calls:
logger.warning(
f"[extractor] JSON parse failed: {raw_text[:200]}"
f"[extractor] JSON parse failed ({len(raw_text)} chars)"
)
logger.debug(f"[extractor] unparsable output: {raw_text[:200]!r}")
return ExtractResult(
success=False,
error=f"JSON parse failed: {raw_text[:200]}",
error=f"JSON parse failed ({len(raw_text)} chars)",
error_code="JSON_PARSE_FAILED",
raw_response=raw_text,
tokens_used=total_tokens,
+1 -2
View File
@@ -129,8 +129,7 @@ def _parse_tool_calls_list(raw_calls: list, _json) -> List[Dict[str, Any]]:
tc_id = tc.get("id") or ""
if not tc_id:
logger.error(
f"[llm] tool_call missing 'id'! raw: "
f"{_json.dumps(tc, ensure_ascii=False, default=str)[:500]}"
f"[llm] tool_call missing 'id'! keys={list(tc.keys())}"
)
# 保留原始 tc 完整结构,只规范化 arguments
+1 -1
View File
@@ -234,7 +234,7 @@ class MemAgent:
f"identity={len(identity)} facts={len(facts)} "
f"basic_info={basic_field_count}fields"
)
logger.info(f"MemAgent: extracted_info={extract_result.info}")
logger.debug(f"MemAgent: extracted_info={extract_result.info}")
# 3. 冲突检测(如果有已有记忆)
if mode == ProcessMode.FULL and existing_memories:
+31
View File
@@ -365,6 +365,13 @@ EXTRACT_PROMPT_ZH = """## 最近 k 条消息
* basic_info:**只装用户本人的画像**。
仅当 NEW user messages 中**明确**陈述稳定画像字段(如 姓名、年龄、所在地、职业、雇主)时才填写。
绝不推断、绝不编造。没有任何字段被明确陈述时输出 `{{}}`。
* **硬性规则**:只有用户在 `user` 回合里**明确说出**的字段才能填写。只在
assistant 回合出现的内容(包括 assistant 引用旧记忆,如"The user's name is X"
/ "用户名是X")一律不算。没有符合条件的字段时,必须输出 `"basic_info": {{}}`。
* **绝不**把字段名本身或占位符当值写入("用户"、"The user"、"user"、"name"、
"姓名"、"N/A"、"unknown"、"未知"、"未提供"、"-"),宁可省略该字段。
* 负例:寒暄如 `[user]: 你好` / `[assistant]: 我可以帮你。`,或
`[user]: Reply with exactly: PONG` → `"basic_info": {{}}`。
* `name` 的例外:用户自己那条消息上的 `name` 字段,或文首形如
"用户名叫X" / "The user's name is X" 的那一行,都是 `name` 的权威来源
——即使用户本人从未开口自报姓名。
@@ -440,6 +447,13 @@ EXTRACT_PROMPT_ZH_NO_INTENTION = """## 最近 k 条消息
* basic_info:**只装用户本人的画像**。
仅当 NEW user messages 中**明确**陈述稳定画像字段(如 姓名、年龄、所在地、职业、雇主)时才填写。
绝不推断、绝不编造。没有任何字段被明确陈述时输出 `{{}}`。
* **硬性规则**:只有用户在 `user` 回合里**明确说出**的字段才能填写。只在
assistant 回合出现的内容(包括 assistant 引用旧记忆,如"The user's name is X"
/ "用户名是X")一律不算。没有符合条件的字段时,必须输出 `"basic_info": {{}}`。
* **绝不**把字段名本身或占位符当值写入("用户"、"The user"、"user"、"name"、
"姓名"、"N/A"、"unknown"、"未知"、"未提供"、"-"),宁可省略该字段。
* 负例:寒暄如 `[user]: 你好` / `[assistant]: 我可以帮你。`,或
`[user]: Reply with exactly: PONG` → `"basic_info": {{}}`。
* `name` 的例外:用户自己那条消息上的 `name` 字段,或文首形如
"用户名叫X" / "The user's name is X" 的那一行,都是 `name` 的权威来源
——即使用户本人从未开口自报姓名。
@@ -669,6 +683,23 @@ Memory Date:2025-09-02
```
职业和所在地是 **John 的**属性,所以进 `memory` 并以"John"作主体,**绝不能**落到 `basic_info` 的 `occupation` / `location` —— `basic_info` 专属于用户本人。只有 `name`(Tim,来自用户消息的 `name` 字段)才该放进去。注意输出语言随输入语言(此例为英文)。
## 示例 11 —— 反例:寒暄没有 basic_info
新消息:
[{"role": "user", "content": "你好"},
{"role": "assistant", "content": "你好!我可以帮你。之前记得 The user's name is 用户."}]
Memory Date:2025-09-03
输出:
```json
{
"memory": [],
"intentions": [],
"basic_info": {}
}
```
用户没有说任何关于自己的信息。assistant 那句不是 user 回合,"用户" 也是占位符而不是名字。必须输出 `{}`——**绝不能**输出 `{"name": "用户"}`。
---
"""
+11 -9
View File
@@ -50,7 +50,7 @@ def is_canonical_state_node(node: Any) -> bool:
def _log_ops_detail(ops: List["ReconcileOp"]) -> None:
"""以单条 INFO(JSON list)输出 reconcile 的逐条结果,便于线上 debug。"""
"""以单条 DEBUG(JSON list)输出 reconcile 的逐条结果;含记忆正文,不进 INFO。"""
if not ops:
return
view = [
@@ -68,7 +68,7 @@ def _log_ops_detail(ops: List["ReconcileOp"]) -> None:
}
for o in ops
]
logger.info(
logger.debug(
f"[reconciler] ops detail: {json.dumps(view, ensure_ascii=False, default=str)}"
)
@@ -1514,9 +1514,10 @@ class MemoryReconciler:
logger.warning(
f"[reconciler] RECONCILE JSON parse failed "
f"(attempt {_attempt+1}/{_MAX_PARSE_ATTEMPTS}), "
f"retrying: raw="
f"{raw_response[:200] if _log_llm_content_enabled() else '<redacted>'}"
f"retrying: raw({len(raw_response or '')} chars)"
)
if _log_llm_content_enabled():
logger.debug(f"[reconciler] unparsable raw={raw_response[:200]!r}")
recon_elapsed = (_time.perf_counter() - t_recon) * 1000
@@ -1528,8 +1529,8 @@ class MemoryReconciler:
error_code = "LLM_UNAVAILABLE"
elif raw_response.strip():
error_msg = (
f"RECONCILE JSON parse failed after {_n_attempts} attempts: "
f"{raw_response[:200] if _log_llm_content_enabled() else '<redacted>'}"
f"RECONCILE JSON parse failed after {_n_attempts} attempts "
f"({len(raw_response)} chars)"
)
error_code = "JSON_PARSE_FAILED"
else:
@@ -1806,7 +1807,8 @@ class MemoryReconciler:
except Exception as _re:
logger.debug(f"[reconciler] json_repair failed: {_re}")
if data is None:
logger.warning(f"[reconciler] failed to parse ops JSON: {text[:200]}")
logger.warning(f"[reconciler] failed to parse ops JSON ({len(text)} chars)")
logger.debug(f"[reconciler] unparsable ops output: {text[:200]!r}")
return []
if not isinstance(data, list):
@@ -2018,11 +2020,11 @@ class MemoryReconciler:
if clash is not None:
logger.warning(
f"[reconciler] memory_id {clash} already touched by '{touched[clash]}', "
f"current op '{op.op} ({op.reason[:60]})' dropped"
f"current op '{op.op}' dropped"
)
continue
for t in target_ids:
touched[t] = f"{op.op}({op.reason[:40]})"
touched[t] = op.op
validated.append(op)
return validated
+3 -1
View File
@@ -211,7 +211,9 @@ def parse_tool_calls_from_json(raw: Any) -> List[ToolCall]:
try:
args = json.loads(args_raw) if args_raw.strip() else {}
except json.JSONDecodeError:
logger.warning(f"[tools] failed to parse arguments as JSON: {args_raw[:120]}")
logger.warning(
f"[tools] failed to parse arguments as JSON ({len(args_raw)} chars)"
)
args = {}
elif isinstance(args_raw, dict):
args = args_raw
+77
View File
@@ -26,6 +26,8 @@ from __future__ import annotations
import asyncio
import logging
import re
import unicodedata
import uuid
import weakref
from dataclasses import dataclass
@@ -208,6 +210,76 @@ def _sanitize_kv(
return result
# ================================================================
# 占位符守卫(P0-1)
# ================================================================
# 弱模型(如 qwen3.5-2b)在用户什么都没说时,会把 schema 字段说明里的词当值
# 填进 basic_info:{"name": "用户"} / "The user" / "N/A"。每个占位符都会
# supersede 掉真实值,演化链变成 费天翔 → Fei → 用户 → The user。
# 规则:占位符永远不入库;读侧组装 KV 时占位符也不覆盖已有真实值。
_PLACEHOLDER_VALUES = frozenset({
"用户", "该用户", "本用户", "用户本人",
"the user", "the user's", "user", "a user", "this user",
"n/a", "na", "none", "null", "nil", "unknown", "unspecified",
"not provided", "not specified", "not mentioned",
"未提供", "未知", "不详", "未说明", "未提及", "无",
"-", "?",
})
# 内置字段的中文标签:弱模型也会把字段说明的中文词当值。
_FIELD_LABELS: Dict[str, tuple] = {
"name": ("姓名", "名字", "用户名", "用户姓名", "用户名字"),
"age": ("年龄", "用户年龄"),
"location": ("所在地", "地点", "城市", "用户所在地"),
"occupation": ("职业", "工作", "用户职业"),
"employer": ("雇主", "公司", "用户雇主"),
}
# 去掉标点/符号后再比较,让 "The user." / "用户。" / "N/A" 都命中。
# "-" / "?" 本身就是纯标点,剥完为空串,同样判为占位符。
_PUNCT_RE = re.compile(r"[\s\.,;:!?'\"`~()\[\]{}<>/\\|_\-\u3000-\u303f\uff00-\uff0f\uff1a-\uff20]+")
def _placeholder_key(value: str) -> str:
s = unicodedata.normalize("NFKC", str(value or "")).strip().lower()
s = s.replace("\u2019", "'")
s = _PUNCT_RE.sub(" ", s)
return " ".join(s.split())
_PLACEHOLDER_KEYS = frozenset(_placeholder_key(v) for v in _PLACEHOLDER_VALUES)
def is_placeholder_value(field_name: str, value: Any) -> bool:
"""value 是否为占位符(空 / 用户 / The user / N/A / 字段名本身 等)。
大小写不敏感、首尾空白与标点不敏感。数字(如 age=30)永远不是占位符。
"""
if value is None:
return True
if isinstance(value, bool):
return False
if isinstance(value, (int, float)):
return False
key = _placeholder_key(value)
if not key:
return True
if key in _PLACEHOLDER_KEYS:
return True
field_key = _placeholder_key(field_name)
if field_key and key in (field_key, field_key.replace("_", " ")):
return True
for label in _FIELD_LABELS.get(field_name, ()):
if key == _placeholder_key(label):
return True
return False
def _drop_placeholders(kv: Dict[str, Any]) -> Dict[str, Any]:
return {k: v for k, v in kv.items() if not is_placeholder_value(k, v)}
# ================================================================
# 公共入口:upsert_basic_profile
# ================================================================
@@ -287,6 +359,8 @@ async def _upsert_basic_profile_unlocked(
# L0 Basic Info 从输入、唯一性到落盘都只属于 user_id。
new_kv = _sanitize_kv(kv or {}, allowed_fields=allowed_fields)
# 单调性守卫:占位符永远不覆盖已有值,过滤后为空就不写新版本、不产生新链头。
new_kv = _drop_placeholders(new_kv)
if not new_kv:
return BasicProfileUpsertResult(
success=True, changed=False, reason="no valid fields after sanitize"
@@ -513,5 +587,8 @@ def _assemble_full_kv_from_nodes(
for k, v in kv.items():
if allowed_set is not None and k not in allowed_set:
continue
# 历史上已落盘的占位符版本(如 name=用户)不覆盖更早的真实值。
if is_placeholder_value(k, v):
continue
full_kv[k] = v
return full_kv
+38 -16
View File
@@ -744,7 +744,7 @@ class HyMemoryClient:
def _install_pipeline_observability_hooks(self) -> None:
"""
包装 cache.store_pipeline_log:
- Log:JSONL 文件,始终写入(环节级,不依赖 Trace 开关)
- Log:JSONL 文件,由 MEMORY_PIPELINE_LOG_ENABLED 控制(默认开启)
- Trace:SQLite pipeline_logs,由 MEMORY_PIPELINE_TRACE_ENABLED 控制(Inspector)
- 落盘 fire-and-forget,不阻塞 search/write 热路径
"""
@@ -958,7 +958,7 @@ class HyMemoryClient:
parsed_tool_calls = json.loads(raw_tool_calls) if raw_tool_calls else []
except Exception:
logger.warning(
"tool_calls 是非法 JSON 字符串,已忽略: %r", raw_tool_calls[:200]
"tool_calls 是非法 JSON 字符串,已忽略 (%d chars)", len(raw_tool_calls)
)
parsed_tool_calls = []
raw_tool_calls = parsed_tool_calls if isinstance(parsed_tool_calls, list) else []
@@ -1544,6 +1544,7 @@ class HyMemoryClient:
# LLM 写的"这个结论一路怎么改成现在这样",直接进提示词;
# 没被覆盖过就是空串。不参与检索——召回认的是 statement 的当前值。
"fact_chain_summary": fact.fact_chain_summary,
"memory_at": _epoch_seconds(fact.memory_at),
"created_at": fact.created_at.isoformat() if fact.created_at else None,
"score": round(float(score), 4),
"source": fact.source,
@@ -1652,6 +1653,8 @@ class HyMemoryClient:
"score": (round(float(score_by_id[d.experience_id]), 4)
if d.experience_id in score_by_id else None),
"confidence": d.confidence,
"memory_at": _epoch_seconds(d.memory_at),
"created_at": d.created_at.isoformat() if d.created_at else None,
"updated_at": d.updated_at.isoformat() if d.updated_at else None,
"source": d.source,
"extraction_source": d.extraction_source,
@@ -1953,6 +1956,7 @@ class HyMemoryClient:
"content" chat 正文;经验为 "task\nsolution";事实为 statement
"score" RRF 融合分;"lane_score" 为该路原始分;"lane_rank" 为该路名次
"source", "created_at", "updated_at"(秒级 epoch,取不到为 None)
"memory_at" 对话时间(秒级 epoch),各路原有字段,三路同名同格式
chat 路内部名次:normal → schema → proactive 依次排;working/coding 路
内部名次:事实与经验各按原顺序交替(事实在前),两层谁也不压谁。
"""
@@ -2144,7 +2148,8 @@ class HyMemoryClient:
"limit", "offset", "request_id", "elapsed_ms"}
每条带 "scene"、"kind"(chat | experience)、"memory_id"、"content"
(经验为 "task\nsolution")、"source"、"status"、"created_at"、
"updated_at"(秒级 epoch),以及该路原有字段。
"updated_at"(秒级 epoch),以及该路原有字段(含对话时间
"memory_at",秒级 epoch,三路同名同格式)。
"""
return self._loop_thread.run(
self.async_list_memories(
@@ -2967,6 +2972,7 @@ class HyMemoryClient:
session_id=session_id,
request_id=request_id,
source=source,
memory_at=memory_at,
)
# Public receipt always identifies the productivity namespace.
resp["variant"] = scene_variant
@@ -3002,13 +3008,13 @@ class HyMemoryClient:
f"[add] writing {preview} user={uid} "
f"agent_id={agent_id} session_id={session_id} mode={self._mode}"
)
logger.info(f"[add] messages detail: {[m.to_dict() for m in chat_messages]}")
logger.debug(f"[add] messages detail: {[m.to_dict() for m in chat_messages]}")
else:
logger.info(
f"[add] writing text({len(content)} chars) user={uid} "
f"agent_id={agent_id} session_id={session_id} mode={self._mode} "
f"content={content}"
f"agent_id={agent_id} session_id={session_id} mode={self._mode}"
)
logger.debug(f"[add] text content={content!r}")
# mode → agent_mode 映射
# lite: 不调 LLM agent
@@ -3083,9 +3089,18 @@ class HyMemoryClient:
if self._history_store is not None:
try:
isolation_key = self._build_isolation_key(uid, agent_id)
# 从 content 或 messages 中提取用于记录的文本
# 从 content 或 messages 中提取用于记录的文本。messages 与 L1_RAW
# 同一规则先裁剪再落审计库(工具输出 / 长回复不原样长期保存)。
from .utils.raw_trim import trim_raw_messages
history_content = content or "; ".join(
f"{m.role}: {m.content}" for m in chat_messages
f"{m.role}: {m.content}"
for m in trim_raw_messages(
chat_messages,
tool_max_bytes=getattr(
getattr(self._config, "coding", None),
"tool_result_max_bytes", None,
),
)
)
await self._history_store.record_add(
memory_id=result.memory_id,
@@ -4028,14 +4043,14 @@ class HyMemoryClient:
decided_variant = HyMemoryClient._scene_variant(scene)
logger.info(
f"[search] scene explicit={decided_scene} (no LLM judge); "
f"target={effective_target_query!r}"
f"target({len(effective_target_query or '')} chars)"
)
elif not agentic_enabled and not queries_supplied:
# 默认 light search 仅指 query="...":固定 normal,绝不调用 LLM。
decided_scene = "normal"
logger.info(
f"[search] search_mode=default single-query scene=normal (no LLM); "
f"target={target_query!r}"
f"target({len(target_query or '')} chars)"
)
elif agentic_enabled:
# agentic:单/多 query 都强制一次 LLM 完成三分类 + rewrite + 时间解析。
@@ -4070,14 +4085,18 @@ class HyMemoryClient:
rewrite_created_after = ar.get("created_after")
logger.info(
f"[search] agentic scene={decided_scene} via rewrite-LLM "
f"(classify={want_classify}); orig={target_query!r} "
f"effective={effective_target_query!r} rewrite_used={rewrite_used} "
f"(classify={want_classify}); rewrite_used={rewrite_used} "
f"created_after={rewrite_created_after}"
)
logger.debug(
f"[search] agentic rewrite: orig={target_query!r} "
f"effective={effective_target_query!r}"
)
else:
decided_scene = "normal"
logger.info(
f"[search] agentic rewrite-LLM fallback=normal; target={target_query!r}"
f"[search] agentic rewrite-LLM fallback=normal; "
f"target({len(target_query or '')} chars)"
)
elif rewrite_enabled and not _too_long:
# default + queries= 的原逻辑:配置开启时一次 LLM 做 rewrite/time,
@@ -4156,7 +4175,7 @@ class HyMemoryClient:
decided_scene = "normal"
logger.info(
f"[search] default multi-query features disabled; no LLM; "
f"target={target_query!r}"
f"target({len(target_query or '')} chars)"
)
# ────────────────────────────────────────────────────────────
@@ -4209,7 +4228,7 @@ class HyMemoryClient:
else:
logger.info(
f"[search] routing to PRODUCTIVITY path variant={decided_variant}: "
f"target={effective_target_query!r} "
f"target({len(effective_target_query or '')} chars) "
f"workspace_id={workspace_id!r} branch={branch!r}"
)
return await self._search_coding(
@@ -4237,9 +4256,11 @@ class HyMemoryClient:
# Chat 召回链路(原有逻辑)
# ────────────────────────────────────────────────────────────
logger.info(
f"[search] query='{effective_target_query}' user_ids={effective_user_ids} "
f"[search] query({len(effective_target_query or '')} chars) "
f"user_ids={effective_user_ids} "
f"agent_ids={effective_agent_ids} session_ids={effective_session_ids} limit={limit}"
)
logger.debug(f"[search] query={effective_target_query!r}")
# rewrite 抽出的时间下界与用户显式 created_after 取更晚者(更严格)
effective_created_after = created_after
@@ -5920,6 +5941,7 @@ class HyMemoryClient:
"source_fact_ids": list(d.source_fact_ids),
"confidence": d.confidence,
"status": MemoryStatus.ACTIVE.value,
"memory_at": _epoch_seconds(d.memory_at),
"created_at": d.created_at.isoformat() if d.created_at else None,
"updated_at": d.updated_at.isoformat() if d.updated_at else None,
"source": d.source,
+4 -1
View File
@@ -1166,7 +1166,10 @@ class CodingMemoryExtractor:
obj = None
if not isinstance(obj, dict):
logger.warning(f"[coding-extract] failed to parse output: {text!r}")
logger.warning(
f"[coding-extract] failed to parse output ({len(text or '')} chars)"
)
logger.debug(f"[coding-extract] unparsable output: {text!r}")
return [], []
# "memories" 是双层化之前的 key,老 prompt 缓存 / 旧模型可能还这么回
+20 -7
View File
@@ -454,9 +454,10 @@ class WorkingFactReconciler:
workspace_id: Optional[str] = None,
branch: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
) -> Tuple[List[WorkingFact], List[FactReconcileOp]]:
"""
执行 reconcile + 持久化。
执行 reconcile + 持久化。memory_at 是这段对话的时间,记到新事实上。
Returns:
(真正落库的新事实, 执行过的 ops)。LLM 挂掉时退化为全 INSERT 不覆盖,
@@ -493,6 +494,7 @@ class WorkingFactReconciler:
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
)
logger.info(
f"[fact-reconcile] drafts={len(drafts)} inserted={len(facts)} "
@@ -568,9 +570,12 @@ class WorkingFactReconciler:
kept.append(fact)
if blocked:
logger.info(
f"[fact-reconcile] subject gate blocked {len(blocked)} candidate(s) "
f"for subject={draft.subject!r}: "
+ ", ".join(f"{f.fact_id}({f.subject!r})" for f in blocked)
f"[fact-reconcile] subject gate blocked {len(blocked)} candidate(s): "
+ ", ".join(f.fact_id for f in blocked)
)
logger.debug(
f"[fact-reconcile] subject gate: draft subject={draft.subject!r} "
f"blocked subjects={[f.subject for f in blocked]!r}"
)
result.append(kept)
return result
@@ -743,7 +748,10 @@ class WorkingFactReconciler:
obj = None
if not isinstance(obj, dict):
logger.warning(f"[fact-reconcile] failed to parse ops output: {text!r}")
logger.warning(
f"[fact-reconcile] failed to parse ops output ({len(text or '')} chars)"
)
logger.debug(f"[fact-reconcile] unparsable ops output: {text!r}")
return {}
ops_raw = obj.get("ops")
if not isinstance(ops_raw, list):
@@ -781,6 +789,7 @@ class WorkingFactReconciler:
workspace_id: Optional[str],
branch: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
) -> Tuple[List[WorkingFact], List[FactReconcileOp]]:
inserted: List[WorkingFact] = []
ops: List[FactReconcileOp] = []
@@ -801,6 +810,7 @@ class WorkingFactReconciler:
subject=draft.subject,
details=draft.details,
status="active",
memory_at=memory_at,
)
allowed = {c.fact_id for c in (candidates[idx] if idx < len(candidates) else [])}
@@ -831,8 +841,11 @@ class WorkingFactReconciler:
# 别的主体的状态不是。闸门关掉时这道复查一并跳过,否则闸门就没关干净。
if self.subject_gate and _subject_conflict(draft.subject, old.subject):
logger.warning(
f"[fact-reconcile] refused cross-subject SUPERSEDE: "
f"{old.fact_id}({old.subject!r}) by draft subject {draft.subject!r}"
f"[fact-reconcile] refused cross-subject SUPERSEDE: {old.fact_id}"
)
logger.debug(
f"[fact-reconcile] cross-subject: old subject={old.subject!r} "
f"draft subject={draft.subject!r}"
)
continue
try:
+21 -10
View File
@@ -195,8 +195,10 @@ async def classify_messages_scene(
result = _parse_scene_json(resp.content)
if result is None:
logger.warning(
f"[coding-judge] failed to parse classify_messages output: {resp.content!r}; defaulting to chat"
f"[coding-judge] failed to parse classify_messages output "
f"({len(resp.content or '')} chars); defaulting to chat"
)
logger.debug(f"[coding-judge] unparsable classify_messages output: {resp.content!r}")
await _store_coding_log(
cache, request_id=request_id, user_id=user_id, agent_id=agent_id,
session_id=session_id, step="CODING_JUDGE", prompt=prompt,
@@ -207,9 +209,8 @@ async def classify_messages_scene(
return "chat"
scene = result["scene"]
parsed = {**result, "is_coding": scene != "chat"}
logger.info(
f"[coding-judge] scene={scene} reason={result.get('reason', '')!r}"
)
logger.info(f"[coding-judge] scene={scene}")
logger.debug(f"[coding-judge] scene={scene} reason={result.get('reason', '')!r}")
await _store_coding_log(
cache, request_id=request_id, user_id=user_id, agent_id=agent_id,
session_id=session_id, step="CODING_JUDGE", prompt=prompt,
@@ -295,8 +296,10 @@ async def classify_queries_scene(
result = _parse_scene_json(resp.content)
if result is None:
logger.warning(
f"[coding-judge] failed to parse classify_queries output: {resp.content!r}; defaulting to chat"
f"[coding-judge] failed to parse classify_queries output "
f"({len(resp.content or '')} chars); defaulting to chat"
)
logger.debug(f"[coding-judge] unparsable classify_queries output: {resp.content!r}")
return "chat"
scene = result["scene"]
logger.info(f"[coding-judge] queries scene={scene}")
@@ -422,15 +425,18 @@ async def classify_and_rewrite_queries(
scene = _parse_scene_label(head)
if scene is None:
logger.warning(
f"[coding-judge] classify_and_rewrite unrecognized head={head!r}; falling back to chat"
f"[coding-judge] classify_and_rewrite unrecognized head "
f"({len(head)} chars); falling back to chat"
)
logger.debug(f"[coding-judge] classify_and_rewrite unrecognized head={head!r}")
return {"scene": "chat", "is_coding": False, "rewrite_query": rewrite, "ok": False}
is_coding = scene != "chat"
logger.info(
f"[coding-judge] classify_and_rewrite scene={scene} "
f"rewrite={rewrite!r}"
f"rewrite({len(rewrite)} chars)"
)
logger.debug(f"[coding-judge] classify_and_rewrite rewrite={rewrite!r}")
return {"scene": scene, "is_coding": is_coding, "rewrite_query": rewrite, "ok": True}
@@ -569,8 +575,9 @@ async def analyze_queries(
parsed_scene = _parse_scene_label(head)
if parsed_scene is None:
logger.warning(
f"[coding-judge] analyze_queries unrecognized head={head!r}; ok=False"
f"[coding-judge] analyze_queries unrecognized head ({len(head)} chars); ok=False"
)
logger.debug(f"[coding-judge] analyze_queries unrecognized head={head!r}")
return {**fallback, "since_raw": ""}
scene = parsed_scene
idx = 1
@@ -589,9 +596,10 @@ async def analyze_queries(
created_after = _since_to_created_after(since_raw, current_time)
logger.info(
f"[coding-judge] analyze scene={scene} rewrite={rewrite!r} "
f"[coding-judge] analyze scene={scene} rewrite({len(rewrite)} chars) "
f"since={since_raw!r} created_after={created_after}"
)
logger.debug(f"[coding-judge] analyze rewrite={rewrite!r}")
return {
"scene": scene, "is_coding": scene != "chat", "rewrite_query": rewrite,
"created_after": created_after, "since_raw": since_raw, "ok": True,
@@ -614,7 +622,10 @@ def _since_to_created_after(since_raw: str, current_time: str) -> Optional[float
return dt.timestamp()
except (ValueError, TypeError):
continue
logger.warning(f"[coding-judge] analyze_queries unparsable SINCE={since_raw!r}")
logger.warning(
f"[coding-judge] analyze_queries unparsable SINCE ({len(since_raw)} chars)"
)
logger.debug(f"[coding-judge] analyze_queries unparsable SINCE={since_raw!r}")
return None
+18
View File
@@ -11,6 +11,7 @@ Coding Memory 预处理工具(规则,零 LLM)
使用方式与 scene 路由见 docs/scene_coding_usage.md。
"""
import re
from typing import Any, Dict, List, Optional, Tuple
from ..pipelines.base import ChatMessage
@@ -63,6 +64,13 @@ DEFAULT_MAX_TOOL_RESULT_BYTES = 2048
DEFAULT_HEAD_BYTES = 1024
DEFAULT_TAIL_BYTES = 512
# 本模块自己产出的省略标记。截断结果会再次经过截断(落库前一次、读历史时又一次),
# 认出自己的标记才能保证幂等:已截断的文本再截一次原样返回,而不是把省略计数改写。
_TOOL_OMIT_MARKER_RE = re.compile(r"\n\n\[\.\.\.\d+ bytes omitted by truncate\.\.\.\]\n\n")
# 标记本身的字节数上限(数字位数有限),用于判断「已截断且只超出一个标记的长度」。
_TOOL_OMIT_MARKER_MAX_BYTES = 128
_ASSISTANT_OMIT_MARKER_RE = re.compile(r" \[\.\.\.\d+ chars omitted\.\.\.\] ")
def truncate_tool_result_text(
text: str,
@@ -81,6 +89,12 @@ def truncate_tool_result_text(
raw = text.encode("utf-8")
if len(raw) <= max_bytes:
return text
# 已截断过(max_bytes 很小时标记本身会让结果略超上限):原样返回,保证幂等。
if (
len(raw) <= max_bytes + _TOOL_OMIT_MARKER_MAX_BYTES
and _TOOL_OMIT_MARKER_RE.search(text)
):
return text
# 自定义 max_bytes 可能小于默认 head+tail(1024+512)。按原比例缩放内容窗口,
# 给省略标记预留空间,不能因为窗口和过大而把超限原文直接返回。
marker_budget = min(96, max(24, max_bytes // 4))
@@ -161,6 +175,10 @@ def truncate_assistant_text(
text = "" if text is None else str(text)
if len(text) <= head_chars + tail_chars:
return text
# 已截断过:中段恰好是本函数的省略标记 → 原样返回,保证幂等(否则会把
# 省略计数改写成标记自身的长度)。
if _ASSISTANT_OMIT_MARKER_RE.fullmatch(text[head_chars:len(text) - tail_chars]):
return text
omitted = len(text) - head_chars - tail_chars
return (
text[:head_chars]
+22 -10
View File
@@ -388,7 +388,10 @@ class CodingMemoryReconciler:
obj = None
if not obj or not isinstance(obj, dict):
logger.warning(f"[coding-reconcile] failed to parse ops output: {text!r}")
logger.warning(
f"[coding-reconcile] failed to parse ops output ({len(text or '')} chars)"
)
logger.debug(f"[coding-reconcile] unparsable ops output: {text!r}")
return []
ops_raw = obj.get("ops") or []
if not isinstance(ops_raw, list):
@@ -441,10 +444,10 @@ class CodingMemoryReconciler:
for op in ops:
if op.action == "ADD":
if op.draft_idx is None or not (0 <= op.draft_idx < len(drafts)):
logger.warning(f"[coding-reconcile] drop invalid ADD: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop invalid ADD: {_op_ref(op)}")
continue
if op.draft_idx in consumed:
logger.warning(f"[coding-reconcile] drop ADD on consumed draft: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop ADD on consumed draft: {_op_ref(op)}")
continue
d = drafts[op.draft_idx]
memory = self._draft_to_new_memory(d)
@@ -458,18 +461,18 @@ class CodingMemoryReconciler:
elif op.action == "UPDATE":
if op.draft_idx is None or not (0 <= op.draft_idx < len(drafts)):
logger.warning(f"[coding-reconcile] drop invalid UPDATE: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop invalid UPDATE: {_op_ref(op)}")
continue
if not op.target_experience_id:
logger.warning(f"[coding-reconcile] drop UPDATE missing target: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop UPDATE missing target: {_op_ref(op)}")
continue
if not self._target_was_offered(op, candidates_per_draft):
logger.warning(
f"[coding-reconcile] drop UPDATE outside authority scope: {op.to_dict()}"
f"[coding-reconcile] drop UPDATE outside authority scope: {_op_ref(op)}"
)
continue
if op.draft_idx in consumed:
logger.warning(f"[coding-reconcile] drop UPDATE on consumed draft: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop UPDATE on consumed draft: {_op_ref(op)}")
continue
d = drafts[op.draft_idx]
memory = self._draft_to_new_memory(d, experience_id=op.target_experience_id)
@@ -488,17 +491,17 @@ class CodingMemoryReconciler:
elif op.action == "DELETE":
if not op.target_experience_id:
logger.warning(f"[coding-reconcile] drop DELETE missing target: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop DELETE missing target: {_op_ref(op)}")
continue
if not self._target_was_offered(op, candidates_per_draft):
logger.warning(
f"[coding-reconcile] drop DELETE outside authority scope: {op.to_dict()}"
f"[coding-reconcile] drop DELETE outside authority scope: {_op_ref(op)}"
)
continue
# DELETE 需要知道 user_id;通过 draft_idx(如有)或第一个 draft 推断
user_id = self._infer_user_id(op, drafts)
if not user_id:
logger.warning(f"[coding-reconcile] drop DELETE no user_id context: {op.to_dict()}")
logger.warning(f"[coding-reconcile] drop DELETE no user_id context: {_op_ref(op)}")
continue
try:
ok = await self.store.delete(op.target_experience_id, user_id=user_id)
@@ -577,6 +580,7 @@ class CodingMemoryReconciler:
confidence=d.confidence,
source=d.source,
extraction_source=d.extraction_source,
memory_at=d.memory_at,
)
@@ -584,6 +588,14 @@ class CodingMemoryReconciler:
# Helpers
# ================================================================
def _op_ref(op: ReconcileOp) -> str:
"""op 的日志引用:只带动作与编号,不带 LLM 写的 reason。"""
return (
f"action={op.action} draft_idx={op.draft_idx} "
f"target={op.target_experience_id}"
)
def _truncate(s: str, n: int) -> str:
if not s:
return ""
+48 -7
View File
@@ -86,6 +86,7 @@ CREATE TABLE IF NOT EXISTS coding_memory_meta (
confidence REAL NOT NULL DEFAULT 0.7,
source TEXT NOT NULL DEFAULT 'default',
extraction_source TEXT NOT NULL DEFAULT 'auto_extract',
memory_at INTEGER,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
@@ -109,6 +110,7 @@ CREATE TABLE IF NOT EXISTS working_fact (
derived_experience_ids TEXT NOT NULL DEFAULT '[]',
status TEXT NOT NULL DEFAULT 'active',
fact_chain_summary TEXT NOT NULL DEFAULT '',
memory_at INTEGER,
created_at TEXT NOT NULL
);
-- 反查前驱(谁的 fact_link 指向我)走这个索引,写演变总结时要顺着链往回走
@@ -164,6 +166,8 @@ _ADDED_COLUMNS = {
"coding_memory_meta": {
"source_fact_ids": "TEXT NOT NULL DEFAULT '[]'",
"extraction_source": "TEXT NOT NULL DEFAULT 'auto_extract'",
# 对话时间,秒级 epoch(同 chat)。老库里的行是 NULL:那时没存过。
"memory_at": "INTEGER",
},
# 早期版本存过一列 fact_chain(机械累积的历代说法),已废弃:同样的信息顺着
# fact_link 反查就有。老库里那一列留着不管,有默认值不影响插入。
@@ -177,6 +181,7 @@ _ADDED_COLUMNS = {
"subject": "TEXT NOT NULL DEFAULT ''",
"details": "TEXT NOT NULL DEFAULT ''",
"source": "TEXT NOT NULL DEFAULT 'default'",
"memory_at": "INTEGER",
},
}
@@ -290,6 +295,29 @@ def _parse_dt(s: Optional[str]) -> Optional[datetime]:
return None
def _from_epoch(value: Any) -> Optional[datetime]:
"""memory_at 列(秒级 epoch)→ datetime;NULL 或坏值返回 None。"""
if value is None:
return None
try:
return datetime.fromtimestamp(int(value))
except Exception:
return None
def _to_epoch(value: Optional[datetime]) -> Optional[int]:
return int(value.timestamp()) if isinstance(value, datetime) else None
def _later(a: Optional[datetime], b: Optional[datetime]) -> Optional[datetime]:
"""两个对话时间取较晚的;一边缺就取另一边。"""
if a is None:
return b
if b is None:
return a
return a if _to_epoch(a) >= _to_epoch(b) else b
def _row_to_memory(row: sqlite3.Row) -> CodingMemory:
"""SQLite row → CodingMemory"""
return CodingMemory(
@@ -308,6 +336,7 @@ def _row_to_memory(row: sqlite3.Row) -> CodingMemory:
confidence=row["confidence"],
source=row["source"],
extraction_source=row["extraction_source"],
memory_at=_from_epoch(row["memory_at"]),
created_at=_parse_dt(row["created_at"]),
updated_at=_parse_dt(row["updated_at"]),
)
@@ -334,6 +363,7 @@ def _row_to_fact(row: sqlite3.Row) -> WorkingFact:
# 老库刚 ADD COLUMN 时可能取不到这一列,用 keys() 兜一下
fact_chain_summary=(row["fact_chain_summary"] or ""
if "fact_chain_summary" in row.keys() else ""),
memory_at=_from_epoch(row["memory_at"]),
created_at=_parse_dt(row["created_at"]),
)
@@ -575,7 +605,11 @@ class CodingMemoryStore:
await self._upsert_vec_keys(memory)
logger.info(
f"[coding-store] inserted experience_id={memory.experience_id} "
f"task={memory.task!r} keys={1 + len(memory.search_keys)}"
f"keys={1 + len(memory.search_keys)}"
)
logger.debug(
f"[coding-store] inserted experience_id={memory.experience_id} "
f"task={memory.task!r}"
)
async def update(self, memory: CodingMemory) -> None:
@@ -597,6 +631,8 @@ class CodingMemoryStore:
memory.updated_at = datetime.utcnow()
if memory.created_at is None:
memory.created_at = prev.created_at
# 合并后的经验讲的是两段对话,对话时间取较晚的那段
memory.memory_at = _later(prev.memory_at, memory.memory_at)
await _run_in_pool(self._upsert_sql_sync, memory)
# 删旧 VDB
await self._delete_vec_keys_by_count(memory.experience_id, prev_key_count)
@@ -970,7 +1006,8 @@ class CodingMemoryStore:
fact.created_at = datetime.utcnow()
await _run_in_pool(self._upsert_fact_sql_sync, fact)
await self._upsert_fact_vec(fact)
logger.info(
logger.info(f"[coding-store] inserted fact_id={fact.fact_id}")
logger.debug(
f"[coding-store] inserted fact_id={fact.fact_id} "
f"statement={fact.statement!r}"
)
@@ -1199,8 +1236,8 @@ class CodingMemoryStore:
experience_id, user_id, agent_id, task, search_keys, solution,
boundary_envs, workspace_id, branch, session_id,
files, source_fact_ids, confidence, source, extraction_source,
created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
memory_at, created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(experience_id) DO UPDATE SET
user_id = excluded.user_id,
agent_id = excluded.agent_id,
@@ -1216,6 +1253,7 @@ class CodingMemoryStore:
confidence = excluded.confidence,
source = excluded.source,
extraction_source = excluded.extraction_source,
memory_at = excluded.memory_at,
updated_at = excluded.updated_at
""",
(
@@ -1226,6 +1264,7 @@ class CodingMemoryStore:
json.dumps(memory.files, ensure_ascii=False),
json.dumps(memory.source_fact_ids, ensure_ascii=False),
memory.confidence, memory.source, memory.extraction_source,
_to_epoch(memory.memory_at),
(memory.created_at or datetime.utcnow()).isoformat(),
(memory.updated_at or datetime.utcnow()).isoformat(),
),
@@ -1366,8 +1405,8 @@ class CodingMemoryStore:
fact_id, user_id, agent_id, session_id, workspace_id, branch,
source, statement, files, subject, details,
fact_link, derived_experience_ids,
status, fact_chain_summary, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
status, fact_chain_summary, memory_at, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(fact_id) DO UPDATE SET
user_id = excluded.user_id,
agent_id = excluded.agent_id,
@@ -1382,7 +1421,8 @@ class CodingMemoryStore:
fact_link = excluded.fact_link,
derived_experience_ids = excluded.derived_experience_ids,
status = excluded.status,
fact_chain_summary = excluded.fact_chain_summary
fact_chain_summary = excluded.fact_chain_summary,
memory_at = excluded.memory_at
""",
(
fact.fact_id, fact.user_id, fact.agent_id,
@@ -1395,6 +1435,7 @@ class CodingMemoryStore:
json.dumps(fact.derived_experience_ids, ensure_ascii=False),
fact.status,
fact.fact_chain_summary or "",
_to_epoch(fact.memory_at),
(fact.created_at or datetime.utcnow()).isoformat(),
),
)
+14
View File
@@ -25,6 +25,11 @@ FactStatus = Literal["active", "superseded"]
MAX_FACT_CHAIN_ENTRIES = 8
def _epoch(value: Optional[datetime]) -> Optional[int]:
"""memory_at 对外是秒级 epoch,与 chat 一致。"""
return int(value.timestamp()) if isinstance(value, datetime) else None
# ================================================================
# 草稿(LLM extract 阶段输出)
# ================================================================
@@ -53,6 +58,8 @@ class CodingMemoryDraft:
files: List[str] = field(default_factory=list)
source: str = "default"
extraction_source: str = "auto_extract"
# 对话时间(SDK 注入,同 chat 的 memory_at)
memory_at: Optional[datetime] = None
def __post_init__(self) -> None:
from ..utils.source_filter import normalize_source
@@ -73,6 +80,7 @@ class CodingMemoryDraft:
"files": list(self.files),
"source": self.source,
"extraction_source": self.extraction_source,
"memory_at": _epoch(self.memory_at),
}
@@ -111,6 +119,9 @@ class CodingMemory:
extraction_source: str = "auto_extract"
# 时间
# memory_at 是对话时间(同 chat),created_at / updated_at 是写库时间。
# UPDATE 合并时取两者中较晚的那个对话时间,见 store.update。
memory_at: Optional[datetime] = None
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
@@ -135,6 +146,7 @@ class CodingMemory:
"confidence": self.confidence,
"source": self.source,
"extraction_source": self.extraction_source,
"memory_at": _epoch(self.memory_at),
"created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None,
}
@@ -249,6 +261,7 @@ class WorkingFact:
fact_chain_summary: str = ""
# 时间
memory_at: Optional[datetime] = None # 对话时间(同 chat),没传时是收到请求的时间
created_at: Optional[datetime] = None # 产生时间,新的优先
def __post_init__(self) -> None:
@@ -272,6 +285,7 @@ class WorkingFact:
"derived_experience_ids": list(self.derived_experience_ids),
"status": self.status,
"fact_chain_summary": self.fact_chain_summary,
"memory_at": _epoch(self.memory_at),
"created_at": self.created_at.isoformat() if self.created_at else None,
}
+8 -1
View File
@@ -18,6 +18,7 @@ CodingWriter - 写入端协调器
import logging
import time
import uuid
from datetime import datetime
from typing import Any, Dict, List, Optional
from ..agent.llm_provider import LLMProvider
@@ -79,9 +80,11 @@ class CodingWriter:
session_id: Optional[str] = None,
request_id: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
) -> Dict[str, Any]:
"""
working/coding 段写入端到端。返回响应 dict(兼容 client.add 返回结构):
working/coding 段写入端到端。memory_at 是对话时间,记到这次写出的经验和
事实上;不传则用收到请求的时间(与 chat 链路一致)。返回响应 dict(兼容 client.add 返回结构):
{
"success": True,
"scene": "productivity",
@@ -101,6 +104,8 @@ class CodingWriter:
"""
request_id = request_id or str(uuid.uuid4())
t0 = time.perf_counter()
if memory_at is None:
memory_at = datetime.now()
if not messages:
return self._empty_response(request_id, t0)
@@ -172,6 +177,7 @@ class CodingWriter:
# extractor 只负责内容;调用来源由 SDK 入口注入,不能交给 LLM 生成。
for draft in drafts:
draft.source = source
draft.memory_at = memory_at
# 3. 经验层 reconcile + 持久化
ops: List[ReconcileOp] = (
@@ -197,6 +203,7 @@ class CodingWriter:
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
)
experience_ids = [op.target_experience_id for op in ops
+5
View File
@@ -260,6 +260,7 @@ class EmbedderConfig:
max_retries: int = None # 最大重试次数
retry_delay: float = None # 重试间隔(秒)
timeout: int = None # 请求超时(秒)
batch_size: int = None # 单次 HTTP 请求最多带多少条 input(embed_batch 按此切块)
# 内部平台扩展(extra_headers / extra_body 会透传给 OpenAI client)
extra_headers: Optional[Dict[str, str]] = None
@@ -294,6 +295,10 @@ class EmbedderConfig:
# 请求切断并制造重试风暴。SDK 内层重试已关闭,15s 是每个显式 attempt
# 的真实上限;部署可通过 MEMORY_EMBEDDER_TIMEOUT 覆盖。
self.timeout = _get_env_int("MEMORY_EMBEDDER_TIMEOUT", 15)
if self.batch_size is None:
# 大批量(如重新 embed 整个库)不能一次 HTTP 发几千条:多数服务对
# input 数组有上限,超了直接 400。按 batch_size 切块逐块发送。
self.batch_size = _get_env_int("MEMORY_EMBEDDER_BATCH_SIZE", 64)
if self.extra_headers is None:
val = os.getenv("MEMORY_EMBEDDER_EXTRA_HEADERS")
if val:
+120 -23
View File
@@ -38,10 +38,12 @@ import importlib.util
import json
from json import JSONDecodeError
import logging
import random
import threading
import time
from ..config import MemoryConfig
from ..embedding import EmbeddingDimensionMismatchError
from .workload import embedding_slot
from ..utils.log_setup import get_request_id
@@ -80,7 +82,7 @@ def _count_tokens(text: str) -> int:
try:
import tiktoken
return len(tiktoken.get_encoding("cl100k_base").encode(text))
return len(tiktoken.get_encoding("cl100k_base").encode(text, disallowed_special=()))
except Exception:
return len(text) // _CHARS_PER_TOKEN_FALLBACK + 1
@@ -98,7 +100,8 @@ def _truncate_for_embedding(text: str, limit: int = _EMBED_MAX_TOKENS) -> str:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode(text)
# Input is ordinary user text, not a tokenizer control-token stream.
ids = enc.encode(text, disallowed_special=())
if len(ids) <= limit:
return text
dropped = len(ids) - limit
@@ -186,6 +189,41 @@ def _redact_headers(headers: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any
return out
# Retry-After 的上限:服务端给出超长等待(或恶意值)时不让单次调用挂几十分钟。
_RETRY_AFTER_MAX_S = 60.0
def _retry_after_seconds(exc: Exception) -> Optional[float]:
"""从 429 异常的响应头里取 Retry-After(秒数或 HTTP 日期),取不到返回 None。"""
response = getattr(exc, "response", None)
headers = getattr(response, "headers", None)
if not headers:
return None
try:
raw = headers.get("retry-after") or headers.get("Retry-After")
except Exception:
return None
if raw is None:
return None
raw = str(raw).strip()
try:
seconds = float(raw)
except ValueError:
try:
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone
when = parsedate_to_datetime(raw)
if when.tzinfo is None:
when = when.replace(tzinfo=timezone.utc)
seconds = (when - datetime.now(timezone.utc)).total_seconds()
except Exception:
return None
if seconds != seconds: # NaN
return None
return min(max(0.0, seconds), _RETRY_AFTER_MAX_S)
def _embeddings_api_url(base_url: str) -> str:
"""OpenAI SDK posts to ``{base_url}/embeddings`` (base_url usually ends with /v1)."""
base = (base_url or "").rstrip("/")
@@ -204,7 +242,8 @@ class EmbedServiceConfig:
api_key: str = ""
base_url: str = ""
embedding_dims: int = 1536
batch_size: int = 100
# embed_batch 单次 HTTP 最多带多少条 input;超出按块顺序发送
batch_size: int = 64
enable_cache: bool = True
cache_size: int = 10000
max_retries: int = 5
@@ -286,6 +325,7 @@ class EmbedService:
max_retries=embedder.max_retries if embedder.max_retries is not None else 5,
retry_delay=embedder.retry_delay if embedder.retry_delay is not None else 3.0,
timeout=embedder.timeout if embedder.timeout is not None else 60,
batch_size=max(1, int(getattr(embedder, "batch_size", None) or 64)),
queue_batch_size=int(os.getenv("MEMORY_EMBED_BATCH_SIZE", "8")),
queue_batch_window_ms=float(os.getenv("MEMORY_EMBED_BATCH_WINDOW_MS", "100")),
extra_headers=embedder.extra_headers,
@@ -639,6 +679,8 @@ class EmbedService:
self,
texts: List[str],
use_cache: bool = True,
*,
max_retries: Optional[int] = None,
) -> List[List[float]]:
"""
批量向量化
@@ -646,17 +688,23 @@ class EmbedService:
Args:
texts: 文本列表
use_cache: 是否使用缓存
max_retries: 本次调用的重试次数;None 用配置值。长批量任务(重新
embed 整个库)可以给更宽的重试预算,不影响在线路径。
Returns:
向量列表
"""
vectors, _metas = await self.embed_batch_with_meta(texts, use_cache=use_cache)
vectors, _metas = await self.embed_batch_with_meta(
texts, use_cache=use_cache, max_retries=max_retries,
)
return vectors
async def embed_batch_with_meta(
self,
texts: List[str],
use_cache: bool = True,
*,
max_retries: Optional[int] = None,
) -> Tuple[List[List[float]], List[Dict[str, Any]]]:
"""批量向量化,同时返回每条的 provider metadata(非 vector)。"""
if not texts:
@@ -681,7 +729,12 @@ class EmbedService:
indices_to_embed.append(i)
if texts_to_embed:
vectors, batch_metas = await self._embed_texts_with_meta(texts_to_embed)
# max_retries 只在显式给出时透传:保持 `_embed_texts_with_meta(texts)`
# 这个既有调用形状不变(测试与下游替身按它打桩)。
retry_kwargs = {} if max_retries is None else {"max_retries": max_retries}
vectors, batch_metas = await self._embed_texts_with_meta(
texts_to_embed, **retry_kwargs,
)
for j, vector in enumerate(vectors):
idx = indices_to_embed[j]
@@ -699,9 +752,9 @@ class EmbedService:
return results, metas
async def _embed_texts_with_meta(
self, texts: List[str]
self, texts: List[str], *, max_retries: Optional[int] = None,
) -> Tuple[List[List[float]], List[Dict[str, Any]]]:
"""实际的向量化调用:**一次 HTTP 带走整批**;返回 (vectors, metas)。
"""实际的向量化调用:**一次 HTTP 带走一块**;返回 (vectors, metas)。
早期实现是 `for text in texts: await _embed_openai([text])` 逐条串行,
原意是"避免 batch size 限制",但代价极大:线上一批 8 条 × ~5.6s 串行
@@ -718,21 +771,34 @@ class EmbedService:
截断在这里做(发送前逐条过 `_truncate_for_embedding`):所有 embedding
路径(embed / embed_batch / embed_queued / *_with_meta)最终都落到这个
方法 —— 放在各入口会漏。理由与阈值见模块头部 `_EMBED_MAX_TOKENS`。
切块:一次带走整批只适用于小批。几千条的 input 数组会被多数服务直接
400(input 上限),所以按 `batch_size`(默认 64)切块,块之间顺序发送。
不超过一块时行为与切块前逐字相同(`batch_mode="single_request"`)。
"""
if not texts:
return [], []
safe_texts = [_truncate_for_embedding(t) for t in texts]
vectors, meta = await self._embed_openai(safe_texts)
base_meta = dict(meta or {})
base_meta["batch_mode"] = "single_request"
base_meta["batch_input_count"] = len(safe_texts)
chunk = max(1, int(self._embed_config.batch_size or 64))
vectors: List[List[float]] = []
all_metas: List[Dict[str, Any]] = []
for idx in range(len(safe_texts)):
m = dict(base_meta)
m["batch_index"] = idx
all_metas.append(m)
chunk_count = (len(safe_texts) + chunk - 1) // chunk
for start in range(0, len(safe_texts), chunk):
part = safe_texts[start:start + chunk]
retry_kwargs = {} if max_retries is None else {"max_retries": max_retries}
part_vectors, meta = await self._embed_openai(part, **retry_kwargs)
base_meta = dict(meta or {})
base_meta["batch_mode"] = "single_request" if chunk_count == 1 else "chunked"
base_meta["batch_input_count"] = len(part)
if chunk_count > 1:
base_meta["batch_chunk_index"] = start // chunk
base_meta["batch_chunk_count"] = chunk_count
for idx in range(len(part)):
m = dict(base_meta)
m["batch_index"] = start + idx
all_metas.append(m)
vectors.extend(part_vectors)
self._last_embed_meta = all_metas[-1] if all_metas else {}
return vectors, all_metas
@@ -753,16 +819,22 @@ class EmbedService:
"""
打一条 INFO 级 embedding 请求日志(含 request_id / provider / model / api / 入参 / 出参)。
出参只含 vector 维度,不含向量数值。request_id 同时写入 message,
出参只含 vector 维度,不含向量数值。入参的 input 是待向量化的原文
(对话 / 记忆 / 查询),INFO 只记每条的字符数(input_chars),原文仅在
DEBUG 单独输出。request_id 同时写入 message,
并依赖 log formatter 的 %(request_id)s(setup_logging 后生效)。
"""
texts = request_payload.get("input")
info_request = {k: v for k, v in request_payload.items() if k != "input"}
if isinstance(texts, list):
info_request["input_chars"] = [len(str(t)) for t in texts]
payload = {
"request_id": get_request_id(),
"status": status,
"provider": self._embed_config.provider,
"model": self._embed_config.model,
"api": api_url,
"request": request_payload,
"request": info_request,
"response": response_payload,
}
try:
@@ -770,9 +842,13 @@ class EmbedService:
except Exception:
body = str(payload)
logger.info(f"[embed] {body}")
if texts is not None and logger.isEnabledFor(logging.DEBUG):
logger.debug(
f"[embed] input: {json.dumps(texts, ensure_ascii=False, default=str)}"
)
async def _embed_openai(
self, texts: List[str]
self, texts: List[str], *, max_retries: Optional[int] = None,
) -> Tuple[List[List[float]], Dict[str, Any]]:
"""
使用 OpenAI 兼容接口向量化
@@ -862,7 +938,9 @@ class EmbedService:
"endpoint_overridden": bool(_ov),
}
_max_retries = max(0, int(self._embed_config.max_retries or 0))
_max_retries = max(0, int(
(self._embed_config.max_retries or 0) if max_retries is None else max_retries
))
_retry_delay = max(0.0, float(self._embed_config.retry_delay or 0.0))
last_exc: Optional[Exception] = None
@@ -908,6 +986,17 @@ class EmbedService:
f"got={len(vectors)}"
)
dims = [len(v) for v in vectors]
expected_dims = self._embed_config.embedding_dims
if expected_dims:
bad = [d for d in dims if d != int(expected_dims)]
if bad:
# 宽度不对的向量写进集合会把集合的维度钉死在错误值上(Chroma
# 首次 add 定维),或在已有集合上整批失败。确定性错误,不重试。
raise EmbeddingDimensionMismatchError(
"embedding dimension mismatch: "
f"expected={int(expected_dims)} got={bad[0]} "
f"model={self._embed_config.model}"
)
meta = {
"http_status": raw.status_code,
"response_headers": resp_headers,
@@ -943,6 +1032,8 @@ class EmbedService:
},
)
return vectors, meta
except EmbeddingDimensionMismatchError:
raise
except Exception as e:
last_exc = e
_err_name = type(e).__name__
@@ -976,12 +1067,18 @@ class EmbedService:
if _retryable and attempt < _max_retries:
# 逐出可能卡死的 client,下个 attempt 重建干净连接(恢复原防御语义)
self._evict_client(_eff_base_url, _eff_api_key, _eff_timeout)
_delay = _retry_delay
_retry_after = _retry_after_seconds(e) if _status == 429 else None
if _retry_after is not None:
# 限流时服务端明确告诉了要等多久:按它等,再加一点抖动,
# 避免多个 worker 在同一时刻一起醒来再次撞限流。
_delay = max(_delay, _retry_after) + random.uniform(0, 0.25 * _retry_after)
logger.warning(
f"[embed] attempt {attempt + 1}/{_max_retries + 1} failed "
f"({_err_name}): {_err_msg[:200]}; retry in {_retry_delay}s"
f"({_err_name}): {_err_msg[:200]}; retry in {_delay:.2f}s"
)
if _retry_delay > 0:
await asyncio.sleep(_retry_delay)
if _delay > 0:
await asyncio.sleep(_delay)
continue
self._log_embed_call(
status="error",
+18
View File
@@ -22,6 +22,7 @@ import logging
import json
import threading
import time
import weakref
from ..models.memory import MemoryNode, MemoryLayer, MemoryStatus
from ..config import MemoryConfig
@@ -77,6 +78,21 @@ async def _run_in_kuzu_pool(func, *args, **kwargs):
return await run_in_executor_guarded(_kuzu_executor, _measured)
# 本进程里已打开、尚未 close 的 store。drop_graph 据此拒绝删掉一个正被
# 活着的 client 使用的图库。WeakSet:store 被回收后自动消失。
_LIVE_STORES: "weakref.WeakSet[KuzuGraphStore]" = weakref.WeakSet()
def open_graph_paths() -> List[str]:
"""Real paths of the graph databases live stores in this process hold open."""
import os
return [
os.path.realpath(str(store._db_path))
for store in list(_LIVE_STORES)
if store._db is not None
]
def kuzu_executor_snapshot() -> Dict[str, Any]:
with _kuzu_stats_lock:
return {
@@ -350,6 +366,7 @@ class KuzuGraphStore(GraphStoreBase):
)
self._db = db
self._conn = conn
_LIVE_STORES.add(self)
except asyncio.TimeoutError:
logger.warning(
f"GraphStore init timed out after 5s (Kuzu file lock contention). "
@@ -2441,6 +2458,7 @@ class KuzuGraphStore(GraphStoreBase):
async def close(self) -> None:
"""关闭连接"""
conn, db = self._conn, self._db
_LIVE_STORES.discard(self)
self._conn = None
self._db = None
self._available = False
+14 -6
View File
@@ -89,6 +89,17 @@ def passes_created_after(
logger = logging.getLogger(__name__)
def physical_collection_name(base_name: Optional[str], dims: Optional[int]) -> str:
"""The collection name a store opens for ``base_name`` at ``dims``.
The dims suffix keeps two embedding widths from sharing one collection.
`hy_memory.reembed.vector_space_names` uses this too, so callers never
re-derive the rule.
"""
base = base_name or "agent_memories"
return f"{base}_{dims}" if dims else base
class VectorStoreBase(ABC):
"""
向量存储抽象基类
@@ -98,13 +109,10 @@ class VectorStoreBase(ABC):
def __init__(self, config: MemoryConfig):
self.config = config
base_name = config.vector_store.collection_name or "agent_memories"
dims = config.vector_store.embedding_dims
# 自动拼接维度后缀,避免不同 embedding 模型写入同一 collection 导致维度冲突
if dims:
self._collection_name = f"{base_name}_{dims}"
else:
self._collection_name = base_name
self._collection_name = physical_collection_name(
config.vector_store.collection_name, config.vector_store.embedding_dims,
)
_supports_retrieval_clauses: bool = False
_supports_narrow_episode_queries: bool = False
+29 -1
View File
@@ -9,10 +9,12 @@ Agent Memory V2 - VectorStore (Chroma)
- 同步 API → _run_in_vdb_pool() 包装为异步
"""
from typing import Optional, List, Dict, Any
from typing import Optional, List, Dict, Any, Set, Tuple
import concurrent.futures
import functools
import logging
import os
import weakref
from ..models.memory import (
MemoryNode, MemoryLayer, MemoryStatus,
@@ -33,6 +35,30 @@ _vdb_executor = concurrent.futures.ThreadPoolExecutor(
)
# 本进程里已 initialize、尚未 close 的 store。drop_vector_space 据此拒绝删掉
# 一个正被活着的 client 使用的集合。WeakSet:store 被回收后自动消失。
_LIVE_STORES: "weakref.WeakSet[ChromaVectorStore]" = weakref.WeakSet()
def _location_key(vs_config) -> Tuple[str, str]:
if getattr(vs_config, "host", None):
return ("http", f"{vs_config.host}:{vs_config.port or 8000}")
return ("local", os.path.realpath(str(vs_config.persist_directory or "")))
def open_collection_names(location: Tuple[str, str]) -> Set[str]:
"""Collections that live stores in this process hold open at ``location``."""
names: Set[str] = set()
for store in list(_LIVE_STORES):
if store._client is None or _location_key(store.config.vector_store) != location:
continue
names.add(store._collection_name)
if store._entity_collection is not None:
from ..pipelines._retrieval.config import entity_collection_name
names.add(entity_collection_name(store._collection_name))
return names
async def _run_in_vdb_pool(func, *args, **kwargs):
"""Run a synchronous Chroma call in the dedicated VDB executor."""
call = functools.partial(func, *args, **kwargs) if args or kwargs else func
@@ -88,6 +114,7 @@ class ChromaVectorStore(VectorStoreBase):
return client, collection
self._client, self._collection = await _run_in_vdb_pool(_init)
_LIVE_STORES.add(self)
logger.debug(f"VectorStore initialized (Chroma), collection={self._collection_name}")
# ================================================================
@@ -561,6 +588,7 @@ class ChromaVectorStore(VectorStoreBase):
async def close(self) -> None:
"""关闭客户端"""
# Chroma PersistentClient 无需显式关闭
_LIVE_STORES.discard(self)
self._client = None
self._collection = None
self._entity_collection = None
@@ -115,11 +115,16 @@ class ReconcileHybridRetriever:
fused_max = max(fused_all) if fused_all else 0.0
base = (
f"[reconcile-hybrid][{req}] pool={pool_size} q_terms={q_terms!r} "
f"[reconcile-hybrid][{req}] pool={pool_size} q_terms={len(q_terms)} "
f"raw_bm25 max={raw_max:.4f} nonzero={raw_nonzero}/{len(raw_bm25)} "
f"bm25_norm max={norm_max:.4f} nonzero={norm_nonzero}/{len(bm25_norms)} "
f"sem_max={sem_max:.4f} fused_max={fused_max:.4f} "
f"kept={kept}/{len(fused_all)} min_score={self._cfg.min_score} "
f"query({len(query_text or '')} chars)"
)
# query 是待对账的新记忆原文,词项与预览只进 DEBUG
logger.debug(
f"[reconcile-hybrid][{req}] q_terms={q_terms!r} "
f"query={q_preview!r} lemmatized={lem_preview!r}"
)
+2 -1
View File
@@ -692,9 +692,10 @@ class CrossDomainSweeper:
)
logger.info(
f"[sweeper] Created L6 core {core_id[:12]}: '{core_content[:60]}' "
f"[sweeper] Created L6 core {core_id[:12]} "
f"from {len(cluster_ids)} basics, evidence_inherited={len(inherited_evidence_ids)}"
)
logger.debug(f"[sweeper] L6 core {core_id[:12]} content={core_content[:60]!r}")
return 1, 0, [core_id]
except Exception as e:
+2 -1
View File
@@ -3587,8 +3587,9 @@ class BaseReadPipeline(ReadPipeline):
f"hidden_l0={len(hidden_l0_hits)} fwd_l6={len(forward_l6)} rev_l6={len(reverse_l6)} "
f"fused_l6={len(fused_l6)} graph_intent={len(graph_intention_hits)} "
f"returned={len(final_results)} "
f"query='{request.query[:80]}'"
f"query({len(request.query or '')} chars)"
)
logger.debug(f"[read/{self.VERSION}] query={request.query[:80]!r}")
response.success = True
+2 -1
View File
@@ -2516,7 +2516,8 @@ def _parse_operations_json(text: str) -> Optional[List[Dict[str, Any]]]:
return result
return None
except json.JSONDecodeError:
logger.warning(f"[S2-agent] JSON parse failed: {json_str[:200]}")
logger.warning(f"[S2-agent] JSON parse failed ({len(json_str)} chars)")
logger.debug(f"[S2-agent] unparsable output: {json_str[:200]!r}")
return None
+56 -18
View File
@@ -280,6 +280,22 @@ class MemoryWriter(WritePipeline):
custom["source_raw_memory_ids"] = source_ids
return custom
def _flat_trimmed_raw(self, request: WriteRequest) -> str:
"""把 request.messages 按 raw 统一规则裁剪后拼成 L1_RAW 文本。
只裁落库副本;request.messages 不动(本次 extractor 输入仍用完整 messages)。
"""
import copy as _copy
from ..utils.raw_trim import trim_raw_messages
_coding_cfg = getattr(getattr(self, "config", None), "coding", None)
_trimmed_req = _copy.copy(request)
_trimmed_req.messages = trim_raw_messages(
request.messages,
tool_max_bytes=getattr(_coding_cfg, "tool_result_max_bytes", None),
)
return _trimmed_req.get_flat_content()
@staticmethod
def _attach_client_meta(custom: Dict[str, Any], request: Optional[WriteRequest]) -> None:
"""给新建节点挂上调用方 meta(单份 list)。
@@ -2554,9 +2570,14 @@ class MemoryWriter(WritePipeline):
try:
# 参数校验
content = request.content
# raw 在写入时就按统一规则裁好(见 utils/raw_trim.py):工具输出、长
# assistant 回复不再原样长期存储;历史拼接直接用存下来的文本,不再二次裁剪。
# request.messages 保持原样——本次 extractor 输入仍是完整 messages。
_raw_trimmed = False
if not content and request.has_messages():
content = request.get_flat_content()
content = self._flat_trimmed_raw(request)
request.content = content
_raw_trimmed = True
if not content:
response.error_code = 400
@@ -2627,6 +2648,9 @@ class MemoryWriter(WritePipeline):
vector_store = await self._get_vector_store()
layer_enum = MemoryLayer.from_string(suggested_layer)
_raw_custom: Dict[str, Any] = {}
if _raw_trimmed:
from ..utils.raw_trim import RAW_TRIMMED_KEY
_raw_custom[RAW_TRIMMED_KEY] = True
self._attach_client_meta(_raw_custom, request)
mem_node = MemoryNode(
user_id=request.user_id,
@@ -2929,13 +2953,16 @@ class MemoryWriter(WritePipeline):
agent_id: Optional[str] = None,
exclude_memory_id: str = "",
max_turns: int = 20,
max_chars_assistant: int = 500,
max_chars_assistant: Optional[int] = None,
) -> str:
"""
获取该用户最近的 L1_RAW 对话记录,按轮次拆分后拼成 history context。
每条 L1_RAW 的 content 是多轮对话([user]: ...\n[assistant]: ...)。
拆成单条 message 后,user 消息完整保留,assistant 消息截取前 max_chars_assistant 字符。
raw 在写入时已按 utils/raw_trim.py 的规则裁好(custom[RAW_TRIMMED_KEY]),
这里拆成单条 message 后直接拼接,不再二次裁剪。该规则之前写入的旧 raw
没有这个标记,仍按同一(幂等)规则裁剪后再拼,避免旧的长回复 / 工具输出
撑大 prompt。max_chars_assistant 只作用于这些旧 raw,默认同写入规则。
最终取最近 max_turns 条 message(注意单位是 message 条数,非对话轮数;
调用方按 k 轮 × 2 传入)。
@@ -2977,23 +3004,33 @@ class MemoryWriter(WritePipeline):
# 避免把正文里形如 "[注]: xxx" 的普通文本误判成消息边界。
import re as _re
from .base import parse_user_name_line
_KNOWN_ROLES = {"user", "assistant", "system", "tool"}
# 带方括号的任意 label(含对方真人名字)
_labeled_prefix = _re.compile(r"^\[([^\]\n]{1,32})\]:\s?")
# 无方括号的已知 role(历史数据容错,沿用旧正则的 `\[?...\]?` 宽容度)
_bare_role_prefix = _re.compile(r"^(user|assistant|system|tool):\s?")
# (label, content) 元组列表;label 为 role 或对方真人名字
parsed: List[Tuple[str, str]] = []
from ..utils.raw_trim import (
DEFAULT_RAW_ASSISTANT_MAX_CHARS,
RAW_TRIMMED_KEY,
trim_raw_body,
)
if max_chars_assistant is None:
max_chars_assistant = DEFAULT_RAW_ASSISTANT_MAX_CHARS
_coding_cfg = getattr(getattr(self, "config", None), "coding", None)
_tool_max_bytes = getattr(_coding_cfg, "tool_result_max_bytes", None)
# (label, content, 是否旧 raw 需兼容裁剪) 元组列表;label 为 role 或对方真人名字
parsed: List[Tuple[str, str, bool]] = []
user_name: str = ""
for node in nodes:
content = node.content or ""
_legacy = not bool((node.custom or {}).get(RAW_TRIMMED_KEY))
for _line_idx, line in enumerate(content.split("\n")):
m = _labeled_prefix.match(line) or _bare_role_prefix.match(line)
if m:
label = m.group(1)
body = line[m.end():]
parsed.append((label, body))
parsed.append((label, body, _legacy))
continue
# 名字行只可能是每个 node content 的第一行(get_flat_content 插在文首)。
# 逐 node 判断而非用 `not parsed`:parsed 跨 node 累积,第二个 node
@@ -3006,18 +3043,18 @@ class MemoryWriter(WritePipeline):
continue
if parsed:
# 续行:拼回上一条 message(保留换行)
prev_label, prev_body = parsed[-1]
parsed[-1] = (prev_label, prev_body + "\n" + line)
prev_label, prev_body, prev_legacy = parsed[-1]
parsed[-1] = (prev_label, prev_body + "\n" + line, prev_legacy)
else:
# 开头就是无前缀行(异常/老数据):当成 user 消息兜底
parsed.append(("user", line))
parsed.append(("user", line, _legacy))
# 取最近 max_turns 条 message(按 message 截断,绝不从消息中间切开)
recent = parsed[-max_turns:] if len(parsed) > max_turns else parsed
if not recent:
return ""
# 格式化:非 user 侧整条截断(含对方真人),user 完整保留
# 格式化:新 raw 直接拼接;旧 raw 按写入同一规则兼容裁剪
all_messages: List[str] = []
if user_name:
from .base import render_user_name_line
@@ -3025,17 +3062,18 @@ class MemoryWriter(WritePipeline):
_name_line = render_user_name_line(
user_name,
is_chinese("\n".join(b for _, b in recent)),
is_chinese("\n".join(b for _, b, _ in recent)),
)
if _name_line:
all_messages.append(_name_line)
for label, body in recent:
for label, body, legacy in recent:
if legacy:
body = trim_raw_body(
label, body,
assistant_max_chars=max_chars_assistant,
tool_max_bytes=_tool_max_bytes,
)
body = body.strip()
# 截断口径:assistant + 对方真人(非已知 role 的 label)。
# user 完整保留;system / tool 沿用历史行为(不截断)。
_truncate = label == "assistant" or label not in _KNOWN_ROLES
if _truncate and len(body) > max_chars_assistant:
body = body[:max_chars_assistant] + "..."
all_messages.append(f"[{label}]: {body}")
return "\n".join(all_messages)
+1777
View File
File diff suppressed because it is too large. Load diff
+4 -1
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
import asyncio
import concurrent.futures
import contextvars
import functools
from concurrent.futures import Executor
from typing import Any, Callable, Optional
@@ -39,7 +40,9 @@ async def run_in_executor_guarded(
async def to_thread_guarded(func: Callable[..., Any], *args: Any, **kwargs: Any) -> Any:
"""Guarded equivalent of ``asyncio.to_thread`` using the default pool."""
return await run_in_executor_guarded(None, func, *args, **kwargs)
context = contextvars.copy_context()
call = functools.partial(func, *args, **kwargs)
return await run_in_executor_guarded(None, context.run, call)
def concurrent_result_guarded(
+2 -1
View File
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
"""
Pipeline Log Writer — 环节级 JSONL 日志(始终开启,与 SQLite Trace 分离)。
Pipeline Log Writer — 环节级 JSONL 日志(默认开启,MEMORY_PIPELINE_LOG_ENABLED=false 关闭;
与 SQLite Trace 分离)。
目录: {log_dir}/{subdir}/{date}.log
每行: request_id, namespace, user_id, agent_id, step, prompt, content, parsed,
+71 -30
View File
@@ -1,16 +1,20 @@
# -*- coding: utf-8 -*-
"""
Pipeline 可观测性:Log(文件 JSONL,始终开启)与 Trace(SQLite pipeline_logs,可关闭)。
Pipeline 可观测性:Log(文件 JSONL,默认开启,可关闭)与 Trace(SQLite pipeline_logs,可关闭)。
- Log:运维 tail/grep,每个环节即时一行,不依赖后续步骤成功与否。
- Trace:Inspector / 调试,落库 pipeline_logs(需 MEMORY_CACHE_BACKEND=sqlite 等持久化后端)。
环境变量:
MEMORY_PIPELINE_LOG_ENABLED — Log 写 JSONL 文件,默认 true;false 时完全不写
MEMORY_PIPELINE_TRACE_ENABLED — Trace 落库,默认 true;OpenClaw C 端可设 false
MEMORY_MEMORY_OPERATIONS_ENABLED — memory_operations 审计表落库,默认 false
MEMORY_LOG_DIR — Log 目录(优先)
MEMORY_TRACE_LOG_DIR — Log 目录(兼容旧名)
Log 与 Trace 都关闭时不包装 store_pipeline_log,而是换成立即返回 True 的
空操作:热路径不做任何计时、脱敏或后台任务,也不写库。
子目录({log_dir}/{subdir}/{date}.log)由 PipelineLogWriter 从每条记录的
user_id 前缀(第一个 "__" 左侧)切分;无前缀回落 "default"。该前缀是
通用 namespace,可用于项目、部署或数据分桶。
@@ -94,6 +98,16 @@ def is_pipeline_trace_enabled() -> bool:
)
def is_pipeline_log_enabled() -> bool:
"""是否把 pipeline step 写成 JSONL 文件({log_dir}/<subdir>/<date>.log)。默认开启。"""
return os.getenv("MEMORY_PIPELINE_LOG_ENABLED", "true").strip().lower() in (
"true",
"1",
"yes",
"on",
)
def is_memory_operations_enabled() -> bool:
"""是否将知识库变动写入 memory_operations 审计表。默认关闭(false)。"""
return os.getenv("MEMORY_MEMORY_OPERATIONS_ENABLED", "false").lower() in (
@@ -125,7 +139,7 @@ def redact_content(value: str) -> str:
def resolve_pipeline_log_dir() -> str:
"""
JSONL pipeline log 根目录(始终使用,有默认值)。
JSONL pipeline log 根目录(MEMORY_PIPELINE_LOG_ENABLED 开启时使用,有默认值)。
优先级: MEMORY_LOG_DIR > MEMORY_TRACE_LOG_DIR > {MEMORY_DATA_DIR}/logs/pipeline
"""
@@ -228,6 +242,7 @@ def wrap_store_pipeline_log(
) -> Callable[..., Any]:
"""包装 cache.store_pipeline_log:计时同步、JSONL/DB 异步落盘,不堵主请求。
- log_writer 为 None 时不写 JSONL(MEMORY_PIPELINE_LOG_ENABLED=false)
- step-to-step elapsed_ms / event_at_ms 在调用瞬间打点(与业务墙钟对齐)
- event_at_ms:调用方传入非 0 值则直接复用(App 已在自己被调用的那一刻
同步打点,比这里更接近真实事件时刻);传 0 / 不传才用本地 time.time()
@@ -410,7 +425,8 @@ def wrap_store_pipeline_log(
try:
loop = asyncio.get_running_loop()
# JSONL 用线程池,与 loop 无亲和,留在当前 loop。
_track_task(loop.create_task(_persist_jsonl()))
if log_writer is not None:
_track_task(loop.create_task(_persist_jsonl()))
# DB trace 用 cache pool(可能 aiomysql,绑定 owner_loop)。若当前不在
# owner_loop(如 search 走独立 loop),派发回 owner_loop 执行,避免跨 loop。
if trace_enabled:
@@ -446,25 +462,26 @@ def wrap_store_pipeline_log(
)
except RuntimeError:
# 无 running loop(极少见):退化为同步写,保证不丢日志
try:
log_writer.write_step(
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
step=step,
prompt=prompt,
response=response,
parsed=parsed,
memory_ids=mem_ids,
elapsed_ms=step_elapsed_ms,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=total_tokens,
event_at_ms=event_at_ms,
)
except Exception:
pass
if log_writer is not None:
try:
log_writer.write_step(
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
step=step,
prompt=prompt,
response=response,
parsed=parsed,
memory_ids=mem_ids,
elapsed_ms=step_elapsed_ms,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=total_tokens,
event_at_ms=event_at_ms,
)
except Exception:
pass
if trace_enabled:
try:
result = original_store(
@@ -496,6 +513,11 @@ def wrap_store_pipeline_log(
return _store_pipeline_observability
async def _noop_store_pipeline_log(*args: Any, **kwargs: Any) -> bool:
"""Log 与 Trace 都关闭时的 store_pipeline_log:不计时、不写文件、不写库。"""
return True
def install_pipeline_observability(
cache: Any,
*,
@@ -503,6 +525,9 @@ def install_pipeline_observability(
) -> None:
"""给 cache 安装 JSONL + Trace 包装(fire-and-forget)。
MEMORY_PIPELINE_LOG_ENABLED=false 时不写 JSONL;若 Trace 也关闭,
store_pipeline_log 被换成空操作(不包装、不写库)。
在 cache 所属 loop(write loop / _LoopThread)上调用,本函数会捕获该 loop
作为 owner_loop 传给包装器:search 走独立 loop 后,从 search loop 触发的 DB
落库会被派发回这个 owner_loop,避免跨 loop 用 aiomysql 污染连接池。
@@ -514,15 +539,31 @@ def install_pipeline_observability(
except RuntimeError:
owner_loop = None
log_dir = resolve_pipeline_log_dir()
log_writer = PipelineLogWriter(log_dir)
log_enabled = is_pipeline_log_enabled()
trace_enabled = is_pipeline_trace_enabled()
logger.info(
f"[{log_label}] Pipeline log (JSONL) enabled: {log_dir}/<subdir>/<date>.log "
f"(subdir from user_id prefix before '__', else default) ; "
f"pipeline trace (DB)={'on' if trace_enabled else 'off'} ; "
f"persist=async(fire-and-forget)"
)
if not log_enabled and not trace_enabled:
logger.info(
f"[{log_label}] Pipeline log (JSONL) disabled ; pipeline trace (DB)=off ; "
f"store_pipeline_log=no-op"
)
cache.store_pipeline_log = _noop_store_pipeline_log
return
log_writer = None
if log_enabled:
log_dir = resolve_pipeline_log_dir()
log_writer = PipelineLogWriter(log_dir)
logger.info(
f"[{log_label}] Pipeline log (JSONL) enabled: {log_dir}/<subdir>/<date>.log "
f"(subdir from user_id prefix before '__', else default) ; "
f"pipeline trace (DB)={'on' if trace_enabled else 'off'} ; "
f"persist=async(fire-and-forget)"
)
else:
logger.info(
f"[{log_label}] Pipeline log (JSONL) disabled ; "
f"pipeline trace (DB)=on ; persist=async(fire-and-forget)"
)
cache.store_pipeline_log = wrap_store_pipeline_log(
cache.store_pipeline_log,
log_writer=log_writer,
+91
View File
@@ -0,0 +1,91 @@
# -*- coding: utf-8 -*-
"""
原始对话(raw)落库前的统一裁剪规则(规则,零 LLM)。
raw 会被长期保存(MemHub 已取消 raw 的保留期限;lite 模式下 raw 是唯一存储),
不裁剪会把工具输出、大段粘贴日志原样存下来。因此 raw 在**写入时**就裁好,
读取方(extractor 的「最近 k 条」历史注入)直接拼接,不再二次裁剪。
规则按说话人标签(ChatMessage.speaker_label() / L1_RAW 里的 ``[label]``):
- ``user``:完整保留;
- ``system``:完整保留(与历史注入原口径一致);
- ``tool``:头尾保留、中段替换为 ``[...N bytes omitted by truncate...]``,
上限为 ``coding.tool_result_max_bytes``(env ``MEMORY_CODING_TOOL_RESULT_MAX_BYTES``,
默认 2048),与 coding 链路同一个函数;
- 其余(``assistant``、对方真人名字等非已知 role 的 label):strip 后超过
``assistant_max_chars``(默认 500,即历史注入原来的 ``max_chars_assistant``)
时截成前 N 个字符 + ``...``。
所有规则确定且幂等:对已裁剪的文本再裁一次结果不变。
"""
from typing import List, Optional
from ..pipelines.base import ChatMessage
# 历史注入原 `_get_recent_history(max_chars_assistant=500)` 的默认值,现为唯一来源。
DEFAULT_RAW_ASSISTANT_MAX_CHARS = 500
# L1_RAW 节点 custom 上的标记:内容已按本模块规则在写入时裁剪。
# 历史拼接据此区分「新 raw 直接拼接」与「本规则之前写入的旧 raw 需兼容裁剪」。
RAW_TRIMMED_KEY = "raw_trimmed"
# 不裁剪的标签(tool 单独处理)
_KEEP_LABELS = frozenset({"user", "system"})
def trim_raw_body(
label: str,
body: str,
*,
assistant_max_chars: int = DEFAULT_RAW_ASSISTANT_MAX_CHARS,
tool_max_bytes: Optional[int] = None,
) -> str:
"""按说话人标签裁剪单条消息正文。幂等。"""
if not isinstance(body, str):
body = "" if body is None else str(body)
if label in _KEEP_LABELS:
return body
if label == "tool":
from ..coding.preproc import (
DEFAULT_MAX_TOOL_RESULT_BYTES,
truncate_tool_result_text,
)
return truncate_tool_result_text(
body, max_bytes=int(tool_max_bytes or DEFAULT_MAX_TOOL_RESULT_BYTES),
)
stripped = body.strip()
if len(stripped) > assistant_max_chars:
return stripped[:assistant_max_chars] + "..."
return body
def trim_raw_messages(
messages: List[ChatMessage],
*,
assistant_max_chars: int = DEFAULT_RAW_ASSISTANT_MAX_CHARS,
tool_max_bytes: Optional[int] = None,
) -> List[ChatMessage]:
"""返回裁剪后的新列表;不修改入参(messages 仍要原样给 extractor)。"""
out: List[ChatMessage] = []
for m in messages:
new_content = trim_raw_body(
m.speaker_label(),
m.content,
assistant_max_chars=assistant_max_chars,
tool_max_bytes=tool_max_bytes,
)
if new_content == m.content:
out.append(m)
continue
out.append(ChatMessage(
role=m.role,
content=new_content,
name=m.name,
tool_calls=list(m.tool_calls),
tool_call_id=m.tool_call_id,
tool_name=m.tool_name,
))
return out
+2 -1
View File
@@ -36,7 +36,8 @@ def count_tokens(text: str) -> int:
enc = _get_encoder()
if enc:
try:
return len(enc.encode(text))
# Quoted control-token spellings are ordinary input text.
return len(enc.encode(text, disallowed_special=()))
except Exception as e:
logger.debug(f"[token_count] encode failed, char fallback: {e}")
return max(1, len(text) // 4)
+2 -2
View File
@@ -13,8 +13,8 @@
接口先补 SDK。Runtime writers 停止后的 App 启动迁移可直接操作底层存储,
但不得把迁移专用方法加入公共 Client。
3. MemHub **不存 memory 实体**;**可以存 `memory_id`** 做组织/回执。原文进入 **SDK raw**。
4. SDK raw 只保留滚动三天;一条 raw 对应一个完整 turn,只含真人 user 与
assistant,tool 不持久化。提取后的高层 memory 不受该窗口影响。
4. SDK raw 全部保留,任何模式(lite / pro / ultra)都不删除;一条 raw 对应一个
完整 turn,只含真人 user 与 assistant,tool 不持久化。
5. `source_raw_ids` 是 SDK 内部溯源关系,不是产品 Evidence 合同;记忆详情不提供
“原始依据”或按 raw ID 展开原文的入口。
+6 -4
View File
@@ -29,10 +29,12 @@ prompt/response、密钥、访问令牌或完整本地路径。结构化日志
文件。详细合同见
[`docs/contracts/diagnostic-logging.md`](docs/contracts/diagnostic-logging.md)。
用户显式把诊断级别切换为 `debug` 时,Core 会启用 HY pipeline trace,并将
`MEMORY_LOG_LLM_CONTENT` 设为 `true`。该模式会把完整模型 prompt/response 写入本地调试
日志,其中可能包含获授权的对话正文和模型输出;它只应用于有意识的故障排查,不应长期
开启,也不应在未经检查和脱敏时分享相关日志。切回 `info` 会停止后续正文记录,但不会
HY pipeline 日志与 trace 在任何级别下都关闭。HY SDK 的文本日志只写入
`hy-memory/governed-v2/logs/hy_memory.log`,其中的 WARNING 及以上同时写入 `core.log`;
SDK 不在 INFO 及以上级别记录对话或记忆正文。用户显式把诊断级别切换为 `debug` 时,Core 将
`MEMORY_LOG_LLM_CONTENT` 设为 `true`。该模式会把完整模型 prompt/response 写入
`hy_memory.log` 的 DEBUG 行,其中可能包含获授权的对话正文和模型输出;它只应用于有意识的
故障排查,不应长期开启,也不应在未经检查和脱敏时分享相关日志。切回 `info` 会停止后续正文记录,但不会
自动删除此前已写入的调试文件。
具体行为以当前源码、网络请求实现和发布说明为准。发现实现与本说明不一致时,请按
+4 -4
View File
@@ -18,13 +18,13 @@ If you run into a problem, please open an Issue with the version, platform,repro
## 发行与平台合同
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.6`。MemHub 源码版本为 `1.0.5`
(build 14),GitHub 仓库为
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.9`。MemHub 源码版本为 `1.0.5`
(build 36),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
`v1.0.5` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
`v1.0.5` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64 App 和 Runtime。
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
@@ -118,7 +118,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
## Hy Memory
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.6`。SDK 配置、权限边界及 search
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.9`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+1 -1
View File
@@ -2352,7 +2352,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.6",
"versionInfo": "2.0.9",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+1 -1
View File
@@ -9,7 +9,7 @@ Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.6
## hy-memory 2.0.9
License: Apache-2.0
@@ -315,6 +315,8 @@ describe("catalog exit codes", () => {
"models",
"memories",
"profile",
"settings",
"evolve",
"graph-width",
"host",
"recall",
+45 -1
View File
@@ -162,10 +162,54 @@ it("never emits a secret from a malformed manually written event", async () => {
command: "logs",
state: "empty",
events: 0,
invalid_lines: 1,
invalid_lines: 0,
unrecognized_lines: 1,
});
});
// After an upgrade the files still hold an older build's records. They are
// whole JSON lines the reader leaves out, not damage, so the text output must
// not warn about them; only a line that is not JSON is called damaged.
it("warns only about lines that are not JSON, not about older records", async () => {
const dataDir = await fixture();
const older = {
schema_version: 0,
service: "desktop",
component: "desktop.lifecycle",
event: "desktop.started",
};
await writeFile(
path.join(dataDir, "logs", "diagnostic-desktop.jsonl"),
`${JSON.stringify(older)}\n${JSON.stringify({ ...older, schema_version: 1 })}\n`,
{ mode: 0o600 },
);
let stdout = "";
let stderr = "";
vi.spyOn(process.stdout, "write").mockImplementation((value) => {
stdout += String(value);
return true;
});
vi.spyOn(process.stderr, "write").mockImplementation((value) => {
stderr += String(value);
return true;
});
expect(await run(["logs"], { ONE_MEMORY_DATA_DIR: dataDir })).toBe(0);
expect(stdout).toContain("core.start_failed");
expect(stdout).not.toContain("损坏");
expect(stderr).toBe("");
await writeFile(
path.join(dataDir, "logs", "diagnostic-cli.jsonl"),
'{"schema_version": 1, "event": \n',
{ mode: 0o600 },
);
stdout = "";
expect(await run(["logs"], { ONE_MEMORY_DATA_DIR: dataDir })).toBe(0);
expect(stdout).toContain("跳过 1 行损坏的诊断记录。");
expect(stderr).toBe("memhub: 跳过 1 行损坏的诊断记录。\n");
});
it("rejects invalid filters without touching the log directory", async () => {
const dataDir = path.join(
await mkdtemp(path.join(os.tmpdir(), "om-cli-no-log-")),
+364 -13
View File
@@ -188,10 +188,31 @@ import {
removeManagedCliLink,
rollbackManagedCliLink,
} from "./managed-cli-link.js";
import {
type FeaturesDocument,
featuresExitCode,
parseFeaturesCommand,
renderFeaturesText,
runFeaturesCommand,
} from "./memory-features-flow.js";
import {
type EvolveDocument,
evolveExitCode,
type ModeDocument,
modeExitCode,
parseEvolveCommand,
parseSettingsCommand,
renderEvolveText,
renderModeText,
runEvolveCommand,
runModeCommand,
} from "./memory-mode-flow.js";
import {
addMemory,
DEFAULT_USER_ID,
isMemoryId,
isRequestId,
isUserId,
MEMORY_KINDS,
MEMORY_SCENES,
MEMORY_VARIANTS,
@@ -209,6 +230,7 @@ import {
renderMemoryAddText,
renderMemoryDetailText,
renderMemoryPageText,
userIdParameters,
validMemoryText,
} from "./memory-report.js";
import {
@@ -322,6 +344,8 @@ export interface RunControls {
watchSignal?: AbortSignal;
watchWait?: (milliseconds: number, signal: AbortSignal) => Promise<void>;
watchNow?: () => string;
/** `memhub evolve --wait` polls on this; tests inject an instant one. */
sleep?: (milliseconds: number) => Promise<void>;
readStdin?: () => Promise<Buffer | null>;
stdinIsTTY?: boolean;
/** Injectable onboarding prompter so the wizard can be driven in tests
@@ -538,6 +562,23 @@ async function runCommand(
environment = { ...environment, ONE_MEMORY_DATA_DIR: lifted.dataDir };
}
}
// `--user-id <id>` is global to the memory commands: it names whose
// memories are read or written. It is lifted before dispatch so each
// command's own parser keeps seeing exactly the arguments it always did.
// `sources connect` gives the flag its own meaning and parses it itself.
let userId: string | undefined;
if (USER_ID_COMMANDS.has(arguments_[0] ?? "")) {
const lifted = liftUserIdOption(arguments_);
if (lifted === null) {
return reject(
findCommand(arguments_[0] ?? ""),
asJson,
"--user-id 无效:只能是 1–64 位字母、数字、点、下划线或连字符。",
);
}
arguments_ = lifted.arguments_;
userId = lifted.userId;
}
environment = await environmentWithManagedDataDir(environment);
const command = arguments_[0];
if (command === undefined) return reject(undefined, asJson, "缺少命令。");
@@ -776,9 +817,11 @@ async function runCommand(
const attachment = await attachToRuntimeOrNull(environment);
if (attachment === null) return runtimeUnavailable(descriptor, asJson);
if (parsed.kind === "status") {
const receipt = await readMemoryWrite(attachment, parsed.id).catch(
() => undefined,
);
const receipt = await readMemoryWrite(
attachment,
parsed.id,
userId,
).catch(() => undefined);
if (receipt === undefined) {
return emitFailure(asJson, "write status", "memory_write_read_failed");
}
@@ -812,6 +855,7 @@ async function runCommand(
? {}
: { memory_at: parsed.memory_at }),
...(parsed.priority === undefined ? {} : { priority: parsed.priority }),
...(userId === undefined ? {} : { user_id: userId }),
});
if (outcome.state !== "accepted") {
emit(
@@ -957,9 +1001,12 @@ async function runCommand(
state: "empty",
events: 0,
invalid_lines: report.invalid_lines,
unrecognized_lines: report.unrecognized_lines,
})}\n`
: renderLogText(report.events, report.invalid_lines),
);
// Only lines that are not JSON are damage. Records an older build wrote
// are left out without a warning: after an upgrade they are expected.
if (report.invalid_lines) {
process.stderr.write(
`${PRODUCT}: 跳过 ${report.invalid_lines} 行损坏的诊断记录。\n`,
@@ -1582,6 +1629,7 @@ async function runCommand(
limit: parsed.limit,
character_budget: parsed.characterBudget,
timeout_ms: parsed.timeoutMs,
...userIdParameters(userId),
})
: ({
schema_version: 2,
@@ -1780,6 +1828,16 @@ async function runCommand(
}
if (named === "sources" && arguments_[1] !== "list") {
const parsed = parseSourcesCommand(arguments_.slice(1));
if (
!parsed &&
arguments_.some((item) => item === "--user-id" || item === "--root")
) {
return reject(
descriptor,
asJson,
"--user-id 和 --root 只能用于接入单个 hermes 来源(Hermes profile),用户 ID 只能是 1–64 位字母、数字、点、下划线或连字符。",
);
}
if (!parsed) return usage();
if (parsed.kind === "receipt") {
const connected = await connectRuntime(environment);
@@ -1981,12 +2039,17 @@ async function runCommand(
return sourceProgressExitCode(document);
}
if (named === "profile") {
const profileFlags = arguments_.slice(1);
if (
arguments_.filter((item) => item === "--json").length > 1 ||
arguments_.slice(1).some((value) => value !== "--json")
profileFlags.filter((item) => item === "--json").length > 1 ||
profileFlags.filter((item) => item === "--versions").length > 1 ||
profileFlags.some((value) => value !== "--json" && value !== "--versions")
) {
return usage();
}
if (profileFlags.includes("--versions")) {
return await profileVersionsCommand(environment, { asJson, userId });
}
const connected = await connectRuntime(environment, {
readerTimeoutMs: 15_000,
});
@@ -2004,7 +2067,11 @@ async function runCommand(
);
return connected.exitCode;
}
const outcome = await connected.reader<unknown>("/api/v2/user-profile");
const profileUser = userIdParameters(userId);
const outcome =
profileUser.user_id === undefined
? await connected.reader<unknown>("/api/v2/user-profile")
: await connected.reader<unknown>("/api/v2/user-profile", profileUser);
const value =
outcome.state === "ok"
? (outcome.data as {
@@ -2109,7 +2176,7 @@ async function runCommand(
}
const { reader, operations } = connected;
if (parsed.kind === "list") {
const document = await readMemoryPage(reader, parsed.query);
const document = await readMemoryPage(reader, parsed.query, userId);
emit(
asJson
? `${JSON.stringify(document)}\n`
@@ -2122,6 +2189,7 @@ async function runCommand(
reader,
parsed.memoryId,
parsed.namespace,
userId,
);
emit(
asJson
@@ -2133,8 +2201,8 @@ async function runCommand(
if (parsed.kind === "add" || parsed.kind === "add-status") {
const document =
parsed.kind === "add"
? await addMemory(operations, text ?? "", parsed.requestId)
: await readMemoryAddStatus(reader, parsed.requestId);
? await addMemory(operations, text ?? "", parsed.requestId, userId)
: await readMemoryAddStatus(reader, parsed.requestId, userId);
emit(
asJson
? `${JSON.stringify(document)}\n`
@@ -2149,9 +2217,13 @@ async function runCommand(
content: text,
...namespaceParameters(parsed.namespace),
...(parsed.task === undefined ? {} : { task: parsed.task }),
...userIdParameters(userId),
})
: await operations.delete<unknown>(
withQuery(memoryPath, namespaceParameters(parsed.namespace)),
withQuery(memoryPath, {
...namespaceParameters(parsed.namespace),
...userIdParameters(userId),
}),
);
const document = {
schema_version: 2,
@@ -2177,6 +2249,89 @@ async function runCommand(
);
return outcome.state === "committed" ? EXIT_OK : EXIT_FAILED;
}
if (
named === "settings" &&
arguments_.slice(1).find((value) => value !== "--json") === "features"
) {
const parsed = parseFeaturesCommand(
arguments_
.slice(1)
.filter((value, index, all) => index !== all.indexOf("features")),
);
if (!parsed) return usage();
const connected = await connectRuntime(environment);
if (!connected.ok) {
const document: FeaturesDocument = {
schema_version: 1,
product: PRODUCT,
command: "settings features",
state: "unavailable",
code: connected.code,
};
emit(
asJson ? `${JSON.stringify(document)}\n` : renderFeaturesText(document),
);
return connected.exitCode;
}
const document = await runFeaturesCommand(
parsed,
connected.reader,
connected.operations,
);
emit(
asJson ? `${JSON.stringify(document)}\n` : renderFeaturesText(document),
);
return featuresExitCode(document);
}
if (named === "settings") {
const parsed = parseSettingsCommand(arguments_.slice(1));
if (!parsed) return usage();
const connected = await connectRuntime(environment);
if (!connected.ok) {
const document: ModeDocument = {
schema_version: 1,
product: PRODUCT,
command: "settings mode",
state: "unavailable",
code: connected.code,
};
emit(asJson ? `${JSON.stringify(document)}\n` : renderModeText(document));
return connected.exitCode;
}
const document = await runModeCommand(
parsed,
connected.reader,
connected.operations,
);
emit(asJson ? `${JSON.stringify(document)}\n` : renderModeText(document));
return modeExitCode(document);
}
if (named === "evolve") {
const parsed = parseEvolveCommand(arguments_.slice(1));
if (!parsed) return usage();
const connected = await connectRuntime(environment);
if (!connected.ok) {
const document: EvolveDocument = {
schema_version: 1,
product: PRODUCT,
command: "evolve",
state: "unavailable",
code: connected.code,
};
emit(
asJson ? `${JSON.stringify(document)}\n` : renderEvolveText(document),
);
return connected.exitCode;
}
const document = await runEvolveCommand(parsed, {
userId: userIdParameters(userId).user_id,
reader: connected.reader,
operations: connected.operations,
sleep: controls.sleep,
});
emit(asJson ? `${JSON.stringify(document)}\n` : renderEvolveText(document));
return evolveExitCode(document);
}
if (named === "graph-width") {
const parsed = parseGraphWidthCommand(arguments_.slice(1));
if (!parsed) return usage();
@@ -3675,13 +3830,21 @@ async function connectSources(
asJson,
);
}
if (parsed.importHistory) {
// Another user's Hermes profile is granted and imported in one connect
// request: only that request can carry a user and a root.
if (
parsed.importHistory ||
parsed.userId !== undefined ||
parsed.root !== undefined
) {
const document = await connectSourcesWithImport(reader, operations, {
sourceIds: parsed.sourceIds,
confirmOverrides: parsed.confirmOverrides,
...(parsed.requestId === undefined
? {}
: { requestId: parsed.requestId }),
...(parsed.userId === undefined ? {} : { userId: parsed.userId }),
...(parsed.root === undefined ? {} : { root: parsed.root }),
});
emitFlowDocument(document, asJson);
return flowExitCode(document);
@@ -3834,6 +3997,108 @@ async function emitModelReceipt(
/** Attach to a verified Runtime, telling a stopped one (exit 3) from one
* that is up but unusable (exit 1). */
/**
* `memhub profile --versions`: the long-term profile's L0 version chains,
* latest first, so a value that replaced a better one can be found by id.
*/
async function profileVersionsCommand(
environment: NodeJS.ProcessEnv,
options: { asJson: boolean; userId?: string },
): Promise<number> {
const fail = (state: string, code: string, exitCode: number): number => {
emit(
options.asJson
? `${JSON.stringify({ schema_version: 2, product: PRODUCT, state, code })}\n`
: `画像版本读取失败:${code}\n`,
);
return exitCode;
};
const connected = await connectRuntime(environment, {
readerTimeoutMs: 15_000,
});
if (!connected.ok) {
return fail("unavailable", connected.code, connected.exitCode);
}
const outcome = await connected.reader<unknown>(
"/api/v2/user-profile/versions",
userIdParameters(options.userId),
);
const chains = outcome.state === "ok" ? profileChains(outcome.data) : null;
if (outcome.state !== "ok" || chains === null) {
return fail(
outcome.state === "ok" ? "invalid" : outcome.state,
outcome.state === "ok" ? "response_invalid" : outcome.code,
EXIT_FAILED,
);
}
if (options.asJson) {
emit(
`${JSON.stringify({
schema_version: 2,
product: PRODUCT,
state: "ok",
...(outcome.data as object),
})}\n`,
);
return EXIT_OK;
}
if (chains.length === 0) {
emit("尚未形成用户画像版本。\n");
return EXIT_OK;
}
const lines: string[] = [];
for (const chain of chains) {
lines.push(`画像 ${chain.memory_id}(${chain.status})`);
for (const version of chain.versions) {
const when =
typeof version.gmt_created === "number"
? new Date(version.gmt_created * 1000).toISOString()
: "时间未知";
lines.push(
` ${version.current ? "*" : " "} ${version.memory_id} ${when} ${version.content}`,
);
}
}
lines.push(
"* 表示当前生效的版本;删除错误版本:memhub memories delete <memory-id> --yes",
);
emit(`${lines.join("\n")}\n`);
return EXIT_OK;
}
type ProfileChainView = {
memory_id: string;
status: string;
versions: {
memory_id: string;
content: string;
gmt_created?: number | null;
current: boolean;
}[];
};
function profileChains(value: unknown): ProfileChainView[] | null {
if (!value || typeof value !== "object") return null;
const chains = (value as { chains?: unknown }).chains;
if (!Array.isArray(chains)) return null;
const valid = chains.every(
(chain) =>
chain &&
typeof chain === "object" &&
typeof (chain as ProfileChainView).memory_id === "string" &&
typeof (chain as ProfileChainView).status === "string" &&
Array.isArray((chain as ProfileChainView).versions) &&
(chain as ProfileChainView).versions.every(
(version) =>
version &&
typeof version.memory_id === "string" &&
typeof version.content === "string" &&
typeof version.current === "boolean",
),
);
return valid ? (chains as ProfileChainView[]) : null;
}
async function connectRuntime(
environment: NodeJS.ProcessEnv,
options?: { readerTimeoutMs?: number },
@@ -4101,6 +4366,8 @@ const SDK_MIGRATION_FAILURES: Record<string, string> = {
"核对发现新版本还读不到你的全部记忆,没有切换数据源。",
sdk_migration_model_profile_required:
"迁移需要先有可用的模型配置,用来核对新版本能否读回记忆。请先用 memhub models setup 完成配置,再运行 memhub restart 重试。",
sdk_migration_vector_profile_missing:
"记忆的向量来自一个已删除的模型配置,当前配置读不回它们,所以没有迁移。请重新添加并启用那个模型配置,再运行 memhub restart 重试。",
};
const MIGRATION_ISSUES_URL =
@@ -5131,14 +5398,23 @@ type SourcesCommand =
importHistory: boolean;
confirmOverrides: boolean;
requestId?: string;
/** `--user-id`: whose grant (Hermes only). Absent for `local`. */
userId?: string;
/** `--root`: an absolute root other than the candidate (Hermes only). */
root?: string;
}
| { kind: "all"; importHistory: boolean; confirmOverrides: boolean }
| { kind: "receipt"; requestId: string };
const SOURCE_ID_SHAPE = /^[a-z0-9][a-z0-9-]{0,127}$/;
/** Only Hermes keeps one history per profile, so only Hermes may be
* connected for another user or from another root. */
const MULTI_USER_SOURCES = new Set(["hermes"]);
/** `sources connect <id> [<id> ...] | --all-detected [--import]
* [--confirm-overrides] [--request-id <uuid>]` and `sources receipt <id>`.
* [--confirm-overrides] [--request-id <uuid>]
* [--user-id <id>] [--root <path>]` and `sources receipt <id>`.
*
* Sources may be named positionally or with repeated `--source` (the old
* `init --source` spelling is rewritten to this command). */
@@ -5154,6 +5430,8 @@ function parseSourcesCommand(rest: readonly string[]): SourcesCommand | null {
const sourceIds: string[] = [];
const switches = new Set<string>();
let requestId: string | undefined;
let userId: string | undefined;
let root: string | undefined;
for (let index = 1; index < tokens.length; index += 1) {
const token = tokens[index];
if (["--import", "--confirm-overrides", "--all-detected"].includes(token)) {
@@ -5161,6 +5439,21 @@ function parseSourcesCommand(rest: readonly string[]): SourcesCommand | null {
switches.add(token);
continue;
}
if (token === "--user-id" || token === "--root") {
const value = tokens[index + 1];
if (value === undefined || value.startsWith("--")) return null;
if (token === "--user-id") {
if (userId !== undefined || !isUserId(value)) return null;
userId = value;
} else {
if (root !== undefined) return null;
const expanded = expandRootPath(value);
if (expanded === null) return null;
root = expanded;
}
index += 1;
continue;
}
if (token === "--source" || token === "--request-id") {
const value = tokens[index + 1];
if (value === undefined || value.startsWith("--")) return null;
@@ -5184,10 +5477,23 @@ function parseSourcesCommand(rest: readonly string[]): SourcesCommand | null {
const confirmOverrides = switches.has("--confirm-overrides");
if (
requestId !== undefined &&
(!OPERATION_ID.test(requestId) || !importHistory)
(!OPERATION_ID.test(requestId) ||
!(importHistory || userId !== undefined || root !== undefined))
) {
return null;
}
// `--user-id local` is the default and changes nothing.
if (userId === DEFAULT_USER_ID) userId = undefined;
if (userId !== undefined || root !== undefined) {
// One Hermes profile per connect: the user and root name one grant.
if (
switches.has("--all-detected") ||
sourceIds.length !== 1 ||
!MULTI_USER_SOURCES.has(sourceIds[0])
) {
return null;
}
}
if (switches.has("--all-detected")) {
// One names exact sources, the other means "whatever is detected".
if (sourceIds.length > 0 || requestId !== undefined) return null;
@@ -5200,9 +5506,24 @@ function parseSourcesCommand(rest: readonly string[]): SourcesCommand | null {
importHistory,
confirmOverrides,
...(requestId === undefined ? {} : { requestId }),
...(userId === undefined ? {} : { userId }),
...(root === undefined ? {} : { root }),
};
}
/** `--root` as an absolute path: a leading `~` is the home directory, a
* relative path is taken from the working directory. Core expands nothing. */
function expandRootPath(value: string): string | null {
if (value.length === 0 || /[\0\r\n]/.test(value)) return null;
const expanded =
value === "~"
? os.homedir()
: value.startsWith("~/")
? path.join(os.homedir(), value.slice(2))
: value;
return path.resolve(expanded);
}
function parseGraphWidthCommand(
rest: readonly string[],
): { decision: GraphWidthDecision; confirmed: boolean } | null {
@@ -6596,6 +6917,36 @@ const OWN_DATA_DIR_COMMANDS = new Set([
/** Remove one `--data-dir <absolute path>` from argv; null when it is
* repeated, missing its value, or not an absolute path. */
/** Memory commands that honour the global `--user-id`. */
const USER_ID_COMMANDS = new Set([
"memories",
"recall",
"profile",
"write",
"evolve",
]);
/**
* Take `--user-id <id>` out of the arguments. The default user (`local`) is
* returned as undefined, so a default request names no user at all and stays
* byte-identical to one sent before multi-user support.
*/
function liftUserIdOption(
arguments_: readonly string[],
): { arguments_: string[]; userId?: string } | null {
const index = arguments_.indexOf("--user-id");
if (index === -1) return { arguments_: [...arguments_] };
if (arguments_.indexOf("--user-id", index + 1) !== -1) return null;
const value = arguments_[index + 1];
if (value === undefined || value.startsWith("--") || !isUserId(value)) {
return null;
}
return {
arguments_: [...arguments_.slice(0, index), ...arguments_.slice(index + 2)],
...(value === DEFAULT_USER_ID ? {} : { userId: value }),
};
}
function liftDataDirOption(
arguments_: readonly string[],
): { arguments_: string[]; dataDir?: string } | null {
+86 -8
View File
@@ -68,6 +68,12 @@ const JSON_OPTION: CommandOption = {
flag: "--json",
summary: "以 JSON 输出到 stdout,供脚本与 Agent 消费。",
};
/** Global to the memory commands: whose memories are read or written. */
const USER_ID_OPTION: CommandOption = {
flag: "--user-id <id>",
summary:
"读写哪个用户的记忆,默认 local(App 用的就是 local)。1–64 位字母、数字、点、下划线或连字符;其他用户的记忆来自用 sources connect hermes --user-id 接入的 Hermes profile。",
};
const WATCH_OPTION: CommandOption = {
flag: "--watch",
summary: "串行持续刷新;Ctrl-C 只停止观察,不停止 Runtime。",
@@ -252,15 +258,25 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "sources",
summary:
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。",
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。Hermes 的每个 profile 是一个独立用户:用 --user-id 和 --root 接入,授权并导入历史(会自动排除 ~/.hermes/profiles/**,各用户互不读取)。",
usage:
"memhub sources list [--json]\n memhub sources connect <id> [<id> ...] [--import] [--confirm-overrides] [--request-id <uuid>] [--json]\n memhub sources connect --all-detected [--import] [--confirm-overrides] [--json]\n memhub sources receipt <request-id> [--json]",
"memhub sources list [--json]\n memhub sources connect <id> [<id> ...] [--import] [--confirm-overrides] [--request-id <uuid>] [--json]\n memhub sources connect hermes --user-id <id> --root <path> [--import] [--request-id <uuid>] [--json]\n memhub sources connect --all-detected [--import] [--confirm-overrides] [--json]\n memhub sources receipt <request-id> [--json]",
options: [
{
flag: "--import",
summary:
"接入后立即导入历史会话并开启持续同步;不加时只授权,处理保持冻结。",
},
{
flag: "--user-id <id>",
summary:
"只用于 hermes:这条接入属于哪个用户(默认 local)。和 --root 一起接入一个 Hermes profile,例如 --user-id wife --root ~/.hermes/profiles/wife;其他来源只能是 local。",
},
{
flag: "--root <path>",
summary:
"只用于 hermes:读取的根目录(~ 会展开为主目录)。默认是 Hermes 自己的目录 ~/.hermes。",
},
{
flag: "--all-detected",
summary:
@@ -373,7 +389,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
summary:
"列出、搜索、查看、添加、修改或物理删除记忆(对话、工作、编码三个场景)。add 和 App 的「添加记忆」是同一个入口;delete 不可逆,需 --yes。",
usage:
"memhub memories list [--scene <all|chat|working|coding>] [--source <id>] [--query <text>] [--status <state>] [--offset <n>] [--limit <n>] [--json]\n memhub memories show <memory-id> [--kind <kind>] [--variant <variant>] [--json]\n memhub memories add <text> [--request-id <uuid>] [--json]\n memhub memories add --content-file <path> [--request-id <uuid>] [--json]\n memhub memories add-status <request-id> [--json]\n memhub memories update <memory-id> --content-file <path> [--kind <kind>] [--variant <variant>] [--task <text>] [--json]\n memhub memories delete <memory-id> [--kind <kind>] [--variant <variant>] [--yes] [--json]",
"memhub memories list [--scene <all|chat|working|coding>] [--source <id>] [--query <text>] [--status <state>] [--offset <n>] [--limit <n>] [--user-id <id>] [--json]\n memhub memories show <memory-id> [--kind <kind>] [--variant <variant>] [--user-id <id>] [--json]\n memhub memories add <text> [--request-id <uuid>] [--user-id <id>] [--json]\n memhub memories add --content-file <path> [--request-id <uuid>] [--user-id <id>] [--json]\n memhub memories add-status <request-id> [--user-id <id>] [--json]\n memhub memories update <memory-id> --content-file <path> [--kind <kind>] [--variant <variant>] [--task <text>] [--user-id <id>] [--json]\n memhub memories delete <memory-id> [--kind <kind>] [--variant <variant>] [--yes] [--user-id <id>] [--json]",
options: [
{
flag: "--scene <all|chat|working|coding>",
@@ -411,15 +427,75 @@ export const COMMANDS: readonly CommandDescriptor[] = [
"add 的幂等键;用同一个 id 重试只会读回第一次的结果,不会重复添加。不给时自动生成并输出。",
},
DESTRUCTIVE_CONFIRM_OPTION,
USER_ID_OPTION,
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
},
{
name: "profile",
summary: "读取 SDK 组装后的用户长期画像(L0 + 已启用的长期画像补充)。",
usage: "memhub profile [--json]",
options: [JSON_OPTION],
summary:
"读取 SDK 组装后的用户长期画像(L0 + 已启用的长期画像补充);--versions 列出 L0 画像的版本链(最新在前,* 为当前版本)。",
usage: "memhub profile [--versions] [--user-id <id>] [--json]",
options: [
{
flag: "--versions",
summary:
"列出 L0 基本信息的版本链(get_user_basic_info),用于找出覆盖了正确值的错误版本。",
},
USER_ID_OPTION,
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
},
{
name: "settings",
summary:
"查看或切换记忆模式(lite / pro / ultra)。lite 不调用模型,只在原始对话里检索;pro 由模型提取记忆(System 1);ultra 在 pro 之上每晚自动进化(System 2,使用知识图谱)。任何模式都保留全部原始对话。切换不迁移数据:已处理的不会重做,排队中的和新的按新模式处理;需 --yes,不加时只显示计划。环境变量 ONE_MEMORY_GRAPH_STORE 会强制 ultra。settings features 查看或开关 SDK 的可选检索与提取功能(全局设置,默认全部保持 SDK 默认值):graph_rerank 图谱重排(仅 ultra)、same_series_timeline 同系列时间线(需 graph_rerank)、episode_recall 片段召回(pro / ultra,需先开 session_episodes 写入片段)、session_episodes 片段写入、episode_summary 片段摘要(每个片段一次模型调用)、intention 意图提取(默认开)、few_shot 提取示例;set 需 --yes,不加时只显示计划;reset 恢复默认值。",
usage:
"memhub settings mode [lite|pro|ultra] [--yes] [--json]\n" +
" memhub settings features [--json]\n" +
" memhub settings features set <name>=<true|false> [...] [--yes] [--json]\n" +
" memhub settings features reset [<name> ...] [--json]",
options: [
{
flag: "mode [lite|pro|ultra]",
summary: "不带模式只显示当前模式;带模式时先显示切换计划。",
},
{
flag: "features set <name>=<true|false>",
summary:
"开关一个或多个 SDK 功能;功能在当前模式下不生效时会注明原因(例如 graph_rerank 需要 ultra)。",
},
{
flag: "features reset [<name> ...]",
summary: "把指定功能(不写则全部)恢复为 SDK 默认值。",
},
CONFIRM_OPTION,
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
},
{
name: "evolve",
summary:
"立即为一个用户运行记忆进化(System 2,仅 ultra 模式)。在后台运行并返回运行 ID;--wait 等待结果。同一用户同时只运行一次;每晚的自动进化由 Runtime 按设置安排。",
usage:
"memhub evolve [--user-id <id>] [--agent-id <id>] [--max-batches <n>] [--wait] [--json]",
options: [
USER_ID_OPTION,
{
flag: "--agent-id <id>",
summary:
"进化哪个 agent 的记忆,默认 all-apps(MemHub 写入记忆用的 agent)。",
},
{
flag: "--max-batches <n>",
summary: "每轮最多处理的批次数(1–1000),默认由 SDK 决定。",
},
{ flag: "--wait", summary: "等待这次进化结束再返回。" },
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
},
{
@@ -475,7 +551,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
summary:
"从原始对话与已沉淀记忆统一检索上下文;本地只读,不调用模型、不触发补索引。文本输出标明每条是「原始对话摘录」还是「记忆」(JSON 里是 kind: raw|memory)。",
usage:
"memhub recall <query> [--kind <raw|memory>] [--source <id>] [--limit <n>] [--character-budget <n>] [--timeout-ms <n>] [--json]\n memhub recall --query-file <path> [...]\n memhub recall --stdin [...]",
"memhub recall <query> [--kind <raw|memory>] [--source <id>] [--limit <n>] [--character-budget <n>] [--timeout-ms <n>] [--user-id <id>] [--json]\n memhub recall --query-file <path> [...]\n memhub recall --stdin [...]",
options: [
{
flag: "--query-file <path>",
@@ -502,6 +578,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
flag: "--timeout-ms <n>",
summary: "Core 查询时限 50..30000 毫秒,默认 2000。",
},
USER_ID_OPTION,
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
@@ -650,7 +727,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
summary:
"提交一段 OpenAI 格式的对话(messages)到写入队列,异步加工成记忆;或按 request id 查询状态。request id 是幂等键:不给时自动生成并输出,重试请带上同一个。新增一条结论式记忆请用 memories add。",
usage:
"memhub write submit --messages-file <path> --session <id> [--source <id>] [--request-id <id>] [--memory-at <RFC3339>] [--live] [--json]\n memhub write status <request-id> [--json]",
"memhub write submit --messages-file <path> --session <id> [--source <id>] [--request-id <id>] [--memory-at <RFC3339>] [--live] [--user-id <id>] [--json]\n memhub write status <request-id> [--user-id <id>] [--json]",
options: [
{
flag: "--messages-file <path>",
@@ -666,6 +743,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
},
{ flag: "--memory-at <RFC3339>", summary: "对话发生时间,必须带时区。" },
{ flag: "--live", summary: "优先处理。" },
USER_ID_OPTION,
JSON_OPTION,
],
exit_codes: RUNTIME_EXITS,
@@ -76,6 +76,8 @@ const COMMANDS = [
"models",
"memories",
"profile",
"settings",
"evolve",
"graph-width",
"host",
"recall",
+2
View File
@@ -10,6 +10,8 @@ export interface ContextSearchInput {
limit: number;
character_budget: number;
timeout_ms: number;
/** Whose memories to search. Omitted for the default user (`local`). */
user_id?: string;
}
/** The CLI's JSON document mirrors Core's SDK-native context schema v2. */
+19
View File
@@ -12,11 +12,19 @@ const READ_ONLY_PATHS = [
"/api/v1/live-sync",
"/api/v1/memory-processing",
"/api/v1/model-profiles",
"/api/v1/source-grants",
"/api/v1/sources",
"/api/v2/home-summary",
"/api/v2/memory-library",
"/api/v2/user-profile",
"/api/v2/user-profile/versions",
"/api/v1/settings/memory-mode",
"/api/v1/settings/memory-features",
"/api/v2/evolve",
] as const;
/** One evolve run's receipt (`memhub evolve --wait`). */
const EVOLVE_RUN =
/^\/api\/v2\/evolve\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const SDK_MEMORY_DETAIL = /^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}$/;
/** Status of one `POST /api/v2/memories` addition, by its request id. */
const MEMORY_ADDITION =
@@ -219,6 +227,7 @@ export function isReadOnlyPath(path: string): boolean {
if (IMPORT_JOB.test(path)) return true;
if (OPERATION_RECEIPT.test(path)) return true;
if (MEMORY_ADDITION.test(path)) return true;
if (EVOLVE_RUN.test(path)) return true;
if (!SDK_MEMORY_DETAIL.test(path)) return false;
const segment = path.slice("/api/v2/memories/".length);
return !(RESERVED_MEMORY_SEGMENTS as readonly string[]).includes(segment);
@@ -310,6 +319,8 @@ export async function submitMemoryWrite(
messages: readonly OpenAIMessage[];
memory_at?: string;
priority?: "live" | "normal";
/** Whose memory this is. Omitted for the default user (`local`). */
user_id?: string;
},
): Promise<MemoryWriteOutcome> {
let response: { status: number; data: unknown };
@@ -327,6 +338,9 @@ export async function submitMemoryWrite(
...(request.priority === undefined
? {}
: { priority: request.priority }),
...(request.user_id === undefined || request.user_id === "local"
? {}
: { user_id: request.user_id }),
},
{
headers: { Authorization: `Bearer ${attachment.apiToken}` },
@@ -469,11 +483,16 @@ export async function deleteModelProfile(
export async function readMemoryWrite(
attachment: RuntimeAttachmentLike,
requestId: string,
/** A receipt belongs to the user that submitted it; omit for `local`. */
userId?: string,
): Promise<MemoryWriteReceipt | null> {
try {
const response = await axios.get<MemoryWriteReceipt>(
`${baseUrlOf(attachment)}/api/v2/memory-writes/${encodeURIComponent(requestId)}`,
{
...(userId === undefined || userId === "local"
? {}
: { params: { user_id: userId } }),
headers: { Authorization: `Bearer ${attachment.apiToken}` },
timeout: 10_000,
proxy: false,
+14 -3
View File
@@ -6,9 +6,10 @@ const WRITE_PATHS = [
] as const;
/** `POST /api/v2/memories` adds one memory (the app's 添加记忆). */
const MEMORY_ADD_PATH = "/api/v2/memories";
/** One SDK memory; delete names its namespace in the query string. */
/** One SDK memory; delete names its namespace, and a non-default user, in
* the query string. */
const SDK_MEMORY_PATH =
/^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}(?:\?kind=(?:chat|experience|fact)(?:&variant=(?:working|coding))?)?$/;
/^\/api\/v2\/memories\/[A-Za-z0-9][\w.-]{0,127}(?:\?(?:kind=(?:chat|experience|fact)(?:&variant=(?:working|coding))?(?:&user_id=[A-Za-z0-9._-]{1,64})?|user_id=[A-Za-z0-9._-]{1,64}))?$/;
const MODEL_PROBE_PATH =
/^\/api\/v1\/model-profiles\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\/probe$/;
const MODEL_PROFILE_PATH =
@@ -25,6 +26,11 @@ const SOURCE_IMPORT_JOB_PATH = "/api/v1/import-jobs";
const IMPORT_JOB_ACTION_PATH =
/^\/api\/v1\/import-jobs\/[A-Za-z0-9][\w.-]{0,127}\/(?:pause|resume|cancel)$/;
const GRAPH_WIDTH_PATH = "/api/v1/graph-width";
/** `memhub evolve` starts a run; `memhub settings mode <m> --yes` sets the mode. */
const EVOLVE_PATH = "/api/v2/evolve";
const MEMORY_MODE_PATH = "/api/v1/settings/memory-mode";
/** `memhub settings features set|reset` sets the SDK features. */
const MEMORY_FEATURES_PATH = "/api/v1/settings/memory-features";
const ONBOARDING_COMPLETE_PATH = "/api/v1/onboarding/complete";
const MAX_BYTES = 4 * 1024 * 1024;
@@ -96,6 +102,7 @@ export function createCoreOperations(attachment: {
!SOURCE_AUTHORIZE_PATH.test(path) &&
!SOURCE_REVOKE_PATH.test(path) &&
path !== GRAPH_WIDTH_PATH &&
path !== EVOLVE_PATH &&
path !== ONBOARDING_COMPLETE_PATH &&
!MODEL_PROFILE_ACTIVATE.test(path)
) {
@@ -131,7 +138,11 @@ export function createCoreOperations(attachment: {
body: Record<string, unknown>,
signal?: AbortSignal,
) {
if (!SDK_MEMORY_PATH.test(path) || path.includes("?")) {
if (
path !== MEMORY_MODE_PATH &&
path !== MEMORY_FEATURES_PATH &&
(!SDK_MEMORY_PATH.test(path) || path.includes("?"))
) {
throw new Error("core_operation_path_not_allowed");
}
let response: { status: number; data: unknown };
+29 -7
View File
@@ -110,9 +110,18 @@ export async function connectSourcesWithImport(
sourceIds: readonly string[];
confirmOverrides: boolean;
requestId?: string;
/** Whose grant this is. Omitted for the default user (`local`). */
userId?: string;
/** An absolute root other than the source's candidate (a Hermes
* profile). Core reads its identity itself, so the source's candidate
* root id and revision are not sent for it. */
root?: string;
},
): Promise<FlowDocument> {
const sourceIds = input.sourceIds;
// A second user's grant is not a first run, and a custom root has no
// candidate the app showed: both are a plain connect.
const profile = input.root !== undefined || input.userId !== undefined;
if (
sourceIds.length < 1 ||
sourceIds.length > 32 ||
@@ -134,9 +143,14 @@ export async function connectSourcesWithImport(
return invalid("source_unavailable");
}
const items = selected.map(([candidate]) => candidate);
const needsOverride = items
.filter((item) => item.requires_override_confirmation)
.map((item) => item.source_id);
// A custom root is the caller's own choice, not the environment override
// the confirmation exists for.
const needsOverride =
input.root !== undefined
? []
: items
.filter((item) => item.requires_override_confirmation)
.map((item) => item.source_id);
if (needsOverride.length > 0 && !input.confirmOverrides) {
return {
schema_version: 1,
@@ -154,18 +168,22 @@ export async function connectSourcesWithImport(
request = buildSourceImportRequest({
sourceIds: [...sourceIds],
sourceRoots: Object.fromEntries(
items.map((item) => [item.source_id, item.candidate_root_id]),
items.flatMap((item) =>
input.root === undefined
? [[item.source_id, item.candidate_root_id]]
: [],
),
),
sourceCandidateRevisions: Object.fromEntries(
items.flatMap((item) =>
item.candidate_revision === undefined
item.candidate_revision === undefined || input.root !== undefined
? []
: [[item.source_id, item.candidate_revision]],
),
),
confirmEnvironmentOverrides: needsOverride.length > 0,
fullRescan: false,
onboarding: true,
onboarding: !profile,
enableSync: true,
requestId,
});
@@ -175,7 +193,11 @@ export async function connectSourcesWithImport(
}
throw error;
}
const result = await operations.post<unknown>(request.path, request.body);
const result = await operations.post<unknown>(request.path, {
...request.body,
...(input.userId === undefined ? {} : { user_id: input.userId }),
...(input.root === undefined ? {} : { root: input.root }),
});
if (result.state === "rejected") {
return rejected(requestId, result.code);
}
+6
View File
@@ -107,7 +107,11 @@ export async function readLogReport(
query: LogQuery,
): Promise<{
events: DiagnosticEventV1[];
/** Lines that are not JSON: the only ones `memhub logs` calls damaged. */
invalid_lines: number;
/** Whole JSON lines the reader leaves out, such as an older build's
* records. Nothing is wrong with them, so the text output says nothing. */
unrecognized_lines: number;
failure?: "log_directory_unreadable";
}> {
// `readDiagnosticEvents` treats an unreadable logs path as "no events". That
@@ -119,6 +123,7 @@ export async function readLogReport(
return {
events: [],
invalid_lines: 0,
unrecognized_lines: 0,
failure: "log_directory_unreadable",
};
}
@@ -129,6 +134,7 @@ export async function readLogReport(
return {
events: [],
invalid_lines: 0,
unrecognized_lines: 0,
failure: "log_directory_unreadable",
};
}
@@ -0,0 +1,318 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
import { parseFeaturesCommand } from "./memory-features-flow.js";
import { composeStatusReport, renderStatusText } from "./status-report.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
}));
vi.mock("axios", () => ({
default: {
create: vi.fn(() => transport),
get: vi.fn(),
post: vi.fn(),
isAxiosError: () => false,
},
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
const DEFAULTS: Record<string, boolean> = {
graph_rerank: false,
same_series_timeline: false,
episode_recall: false,
session_episodes: false,
episode_summary: false,
intention: true,
few_shot: false,
};
function featuresState(
configured: Record<string, boolean> = {},
mode = "pro",
extra: Record<string, unknown> = {},
) {
const nonDefault: Record<string, boolean> = {};
const features = Object.entries(DEFAULTS).map(([name, fallback]) => {
const enabled = configured[name] ?? fallback;
if (enabled !== fallback) nonDefault[name] = enabled;
const needsUltra = name === "graph_rerank";
const reason = !enabled
? "disabled"
: needsUltra && mode !== "ultra"
? "requires_ultra"
: null;
return {
name,
enabled,
configured: name in configured ? configured[name] : null,
default: fallback,
active: reason === null,
reason,
side: "read",
requires_mode: needsUltra ? "ultra" : "pro",
requires_feature: null,
cost: "none",
sdk_field: `recall.${name}`,
env: `MEMORY_${name.toUpperCase()}`,
description: name,
};
});
return { mode, features, non_default: nonDefault, ...extra };
}
beforeEach(() => {
vi.clearAllMocks();
for (const method of Object.values(transport)) method.mockReset();
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue({
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
});
transport.get.mockImplementation(async (url: string) =>
url === "/api/v1/settings/memory-features"
? { status: 200, data: featuresState() }
: { status: 404, data: {} },
);
});
describe("memhub settings features", () => {
it("lists every feature with its value and whether it acts", async () => {
const result = await capturedRun(["settings", "features", "--json"]);
expect(result.code).toBe(0);
const json = JSON.parse(result.stdout);
expect(json.state).toBe("ok");
expect(json.features.features).toHaveLength(7);
expect(transport.put).not.toHaveBeenCalled();
const text = (await capturedRun(["settings", "features"])).stdout;
expect(text).toContain("graph_rerank:关(默认)");
expect(text).toContain("intention:开(默认)");
expect(text).toContain("先开启 session_episodes");
});
it("prints the plan and changes nothing without --yes", async () => {
const result = await capturedRun([
"settings",
"features",
"set",
"graph_rerank=true",
"episode_recall=true",
]);
expect(result.code).toBe(0);
expect(transport.put).not.toHaveBeenCalled();
expect(result.stdout).toContain(
"memhub settings features set graph_rerank=true episode_recall=true --yes",
);
});
it("sets features with --yes and reports the ones inactive in this mode", async () => {
transport.put.mockResolvedValue({
status: 200,
data: featuresState({ graph_rerank: true, episode_recall: true }, "pro", {
changed: ["graph_rerank", "episode_recall"],
}),
});
const result = await capturedRun([
"settings",
"features",
"set",
"graph_rerank=true",
"episode_recall=true",
"--yes",
]);
expect(result.code).toBe(0);
expect(transport.put).toHaveBeenCalledWith(
"/api/v1/settings/memory-features",
{ graph_rerank: true, episode_recall: true },
undefined,
);
expect(result.stdout).toContain("已更新:graph_rerank、episode_recall");
expect(result.stdout).toContain(
"graph_rerank:开(已设置,当前模式下不生效(需要 ultra))",
);
});
it("does nothing when the values are already set", async () => {
transport.get.mockResolvedValue({
status: 200,
data: featuresState({ few_shot: true }),
});
const result = await capturedRun([
"settings",
"features",
"set",
"few_shot=true",
"--yes",
]);
expect(result.code).toBe(0);
expect(result.stdout).toContain("没有需要改动的设置");
expect(transport.put).not.toHaveBeenCalled();
});
it("resets the named features, or every set one, to the default", async () => {
transport.get.mockResolvedValue({
status: 200,
data: featuresState({ few_shot: true, intention: false }),
});
transport.put.mockResolvedValue({
status: 200,
data: featuresState({}, "pro", { changed: ["intention", "few_shot"] }),
});
expect(
(await capturedRun(["settings", "features", "reset", "few_shot"])).code,
).toBe(0);
expect(transport.put).toHaveBeenLastCalledWith(
"/api/v1/settings/memory-features",
{ few_shot: null },
undefined,
);
const all = await capturedRun(["settings", "features", "reset", "--json"]);
expect(JSON.parse(all.stdout).state).toBe("committed");
expect(transport.put).toHaveBeenLastCalledWith(
"/api/v1/settings/memory-features",
{ intention: null, few_shot: null },
undefined,
);
});
it("refuses unknown names and malformed values before the Runtime", async () => {
for (const bad of [
["settings", "features", "set"],
["settings", "features", "set", "local_rerank=true"],
["settings", "features", "set", "graph_rerank=yes"],
["settings", "features", "set", "graph_rerank"],
["settings", "features", "set", "few_shot=true", "few_shot=false"],
["settings", "features", "reset", "nope"],
["settings", "features", "toggle"],
]) {
const result = await capturedRun(bad);
expect(result.code, bad.join(" ")).toBe(2);
}
expect(transport.put).not.toHaveBeenCalled();
});
it("surfaces a 422 from the Runtime as rejected", async () => {
transport.put.mockResolvedValue({
status: 422,
data: { detail: "invalid_request" },
});
const result = await capturedRun([
"settings",
"features",
"set",
"few_shot=true",
"--yes",
"--json",
]);
expect(result.code).toBe(1);
expect(JSON.parse(result.stdout).state).toBe("rejected");
});
it("parses the forms the help shows", () => {
expect(parseFeaturesCommand([])).toEqual({ kind: "show" });
expect(parseFeaturesCommand(["set", "intention=false", "--yes"])).toEqual({
kind: "set",
values: { intention: false },
confirmed: true,
});
expect(parseFeaturesCommand(["reset"])).toEqual({
kind: "reset",
names: [],
});
});
});
describe("status pipeline features", () => {
it("shows the features that differ from the defaults", async () => {
const processing = {
pending: 0,
fresh_pending: 0,
backlog_pending: 0,
outside_window_pending: 0,
running: 0,
waiting_for_model: 0,
completed: 3,
failed: 0,
blocked_stages: [],
pipeline: {
mode: "pro",
configured_mode: "pro",
forced_by_env: false,
last_evolve_at: null,
graph_build: { state: "idle", users: {} },
evolve: { mode: "pro", schedule: null, running: [], users: {} },
features: { episode_recall: true, intention: false },
},
};
const report = await composeStatusReport({
dataDir: "/tmp/isolated",
runtime: { state: "running", code: "ok", host: "127.0.0.1", port: 1 },
background: null,
reader: (async (path: string) =>
path === "/api/v1/memory-processing"
? { state: "ok", data: processing }
: { state: "unavailable", code: "x" }) as never,
});
expect(report.pipeline?.features).toEqual({
episode_recall: true,
intention: false,
});
const text = renderStatusText(report, "MemHub");
expect(text).toContain("SDK 功能:episode_recall=开、intention=关");
});
});
async function capturedRun(arguments_: string[]) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return {
code: await run(arguments_, { ONE_MEMORY_DATA_DIR: "/synthetic/data" }),
stdout,
stderr,
};
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
+273
View File
@@ -0,0 +1,273 @@
/**
* `memhub settings features [set <name>=<true|false> ... | reset [<name> ...]]`.
*
* The SDK's optional retrieval and extraction features (graph rerank, episode
* recall, the episode write side, intention, few-shot). One global setting in
* the Runtime; an unset feature keeps the SDK default, so an install nobody
* touched behaves as before. `set` prints its plan first and needs `--yes`,
* like `settings mode`, because some features spend model calls; `reset`
* only returns features to their defaults and applies at once.
*/
import type { CoreReader } from "./core-client.js";
import type { CorePutOperations } from "./core-operations.js";
import { PRODUCT } from "./version.js";
export const MEMORY_FEATURES = [
"graph_rerank",
"same_series_timeline",
"episode_recall",
"session_episodes",
"episode_summary",
"intention",
"few_shot",
] as const;
export type MemoryFeature = (typeof MEMORY_FEATURES)[number];
const FEATURE_LABEL: Record<MemoryFeature, string> = {
graph_rerank: "图谱重排(沿知识图谱的边传播支持度;仅 ultra)",
same_series_timeline:
"同系列时间线(同一系列的记忆按时间放在一起;需图谱重排)",
episode_recall:
"片段召回(补入同一片段的记忆并多通道打分;pro / ultra,需片段写入)",
session_episodes: "片段写入(同一会话提取的记忆归入同一片段)",
episode_summary: "片段摘要(片段结束时总结一次;每个片段一次模型调用)",
intention: "意图提取(随事实一起提取用户的计划与意图)",
few_shot: "提取示例(在提取提示里附上示例;不加调用,输入更长)",
};
const REASON_TEXT: Record<string, string> = {
disabled: "未开启",
requires_pro: "lite 模式下不生效(需要 pro 或 ultra)",
requires_ultra: "当前模式下不生效(需要 ultra)",
requires_graph_rerank: "需要先开启并生效 graph_rerank",
};
export type FeaturesCommand =
| { kind: "show" }
| {
kind: "set";
values: Partial<Record<MemoryFeature, boolean>>;
confirmed: boolean;
}
| { kind: "reset"; names: MemoryFeature[] };
function isFeature(value: string): value is MemoryFeature {
return (MEMORY_FEATURES as readonly string[]).includes(value);
}
/** Arguments after `settings features`; null = usage. */
export function parseFeaturesCommand(
values: readonly string[],
): FeaturesCommand | null {
const rest = values.filter((value) => value !== "--json");
if (rest.length === 0) return { kind: "show" };
const [verb, ...tail] = rest;
const yes = tail.filter((value) => value === "--yes").length;
if (yes > 1) return null;
const positional = tail.filter((value) => value !== "--yes");
if (verb === "set") {
if (positional.length === 0) return null;
const parsed: Partial<Record<MemoryFeature, boolean>> = {};
for (const item of positional) {
const match = /^([a-z_]+)=(true|false)$/.exec(item);
if (!match || !isFeature(match[1]) || match[1] in parsed) return null;
parsed[match[1]] = match[2] === "true";
}
return { kind: "set", values: parsed, confirmed: yes === 1 };
}
if (verb === "reset") {
if (new Set(positional).size !== positional.length) return null;
if (!positional.every(isFeature)) return null;
return { kind: "reset", names: positional as MemoryFeature[] };
}
return null;
}
export interface FeatureState {
name: MemoryFeature;
enabled: boolean;
configured: boolean | null;
default: boolean;
active: boolean;
reason: string | null;
cost: string;
requires_mode: string;
description: string;
}
export interface FeaturesState {
mode: string;
features: FeatureState[];
non_default: Partial<Record<MemoryFeature, boolean>>;
changed?: MemoryFeature[];
}
function featuresState(value: unknown): FeaturesState | null {
if (!value || typeof value !== "object") return null;
const item = value as Record<string, unknown>;
if (
typeof item.mode !== "string" ||
!Array.isArray(item.features) ||
!item.non_default ||
typeof item.non_default !== "object"
) {
return null;
}
for (const feature of item.features) {
if (
!feature ||
typeof feature !== "object" ||
typeof (feature as FeatureState).name !== "string" ||
typeof (feature as FeatureState).enabled !== "boolean" ||
typeof (feature as FeatureState).active !== "boolean"
) {
return null;
}
}
return item as unknown as FeaturesState;
}
export type FeaturesDocument = {
schema_version: 1;
product: string;
command: "settings features";
} & (
| { state: "ok"; features: FeaturesState }
| {
state: "preview";
features: FeaturesState;
target: Partial<Record<MemoryFeature, boolean>>;
applied: false;
next_step: string;
}
| { state: "unchanged"; features: FeaturesState }
| { state: "committed"; features: FeaturesState; changed: MemoryFeature[] }
| { state: "rejected" | "unknown" | "unavailable"; code: string }
);
const BASE = {
schema_version: 1 as const,
product: PRODUCT,
command: "settings features" as const,
};
const PATH = "/api/v1/settings/memory-features";
export async function runFeaturesCommand(
command: FeaturesCommand,
reader: CoreReader,
operations: CorePutOperations,
): Promise<FeaturesDocument> {
const read = await reader<unknown>(PATH);
if (read.state !== "ok") {
return { ...BASE, state: "unavailable", code: read.code };
}
const current = featuresState(read.data);
if (current === null) {
return { ...BASE, state: "unavailable", code: "response_invalid" };
}
if (command.kind === "show") {
return { ...BASE, state: "ok", features: current };
}
let body: Record<string, unknown>;
if (command.kind === "set") {
const configured = new Map(
current.features.map((feature) => [feature.name, feature.configured]),
);
const differs = Object.entries(command.values).some(
([name, value]) => configured.get(name as MemoryFeature) !== value,
);
if (!differs) return { ...BASE, state: "unchanged", features: current };
if (!command.confirmed) {
const pairs = Object.entries(command.values)
.map(([name, value]) => `${name}=${value}`)
.join(" ");
return {
...BASE,
state: "preview",
features: current,
target: command.values,
applied: false,
next_step: `${PRODUCT} settings features set ${pairs} --yes`,
};
}
body = { ...command.values };
} else {
const names =
command.names.length > 0
? command.names
: current.features
.filter((feature) => feature.configured !== null)
.map((feature) => feature.name);
if (
names.every(
(name) =>
current.features.find((feature) => feature.name === name)
?.configured === null,
)
) {
return { ...BASE, state: "unchanged", features: current };
}
body = Object.fromEntries(names.map((name) => [name, null]));
}
const result = await operations.put<unknown>(PATH, body);
if (result.state !== "committed") {
return { ...BASE, state: result.state, code: result.code };
}
const changed = featuresState(result.data);
if (changed === null) {
return { ...BASE, state: "unknown", code: "response_invalid" };
}
return {
...BASE,
state: "committed",
features: changed,
changed: changed.changed ?? [],
};
}
export function featuresExitCode(document: FeaturesDocument): number {
return ["ok", "preview", "unchanged", "committed"].includes(document.state)
? 0
: 1;
}
function featureLine(feature: FeatureState): string {
const label = FEATURE_LABEL[feature.name] ?? feature.name;
const value = feature.enabled ? "开" : "关";
const origin = feature.configured === null ? "默认" : "已设置";
const activity =
feature.enabled && !feature.active && feature.reason
? `,${REASON_TEXT[feature.reason] ?? feature.reason}`
: "";
return `${feature.name}:${value}(${origin}${activity})— ${label}`;
}
export function renderFeaturesText(document: FeaturesDocument): string {
if ("code" in document) {
return `SDK 功能未能${document.state === "unavailable" ? "读取" : "修改"}:${document.code}\n`;
}
const state = document.features;
const lines = [`当前模式:${state.mode}`];
for (const feature of state.features) lines.push(featureLine(feature));
lines.push(
"片段召回(episode_recall)只对带片段的记忆有用:先开启 session_episodes,之后提取的记忆才会带片段。",
);
if (document.state === "unchanged") lines.push("没有需要改动的设置。");
if (document.state === "preview") {
lines.push("将设置:");
for (const [name, value] of Object.entries(document.target)) {
lines.push(`- ${name} = ${value ? "开" : "关"}`);
}
lines.push(`未执行任何改动。确认后运行:${document.next_step}`);
}
if (document.state === "committed") {
lines.push(
document.changed.length > 0
? `已更新:${document.changed.join("、")}。正在进行的写入按原设置完成,之后的写入和检索使用新设置。`
: "已保存,生效的值没有变化。",
);
}
return `${lines.join("\n")}\n`;
}
+309
View File
@@ -0,0 +1,309 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
import { modeEffects, parseSettingsCommand } from "./memory-mode-flow.js";
import { composeStatusReport, renderStatusText } from "./status-report.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
}));
vi.mock("axios", () => ({
default: {
create: vi.fn(() => transport),
get: vi.fn(),
post: vi.fn(),
isAxiosError: () => false,
},
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
const RUN_ID = "11111111-2222-4333-8444-555555555555";
function modeState(mode: string, configured = mode) {
return {
mode,
configured_mode: configured,
forced_by_env: false,
modes: ["lite", "pro", "ultra"],
graph_build: { state: "idle", users: {} },
};
}
function receipt(state: string, extra: Record<string, unknown> = {}) {
return {
run_id: RUN_ID,
user_id: "wife",
agent_id: "all-apps",
state,
trigger: "manual",
started_at: "2026-11-01T03:00:00Z",
rounds: state === "running" ? 0 : 2,
tasks_processed: state === "running" ? 0 : 5,
batch_failures: 0,
capped: false,
elapsed_ms: 10,
results: { schemas: ["never printed"] },
...extra,
};
}
beforeEach(() => {
vi.clearAllMocks();
for (const method of Object.values(transport)) method.mockReset();
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue({
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
});
transport.get.mockImplementation(async (url: string) =>
url === "/api/v1/settings/memory-mode"
? { status: 200, data: modeState("pro") }
: { status: 404, data: {} },
);
});
describe("memhub settings mode", () => {
it("shows the current mode without changing it", async () => {
const result = await capturedRun(["settings", "mode", "--json"]);
expect(result.code).toBe(0);
expect(JSON.parse(result.stdout).mode.mode).toBe("pro");
expect(transport.put).not.toHaveBeenCalled();
});
it("prints the plan and changes nothing without --yes", async () => {
const result = await capturedRun(["settings", "mode", "lite"]);
expect(result.code).toBe(0);
expect(transport.put).not.toHaveBeenCalled();
expect(result.stdout).toContain("更高模式生成的记忆仍保存着");
expect(result.stdout).toContain("memhub settings mode lite --yes");
const json = JSON.parse(
(await capturedRun(["settings", "mode", "ultra", "--json"])).stdout,
);
expect(json.state).toBe("preview");
expect(json.applied).toBe(false);
expect(json.effects).toEqual(["existing_memories_added_to_graph"]);
});
it("changes the mode with --yes and reports what changed", async () => {
transport.put.mockResolvedValue({
status: 200,
data: {
...modeState("lite"),
previous_mode: "pro",
effects: ["higher_mode_memories_hidden"],
},
});
const result = await capturedRun(["settings", "mode", "lite", "--yes"]);
expect(result.code).toBe(0);
expect(transport.put).toHaveBeenCalledWith(
"/api/v1/settings/memory-mode",
{ mode: "lite" },
undefined,
);
expect(result.stdout).toContain("已切换");
expect(result.stdout).toContain("原始对话:全部保留");
});
it("does nothing for the mode already set, and refuses bad input", async () => {
const same = await capturedRun(["settings", "mode", "pro", "--yes"]);
expect(same.code).toBe(0);
expect(same.stdout).toContain("已经是这个模式");
expect(transport.put).not.toHaveBeenCalled();
for (const bad of [
["settings"],
["settings", "mode", "max"],
["settings", "mode", "--yes"],
["settings", "mode", "lite", "pro"],
["settings", "language", "en"],
]) {
expect((await capturedRun(bad)).code, bad.join(" ")).toBe(2);
}
});
it("plans the same effects Core reports", () => {
expect(modeEffects("lite", "pro")).toEqual([
"earlier_raw_not_extracted",
"lite_search_stops",
]);
expect(modeEffects("ultra", "lite")).toEqual([
"higher_mode_memories_hidden",
"evolve_stopped",
]);
expect(modeEffects("pro", "pro")).toEqual([]);
expect(parseSettingsCommand(["mode", "ultra", "--yes"])).toEqual({
kind: "set",
mode: "ultra",
confirmed: true,
});
});
});
describe("memhub evolve", () => {
it("starts a run for the named user and returns at once", async () => {
transport.post.mockResolvedValue({ status: 202, data: receipt("running") });
const result = await capturedRun([
"evolve",
"--user-id",
"wife",
"--agent-id",
"all-apps",
"--max-batches",
"3",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
"/api/v2/evolve",
{ user_id: "wife", agent_id: "all-apps", max_batches: 3 },
undefined,
);
const document = JSON.parse(result.stdout);
expect(document.state).toBe("running");
expect(document.run.results).toBeUndefined();
});
it("follows the run with --wait until it ends", async () => {
transport.post.mockResolvedValue({ status: 202, data: receipt("running") });
const reads = [receipt("running"), receipt("succeeded")];
transport.get.mockImplementation(async (url: string) =>
url === `/api/v2/evolve/${RUN_ID}`
? { status: 200, data: reads.shift() }
: { status: 404, data: {} },
);
const result = await capturedRun(["evolve", "--wait"], {
sleep: async () => undefined,
});
expect(result.code).toBe(0);
expect(transport.post).toHaveBeenCalledWith(
"/api/v2/evolve",
{},
undefined,
);
expect(result.stdout).toContain("进化完成");
expect(result.stdout).toContain("2 轮");
});
it("names the mode when Core refuses outside ultra", async () => {
transport.post.mockResolvedValue({
status: 409,
data: { detail: "memory_mode_not_ultra" },
});
const result = await capturedRun(["evolve"]);
expect(result.code).toBe(1);
expect(result.stdout).toContain("memhub settings mode ultra --yes");
expect((await capturedRun(["evolve", "--max-batches", "0"])).code).toBe(2);
expect((await capturedRun(["evolve", "--now"])).code).toBe(2);
});
});
describe("status pipeline", () => {
it("reports pipeline.mode and the last evolve from memory-processing", async () => {
const processing = {
pending: 0,
fresh_pending: 0,
backlog_pending: 0,
outside_window_pending: 0,
running: 0,
waiting_for_model: 0,
completed: 3,
failed: 0,
blocked_stages: [],
pipeline: {
mode: "ultra",
configured_mode: "ultra",
forced_by_env: false,
last_evolve_at: "2026-11-01T03:00:05+08:00",
graph_build: { state: "succeeded", users: {} },
evolve: {
mode: "ultra",
schedule: { enabled: true, time: "03:00" },
running: [],
users: {
local: {
last_run_at: "2026-11-01T03:00:05+08:00",
last_result: { state: "succeeded", tasks_processed: 4 },
last_error: null,
},
},
},
},
};
const report = await composeStatusReport({
dataDir: "/tmp/isolated",
runtime: { state: "running", code: "ok", host: "127.0.0.1", port: 1 },
background: null,
reader: (async (path: string) =>
path === "/api/v1/memory-processing"
? { state: "ok", data: processing }
: { state: "unavailable", code: "x" }) as never,
});
expect(report.pipeline?.mode).toBe("ultra");
expect(report.pipeline?.last_evolve_at).toBe("2026-11-01T03:00:05+08:00");
expect(report.pipeline?.evolve_users.local?.state).toBe("succeeded");
const text = renderStatusText(report, "MemHub");
expect(text).toContain("记忆模式:ultra");
expect(text).toContain("每日 03:00 自动进化");
});
});
async function capturedRun(
arguments_: string[],
controls: Parameters<typeof run>[2] = {},
) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return {
code: await run(
arguments_,
{ ONE_MEMORY_DATA_DIR: "/synthetic/data" },
controls,
),
stdout,
stderr,
};
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
+372
View File
@@ -0,0 +1,372 @@
/**
* `memhub settings mode` and `memhub evolve`.
*
* The mode (lite / pro / ultra) decides what the SDK does with every
* conversation, and switching it migrates nothing: what one mode made is not
* shown by a lower one, and raw kept by lite is not extracted until it is
* imported again. So a change prints its plan first and needs `--yes`, like
* every other command whose effect a person cannot simply take back.
*
* `memhub evolve` starts System 2 (ultra only) and, with `--wait`, follows
* the run until it ends.
*/
import type { CoreReader } from "./core-client.js";
import type { CoreOperations, CorePutOperations } from "./core-operations.js";
import { PRODUCT } from "./version.js";
export const MEMORY_MODES = ["lite", "pro", "ultra"] as const;
export type MemoryMode = (typeof MEMORY_MODES)[number];
export type ModeEffect =
| "earlier_raw_not_extracted"
| "higher_mode_memories_hidden"
| "existing_memories_added_to_graph"
| "evolve_stopped"
| "lite_search_stops"
| "graph_backfill_already_running";
const MODE_SUMMARY: Record<MemoryMode, string> = {
lite: "lite:不调用模型,保留全部原始对话,只在原始对话里检索",
pro: "pro:模型从对话中提取记忆(System 1)",
ultra: "ultra:pro + 每晚自动进化(System 2,使用知识图谱)",
};
const EFFECT_TEXT: Record<ModeEffect, string> = {
earlier_raw_not_extracted:
"lite 下已处理完的对话不会再被提取;只有尚未处理的和之后的新对话会由模型整理成记忆,检索立即改为记忆检索",
higher_mode_memories_hidden:
"更高模式生成的记忆仍保存着,但在这个模式下不会显示",
existing_memories_added_to_graph:
"已有记忆会在后台加入知识图谱(每个用户一次)",
evolve_stopped: "每晚的自动进化(System 2)停止",
lite_search_stops: "lite 只在原始对话里的检索立即停止,改为检索记忆",
graph_backfill_already_running: "知识图谱的补建已在进行中",
};
/** The same rule Core applies (`memory_mode_effects`), for the plan. */
export function modeEffects(
previous: MemoryMode,
next: MemoryMode,
): ModeEffect[] {
if (previous === next) return [];
const rank: Record<MemoryMode, number> = { lite: 0, pro: 1, ultra: 2 };
const effects: ModeEffect[] = [];
if (previous === "lite") {
effects.push("earlier_raw_not_extracted", "lite_search_stops");
}
if (rank[next] < rank[previous]) effects.push("higher_mode_memories_hidden");
if (previous === "ultra") effects.push("evolve_stopped");
if (next === "ultra") effects.push("existing_memories_added_to_graph");
return effects;
}
export type ModeCommand =
| { kind: "show" }
| { kind: "set"; mode: MemoryMode; confirmed: boolean };
/** `settings mode [lite|pro|ultra] [--yes]` after `settings`; null = usage. */
export function parseSettingsCommand(
values: readonly string[],
): ModeCommand | null {
const rest = values.filter((value) => value !== "--json");
if (rest[0] !== "mode") return null;
const tail = rest.slice(1);
const confirmed = tail.includes("--yes");
const positional = tail.filter((value) => value !== "--yes");
if (tail.filter((value) => value === "--yes").length > 1) return null;
if (positional.length === 0) return confirmed ? null : { kind: "show" };
if (positional.length !== 1) return null;
const mode = positional[0] as MemoryMode;
if (!(MEMORY_MODES as readonly string[]).includes(mode)) return null;
return { kind: "set", mode, confirmed };
}
export interface ModeState {
mode: MemoryMode;
configured_mode: MemoryMode;
forced_by_env: boolean;
graph_build?: { state?: string } | null;
previous_mode?: MemoryMode;
effects?: ModeEffect[];
}
function modeState(value: unknown): ModeState | null {
if (!value || typeof value !== "object") return null;
const item = value as Record<string, unknown>;
if (
!(MEMORY_MODES as readonly unknown[]).includes(item.mode) ||
!(MEMORY_MODES as readonly unknown[]).includes(item.configured_mode) ||
typeof item.forced_by_env !== "boolean"
) {
return null;
}
return item as unknown as ModeState;
}
export type ModeDocument = {
schema_version: 1;
product: string;
command: "settings mode";
} & (
| { state: "ok"; mode: ModeState }
| {
state: "preview";
mode: ModeState;
target: MemoryMode;
effects: ModeEffect[];
applied: false;
next_step: string;
}
| { state: "unchanged"; mode: ModeState }
| { state: "committed"; mode: ModeState; effects: ModeEffect[] }
| { state: "rejected" | "unknown" | "unavailable"; code: string }
);
const BASE = {
schema_version: 1 as const,
product: PRODUCT,
command: "settings mode" as const,
};
export async function runModeCommand(
command: ModeCommand,
reader: CoreReader,
operations: CorePutOperations,
): Promise<ModeDocument> {
const read = await reader<unknown>("/api/v1/settings/memory-mode");
if (read.state !== "ok") {
return { ...BASE, state: "unavailable", code: read.code };
}
const current = modeState(read.data);
if (current === null) {
return { ...BASE, state: "unavailable", code: "response_invalid" };
}
if (command.kind === "show") return { ...BASE, state: "ok", mode: current };
if (current.configured_mode === command.mode) {
return { ...BASE, state: "unchanged", mode: current };
}
if (!command.confirmed) {
return {
...BASE,
state: "preview",
mode: current,
target: command.mode,
effects: modeEffects(current.mode, command.mode),
applied: false,
next_step: `${PRODUCT} settings mode ${command.mode} --yes`,
};
}
const result = await operations.put<unknown>("/api/v1/settings/memory-mode", {
mode: command.mode,
});
if (result.state !== "committed") {
return { ...BASE, state: result.state, code: result.code };
}
const changed = modeState(result.data);
if (changed === null) {
return { ...BASE, state: "unknown", code: "response_invalid" };
}
return {
...BASE,
state: "committed",
mode: changed,
effects: changed.effects ?? [],
};
}
export function modeExitCode(document: ModeDocument): number {
return ["ok", "preview", "unchanged", "committed"].includes(document.state)
? 0
: 1;
}
export function renderModeText(document: ModeDocument): string {
if ("code" in document) {
return `记忆模式未能${document.state === "unavailable" ? "读取" : "修改"}:${document.code}\n`;
}
const mode = document.mode;
const lines = [`当前模式:${MODE_SUMMARY[mode.mode]}`];
if (mode.forced_by_env) {
lines.push(
`设置里是 ${mode.configured_mode};环境变量 ONE_MEMORY_GRAPH_STORE 强制使用 ultra。`,
);
}
lines.push("原始对话:全部保留(任何模式都不删除)");
if (document.state === "unchanged") {
lines.push("已经是这个模式,未做改动。");
}
if (document.state === "preview") {
lines.push(`将切换到:${MODE_SUMMARY[document.target]}`);
for (const effect of document.effects)
lines.push(`- ${EFFECT_TEXT[effect]}`);
lines.push(`未执行任何改动。确认后运行:${document.next_step}`);
}
if (document.state === "committed") {
lines.push("已切换。");
for (const effect of document.effects)
lines.push(`- ${EFFECT_TEXT[effect]}`);
}
return `${lines.join("\n")}\n`;
}
export type EvolveCommand = {
agentId?: string;
maxBatches?: number;
wait: boolean;
};
/** `evolve [--agent-id <id>] [--max-batches <n>] [--wait]`; null = usage. */
export function parseEvolveCommand(
values: readonly string[],
): EvolveCommand | null {
const parsed: EvolveCommand = { wait: false };
const rest = values.filter((value) => value !== "--json");
for (let index = 0; index < rest.length; index += 1) {
const flag = rest[index];
if (flag === "--wait" && !parsed.wait) {
parsed.wait = true;
} else if (flag === "--agent-id" && parsed.agentId === undefined) {
const value = rest[index + 1];
if (!value || !/^[A-Za-z0-9._:-]{1,128}$/.test(value)) return null;
parsed.agentId = value;
index += 1;
} else if (flag === "--max-batches" && parsed.maxBatches === undefined) {
const value = Number(rest[index + 1]);
if (!Number.isInteger(value) || value < 1 || value > 1000) return null;
parsed.maxBatches = value;
index += 1;
} else {
return null;
}
}
return parsed;
}
export type EvolveDocument = {
schema_version: 1;
product: string;
command: "evolve";
} & (
| { state: "running" | "succeeded" | "failed"; run: Record<string, unknown> }
| { state: "rejected" | "unknown" | "unavailable"; code: string }
);
const RUN_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
function evolveRun(value: unknown): Record<string, unknown> | null {
if (!value || typeof value !== "object") return null;
const run = value as Record<string, unknown>;
if (
typeof run.run_id !== "string" ||
!RUN_ID.test(run.run_id) ||
!["running", "succeeded", "failed"].includes(String(run.state))
) {
return null;
}
return run;
}
export async function runEvolveCommand(
command: EvolveCommand,
options: {
userId?: string;
reader: CoreReader;
operations: CoreOperations;
sleep?: (ms: number) => Promise<void>;
pollMs?: number;
},
): Promise<EvolveDocument> {
const base = {
schema_version: 1 as const,
product: PRODUCT,
command: "evolve" as const,
};
const body: Record<string, unknown> = {};
if (options.userId !== undefined) body.user_id = options.userId;
if (command.agentId !== undefined) body.agent_id = command.agentId;
if (command.maxBatches !== undefined) body.max_batches = command.maxBatches;
const started = await options.operations.post<unknown>(
"/api/v2/evolve",
body,
);
if (started.state !== "committed") {
return { ...base, state: started.state, code: started.code };
}
let run = evolveRun(started.data);
if (run === null)
return { ...base, state: "unknown", code: "response_invalid" };
const sleep =
options.sleep ??
((ms: number) => new Promise((done) => setTimeout(done, ms)));
while (command.wait && run.state === "running") {
await sleep(options.pollMs ?? 2_000);
const read = await options.reader<unknown>(
`/api/v2/evolve/${run.run_id}`,
options.userId === undefined ? {} : { user_id: options.userId },
);
if (read.state !== "ok") {
return { ...base, state: "unavailable", code: read.code };
}
const next = evolveRun(read.data);
if (next === null)
return { ...base, state: "unknown", code: "response_invalid" };
run = next;
}
return {
...base,
state: run.state as "running" | "succeeded" | "failed",
run: projectRun(run),
};
}
/** Counts and codes; SDK result bodies stay in the Core receipt. */
function projectRun(run: Record<string, unknown>): Record<string, unknown> {
const keep = [
"run_id",
"user_id",
"agent_id",
"state",
"trigger",
"started_at",
"finished_at",
"rounds",
"tasks_processed",
"batch_failures",
"cycle_status",
"should_continue",
"continuation_reason",
"remaining_clusters",
"remaining_actionable_clusters",
"remaining_fresh_facts",
"capped",
"elapsed_ms",
"error_code",
"failure_kind",
];
return Object.fromEntries(
keep.filter((key) => key in run).map((key) => [key, run[key]]),
);
}
export function evolveExitCode(document: EvolveDocument): number {
return document.state === "running" || document.state === "succeeded" ? 0 : 1;
}
export function renderEvolveText(document: EvolveDocument): string {
if ("code" in document) {
return document.code === "memory_mode_not_ultra"
? "进化只在 ultra 模式下可用。切换:memhub settings mode ultra --yes\n"
: `进化未能开始:${document.code}\n`;
}
const run = document.run;
if (document.state === "running") {
return `进化已开始(${run.run_id}),在后台运行;加 --wait 等待结果。\n`;
}
if (document.state === "failed") {
return `进化失败(${run.run_id}):${String(run.error_code ?? "unknown")}\n`;
}
return `进化完成(${run.run_id}):${run.rounds} 轮,处理 ${run.tasks_processed} 项${
run.cycle_status ? `,周期 ${run.cycle_status}` : ""
}${run.capped ? ",已到轮数上限,可再次运行继续" : ""}\n`;
}
+37 -3
View File
@@ -14,6 +14,25 @@ import { type CoreReader, RESERVED_MEMORY_SEGMENTS } from "./core-client.js";
import type { CoreOperations } from "./core-operations.js";
import { PRODUCT } from "./version.js";
/** A MemHub user id, the same shape Core accepts. `local` is the default. */
const USER_ID = /^[A-Za-z0-9._-]{1,64}$/;
export const DEFAULT_USER_ID = "local";
export function isUserId(value: string): boolean {
return USER_ID.test(value);
}
/**
* The `user_id` to send, or nothing. The default user is never named on the
* wire, so a request without `--user-id` stays exactly what it was before
* multi-user support and still works against an older Core.
*/
export function userIdParameters(userId?: string): { user_id?: string } {
return userId === undefined || userId === DEFAULT_USER_ID
? {}
: { user_id: userId };
}
/** Output budget: a long memory is shortened visibly, never dropped silently. */
const CONTENT_BUDGET = 240;
@@ -101,10 +120,12 @@ export interface MemoryDetailDocument {
export async function readMemoryPage(
reader: CoreReader,
query: MemoryQuery,
userId?: string,
): Promise<MemoryPageDocument> {
const parameters: Record<string, string | number> = {
offset: query.offset,
limit: query.limit,
...userIdParameters(userId),
};
if (query.status !== undefined) parameters.statuses = query.status;
if (query.source !== undefined) parameters.source = query.source;
@@ -156,8 +177,12 @@ export async function readMemoryDetail(
reader: CoreReader,
memoryId: string,
namespace: MemoryNamespace = {},
userId?: string,
): Promise<MemoryDetailDocument> {
const parameters = namespaceParameters(namespace);
const parameters = {
...namespaceParameters(namespace),
...userIdParameters(userId),
};
const outcome = await reader<unknown>(
`/api/v2/memories/${memoryId}`,
Object.keys(parameters).length > 0 ? parameters : undefined,
@@ -236,6 +261,7 @@ export async function addMemory(
operations: CoreOperations,
text: string,
requestId: string = randomUUID(),
userId?: string,
): Promise<MemoryAddDocument> {
const base = {
schema_version: 2 as const,
@@ -259,7 +285,10 @@ export async function addMemory(
}
throw error;
}
const result = await operations.post<unknown>(request.path, request.body);
const result = await operations.post<unknown>(request.path, {
...request.body,
...userIdParameters(userId),
});
if (result.state !== "committed") {
return { ...base, state: result.state, code: result.code };
}
@@ -273,6 +302,7 @@ export async function addMemory(
export async function readMemoryAddStatus(
reader: CoreReader,
requestId: string,
userId?: string,
): Promise<MemoryAddDocument> {
const base = {
schema_version: 2 as const,
@@ -283,7 +313,11 @@ export async function readMemoryAddStatus(
if (!UUID.test(requestId)) {
return { ...base, state: "invalid", code: "invalid_request_id" };
}
const outcome = await reader<unknown>(memoryAddStatusPath(requestId));
const user = userIdParameters(userId);
const outcome =
user.user_id === undefined
? await reader<unknown>(memoryAddStatusPath(requestId))
: await reader<unknown>(memoryAddStatusPath(requestId), user);
if (outcome.state === "not_found") {
return { ...base, state: "not_found", code: "memory_add_not_found" };
}
@@ -0,0 +1,166 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
}));
vi.mock("axios", () => ({
default: {
create: vi.fn(() => transport),
get: vi.fn(),
post: vi.fn(),
isAxiosError: () => false,
},
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
const VERSIONS = {
user_id: "wife",
request_id: "r-versions",
elapsed_ms: 2,
chains: [
{
memory_id: "l0-head",
content: "The user's name is Fei.",
status: "active",
memory_at: null,
gmt_created: 1_790_000_100,
basic_info: { name: "Fei" },
versions: [
{
memory_id: "l0-head",
content: "The user's name is Fei.",
memory_at: null,
gmt_created: 1_790_000_100,
current: true,
},
{
memory_id: "l0-old",
content: "The user's name is 费天翔.",
memory_at: null,
gmt_created: 1_790_000_000,
current: false,
},
],
},
],
};
beforeEach(() => {
vi.clearAllMocks();
for (const method of Object.values(transport)) method.mockReset();
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue({
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
});
transport.get.mockImplementation(async (url: string) =>
url === "/api/v2/user-profile/versions"
? { status: 200, data: VERSIONS }
: { status: 404, data: {} },
);
});
describe("memhub profile --versions", () => {
it("reads the L0 version chain for the named user", async () => {
const result = await capturedRun([
"profile",
"--versions",
"--user-id",
"wife",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.get).toHaveBeenCalledWith(
"/api/v2/user-profile/versions",
{ params: { user_id: "wife" } },
);
const document = JSON.parse(result.stdout);
expect(document.state).toBe("ok");
expect(document.chains[0].versions).toHaveLength(2);
});
it("marks the current version in text and names no user for local", async () => {
const result = await capturedRun(["profile", "--versions"]);
expect(result.code).toBe(0);
const [url, config] = transport.get.mock.calls[0] ?? [];
expect(url).toBe("/api/v2/user-profile/versions");
expect(config?.params?.user_id).toBeUndefined();
expect(result.stdout).toContain("* l0-head");
expect(result.stdout).toContain(" l0-old");
expect(result.stdout).toContain("费天翔");
});
it("refuses an unknown flag and a repeated --versions", async () => {
expect((await capturedRun(["profile", "--version-chain"])).code).toBe(2);
expect(
(await capturedRun(["profile", "--versions", "--versions"])).code,
).toBe(2);
expect(transport.get).not.toHaveBeenCalled();
});
it("reports a malformed answer as invalid", async () => {
transport.get.mockResolvedValue({ status: 200, data: { chains: "x" } });
const result = await capturedRun(["profile", "--versions", "--json"]);
expect(result.code).toBe(1);
expect(JSON.parse(result.stdout).code).toBe("response_invalid");
});
});
async function capturedRun(arguments_: string[]) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return {
code: await run(arguments_, {
ONE_MEMORY_DATA_DIR: "/synthetic/data",
}),
stdout,
stderr,
};
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
+107 -1
View File
@@ -14,6 +14,11 @@ import {
validateProcessing,
validateSources,
} from "./core-responses.js";
import {
describePipeline,
type PipelineSummary,
pipelineLine,
} from "./status-report.js";
import { PRODUCT } from "./version.js";
type ReadState =
@@ -38,6 +43,21 @@ export interface SourcesDocument {
state: ReadState["state"];
code?: string;
sources: Array<Record<string, unknown>>;
/** Every grant of every user (`GET /api/v1/source-grants`). Absent when
* Core is too old to list them; `sources` is the default user's view. */
grants?: SourceGrant[];
}
/** One source grant. Only Hermes has grants for users other than `local`,
* one per Hermes profile root. */
export interface SourceGrant {
user_id: string;
source_id: string;
root_label: string | null;
root_path: string | null;
authorized: boolean;
revoked_at: string | null;
sync_enabled: boolean;
}
export interface SyncDocument {
@@ -69,6 +89,8 @@ export interface ProgressDocument {
summary_state: ReadState;
summary: Record<string, unknown> | null;
};
/** The SDK mode and last evolve per user; absent on an older Core. */
pipeline?: PipelineSummary | null;
}
export async function readSources(
@@ -86,14 +108,66 @@ export async function readSources(
sources: [],
};
}
const grants = await readSourceGrants(reader);
return {
schema_version: SCHEMA_VERSION,
product: PRODUCT,
state: "ok",
sources: result.data.map(projectSource),
// `/api/v1/sources` is the default user's view of every source.
sources: result.data.map((source) => ({
...projectSource(source),
user_id: "local",
})),
...(grants === null ? {} : { grants }),
};
}
/** The grants of every user, or null when Core cannot list them (an older
* Core answers 404): the source list is then shown as before. */
async function readSourceGrants(
reader: CoreReader,
): Promise<SourceGrant[] | null> {
let outcome: ReadOutcome<unknown>;
try {
outcome = await reader<unknown>("/api/v1/source-grants");
} catch {
return null;
}
if (outcome.state !== "ok") return null;
const value = outcome.data;
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
const items = (value as { grants?: unknown }).grants;
if (!Array.isArray(items)) return null;
const grants: SourceGrant[] = [];
const text = (candidate: unknown): string | null =>
typeof candidate === "string" && candidate.length <= 4096
? candidate
: null;
for (const item of items) {
if (!item || typeof item !== "object" || Array.isArray(item)) return null;
const grant = item as Record<string, unknown>;
if (
typeof grant.user_id !== "string" ||
!/^[A-Za-z0-9._-]{1,64}$/.test(grant.user_id) ||
typeof grant.source_id !== "string" ||
!/^[a-z0-9][a-z0-9-]{0,127}$/.test(grant.source_id) ||
typeof grant.authorized !== "boolean"
) {
return null;
}
grants.push({
user_id: grant.user_id,
source_id: grant.source_id,
root_label: text(grant.root_label),
root_path: text(grant.root_path),
authorized: grant.authorized,
revoked_at: text(grant.revoked_at),
sync_enabled: grant.sync_enabled === true,
});
}
return grants;
}
export async function readSync(reader: CoreReader): Promise<SyncDocument> {
const result = validated(
await reader<unknown>("/api/v1/live-sync"),
@@ -216,6 +290,10 @@ export async function readProgress(
summary:
processing.state === "ok" ? projectProcessing(processing.data) : null,
},
pipeline:
processingRead.state === "ok"
? describePipeline(processingRead.data)
: null,
};
}
@@ -231,6 +309,22 @@ export function renderSourcesText(document: SourcesDocument): string {
);
}
if (document.sources.length === 0) lines.push("(没有来源)");
// The list above is the default user's (`local`). Another user's grant,
// a Hermes profile, is listed with its user so it is not mistaken for it.
const others = (document.grants ?? []).filter(
(grant) => grant.user_id !== "local",
);
if (others.length > 0) {
lines.push("其他用户的连接:");
for (const grant of others) {
const name =
document.sources.find((source) => source.source_id === grant.source_id)
?.display_name ?? grant.source_id;
lines.push(
`- ${name}(用户 ${grant.user_id}):${grant.authorized ? "已连接" : grant.revoked_at ? "已撤销" : "未连接"};处理${grant.sync_enabled ? "已开启" : "已冻结"}${grant.root_path ? `;目录 ${grant.root_path}` : ""}`,
);
}
}
lines.push("");
return lines.join("\n");
}
@@ -270,6 +364,18 @@ export function renderProgressText(document: ProgressDocument): string {
? `记忆整理:待处理 ${document.generation.summary.pending} · 进行中 ${document.generation.summary.running} · 等待模型 ${document.generation.summary.waiting_for_model} · 失败 ${document.generation.summary.failed}`
: "记忆整理:暂时无法读取",
);
if (document.pipeline) {
lines.push(`记忆模式:${pipelineLine(document.pipeline)}`);
for (const [userId, user] of Object.entries(
document.pipeline.evolve_users,
)) {
lines.push(
` 进化 ${userId}:${user.last_run_at ?? "尚未进化"}${
user.last_error ? `(失败:${user.last_error})` : ""
}`,
);
}
}
if (document.imports.output_truncated) {
lines.push("这里只显示汇总;完整记录请使用 --json。");
}
+62 -1
View File
@@ -179,12 +179,72 @@ describe("CLI-07/10 source and synchronization reads", () => {
sync_enabled: true,
sync_completeness: "additions_only",
});
expect(requestedPaths()).toEqual(["/api/v1/sources"]);
// The grants list answers 404 here, as an older Core does: the source
// list is shown as before, every row the default user's.
expect(report.sources[0].user_id).toBe("local");
expect(report.grants).toBeUndefined();
expect(requestedPaths()).toEqual([
"/api/v1/sources",
"/api/v1/source-grants",
]);
expect(requestedPaths().some((path) => path.includes("detect"))).toBe(
false,
);
});
it("shows every user's grant, a Hermes profile with its user", async () => {
const grants = {
schema_version: 1,
grants: [
{
user_id: "local",
source_id: "codex",
root_label: ".codex",
root_path: "/home/synthetic/.codex",
authorized: true,
revoked_at: null,
sync_enabled: true,
},
{
user_id: "wife",
source_id: "hermes",
root_label: "wife",
root_path: "/home/synthetic/.hermes/profiles/wife",
authorized: true,
revoked_at: null,
sync_enabled: true,
},
],
};
transport.get.mockImplementation(async (url: string) =>
url === "/api/v1/source-grants"
? { status: 200, data: grants }
: (RESPONSES.get(url) ?? { status: 404, data: {} }),
);
let streams = captureStreams();
try {
expect(await run(["sources", "list", "--json"], {})).toBe(0);
} finally {
streams.restore();
}
const report = JSON.parse(streams.stdout());
expect(report.grants).toEqual(grants.grants);
expect(report.sources[0]).toMatchObject({
source_id: "codex",
user_id: "local",
});
streams = captureStreams();
try {
expect(await run(["sources", "list"], {})).toBe(0);
} finally {
streams.restore();
}
expect(streams.stdout()).toContain("其他用户的连接:");
expect(streams.stdout()).toContain(
"hermes(用户 wife):已连接;处理已开启;目录 /home/synthetic/.hermes/profiles/wife",
);
});
it("keeps degraded synchronization distinct and prints its error code", async () => {
const streams = captureStreams();
try {
@@ -552,6 +612,7 @@ describe("CLI-19 progress and issue reads", () => {
requestedPaths().every((path) =>
[
"/api/v1/sources",
"/api/v1/source-grants",
"/api/v1/live-sync",
"/api/v1/import-jobs",
"/api/v1/memory-processing",
+131
View File
@@ -57,6 +57,9 @@ export interface StatusDocument {
broker?: BackgroundHealth["broker"];
};
sections: Record<string, Section>;
/** R10: the SDK mode and System 2 state; null when the Runtime is not read
* or is older than the setting. */
pipeline?: PipelineSummary | null;
summary: {
state: "ok" | "partial";
degraded: string[];
@@ -64,6 +67,99 @@ export interface StatusDocument {
};
}
export interface PipelineSummary {
mode: "lite" | "pro" | "ultra";
configured_mode: "lite" | "pro" | "ultra";
forced_by_env: boolean;
last_evolve_at: string | null;
evolve_schedule: { enabled: boolean; time: string } | null;
evolve_running: number;
evolve_users: Record<
string,
{
last_run_at: string | null;
last_error: string | null;
state: string | null;
}
>;
graph_build: string | null;
/** SDK features (`memory_features`) whose value differs from the default. */
features: Record<string, boolean>;
}
const MODES = new Set(["lite", "pro", "ultra"]);
/** Projection of `pipeline` in `/api/v1/memory-processing`; null if absent or malformed. */
export function describePipeline(value: unknown): PipelineSummary | null {
if (!value || typeof value !== "object") return null;
const pipeline = (value as { pipeline?: unknown }).pipeline;
if (!pipeline || typeof pipeline !== "object") return null;
const item = pipeline as Record<string, unknown>;
if (
!MODES.has(String(item.mode)) ||
!MODES.has(String(item.configured_mode))
) {
return null;
}
const evolve =
item.evolve && typeof item.evolve === "object"
? (item.evolve as Record<string, unknown>)
: {};
const schedule =
evolve.schedule && typeof evolve.schedule === "object"
? (evolve.schedule as Record<string, unknown>)
: null;
const users: PipelineSummary["evolve_users"] = {};
if (evolve.users && typeof evolve.users === "object") {
for (const [userId, raw] of Object.entries(
evolve.users as Record<string, unknown>,
)) {
const user =
raw && typeof raw === "object" ? (raw as Record<string, unknown>) : {};
const result =
user.last_result && typeof user.last_result === "object"
? (user.last_result as Record<string, unknown>)
: {};
users[userId] = {
last_run_at:
typeof user.last_run_at === "string" ? user.last_run_at : null,
last_error:
typeof user.last_error === "string" ? user.last_error : null,
state: typeof result.state === "string" ? result.state : null,
};
}
}
const features: Record<string, boolean> = {};
if (item.features && typeof item.features === "object") {
for (const [name, value] of Object.entries(
item.features as Record<string, unknown>,
)) {
if (typeof value === "boolean") features[name] = value;
}
}
const graph =
item.graph_build && typeof item.graph_build === "object"
? (item.graph_build as Record<string, unknown>)
: {};
return {
mode: item.mode as PipelineSummary["mode"],
configured_mode: item.configured_mode as PipelineSummary["mode"],
forced_by_env: item.forced_by_env === true,
last_evolve_at:
typeof item.last_evolve_at === "string" ? item.last_evolve_at : null,
evolve_schedule:
schedule &&
typeof schedule.enabled === "boolean" &&
typeof schedule.time === "string"
? { enabled: schedule.enabled, time: schedule.time }
: null,
evolve_running: Array.isArray(evolve.running) ? evolve.running.length : 0,
evolve_users: users,
graph_build: typeof graph.state === "string" ? graph.state : null,
features,
};
}
const SECTION_NAMES = [
"sources",
"sync",
@@ -97,6 +193,7 @@ export async function composeStatusReport(options: {
attachFailure?: string;
}): Promise<StatusDocument> {
const sections: Record<string, Section> = {};
let pipeline: PipelineSummary | null = null;
if (options.reader === null) {
const state: SectionState =
options.attachFailure === undefined ? "not_checked" : "unavailable";
@@ -118,6 +215,8 @@ export async function composeStatusReport(options: {
validateProcessing,
describeProcessing,
);
pipeline =
processing.state === "ok" ? describePipeline(processing.data) : null;
sections.model = section(model, validateModelProfiles, describeModel);
sections.memories = section(
memories,
@@ -169,6 +268,7 @@ export async function composeStatusReport(options: {
},
bridge,
sections,
pipeline,
summary: {
state:
degraded.length === 0 && notChecked.length === 0 ? "ok" : "partial",
@@ -205,6 +305,7 @@ export function renderStatusText(
`${SECTION_LABELS[name]}:${sectionLine(report.sections[name])}`,
);
}
if (report.pipeline) lines.push(`记忆模式:${pipelineLine(report.pipeline)}`);
if (report.summary.state === "partial") {
const notes = [
report.summary.degraded.length > 0
@@ -220,6 +321,36 @@ export function renderStatusText(
return lines.join("\n");
}
export function pipelineLine(pipeline: PipelineSummary): string {
const parts: string[] = [pipeline.mode];
if (pipeline.forced_by_env)
parts.push("(ONE_MEMORY_GRAPH_STORE 强制 ultra)");
if (pipeline.mode === "ultra") {
const schedule = pipeline.evolve_schedule;
parts.push(
schedule?.enabled
? `,每日 ${schedule.time} 自动进化`
: ",自动进化已关闭",
);
parts.push(
pipeline.last_evolve_at
? `,上次进化 ${pipeline.last_evolve_at}`
: ",尚未进化",
);
if (pipeline.evolve_running > 0)
parts.push(`,进行中 ${pipeline.evolve_running}`);
}
const changed = Object.entries(pipeline.features);
if (changed.length > 0) {
parts.push(
`;SDK 功能:${changed
.map(([name, value]) => `${name}=${value ? "开" : "关"}`)
.join("、")}`,
);
}
return parts.join("");
}
function runtimeLine(report: StatusDocument): string {
if (report.runtime.state !== "running") {
return report.runtime.state === "stopped" ? "未运行" : "暂时无法确认";
+405
View File
@@ -0,0 +1,405 @@
import { mkdtemp, writeFile } from "node:fs/promises";
import os, { tmpdir } from "node:os";
import path from "node:path";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { run } from "./cli.js";
const transport = vi.hoisted(() => ({
get: vi.fn(),
post: vi.fn(),
put: vi.fn(),
patch: vi.fn(),
delete: vi.fn(),
}));
const direct = vi.hoisted(() => ({ get: vi.fn(), post: vi.fn() }));
vi.mock("axios", () => ({
default: {
create: vi.fn(() => transport),
get: direct.get,
post: direct.post,
isAxiosError: () => false,
},
}));
const runtime = vi.hoisted(() => ({
inspectRuntimeStatus: vi.fn(),
discoverRuntimeAttachment: vi.fn(),
}));
vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
...(await importOriginal<typeof import("@one-memory/runtime-control")>()),
...runtime,
}));
const REQUEST = "33333333-3333-4333-8333-333333333333";
const HERMES = {
source_id: "hermes",
display_name: "Hermes",
storage_kinds: ["sqlite"],
available: true,
authorized: true,
detected: null,
root_label: ".hermes",
candidate_path: "~/.hermes",
candidate_kind: "default",
candidate_revision: 0,
candidate_root_id: "hermes-root-identity",
imported_messages: 0,
imported_conversations: 0,
capabilities: {
historical_import: "implemented",
incremental_import: "implemented",
live_sync: "implemented",
recall: "implemented",
installer: "not_implemented",
verification: "fixture_only",
},
sync_completeness: "additions_only",
location_state: "found",
};
const CODEX = {
...HERMES,
source_id: "codex",
display_name: "Codex",
candidate_root_id: "codex-root-identity",
};
function addReceipt() {
return {
evidence_id: "44444444-4444-4444-8444-444444444444",
text: "Prefers tea",
scope: "all-apps",
created_at: "2026-10-27T00:00:00Z",
state: "pending",
task_id: null,
memory_ids: [],
error: null,
};
}
beforeEach(() => {
vi.clearAllMocks();
for (const method of [
...Object.values(transport),
...Object.values(direct),
]) {
method.mockReset();
}
runtime.inspectRuntimeStatus.mockResolvedValue({
state: "running",
code: "runtime_verified",
instanceId: "instance-1",
host: "127.0.0.1",
port: 8765,
version: "1.2.3",
});
runtime.discoverRuntimeAttachment.mockResolvedValue({
instanceId: "instance-1",
apiToken: "synthetic-runtime-token",
host: "127.0.0.1",
port: 8765,
servicePid: 4242,
health: {
status: "ok",
service: "one-memory-core",
version: "1.2.3",
instance_id: "instance-1",
},
});
transport.get.mockImplementation(async (url: string) => {
if (url === "/api/v1/sources") {
return { status: 200, data: [HERMES, CODEX] };
}
if (url === "/api/v2/user-profile") {
return {
status: 200,
data: { request_id: "r-1", profile: null, elapsed_ms: 1 },
};
}
return { status: 404, data: {} };
});
transport.post.mockImplementation(
async (url: string, body: Record<string, unknown>) =>
url === "/api/v1/sources/connect"
? {
status: 200,
data: {
request_id: body.request_id,
accepted_source_ids: ["hermes"],
failed_sources: {},
jobs: [],
},
}
: url === "/api/v2/memories"
? { status: 201, data: addReceipt() }
: { status: 404, data: {} },
);
transport.put.mockResolvedValue({ status: 200, data: {} });
transport.delete.mockResolvedValue({ status: 204, data: "" });
direct.post.mockResolvedValue({
status: 202,
data: {
request_id: REQUEST,
state: "accepted",
created_at: "T1",
updated_at: "T1",
},
});
direct.get.mockResolvedValue({
data: {
request_id: REQUEST,
state: "accepted",
created_at: "T1",
updated_at: "T1",
},
});
});
describe("--user-id on the memory commands", () => {
it("passes the user as a query parameter to memory reads and deletes", async () => {
await capturedRun(["memories", "list", "--user-id", "wife", "--json"]);
expect(transport.get).toHaveBeenCalledWith("/api/v2/memory-library", {
params: { offset: 0, limit: 100, user_id: "wife" },
});
// Global: the flag may come before the subcommand too.
await capturedRun(["memories", "--user-id", "wife", "show", "m-1"]);
expect(transport.get).toHaveBeenCalledWith("/api/v2/memories/m-1", {
params: { user_id: "wife" },
});
await capturedRun(["memories", "add-status", REQUEST, "--user-id", "wife"]);
expect(transport.get).toHaveBeenCalledWith(
`/api/v2/memories/additions/${REQUEST}`,
{ params: { user_id: "wife" } },
);
await capturedRun(["profile", "--user-id", "wife", "--json"]);
expect(transport.get).toHaveBeenCalledWith("/api/v2/user-profile", {
params: { user_id: "wife" },
});
await capturedRun([
"memories",
"delete",
"fact-1",
"--kind",
"fact",
"--yes",
"--user-id",
"wife",
]);
expect(transport.delete).toHaveBeenCalledWith(
"/api/v2/memories/fact-1?kind=fact&user_id=wife",
undefined,
);
await capturedRun([
"memories",
"delete",
"m-1",
"--yes",
"--user-id",
"wife",
]);
expect(transport.delete).toHaveBeenCalledWith(
"/api/v2/memories/m-1?user_id=wife",
undefined,
);
await capturedRun(["write", "status", REQUEST, "--user-id", "wife"]);
expect(direct.get.mock.calls[0][0]).toBe(
`http://127.0.0.1:8765/api/v2/memory-writes/${REQUEST}`,
);
expect(direct.get.mock.calls[0][1].params).toEqual({ user_id: "wife" });
});
it("passes the user in the body of memory writes and searches", async () => {
await capturedRun(["memories", "add", "Prefers tea", "--user-id", "wife"]);
expect(transport.post.mock.calls[0][0]).toBe("/api/v2/memories");
expect(transport.post.mock.calls[0][1]).toMatchObject({
text: "Prefers tea",
scope: "all-apps",
user_id: "wife",
});
const directory = await mkdtemp(path.join(tmpdir(), "om-user-id-"));
const content = path.join(directory, "content.txt");
await writeFile(content, "Prefers green tea\n");
await capturedRun([
"memories",
"update",
"m-1",
"--content-file",
content,
"--user-id",
"wife",
]);
expect(transport.put.mock.calls[0][0]).toBe("/api/v2/memories/m-1");
expect(transport.put.mock.calls[0][1]).toMatchObject({ user_id: "wife" });
await capturedRun(["recall", "tea", "--user-id", "wife", "--json"]);
const search = transport.post.mock.calls.find(
([url]) => url === "/api/v2/context/search",
);
expect(search?.[1]).toMatchObject({ query: "tea", user_id: "wife" });
const messages = path.join(directory, "messages.json");
await writeFile(
messages,
JSON.stringify([{ role: "user", content: "I like tea" }]),
);
await capturedRun([
"write",
"submit",
"--messages-file",
messages,
"--session",
"s-1",
"--user-id",
"wife",
]);
expect(direct.post.mock.calls[0][0]).toBe(
"http://127.0.0.1:8765/api/v2/memory-writes",
);
expect(direct.post.mock.calls[0][1]).toMatchObject({
session_id: "s-1",
user_id: "wife",
});
});
it("names no user by default or for --user-id local", async () => {
for (const extra of [[], ["--user-id", "local"]]) {
transport.get.mockClear();
transport.post.mockClear();
await capturedRun(["memories", "list", ...extra, "--json"]);
expect(transport.get).toHaveBeenCalledWith("/api/v2/memory-library", {
params: { offset: 0, limit: 100 },
});
await capturedRun(["profile", ...extra]);
expect(transport.get).toHaveBeenCalledWith(
"/api/v2/user-profile",
undefined,
);
await capturedRun(["memories", "add", "Prefers tea", ...extra]);
expect(transport.post.mock.calls[0][1]).not.toHaveProperty("user_id");
await capturedRun(["recall", "tea", ...extra, "--json"]);
const search = transport.post.mock.calls.find(
([url]) => url === "/api/v2/context/search",
);
expect(search?.[1]).not.toHaveProperty("user_id");
}
});
it("refuses a malformed user id before contacting Core", async () => {
for (const value of ["bad:id", "", "x".repeat(65), "a b", "--json"]) {
const result = await capturedRun([
"memories",
"list",
"--user-id",
value,
]);
expect(result.code).toBe(2);
}
expect(
(await capturedRun(["profile", "--user-id", "a", "--user-id", "b"])).code,
).toBe(2);
expect((await capturedRun(["recall", "tea", "--user-id"])).code).toBe(2);
expect(transport.get).not.toHaveBeenCalled();
expect(transport.post).not.toHaveBeenCalled();
});
});
describe("sources connect for a Hermes profile", () => {
it("connects hermes for another user from its own root", async () => {
const result = await capturedRun([
"sources",
"connect",
"hermes",
"--user-id",
"wife",
"--root",
"~/.hermes/profiles/wife",
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post).toHaveBeenCalledTimes(1);
const [url, body] = transport.post.mock.calls[0];
expect(url).toBe("/api/v1/sources/connect");
expect(body).toMatchObject({
source_ids: ["hermes"],
// Core reads the custom root's identity itself.
source_roots: {},
source_candidate_revisions: {},
onboarding: false,
user_id: "wife",
root: path.join(os.homedir(), ".hermes/profiles/wife"),
});
});
it("keeps a plain connect unchanged and accepts --import with a profile", async () => {
await capturedRun(["sources", "connect", "hermes", "--import", "--json"]);
const plain = transport.post.mock.calls[0][1];
expect(plain).not.toHaveProperty("user_id");
expect(plain).not.toHaveProperty("root");
expect(plain).toMatchObject({
source_roots: { hermes: "hermes-root-identity" },
onboarding: true,
});
transport.post.mockClear();
const result = await capturedRun([
"sources",
"connect",
"hermes",
"--import",
"--user-id",
"wife",
"--root",
"/profiles/wife",
"--request-id",
REQUEST,
"--json",
]);
expect(result.code).toBe(0);
expect(transport.post.mock.calls[0][1]).toMatchObject({
request_id: REQUEST,
user_id: "wife",
root: "/profiles/wife",
});
});
it("refuses --user-id and --root for any source but hermes", async () => {
for (const argv of [
["sources", "connect", "codex", "--user-id", "wife"],
["sources", "connect", "codex", "--root", "/somewhere", "--import"],
["sources", "connect", "hermes", "codex", "--user-id", "wife"],
["sources", "connect", "--all-detected", "--user-id", "wife"],
["sources", "connect", "hermes", "--user-id", "bad:id"],
]) {
const result = await capturedRun(argv);
expect(result.code).toBe(2);
expect(result.stderr).toContain("hermes");
}
expect(transport.post).not.toHaveBeenCalled();
});
});
async function capturedRun(arguments_: string[]) {
let stdout = "";
let stderr = "";
const stdoutSpy = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk) => {
stdout += String(chunk);
return true;
});
const stderrSpy = vi
.spyOn(process.stderr, "write")
.mockImplementation((chunk) => {
stderr += String(chunk);
return true;
});
try {
return {
code: await run(arguments_, {
ONE_MEMORY_DATA_DIR: "/synthetic/data",
}),
stdout,
stderr,
};
} finally {
stdoutSpy.mockRestore();
stderrSpy.mockRestore();
}
}
+1 -1
View File
@@ -12,7 +12,7 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.6",
"hy-memory==2.0.9",
# chromadb arrives through the HY SDK; locked here so an unrelated upgrade
# cannot move the SDK's store. Built-in embedding consumes numpy directly.
"chromadb==1.5.9",
File diff suppressed because it is too large. Load diff
+3 -2
View File
@@ -584,6 +584,7 @@ def _sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
}
else:
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
from one_memory.hy_sdk_runtime import builtin_retrieval_snapshot
from one_memory.sdk_data_plane_migration import (
MemoryAuthority,
SdkDataPlaneMigrationStore,
@@ -595,14 +596,14 @@ def _sdk_migration_status(options: Sequence[str], stream: TextIO) -> int:
authority = SdkDataPlaneMigrationStore(database_path).authority()
if authority is MemoryAuthority.SDK:
profile = read_active_profile(database_path)
profile = read_active_profile(database_path) or builtin_retrieval_snapshot()
leftovers = bool(
legacy_tables_present(database_path)
or abandoned_staging_present(
data_dir=settings.data_dir,
namespace=HY_MEMORY_NAMESPACE,
)
or (profile is not None and legacy_vector_collections(settings.data_dir, profile))
or legacy_vector_collections(settings.data_dir, profile)
)
state = "required" if leftovers else "not_required"
else:
@@ -5,6 +5,9 @@ from typing import Annotated, Literal
from pydantic import BaseModel, ConfigDict, Field, model_validator
from one_memory.sdk_search_options import SdkSearchOptions
from one_memory.user_scope import DEFAULT_USER_ID, UserId
ContextKind = Literal["raw", "memory"]
ContextSearchChannel = Literal["lexical", "semantic"]
ContextSemanticStatus = Literal["not_searched", "completed", "failed", "timed_out"]
@@ -47,17 +50,24 @@ class ContextSearchRequest(BaseModel):
return self
class SdkContextSearchRequest(ContextSearchRequest):
class SdkContextSearchRequest(ContextSearchRequest, SdkSearchOptions):
"""`POST /api/v2/context/search`: the three SDK recall lanes are set apart.
`limit` is the normal lane's top-k and the cap on returned items;
`schema_limit` and `intention_limit` are the SDK's schema (L6) and
proactive intention (L7) lanes. A caller's values are passed through.
The `SdkSearchOptions` fields reach the memory lane's `client.search`;
the raw lane takes the floor and the source, session, agent and time
filters (never wider than the caller's raw authority). Absent, both
lanes make the call they always made.
"""
limit: int = Field(default=10, ge=1, le=20, strict=True)
schema_limit: int = Field(default=3, ge=0, le=20, strict=True)
intention_limit: int = Field(default=1, ge=0, le=20, strict=True)
#: Whose memories to search. The App never sends it and reads `local`.
user_id: UserId = DEFAULT_USER_ID
class ContextMessage(BaseModel):
+100 -25
View File
@@ -24,13 +24,57 @@ from one_memory.hy_runtime_config import HY_MEMORY_READER
from one_memory.hy_sdk_runtime import sdk_relevance_floor
from one_memory.hy_sdk_types import (
HySdkMemoryRecord,
parse_sdk_lane_search,
parse_sdk_raw_search,
parse_sdk_search,
)
from one_memory.imports.storage import EvidenceStore
from one_memory.sdk_search_options import SdkSearchFilters, SearchLane
from one_memory.semantic_deadline import SemanticMemoryHit
from one_memory.user_scope import owns
CALL_EVENT_RESPONSE_BUDGET_MS = 250
#: Pass-through filters the raw lane honours. Scene, layers, reader and the
#: schema/working floors describe memory lanes; the raw lane is always the
#: chat store's L1 raw layer.
_RAW_LANE_FILTERS = ("created_after", "session_ids", "agent_ids", "exclude_sources", "rerank")
def raw_lane_sources(
allowed: Sequence[str] | frozenset[str] | None,
requested: Sequence[str] | None,
) -> list[str] | None:
"""The raw lane's source filter: the caller's, never wider than its authority.
`allowed` empty or None means every source (the call Core always made).
None back means nothing is left to search.
"""
allowed_list = list(allowed or ())
if requested is None:
return allowed_list
chosen = [
source for source in dict.fromkeys(requested) if not allowed_list or source in allowed_list
]
return chosen or None
def raw_lane_overrides(value: SdkSearchFilters) -> dict[str, object]:
"""The caller's raw-lane overrides: its floor and its row filters."""
overrides = value.filter_kwargs()
found = {name: overrides[name] for name in _RAW_LANE_FILTERS if name in overrides}
floor = value.lane_floor("raw")
if floor is not None:
found["min_score"] = floor
return found
def hit_namespace(hit: SemanticMemoryHit) -> dict[str, str]:
"""The `get` namespace of a hit: nothing for chat, kind and variant otherwise."""
if hit.kind in {"experience", "fact"} and hit.variant is not None:
return {"kind": hit.kind, "variant": hit.variant}
return {}
def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
"""SDK-native context: no Evidence/Governance ids or fabricated revisions."""
@@ -66,26 +110,36 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
client = await client_handle.client()
except Exception:
raise HTTPException(503, "sdk_context_unavailable") from None
memory_mode = getattr(request.app.state, "memory_mode", None)
lite = callable(memory_mode) and memory_mode() == "lite"
async def raw_lane() -> tuple[SdkRawContextItem, ...]:
if "raw" not in value.kinds or not authority.raw_allowed:
return ()
sources = raw_lane_sources(authority.raw_sources, value.include_sources)
if sources is None:
return ()
raw = await asyncio.to_thread(
client.search,
value.query,
user_ids=["local"],
agent_ids=[],
include_sources=list(authority.raw_sources or ()),
limit=value.limit,
min_score=0.0,
schema_min_score=0.0,
schema_limit=0,
intention_limit=0,
scene="chat",
reader=HY_MEMORY_READER,
layers=["l1_raw"],
**{
"user_ids": [value.user_id],
"agent_ids": [],
"include_sources": sources,
"limit": value.limit,
"min_score": 0.0,
"schema_min_score": 0.0,
"schema_limit": 0,
"intention_limit": 0,
"scene": "chat",
"reader": HY_MEMORY_READER,
"layers": ["l1_raw"],
**raw_lane_overrides(value),
},
)
hits = tuple(
hit for hit in parse_sdk_raw_search(raw) if value.passes_floor("raw", hit.score)
)
hits = parse_sdk_raw_search(raw)
return tuple(
SdkRawContextItem(
raw_id=hit.raw_id,
@@ -103,27 +157,48 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
)
async def memory_lane() -> tuple[SdkMemoryContextItem, ...]:
if "memory" not in value.kinds or not authority.memory_allowed:
# lite extracts nothing: its search answers raw turns, which the
# raw lane already reads under the raw authority.
if "memory" not in value.kinds or not authority.memory_allowed or lite:
return ()
overrides = value.search_overrides()
raw = await asyncio.to_thread(
client.search,
value.query,
user_ids=["local"],
agent_ids=[],
limit=value.limit,
min_score=sdk_relevance_floor(),
schema_min_score=sdk_relevance_floor(),
schema_limit=value.schema_limit,
intention_limit=value.intention_limit,
scene="normal",
reader=HY_MEMORY_READER,
**{
"user_ids": [value.user_id],
"agent_ids": [],
"limit": value.limit,
"min_score": sdk_relevance_floor(),
"schema_min_score": sdk_relevance_floor(),
"schema_limit": value.schema_limit,
"intention_limit": value.intention_limit,
"scene": "normal",
"reader": HY_MEMORY_READER,
**overrides,
},
)
hits = (
tuple(
hit
for hit in parse_sdk_lane_search(
raw, error_code="hy_memory_search_shape_unknown"
)
if value.passes_floor(cast(SearchLane, hit.lane), hit.score)
)
if overrides
else parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
)
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
items: list[SdkMemoryContextItem] = []
for hit in hits:
record = await asyncio.to_thread(client.get, hit.memory_id)
record = await asyncio.to_thread(client.get, hit.memory_id, **hit_namespace(hit))
memory = None if record is None else HySdkMemoryRecord.from_sdk(record)
if memory is None or memory.status != "active":
# `get` takes no user: a hit is kept only if it is the caller's.
if (
memory is None
or memory.status != "active"
or not owns(memory.user_id, value.user_id)
):
continue
items.append(
SdkMemoryContextItem(
@@ -71,6 +71,16 @@ def graph_store_enabled() -> bool:
return raw != "none"
def effective_memory_mode(configured: str = "pro") -> str:
"""The SDK mode clients are built with: the setting, unless dev forces ultra.
`ONE_MEMORY_GRAPH_STORE` (any value but empty or `none`) predates the
setting and keeps working for development: it forces ultra whatever the
configured mode. Without it the configured mode (default pro) applies.
"""
return "ultra" if graph_store_enabled() else configured
def inspect_graph_width_mismatch(config: Any) -> GraphWidthMismatch | None:
"""Return a mismatch without renaming or deleting anything."""
db_path = Path(config.graph_store.db_path)
@@ -83,11 +93,12 @@ def inspect_profile_graph_width(
*,
api_key: str | None,
data_dir: Path,
mode: str | None = None,
) -> GraphWidthMismatch | None:
"""Inspect the graph path and width selected by the active profile."""
del api_key
return _inspect_graph_width(
_profile_graph_path(data_dir),
_profile_graph_path(data_dir, mode),
_profile_embedding_width(profile),
)
@@ -98,11 +109,12 @@ def decide_profile_graph_width(
api_key: str | None,
data_dir: Path,
decision: GraphWidthDecision,
mode: str | None = None,
) -> GraphWidthOutcome:
"""Apply the user's decision to the active profile's graph."""
del api_key
config = _GraphAdminConfig(
graph_store=_GraphStoreConfig(str(_profile_graph_path(data_dir))),
graph_store=_GraphStoreConfig(str(_profile_graph_path(data_dir, mode))),
embedder=_EmbeddingWidthConfig(_profile_embedding_width(profile)),
)
return apply_graph_width_decision(
@@ -240,8 +252,9 @@ def _inspect_graph_width(
)
def _profile_graph_path(data_dir: Path) -> Path:
name = "graph" if graph_store_enabled() else "graph-disabled"
def _profile_graph_path(data_dir: Path, mode: str | None = None) -> Path:
ultra = mode == "ultra" if mode is not None else graph_store_enabled()
name = "graph" if ultra else "graph-disabled"
return data_dir / "hy-memory" / HY_MEMORY_NAMESPACE / name
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.6"
HY_MEMORY_VERSION = "2.0.9"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -109,11 +109,30 @@ def apply_diagnostic_log_level(level: DiagnosticLogLevel) -> None:
if any(name == root or name.startswith(f"{root}.") for root in _NOISY_LIBRARY_LOGGERS):
logging.getLogger(name).setLevel(noisy_level)
# HY configures its text log once, on the first client, at the level of
# that moment. Its file handler must follow later changes too, or a live
# switch to debug never reaches hy_memory.log.
for handler in logging.getLogger("hy_memory").handlers:
if isinstance(handler, logging.FileHandler):
handler.setLevel(selected)
os.environ["MEMORY_LOG_LEVEL"] = logging.getLevelName(selected)
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "true" if level == "debug" else "false"
_apply_hy_pipeline_policy()
os.environ["MEMORY_LOG_LLM_CONTENT"] = "true" if level == "debug" else "false"
def _apply_hy_pipeline_policy() -> None:
"""Keep HY's per-step pipeline log and trace off at every level.
Nothing in MemHub reads them: Core's own diagnostics carry the step
timing, and the JSONL/SQLite copies only grew on disk. With both off
the SDK skips the per-step bookkeeping altogether.
"""
os.environ["MEMORY_PIPELINE_LOG_ENABLED"] = "false"
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "false"
def hy_memory_version() -> str:
"""Return the installed SDK version, failing if the pinned dist is absent."""
@@ -131,7 +150,9 @@ def configure_hy_runtime_environment(data_dir: Path) -> None:
os.environ["MEMORY_LOG_DIR"] = str(log_dir)
level = diagnostic_log_level(data_dir)
os.environ["MEMORY_LOG_LEVEL"] = level.upper()
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "true" if level == "debug" else "false"
_apply_hy_pipeline_policy()
# Still meaningful with the pipeline log off: it decides whether HY's
# DEBUG lines in hy_memory.log carry full model prompts and responses.
os.environ["MEMORY_LOG_LLM_CONTENT"] = "true" if level == "debug" else "false"
os.environ["MEMORY_MEMORY_OPERATIONS_ENABLED"] = "false"
os.environ["ANONYMIZED_TELEMETRY"] = "False"
@@ -426,6 +426,7 @@ def _repair_coding(
path = root / name
if not path.is_file():
continue
_ensure_coding_schema(path)
with closing(sqlite3.connect(path)) as database:
tables = {
str(row[0])
@@ -437,10 +438,14 @@ def _repair_coding(
):
if table not in tables:
continue
# _ensure_coding_schema added it; an SDK without the column
# (before 2.0.7) leaves the time to the vector repair alone.
has_memory_at = "memory_at" in _columns(database, table)
rows = database.execute(
f"SELECT {identifier},source,created_at FROM {table}"
f"SELECT {identifier},source,created_at,"
f"{'memory_at' if has_memory_at else 'NULL'} FROM {table}"
).fetchall()
for memory_id, source, created_at in rows:
for memory_id, source, created_at, memory_at in rows:
desired = mapping.get(str(memory_id))
if desired is None:
continue
@@ -449,6 +454,13 @@ def _repair_coding(
if desired.source and source in {None, "", "default"}:
updates.append("source=?")
values.append(desired.source)
if (
has_memory_at
and desired.memory_at is not None
and memory_at != desired.memory_at
):
updates.append("memory_at=?")
values.append(desired.memory_at)
if desired.created_at is not None:
desired_created_at = (
datetime.fromtimestamp(desired.created_at, tz=UTC)
@@ -470,6 +482,32 @@ def _repair_coding(
return repaired
def _ensure_coding_schema(path: Path) -> None:
"""Bring a Coding/Working store to the installed SDK's schema.
A migrated home's stores were written by an older SDK and may lack columns
the repair writes (``memory_at`` since hy-memory 2.0.7). The repair runs
before any SDK client opens them, so the SDK's own schema step is run here
rather than repeating its DDL. ``CodingMemoryStore`` has no public
schema-only entry point: ``initialize()`` also opens its Chroma
collections, which must stay closed during the offline repair. So only its
SQLite step runs, on a store object that carries nothing but that path.
"""
from hy_memory.coding.store import ( # type: ignore[import-untyped]
CodingMemoryStore,
)
# Relies on the pinned SDK's internals (hy-memory==2.0.9 is pinned exactly).
store = CodingMemoryStore.__new__(CodingMemoryStore)
store._db_path = str(path)
store._conn = None
try:
store._init_sqlite_sync()
finally:
if store._conn is not None:
store._conn.close()
def _batches[T](values: list[T] | tuple[T, ...]) -> Iterator[tuple[T, ...]]:
for start in range(0, len(values), _BATCH):
yield tuple(values[start : start + _BATCH])
+119 -16
View File
@@ -11,7 +11,7 @@ import asyncio
import hashlib
import os
import threading
from collections.abc import Callable, Coroutine
from collections.abc import Callable, Coroutine, Iterable, Mapping
from dataclasses import dataclass
from pathlib import Path
from typing import Any, ClassVar
@@ -31,11 +31,18 @@ from one_memory.embedding_assets import (
)
from one_memory.hy_graph_admin import (
GraphWidthMismatchError,
graph_store_enabled,
effective_memory_mode,
inspect_graph_width_mismatch,
)
from one_memory.hy_sdk_types import HySdkRuntimeError
from one_memory.local_embedding import LocalEmbedder
from one_memory.log_file import confine_hy_sdk_logging
from one_memory.memory_features import (
FEATURES,
FEATURES_BY_NAME,
FeatureOverrides,
feature_overrides,
)
from one_memory.model_profiles import (
ModelProfileSnapshot,
ProviderKind,
@@ -66,6 +73,16 @@ _STORE_GATE = threading.Condition()
_OFFLINE_STORES: set[Path] = set()
_OPENING: dict[Path, int] = {}
# One client constructs at a time per home, its first-time store setup
# included. The constructor opens Chroma's `PersistentClient`, and two first
# opens on one directory race over that same unlocked System cache. On a
# clean install the write drain's client and a UI read client started in the
# same second; neither reached `VectorStore ready`, and the first write job
# was lost as unknown. Taken inside `acquire_client`, after the offline gate
# and never while `_STORE_GATE` is held, so it cannot deadlock with
# `hold_stores_offline`, which refuses while `_OPENING` counts a client.
_CONSTRUCTING: dict[Path, threading.Lock] = {}
class LocalOnnxEmbeddingService:
"""Public ``EmbeddingService`` implementation backed by MemHub's ONNX model."""
@@ -159,11 +176,17 @@ class SdkClientHandle:
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None = None,
mode: str | None = None,
features: FeatureOverrides = (),
) -> None:
self._profile = profile
self._api_key = api_key
self._data_dir = data_dir
self._http_transport_factory = http_transport_factory
#: The SDK mode this handle's client is built with, fixed for its life.
self.mode = mode or effective_memory_mode()
#: The SDK features set on its config (`memory_features`), fixed likewise.
self.features = features
self._lease: ClientLease | None = None
self._lock = asyncio.Lock()
@@ -178,6 +201,8 @@ class SdkClientHandle:
api_key=self._api_key,
data_dir=self._data_dir,
http_transport_factory=self._http_transport_factory,
mode=self.mode,
features=self.features,
)
return self._lease.client
@@ -206,6 +231,33 @@ def builtin_retrieval_snapshot() -> ModelProfileSnapshot:
)
def retrieval_view(profile: ModelProfileSnapshot) -> ModelProfileSnapshot:
"""The read side of `profile`: its embedding space, and no generation.
Reads must open the collection writes fill (#242), and the collection is
named by the embedding only. Generation stays the built-in placeholder, so
a read never needs the generation model. A remote embedding still needs
the profile's API key, which `build_sdk_config` uses as the embedding key.
"""
provider = profile.embedding_provider_kind or profile.provider_kind
endpoint: str | None = None
if provider is not ProviderKind.LOCAL_ONNX:
endpoint = profile.embedding_base_url
if not endpoint:
raise HySdkRuntimeError("hy_sdk_model_endpoint_missing")
return builtin_retrieval_snapshot().model_copy(
update={
"profile_id": profile.profile_id,
"revision": profile.revision,
"embedding_provider_kind": provider,
"embedding_model": profile.embedding_model,
"embedding_base_url": endpoint,
"embedding_dimensions": profile.embedding_dimensions,
"has_api_key": profile.has_api_key,
}
)
def sdk_relevance_floor() -> float:
raw = os.environ.get("ONE_MEMORY_HY_RELEVANCE_FLOOR", "").strip()
if not raw:
@@ -217,13 +269,48 @@ def sdk_relevance_floor() -> float:
return min(1.0, max(0.0, value))
def sdk_feature_defaults() -> dict[str, bool]:
"""Each `memory_features` feature's SDK default, as a fresh config has it.
That is the SDK's code default unless the process environment sets the
SDK's own variable (`FeatureSpec.env`), exactly what an unset feature
leaves on a client.
"""
config = MemoryConfig()
return {
spec.name: bool(getattr(getattr(config, spec.section), spec.field)) for spec in FEATURES
}
def apply_sdk_features(
config: MemoryConfig, features: Mapping[str, bool] | Iterable[tuple[str, bool]]
) -> MemoryConfig:
"""Set each feature a person chose on its `MemoryConfig` field.
Every exposed feature is a config field the SDK reads from the config at
call time, never from `os.environ`, so this is per client and
deterministic. Nothing here touches `vector_store`, so the collection
identity never depends on a feature.
"""
values = dict(features.items() if isinstance(features, Mapping) else features)
for name, value in feature_overrides(values):
spec = FEATURES_BY_NAME[name]
setattr(getattr(config, spec.section), spec.field, value)
return config
def build_sdk_config(
profile: ModelProfileSnapshot,
*,
api_key: str | None,
data_dir: Path,
features: Mapping[str, bool] | Iterable[tuple[str, bool]] = (),
) -> MemoryConfig:
"""Map a model snapshot to SDK models, endpoints, backends, and local paths."""
"""Map a model snapshot to SDK models, endpoints, backends, and local paths.
`features` (`memory_features`) are set on top; with none the config is
exactly what it was before the setting existed.
"""
generation_endpoint, generation_key = _generation_access(profile, api_key)
embedding_provider = profile.embedding_provider_kind or profile.provider_kind
embedding_service_is_local = embedding_provider is ProviderKind.LOCAL_ONNX
@@ -238,10 +325,8 @@ def build_sdk_config(
embedding_key = None
dimensions = profile.embedding_dimensions or BUILTIN_EMBEDDING_DIMENSIONS
else:
embedding_endpoint = _openai_endpoint(
embedding_provider,
profile.embedding_base_url or profile.base_url,
)
# Never the generation endpoint: the collection name hashes this one.
embedding_endpoint = _openai_endpoint(embedding_provider, profile.embedding_base_url)
embedding_key = "ollama" if embedding_provider is ProviderKind.OLLAMA else api_key
if embedding_provider is ProviderKind.OPENAI_COMPATIBLE and not embedding_key:
raise HySdkRuntimeError("hy_sdk_embedding_api_key_missing")
@@ -271,7 +356,7 @@ def build_sdk_config(
if thinking_off:
llm["extra_body"] = thinking_off
return MemoryConfig.from_dict(
config = MemoryConfig.from_dict(
{
"llm": llm,
"embedder": {
@@ -308,6 +393,7 @@ def build_sdk_config(
"coding": {"db_path": str(root / "coding.sqlite3")},
}
)
return apply_sdk_features(config, features)
def acquire_client(
@@ -316,9 +402,15 @@ def acquire_client(
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None = None,
mode: str | None = None,
features: FeatureOverrides = (),
) -> ClientLease:
"""Construct a public SDK client and return its explicit runtime lease.
`mode` is the SDK pipeline mode (lite / pro / ultra); None keeps the
pre-setting default (ultra when `ONE_MEMORY_GRAPH_STORE` is set, else pro).
`features` are the `memory_features` set on its config; none keeps every
SDK default.
Waits while `hold_stores_offline` holds this home's stores.
"""
key = data_dir.resolve()
@@ -332,6 +424,8 @@ def acquire_client(
api_key=api_key,
data_dir=data_dir,
http_transport_factory=http_transport_factory,
mode=mode or effective_memory_mode(),
features=features,
)
finally:
with _STORE_GATE:
@@ -370,8 +464,10 @@ def _acquire_client(
api_key: str | None,
data_dir: Path,
http_transport_factory: Callable[[], Any] | None,
mode: str = "pro",
features: FeatureOverrides = (),
) -> ClientLease:
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir)
config = build_sdk_config(profile, api_key=api_key, data_dir=data_dir, features=features)
config.http_transport_factory = http_transport_factory
mismatch = inspect_graph_width_mismatch(config)
if mismatch is not None:
@@ -381,16 +477,23 @@ def _acquire_client(
embedding_service: LocalOnnxEmbeddingService | None = None
if (profile.embedding_provider_kind or profile.provider_kind) is ProviderKind.LOCAL_ONNX:
embedding_service = LocalOnnxEmbeddingService()
with _STORE_GATE:
constructing = _CONSTRUCTING.setdefault(key, threading.Lock())
try:
client = HyMemoryClient(
config=config,
mode="ultra" if graph_store_enabled() else "pro",
runtime=runtime,
embedding_service=embedding_service,
)
with constructing:
client = HyMemoryClient(
config=config,
mode=mode,
runtime=runtime,
embedding_service=embedding_service,
)
except BaseException:
_release_runtime(key)
raise
finally:
# The constructor's first step configures HY's text log (once per
# process), so by here it exists whether or not construction succeeded.
confine_hy_sdk_logging()
return ClientLease(client, key)
@@ -449,7 +552,7 @@ def vector_collection_bases(profile: ModelProfileSnapshot) -> tuple[str, tuple[s
endpoint, dimensions = None, profile.embedding_dimensions or BUILTIN_EMBEDDING_DIMENSIONS
legacy_endpoint = "local-onnx"
else:
endpoint = _openai_endpoint(provider, profile.embedding_base_url or profile.base_url)
endpoint = _openai_endpoint(provider, profile.embedding_base_url)
dimensions = profile.embedding_dimensions
legacy_endpoint = endpoint
legacy_model = profile.embedding_model or "bge-small-zh-v1.5"
@@ -2,6 +2,7 @@
from __future__ import annotations
import re
from collections.abc import Mapping
from datetime import UTC, datetime
from typing import Literal, NamedTuple, cast
@@ -151,6 +152,7 @@ class HySdkMemoryRecord(BaseModel):
if not content.strip():
raise HySdkReadError("hy_memory_record_shape_unknown")
status = str(raw.get("status") or "active")
memory_at = raw.get("memory_at")
created_at = _epoch(raw.get("created_at"))
updated_at = _epoch(raw.get("updated_at"))
try:
@@ -167,6 +169,9 @@ class HySdkMemoryRecord(BaseModel):
str(raw["session_id"]) if isinstance(raw.get("session_id"), str) else None
),
source=str(raw["source"]) if isinstance(raw.get("source"), str) else None,
# Conversation time; hy-memory 2.0.7 stores it for experiences
# and facts. Older stores leave it unknown.
memory_at=memory_at if isinstance(memory_at, int) else None,
gmt_created=created_at,
scene=cast(MemoryScene, variant),
kind=cast(MemoryKindName, kind),
@@ -387,6 +392,8 @@ class HySdkRawRecord(BaseModel):
memory_at: int | None = None
gmt_created: int | None = None
custom: dict[str, object] = Field(default_factory=dict)
#: active, or shadow once pro extracted it; None when the SDK gave none.
status: str | None = None
@classmethod
def from_sdk(cls, raw: object) -> HySdkRawRecord:
@@ -413,11 +420,34 @@ class HySdkRawRecord(BaseModel):
raw.get("gmt_created") if isinstance(raw.get("gmt_created"), int) else None
),
custom=dict(custom),
status=str(raw["status"]) if isinstance(raw.get("status"), str) else None,
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_raw_shape_unknown") from error
_RAW_TURN_LINE = re.compile(r"^\[(user|assistant|tool|system)\]:\s?(.*)$")
def raw_turn_messages(content: str) -> tuple[tuple[str, str], ...]:
"""The `(role, text)` messages of one SDK raw turn.
The SDK stores a turn as `[user]: …\\n[assistant]: …`; a line without a
role prefix continues the message above it. Content that names no role
at all is one message of unknown role (`""`).
"""
messages: list[list[str]] = []
for line in content.splitlines():
match = _RAW_TURN_LINE.match(line)
if match is not None:
messages.append([match.group(1), match.group(2)])
elif messages:
messages[-1][1] += "\n" + line
else:
messages.append(["", line])
return tuple((role, text) for role, text in messages)
class HySdkRawHit(HySdkRawRecord):
score: float = Field(ge=0, le=1)
@@ -512,6 +542,96 @@ class HySdkUserProfile(BaseModel):
)
class HySdkProfileVersion(BaseModel):
"""One version on an L0 basic-info chain."""
model_config = ConfigDict(frozen=True, extra="forbid")
memory_id: str
content: str
#: Conversation time as the SDK renders it on a chain (epoch or text).
memory_at: int | str | None = None
gmt_created: int | None = None
#: The chain head: the version the profile shows now.
current: bool = False
class HySdkProfileChain(BaseModel):
"""One L0 basic-info head and its versions, latest first."""
model_config = ConfigDict(frozen=True, extra="forbid")
memory_id: str
content: str
status: str
memory_at: int | None = None
gmt_created: int | None = None
#: The structured fields this head holds (`basic_info_kv`).
basic_info: dict[str, object] = Field(default_factory=dict)
versions: tuple[HySdkProfileVersion, ...]
class HySdkProfileVersions(BaseModel):
"""`get_user_basic_info`: the long-term profile's L0 version chains."""
model_config = ConfigDict(frozen=True, extra="forbid")
request_id: str
chains: tuple[HySdkProfileChain, ...]
elapsed_ms: float = Field(ge=0)
@classmethod
def from_sdk(cls, raw: object) -> HySdkProfileVersions:
if not isinstance(raw, Mapping):
raise HySdkReadError("hy_memory_profile_versions_shape_unknown")
request_id = raw.get("request_id")
rows = raw.get("memories")
elapsed_ms = raw.get("elapsed_ms")
if (
not isinstance(request_id, str)
or not isinstance(rows, list)
or not isinstance(elapsed_ms, int | float)
or isinstance(elapsed_ms, bool)
):
raise HySdkReadError("hy_memory_profile_versions_shape_unknown")
chains: list[HySdkProfileChain] = []
for row in rows:
if not isinstance(row, Mapping) or not isinstance(row.get("memory_id"), str):
raise HySdkReadError("hy_memory_profile_versions_shape_unknown")
head_id = str(row["memory_id"])
chain = row.get("evolution_chain")
entries = chain if isinstance(chain, list) and chain else [{**row, "node_id": head_id}]
versions = tuple(
HySdkProfileVersion(
memory_id=str(entry.get("node_id") or entry.get("memory_id")),
content=str(entry.get("content") or ""),
memory_at=(
entry.get("memory_at")
if isinstance(entry.get("memory_at"), int | str)
and not isinstance(entry.get("memory_at"), bool)
else None
),
gmt_created=_epoch(entry.get("gmt_created")),
current=str(entry.get("node_id") or entry.get("memory_id")) == head_id,
)
for entry in entries
if isinstance(entry, Mapping) and (entry.get("node_id") or entry.get("memory_id"))
)
basic_info = row.get("basic_info_kv")
chains.append(
HySdkProfileChain(
memory_id=head_id,
content=str(row.get("content") or ""),
status=str(row.get("status") or "active"),
memory_at=_epoch(row.get("memory_at")),
gmt_created=_epoch(row.get("gmt_created")),
basic_info=dict(basic_info) if isinstance(basic_info, Mapping) else {},
versions=versions,
)
)
return cls(request_id=request_id, chains=tuple(chains), elapsed_ms=float(elapsed_ms))
def parse_sdk_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit, ...]:
rows = _reply_rows(raw, error_code)
found: list[SemanticMemoryHit] = []
@@ -538,6 +658,120 @@ def parse_sdk_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit
return tuple(found)
def parse_sdk_lane_search(raw: object, *, error_code: str) -> tuple[SemanticMemoryHit, ...]:
"""Every scene's search reply as hits that name their lane and namespace.
`parse_sdk_search` reads the chat reply only and forgets which lane a hit
came from. A caller may now pick the scene and a floor per lane, so this
reads all three reply shapes:
- chat: `memories` as `{schema, proactive, normal}` (or one list, normal);
- working/coding: `facts` and `experiences` of that variant;
- all: one fused `memories` list whose rows carry `scene`, `kind` and
their own lane's score (`lane_score`). The fused `score` is a rank
fusion (1/(60+rank)), not a relevance, so the lane score is reported.
Highest score first, first occurrence of an id kept.
"""
if not isinstance(raw, Mapping):
raise HySdkReadError(error_code)
scene = raw.get("scene")
found: list[SemanticMemoryHit] = []
if scene == "all":
rows = raw.get("memories")
if not isinstance(rows, list) or any(not isinstance(row, Mapping) for row in rows):
raise HySdkReadError(error_code)
for row in rows:
hit = _lane_hit(row, scene=str(row.get("scene") or "chat"), score_key="lane_score")
if hit is not None:
found.append(hit)
elif scene in ("productivity", "working", "coding"):
variant = raw.get("variant") if raw.get("variant") in {"working", "coding"} else None
lane = variant or (scene if scene in {"working", "coding"} else "working")
for key, kind in (("facts", "fact"), ("experiences", "experience")):
rows = raw.get(key) or []
if not isinstance(rows, list) or any(not isinstance(row, Mapping) for row in rows):
raise HySdkReadError(error_code)
for row in rows:
hit = _lane_hit({**row, "kind": kind}, scene=lane, score_key="score")
if hit is not None:
found.append(hit)
else:
memories = raw.get("memories")
if isinstance(memories, Mapping):
if set(memories) != {"schema", "proactive", "normal"}:
raise HySdkReadError(error_code)
channels = tuple(memories.items())
elif isinstance(memories, list):
channels = (("normal", memories),)
else:
raise HySdkReadError(error_code)
for lane, rows in channels:
if not isinstance(rows, list) or any(not isinstance(row, Mapping) for row in rows):
raise HySdkReadError(error_code)
for row in rows:
hit = _lane_hit(row, scene="chat", score_key="score", lane=str(lane))
if hit is not None:
found.append(hit)
ordered: list[SemanticMemoryHit] = []
seen: set[tuple[str, str, str | None]] = set()
for hit in sorted(found, key=lambda value: value.score, reverse=True):
identity = (hit.memory_id, hit.kind, hit.variant)
if identity in seen:
continue
seen.add(identity)
ordered.append(hit)
return tuple(ordered)
def _lane_hit(
row: Mapping[str, object],
*,
scene: str,
score_key: str,
lane: str | None = None,
) -> SemanticMemoryHit | None:
kind = str(row.get("kind")) if row.get("kind") in {"experience", "fact"} else "chat"
if kind == "chat":
memory_id = row.get("memory_id") or row.get("id")
variant = None
hit_lane = lane or "normal"
else:
id_key = "experience_id" if kind == "experience" else "fact_id"
memory_id = row.get("memory_id") or row.get(id_key)
variant = scene if scene in {"working", "coding"} else None
if variant is None:
return None
hit_lane = variant
if not isinstance(memory_id, str) or not memory_id:
return None
value = row.get(score_key)
if not isinstance(value, int | float) or isinstance(value, bool):
value = row.get("score")
score = (
max(0.0, min(1.0, float(value)))
if isinstance(value, int | float) and not isinstance(value, bool)
else 0.0
)
if kind != "chat":
return SemanticMemoryHit(
memory_id=memory_id, score=score, lane=hit_lane, kind=kind, variant=variant
)
content = row.get("content")
member_ids, formatting_complete = _formatted_member_ids(row)
return SemanticMemoryHit(
memory_id=memory_id,
score=score,
formatted_content=(
content.strip() if isinstance(content, str) and content.strip() else None
),
member_ids=member_ids,
sdk_formatted=True,
formatting_complete=formatting_complete,
lane=hit_lane,
)
class LibraryHit(NamedTuple):
"""One keyword match in a library search window, with its namespace."""
@@ -31,6 +31,16 @@ class ScanIssueCode(StrEnum):
#: import it. Named for the reason, so a reader can tell a setting from
#: a record the parser could not use.
SUBAGENT_TRACE_DISABLED = "subagent_trace_disabled"
#: A host's own notice wearing the user's role — Hermes posts a finished
#: background process, and its gateway marks the post as not written by a
#: human. Nothing in it is the person's, so it never reaches extraction.
INJECTED_MACHINE_NOTICE = "injected_machine_notice"
#: The host could not transcribe a voice message and left a placeholder
#: naming the audio file. The person did speak, but the placeholder holds
#: only a path, which an extractor would otherwise read as a fact.
VOICE_PLACEHOLDER = "voice_placeholder"
#: A health check such as "Reply with exactly: PONG": no one's words.
PROBE = "probe"
SOURCE_BUSY = "source_busy"
UNSUPPORTED_SCHEMA = "unsupported_schema"
BUDGET_EXCEEDED = "budget_exceeded"
@@ -27,6 +27,7 @@ def discovered_off_loop(
*,
suffixes: frozenset[str],
max_targets: int,
excluded_top_directories: frozenset[str] = frozenset(),
) -> AsyncIterator[SourceFileDiscoveryEvent]:
"""The same walk, for callers that have an event loop to keep free.
@@ -41,6 +42,7 @@ def discovered_off_loop(
grant,
suffixes=suffixes,
max_targets=max_targets,
excluded_top_directories=excluded_top_directories,
),
scope_key=grant.source_id,
)
@@ -51,12 +53,15 @@ def discover_source_files(
*,
suffixes: frozenset[str],
max_targets: int,
excluded_top_directories: frozenset[str] = frozenset(),
) -> Iterator[SourceFileDiscoveryEvent]:
"""Metadata only: one lstat per candidate, no open and no per-file grant lookup.
The grant and root identity are checked once before the walk. Each listed
file keeps the grant's check, which runs again when that file is opened;
the open itself refuses symlinks and any inode other than the one listed.
`excluded_top_directories` are directories directly under the root that
are never entered, such as another user's Hermes profiles.
"""
grant.check_root()
inspected = 0
@@ -64,8 +69,12 @@ def discover_source_files(
grant.root,
followlinks=False,
):
at_root = Path(directory) == grant.root
child_directories[:] = sorted(
child for child in child_directories if not (Path(directory) / child).is_symlink()
child
for child in child_directories
if not (Path(directory) / child).is_symlink()
and not (at_root and child in excluded_top_directories)
)
for filename in sorted(filenames):
if Path(filename).suffix.lower() not in suffixes:
@@ -252,6 +252,18 @@ class SourceGrantStore:
).fetchone()
return self._to_receipt(row) if row else None
def grants(self) -> tuple[tuple[SourceGrantReceipt, str], ...]:
"""Every grant of every user, with its granted root path.
One source may hold several grants, one per `(user_id, source_id)`:
a Hermes root for `local` and a Hermes profile for another user.
"""
with self._connect() as connection:
rows = connection.execute(
"SELECT * FROM source_grants ORDER BY source_id, user_id"
).fetchall()
return tuple((self._to_receipt(row), str(row["requested_path"])) for row in rows)
def require_receipt(self, *, user_id: str, source_id: str) -> SourceGrantReceipt:
receipt = self.receipt(user_id=user_id, source_id=source_id)
if receipt is None:
@@ -28,6 +28,7 @@ from one_memory.imports.adapters import (
)
from one_memory.imports.jobs import ImportJob, ImportJobStore, ImportStatus
from one_memory.imports.supervisor import ImportSupervisor
from one_memory.user_scope import DEFAULT_USER_ID, lane_key, split_lane
#: How long an automatic retry will wait for a wedged worker to notice cancel.
ABANDON_TIMEOUT_SECONDS = 5.0
@@ -64,7 +65,12 @@ class LiveSyncHealth(BaseModel):
class PollingLiveSyncSupervisor:
"""Near-live sync that reuses each adapter's persisted incremental cursor."""
"""Near-live sync that reuses each adapter's persisted incremental cursor.
A "source" here is a lane key (`user_scope.lane_key`): the bare source id
for `local`, `source@user` for another user's grant. Cursors, jobs and
locks are therefore per `(user_id, source_id)`.
"""
def __init__(
self,
@@ -92,8 +98,22 @@ class PollingLiveSyncSupervisor:
}
self._locks = {source_id: asyncio.Lock() for source_id in self.source_ids}
self._task: asyncio.Task[None] | None = None
self._group: asyncio.TaskGroup | None = None
self._observers: set[asyncio.Task[None]] = set()
def ensure_source(self, source_id: str) -> None:
"""Follow a lane granted after start, such as a new Hermes profile."""
if source_id in self._locks:
return
self.source_ids = (*self.source_ids, source_id)
self._health.setdefault(
source_id,
LiveSyncHealth(source_id=source_id, mode=self.mode, state=LiveSyncState.IDLE),
)
self._locks[source_id] = asyncio.Lock()
if self._group is not None:
self._group.create_task(self._poll_source(source_id))
def _set_health(self, source_id: str, value: LiveSyncHealth) -> None:
previous = self._health.get(source_id)
self._health[source_id] = value
@@ -152,7 +172,12 @@ class PollingLiveSyncSupervisor:
self._watched_sources = frozenset(source_ids)
def all_health(self) -> tuple[LiveSyncHealth, ...]:
return tuple(self.health(source_id) for source_id in self.source_ids)
"""`local`'s lanes only: what the App has always been shown."""
return tuple(
self.health(source_id)
for source_id in self.source_ids
if split_lane(source_id)[0] == DEFAULT_USER_ID
)
async def initialize_source(
self, source_id: str, *, baseline_missing_files: bool = True
@@ -164,11 +189,12 @@ class PollingLiveSyncSupervisor:
baselines = (
await adapter.live_baseline() if isinstance(adapter, _LiveBaselineAdapter) else {}
)
user_id, bare_source_id = split_lane(source_id)
await in_the_background(
partial(
self.job_store.ensure_live_cursors,
user_id="local",
source_id=source_id,
user_id=user_id,
source_id=bare_source_id,
baselines=baselines,
baseline_missing_files=baseline_missing_files,
)
@@ -228,7 +254,7 @@ class PollingLiveSyncSupervisor:
(
job
for job in reversed(source_jobs)
if job.source_id == source_id
if lane_key(job.user_id, job.source_id) == source_id
and job.operation == "sync"
and job.status
in {
@@ -277,13 +303,13 @@ class PollingLiveSyncSupervisor:
competing = [
job
for job in source_jobs
if job.source_id == source_id
if lane_key(job.user_id, job.source_id) == source_id
and job.operation != "sync"
and job.status
in {ImportStatus.PENDING, ImportStatus.RUNNING, ImportStatus.INTERRUPTED}
]
bounded_history_incomplete = any(
job.source_id == source_id
lane_key(job.user_id, job.source_id) == source_id
and job.operation == "history"
and job.history_after is not None
and job.status is not ImportStatus.COMPLETED
@@ -379,11 +405,12 @@ class PollingLiveSyncSupervisor:
await close()
async def _live_cursors(self, source_id: str, adapter: object) -> dict[str, SourceCursor]:
user_id, bare_source_id = split_lane(source_id)
cursors = await in_the_background(
partial(
self.job_store.get_source_cursors,
user_id="local",
source_id=source_id,
user_id=user_id,
source_id=bare_source_id,
operation="sync",
)
)
@@ -395,8 +422,8 @@ class PollingLiveSyncSupervisor:
return await in_the_background(
partial(
self.job_store.ensure_live_cursors,
user_id="local",
source_id=source_id,
user_id=user_id,
source_id=bare_source_id,
baselines=baselines,
)
)
@@ -520,9 +547,13 @@ class PollingLiveSyncSupervisor:
async def _run(self) -> None:
# A source owns its cadence; one large catch-up cannot delay every
# other source. The group keeps shutdown responsible for every poller.
async with asyncio.TaskGroup() as group:
for source_id in self.source_ids:
group.create_task(self._poll_source(source_id))
try:
async with asyncio.TaskGroup() as group:
self._group = group
for source_id in self.source_ids:
group.create_task(self._poll_source(source_id))
finally:
self._group = None
async def _poll_source(self, source_id: str) -> None:
while True:
@@ -3,7 +3,7 @@ import logging
import math
import sqlite3
from collections import Counter
from collections.abc import Callable
from collections.abc import AsyncIterator, Callable
from contextlib import suppress
from contextvars import ContextVar
from dataclasses import dataclass, field
@@ -227,14 +227,17 @@ class ImportRunner:
heartbeat = asyncio.create_task(self._keep_alive(job_id=job_id, runner_token=runner_token))
pending = _PendingSkips()
writes = _PendingWrites()
stream = adapter.scan(
ScanOptions(
cursors=cursors,
reuse_listing=(
job.operation == "history"
or (job.operation == "sync" and job.source_id == "workbuddy")
),
)
stream = _owned_by(
adapter.scan(
ScanOptions(
cursors=cursors,
reuse_listing=(
job.operation == "history"
or (job.operation == "sync" and job.source_id == "workbuddy")
),
)
),
job.user_id,
)
slice_cursors = dict(cursors)
next_event: asyncio.Future[ScanEvent] | None = None
@@ -924,3 +927,30 @@ async def _settled_work[T](
with suppress(asyncio.CancelledError, Exception):
task.result()
raise
def _owned_by(stream: AsyncIterator[ScanEvent], user_id: str) -> AsyncIterator[ScanEvent]:
"""Stamp every scanned record with the user whose root is being read.
Adapters know nothing of users and emit `local`. Which user a record
belongs to is decided by the grant the job reads (a Hermes profile's
`state.db` is that profile's user), so the job's user is the record's.
`local` passes the stream through untouched.
"""
if user_id == "local":
return stream
return _restamped(stream, user_id)
async def _restamped(stream: AsyncIterator[ScanEvent], user_id: str) -> AsyncIterator[ScanEvent]:
try:
async for event in stream:
if isinstance(event, ScannedEvidence) and event.evidence.user_id != user_id:
event = event.model_copy(
update={"evidence": event.evidence.model_copy(update={"user_id": user_id})}
)
yield event
finally:
close = getattr(stream, "aclose", None)
if close is not None:
await close()
@@ -1,4 +1,5 @@
import json
import re
import sqlite3
from collections.abc import AsyncIterator, Iterator
from functools import partial
@@ -87,6 +88,7 @@ class HermesSourceAdapter:
self.grant,
suffixes=frozenset({".jsonl", ".db"}),
max_targets=10_000,
excluded_top_directories=frozenset({HERMES_PROFILES_DIRECTORY}),
):
if isinstance(item, AuthorizedSourceFile) and _is_hermes_target(
item.relative_path.as_posix()
@@ -100,6 +102,7 @@ class HermesSourceAdapter:
self.grant,
suffixes=frozenset({".jsonl", ".db"}),
max_targets=options.max_scan_targets or 100_000,
excluded_top_directories=frozenset({HERMES_PROFILES_DIRECTORY}),
):
yield ScanProgressEvent(
phase=ScanPhase.DISCOVER,
@@ -214,7 +217,11 @@ def parse_hermes_rollout_record(
message.get("content"),
allowed_block_types=frozenset({"text", "input_text", "output_text"}),
)
text = strip_injected_memory(text)
text = strip_injected_memory(text or "")
if role == "user":
text, injected = gateway_user_text(text)
if injected is not None:
return ParsedSkip(reason_code=injected, conversation_id=conversation_id)
if not text:
return ParsedSkip(
reason_code=ScanIssueCode.NO_TEXT,
@@ -242,8 +249,17 @@ def parse_hermes_rollout_record(
)
#: Hermes keeps each extra profile under `<root>/profiles/<name>/`, with its
#: own `state.db` and sessions. A profile is another person's history: it is
#: read only through its own grant (its own MemHub user), never as part of the
#: root above it. Hermes' own export skips this directory for the same reason.
HERMES_PROFILES_DIRECTORY = "profiles"
def _is_hermes_target(relative_path: str) -> bool:
path = PurePosixPath(relative_path)
if path.parts and path.parts[0] == HERMES_PROFILES_DIRECTORY:
return False
return path.name == "state.db" or (path.suffix == ".jsonl" and "sessions" in path.parts)
@@ -389,6 +405,15 @@ def _read_state_db(
# `tool` rows after it hold only what came back.
calls = _tool_calls_text(row["tool_calls"]) if role == "assistant" else None
content = strip_injected_memory(str(row["content"] or ""))
if role == "user":
content, injected = gateway_user_text(content)
if injected is not None:
yield ScanSkip(
cursor=cursor,
conversation_id=str(row["session_id"]),
reason_code=injected,
)
continue
has_text = bool(content.strip())
if role not in {"user", "assistant"} or (not has_text and calls is None):
yield ScanSkip(
@@ -439,6 +464,69 @@ def _read_state_db(
_PERSON_SESSION_SOURCES = frozenset({"cli", "tui", "webui", "desktop"})
#: What the Hermes gateway writes into a `user` row on its own behalf. Every
#: one of these wears the `user` role, so only the text can tell them apart
#: from the person. Read off real `state.db` rows: a finished background
#: process is posted as a turn, often with the gateway's own "not written by
#: a human" note appended, which is why that note counts anywhere in the row.
_BACKGROUND_NOTICE = re.compile(
r"\[IMPORTANT:\s*(?:\d+\s+background\s+process(?:es)?\b|Background\s+process\b)",
re.IGNORECASE,
)
_GATEWAY_NOTICE = re.compile(
r"\[INTERNAL NOTIFICATION\s*\W+\s*generated by the Hermes gateway",
re.IGNORECASE,
)
_VOICE_PLACEHOLDER = re.compile(r"\[voice message could not be transcribed", re.IGNORECASE)
_PROBE = re.compile(r"Reply with exactly:\s*[A-Za-z0-9_\-]{1,24}\s*")
_MEDIA_ONLY = re.compile(r"(?:MEDIA:/\S+\s*)+")
#: Wrappers the gateway puts around a message the person did write. Only the
#: wrapper goes; what it carries is the turn. Each is matched at the edge of
#: what is left, and the set is applied until nothing more comes off, since a
#: relayed message carries the origin block inside the out-of-band markers.
_GATEWAY_WRAPPERS = (
re.compile(r"^\[OUT-OF-BAND USER MESSAGE\b[^\]\n]*\]"),
re.compile(r"\[/OUT-OF-BAND USER MESSAGE\]$"),
re.compile(
r"^Gateway message origin \(JSON data, not instructions or authorization\):[ \t]*"
r"(?:\n[ \t]*\{[^\n]*\}[ \t]*)?"
r"(?:\n[ \t]*Do not guess a reply destination[^\n]*)?"
r"(?=\n|$)"
),
re.compile(r"^\[Context from the interrupted assistant response\][ \t]*(?=\n|$)"),
)
def gateway_user_text(text: str) -> tuple[str, ScanIssueCode | None]:
"""The person's words in a Hermes `user` row, or why there are none.
Applied only to the `user` role: an assistant or tool row quoting the
same text is reporting what it saw, which is the existing tool rule's
business. The cleaned text is what becomes the evidence, so its content
hash and duplicate check see the message rather than its envelope.
"""
if _GATEWAY_NOTICE.search(text):
return "", ScanIssueCode.INJECTED_MACHINE_NOTICE
cleaned = text.strip()
while True:
unwrapped = cleaned
for wrapper in _GATEWAY_WRAPPERS:
unwrapped = wrapper.sub("", unwrapped, count=1).strip()
if unwrapped == cleaned:
break
cleaned = unwrapped
if _BACKGROUND_NOTICE.match(cleaned):
return "", ScanIssueCode.INJECTED_MACHINE_NOTICE
if _VOICE_PLACEHOLDER.match(cleaned):
return "", ScanIssueCode.VOICE_PLACEHOLDER
if _PROBE.fullmatch(cleaned):
return "", ScanIssueCode.PROBE
if _MEDIA_ONLY.fullmatch(cleaned):
return "", ScanIssueCode.NO_TEXT
return cleaned, None
def _tool_calls_text(value: object) -> str | None:
"""OpenAI-shaped `tool_calls`: `[{"function": {"name", "arguments"}}]`, stored as JSON."""
if isinstance(value, str):
@@ -24,6 +24,7 @@ from one_memory.service_levels import (
ServiceLevelScheduler,
ServiceResource,
)
from one_memory.user_scope import lane_key, split_lane
class UnknownSourceError(KeyError):
@@ -82,36 +83,45 @@ class ImportSupervisor:
self._tasks: dict[str, asyncio.Task[None]] = {}
self._drains: dict[str, asyncio.Task[None]] = {}
def register_adapter(self, adapter: SourceAdapter) -> None:
self.adapters[adapter.descriptor.source_id] = adapter
def register_adapter(self, adapter: SourceAdapter, *, user_id: str = "local") -> None:
"""Adapters are keyed by lane: the source id for `local`, `source@user`
for another user's grant of the same source."""
self.adapters[lane_key(user_id, adapter.descriptor.source_id)] = adapter
def unregister_adapter(self, source_id: str) -> None:
self.adapters.pop(source_id, None)
@staticmethod
def lane_of(job: ImportJob) -> str:
return lane_key(job.user_id, job.source_id)
def active_job_ids(self) -> frozenset[str]:
return frozenset(job_id for job_id, task in self._tasks.items() if not task.done())
def create_job(
self, source_id: str, *, operation: Literal["history", "sync"] = "history"
) -> ImportJob:
"""`source_id` is a lane key: another user's lane creates that user's job."""
if self.source_enabled is not None and not self.source_enabled(source_id):
raise SourceAccessError(SourceAccessErrorCode.GRANT_REQUIRED)
return self.job_store.create(user_id="local", source_id=source_id, operation=operation)
user_id, bare_source_id = split_lane(source_id)
return self.job_store.create(user_id=user_id, source_id=bare_source_id, operation=operation)
def enqueue(self, job_id: str) -> ImportJob:
if job_id in self._drains:
raise InvalidJobTransitionError("import_cleanup_in_progress")
job = self.job_store.require(job_id)
if job.source_id not in self.adapters:
raise UnknownSourceError(job.source_id)
if self.source_enabled is not None and not self.source_enabled(job.source_id):
lane = self.lane_of(job)
if lane not in self.adapters:
raise UnknownSourceError(lane)
if self.source_enabled is not None and not self.source_enabled(lane):
return job
existing = self._tasks.get(job_id)
if existing is None or existing.done():
self._tasks[job_id] = asyncio.create_task(
self._execute(
job_id,
source_id=job.source_id,
source_id=lane,
operation=job.operation,
)
)
@@ -130,8 +140,10 @@ class ImportSupervisor:
# pending history. Each resumed runner will yield after one bounded slice.
for status in (ImportStatus.INTERRUPTED, ImportStatus.PENDING):
for job in self.job_store.list_jobs(status=status):
if job.source_id in self.adapters:
if self.source_enabled is not None and not self.source_enabled(job.source_id):
if self.lane_of(job) in self.adapters:
if self.source_enabled is not None and not self.source_enabled(
self.lane_of(job)
):
continue
if job.status is not ImportStatus.PENDING:
self.job_store.queue_for_resume(job.job_id)
@@ -227,7 +239,7 @@ class ImportSupervisor:
}:
return
if self.source_enabled is not None and not await in_the_background(
self.source_enabled, job.source_id
self.source_enabled, self.lane_of(job)
):
return
runner = ImportRunner(
@@ -238,7 +250,7 @@ class ImportSupervisor:
sdk_queue=self.sdk_queue,
sdk_authority=self.sdk_authority,
)
adapter = self.adapters[job.source_id]
adapter = self.adapters[self.lane_of(job)]
if self.prepare_adapter is not None:
adapter = await self.prepare_adapter(job, adapter)
@@ -273,7 +285,7 @@ class ImportSupervisor:
and retry_attempt < self.max_history_retries
and (
self.source_enabled is None
or await in_the_background(self.source_enabled, current.source_id)
or await in_the_background(self.source_enabled, self.lane_of(current))
)
):
delay = self.retry_base_seconds * (2**retry_attempt)
@@ -7,9 +7,9 @@ between pages. The matches are kept here under an opaque ``window_id`` that
the client passes back with the next offset.
A window is bound to the filters it was searched with (query, source,
statuses, scene). It expires after ``ttl_seconds`` without being read, and the
store keeps at most ``max_windows`` windows, dropping the least recently read
first.
statuses, scene) and to the user whose library it searched. It expires after
``ttl_seconds`` without being read, and the store keeps at most
``max_windows`` windows, dropping the least recently read first.
Only memory identities are kept, never content: each page is read fresh from
the SDK, so an edit or a delete shows on the next page read.
"""
@@ -28,13 +28,24 @@ from one_memory.hy_sdk_types import LibraryHit
DEFAULT_WINDOW_TTL_SECONDS = 300.0
DEFAULT_MAX_WINDOWS = 32
WindowKey = tuple[str, str | None, tuple[str, ...], str]
#: (query, source, statuses, scene, user, other filters as sorted name/values pairs)
WindowKey = tuple[
str, str | None, tuple[str, ...], str, str, tuple[tuple[str, tuple[str, ...]], ...]
]
def window_key(
query: str, source: str | None, statuses: list[str], scene: str = "all"
query: str,
source: str | None,
statuses: list[str],
scene: str = "all",
user_id: str = "local",
filters: dict[str, list[str]] | None = None,
) -> WindowKey:
return (query, source, tuple(sorted(set(statuses))), scene)
extra = tuple(
sorted((name, tuple(sorted(set(values)))) for name, values in (filters or {}).items())
)
return (query, source, tuple(sorted(set(statuses))), scene, user_id, extra)
@dataclass(frozen=True)
+55 -2
View File
@@ -20,8 +20,11 @@ def start_logging_to_file(data_dir: Path) -> Path:
nowhere. That includes the loop watchdog: the one report that names what
stalled the app was unreadable in exactly the situation it exists for.
The active diagnostic collection level controls Core, HY SDK, and provider
client records. Uvicorn keeps its request log on its own non-propagating
The active diagnostic collection level controls Core and provider client
records. The HY SDK keeps its own text log, hy_memory.log under the SDK's
log directory, and stops propagating here once its first client is built;
`confine_hy_sdk_logging` then forwards only its warnings and errors to
this file. Uvicorn keeps its request log on its own non-propagating
loggers.
"""
destination = log_path(data_dir)
@@ -49,3 +52,53 @@ def start_logging_to_file(data_dir: Path) -> Path:
if root.level > logging.INFO or root.level == logging.NOTSET:
root.setLevel(logging.INFO)
return destination
HY_SDK_LOGGER = "hy_memory"
class _CoreLogForwarder(logging.Handler):
"""Hands HY warnings to Core's own core.log handler, rotation and format."""
def __init__(self, target: logging.Handler) -> None:
super().__init__(level=logging.WARNING)
self.target = target
def emit(self, record: logging.LogRecord) -> None:
self.target.handle(record)
def confine_hy_sdk_logging() -> None:
"""Keep HY's text log in hy_memory.log, with only its problems in core.log.
HY's first client sets up hy_memory.log plus a stdout and a stderr stream
on the "hy_memory" logger, and stops it propagating to the root. Under
launchd those streams are service.log and service-error.log, so every SDK
line was written twice, and none reached core.log. Drop the streams and
forward WARNING and above to core.log, which the Debug panel reads; the
SDK keeps conversation text out of those levels.
Idempotent. Does nothing until HY has configured its logger, because a
propagating logger already reaches core.log through the root.
"""
sdk = logging.getLogger(HY_SDK_LOGGER)
if sdk.propagate:
return
for handler in list(sdk.handlers):
if isinstance(handler, logging.StreamHandler) and not isinstance(
handler, logging.FileHandler
):
sdk.removeHandler(handler)
if any(isinstance(handler, _CoreLogForwarder) for handler in sdk.handlers):
return
core = next(
(
handler
for handler in logging.getLogger().handlers
if isinstance(handler, RotatingFileHandler)
and Path(handler.baseFilename).name == "core.log"
),
None,
)
if core is not None:
sdk.addHandler(_CoreLogForwarder(core))
@@ -11,10 +11,11 @@ import asyncio
import sqlite3
from collections.abc import Sequence
from datetime import UTC, datetime
from functools import partial
from typing import Any, Literal, cast
from uuid import UUID
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
from fastapi import APIRouter, BackgroundTasks, HTTPException, Query, Request
from fastapi.params import Depends
from pydantic import BaseModel, ConfigDict, Field, field_validator
@@ -34,6 +35,7 @@ from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
from one_memory.memory_tasks import MemoryTaskStatus, MemoryTaskStore
from one_memory.secrets import SecretStoreUnavailableError
from one_memory.user_scope import DEFAULT_USER_ID, USER_ID_PATTERN, UserId
SOURCE_ID = "one-memory-manual"
MemoryAddState = Literal[
@@ -53,6 +55,8 @@ class MemoryAddInput(BaseModel):
request_id: UUID
text: str = Field(min_length=1, max_length=12_000)
scope: Literal["all-apps"]
#: Whose memory this is. The App never sends it and writes `local`.
user_id: UserId = DEFAULT_USER_ID
@field_validator("text")
@classmethod
@@ -108,7 +112,11 @@ class MemoryAdditionStore:
def save(self, addition: MemoryAddInput) -> EvidenceRecord:
now = datetime.now(UTC)
if addition.user_id != DEFAULT_USER_ID and not self.sdk_authority:
# Legacy authority keeps one person's library only.
raise ValueError("memory_add_user_requires_sdk_authority")
value = EvidenceInput(
user_id=addition.user_id,
source_id=SOURCE_ID,
conversation_id=f"note:{addition.request_id}",
message_id=str(addition.request_id),
@@ -200,7 +208,7 @@ class MemoryAdditionStore:
)
return self.require(value.evidence_id)
def require(self, evidence_id: str) -> EvidenceRecord:
def require(self, evidence_id: str, *, user_id: str = DEFAULT_USER_ID) -> EvidenceRecord:
if self.sdk_authority:
if self.write_queue is None:
raise KeyError(evidence_id)
@@ -219,11 +227,11 @@ class MemoryAdditionStore:
(item for item in payload.input_records if item.evidence_id == evidence_id),
None,
)
if record is None:
if record is None or record.user_id != user_id:
raise KeyError(evidence_id)
return record
record = self.evidence.get(evidence_id)
if record is None or record.user_id != "local" or record.source_id != SOURCE_ID:
if record is None or record.user_id != user_id or record.source_id != SOURCE_ID:
raise KeyError(evidence_id)
metadata = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
if metadata is None or metadata.raw_meta.get("input_mode") != "direct_memory_note":
@@ -237,8 +245,8 @@ class MemoryAdditionStore:
raise ValueError("memory_add_source_revoked")
return record
def receipt(self, evidence_id: str) -> MemoryAddReceipt:
record = self.require(evidence_id)
def receipt(self, evidence_id: str, *, user_id: str = DEFAULT_USER_ID) -> MemoryAddReceipt:
record = self.require(evidence_id, user_id=user_id)
if self.sdk_authority:
assert self.write_queue is not None
with main_database(self.write_queue.database_path).read() as database:
@@ -300,10 +308,15 @@ class MemoryAdditionStore:
error=error,
)
def receipt_for_request(self, request_id: UUID) -> MemoryAddReceipt:
def receipt_for_request(
self, request_id: UUID, *, user_id: str = DEFAULT_USER_ID
) -> MemoryAddReceipt:
identity = str(request_id)
if self.sdk_authority:
# The Evidence id carries the user, so another user's request id
# names a different note and reads as not found.
probe = EvidenceInput(
user_id=user_id,
source_id=SOURCE_ID,
conversation_id=f"note:{identity}",
message_id=identity,
@@ -316,16 +329,16 @@ class MemoryAdditionStore:
)
},
)
return self.receipt(probe.evidence_id)
return self.receipt(probe.evidence_id, user_id=user_id)
with main_database(self.evidence.database_path).read() as db:
row = db.execute(
"SELECT evidence_id FROM evidence WHERE user_id='local' AND source_id=? "
"SELECT evidence_id FROM evidence WHERE user_id=? AND source_id=? "
"AND conversation_id=? AND message_id=?",
(SOURCE_ID, f"note:{identity}", identity),
(user_id, SOURCE_ID, f"note:{identity}", identity),
).fetchone()
if row is None:
raise KeyError(identity)
return self.receipt(str(row[0]))
return self.receipt(str(row[0]), user_id=user_id)
def memory_add_router(dependencies: Sequence[Depends]) -> APIRouter:
@@ -376,14 +389,20 @@ def memory_add_router(dependencies: Sequence[Depends]) -> APIRouter:
record = await asyncio.to_thread(additions.save, addition)
except ValueError as error:
raise HTTPException(409, str(error)) from error
saved = await asyncio.to_thread(additions.receipt, record.evidence_id)
saved = await asyncio.to_thread(
partial(additions.receipt, record.evidence_id, user_id=addition.user_id)
)
background.add_task(schedule, additions, record)
return saved
@router.get("/additions/{request_id}", response_model=MemoryAddReceipt)
def memory_add_status(request_id: UUID, request: Request) -> MemoryAddReceipt:
def memory_add_status(
request_id: UUID,
request: Request,
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> MemoryAddReceipt:
try:
return store(request).receipt_for_request(request_id)
return store(request).receipt_for_request(request_id, user_id=user_id)
except KeyError as error:
raise HTTPException(404, "memory_add_not_found") from error
except ValueError as error:
@@ -0,0 +1,618 @@
"""System 2 for ultra: evolve runs, their receipts, the nightly schedule, graph backfill.
hy-memory 2.0.9's `client.evolve(user_id, agent_id, max_batches)` is ultra
only, holds no lock, and has no scheduler: MemHub owns all three.
- A run is asynchronous: `start` answers a receipt at once and the run goes
on in the background (evolve makes model calls across the user's whole
library and can take minutes, longer than any HTTP timeout). One run per
user at a time: asking again while one runs returns that run. A run holds
the write coordinator's per-user exclusivity, so it never overlaps that
user's write batch, and loops while the SDK answers `should_continue`, up
to `EVOLVE_MAX_ROUNDS`.
- The schedule runs every user that has memories once a day at the
configured local time, and catches up after the machine slept through it.
- Switching to ultra backfills each user's existing facts into the graph
once (`build_memory_graph`).
What is kept on disk is counts and stable codes only, never memory text:
`<data_dir>/evolve-state.json`.
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import re
import threading
from collections.abc import Awaitable, Callable, Iterable, Mapping
from datetime import UTC, datetime, timedelta
from pathlib import Path
from tempfile import NamedTemporaryFile
from typing import Any, Literal, Protocol
from uuid import uuid4
from pydantic import BaseModel, ConfigDict, Field
from one_memory.diagnostic_log import diagnostic_event
from one_memory.memory_server.config_state import EvolveSchedule
from one_memory.memory_server.durable_queue import safe_error_code
from one_memory.sdk_write_processor import PRODUCT_AGENT_ID
logger = logging.getLogger(__name__)
#: Rounds one run makes while the SDK says there is more to do.
EVOLVE_MAX_ROUNDS = 20
#: How often the schedule looks at the clock; also bounds catch-up after wake.
SCHEDULE_CHECK_SECONDS = 300.0
#: Receipts kept in memory for `GET /api/v2/evolve/{run_id}`.
_KEPT_RUNS = 64
_STABLE_CODE = re.compile(r"[a-z][a-z0-9_]{0,95}")
_COUNTS = (
"remaining_clusters",
"remaining_actionable_clusters",
"remaining_fresh_facts",
)
RunState = Literal["running", "succeeded", "failed"]
Trigger = Literal["manual", "scheduled"]
class EvolveModeError(RuntimeError):
"""Evolve asked for outside ultra."""
def __init__(self) -> None:
super().__init__("memory_mode_not_ultra")
class EvolveRunReceipt(BaseModel):
"""One evolve run. Counts and codes; `results` only while in memory."""
model_config = ConfigDict(extra="forbid")
run_id: str
user_id: str
agent_id: str
max_batches: int | None = None
trigger: Trigger = "manual"
state: RunState = "running"
started_at: datetime
finished_at: datetime | None = None
rounds: int = 0
tasks_processed: int = 0
batch_failures: int = 0
cycle_status: str | None = None
should_continue: bool | None = None
continuation_reason: str | None = None
remaining_clusters: int | None = None
remaining_actionable_clusters: int | None = None
remaining_fresh_facts: int | None = None
#: True when the round cap stopped a run the SDK would have continued.
capped: bool = False
elapsed_ms: float = 0.0
error_code: str | None = None
failure_kind: str | None = None
#: The last round's SDK `request_id`, for matching SDK logs.
request_id: str | None = None
#: The last round's SDK `results` and `cross_domain_sweeper`. Kept only
#: in this receipt, never written to disk or diagnostics.
results: Any = None
cross_domain_sweeper: Any = None
def summary(self) -> dict[str, object]:
"""What is persisted and shown in status: no SDK result bodies."""
return self.model_dump(mode="json", exclude={"results", "cross_domain_sweeper"})
class UserEvolveState(BaseModel):
model_config = ConfigDict(extra="forbid")
last_run_at: datetime | None = None
last_result: dict[str, object] | None = None
last_error: str | None = None
class GraphBuildUser(BaseModel):
model_config = ConfigDict(extra="forbid")
state: Literal["pending", "running", "succeeded", "failed"] = "pending"
memories_scanned: int | None = None
nodes: int | None = None
error_code: str | None = None
class GraphBuildState(BaseModel):
"""The one-time backfill of existing facts into the graph after → ultra."""
model_config = ConfigDict(extra="forbid")
state: Literal["idle", "running", "succeeded", "failed"] = "idle"
started_at: datetime | None = None
finished_at: datetime | None = None
users: dict[str, GraphBuildUser] = Field(default_factory=dict)
class EvolveStateDocument(BaseModel):
model_config = ConfigDict(extra="forbid")
schema_version: int = Field(default=1, ge=1, le=1)
users: dict[str, UserEvolveState] = Field(default_factory=dict)
#: When the schedule first ran in ultra: the "last run" of a user that
#: never ran, so enabling ultra at noon does not evolve everyone at once.
schedule_anchor: datetime | None = None
graph_build: GraphBuildState = Field(default_factory=GraphBuildState)
class EvolveStateStore:
"""The only writer of `<data_dir>/evolve-state.json`."""
def __init__(self, data_dir: Path) -> None:
self.path = data_dir / "evolve-state.json"
self._lock = threading.RLock()
self._document = self._read()
def document(self) -> EvolveStateDocument:
with self._lock:
return self._document.model_copy(deep=True)
def user(self, user_id: str) -> UserEvolveState:
with self._lock:
return self._document.users.get(user_id, UserEvolveState()).model_copy()
def record_run(self, receipt: EvolveRunReceipt) -> None:
with self._lock:
users = dict(self._document.users)
users[receipt.user_id] = UserEvolveState(
last_run_at=receipt.started_at,
last_result=receipt.summary(),
last_error=receipt.error_code,
)
self._save(self._document.model_copy(update={"users": users}))
def anchor(self, now: datetime) -> datetime:
"""The schedule anchor, set to `now` the first time it is asked."""
with self._lock:
if self._document.schedule_anchor is None:
self._save(self._document.model_copy(update={"schedule_anchor": now}))
assert self._document.schedule_anchor is not None
return self._document.schedule_anchor
def graph_build(self) -> GraphBuildState:
with self._lock:
return self._document.graph_build.model_copy(deep=True)
def set_graph_build(self, state: GraphBuildState) -> None:
with self._lock:
self._save(self._document.model_copy(update={"graph_build": state}))
def _save(self, document: EvolveStateDocument) -> None:
payload = json.dumps(document.model_dump(mode="json"), indent=2, sort_keys=True) + "\n"
self.path.parent.mkdir(parents=True, exist_ok=True)
temporary: Path | None = None
try:
with NamedTemporaryFile(
mode="w",
encoding="utf-8",
dir=self.path.parent,
prefix=".evolve-state.",
suffix=".tmp",
delete=False,
) as stream:
temporary = Path(stream.name)
stream.write(payload)
stream.flush()
os.fsync(stream.fileno())
os.chmod(temporary, 0o600)
os.replace(temporary, self.path)
except OSError:
if temporary is not None:
temporary.unlink(missing_ok=True)
# Status is best-effort: a run must not fail because its record did.
logger.warning("evolve_state_write_failed")
self._document = document
def _read(self) -> EvolveStateDocument:
try:
return EvolveStateDocument.model_validate_json(self.path.read_text(encoding="utf-8"))
except FileNotFoundError:
return EvolveStateDocument()
except (OSError, ValueError):
# A damaged status file only loses history; start a fresh one.
logger.warning("evolve_state_unreadable")
return EvolveStateDocument()
class EvolveCoordinator(Protocol):
"""The write coordinator surface evolve and the graph backfill use."""
def exclusive_for_evolve(self, user_id: str) -> Any: ...
def use_client(self) -> Any: ...
class EvolveService:
"""Start, serialize and record evolve runs."""
def __init__(
self,
*,
coordinator: EvolveCoordinator,
state: EvolveStateStore,
memory_mode: Callable[[], str],
max_rounds: int = EVOLVE_MAX_ROUNDS,
now: Callable[[], datetime] = lambda: datetime.now(UTC),
) -> None:
self.coordinator = coordinator
self.state = state
self.memory_mode = memory_mode
self.max_rounds = max_rounds
self._now = now
self._runs: dict[str, EvolveRunReceipt] = {}
self._tasks: dict[str, asyncio.Task[None]] = {}
self._active: dict[str, str] = {}
def start(
self,
user_id: str,
*,
agent_id: str | None = None,
max_batches: int | None = None,
trigger: Trigger = "manual",
) -> EvolveRunReceipt:
"""Start a run, or return the one already running for this user."""
if self.memory_mode() != "ultra":
raise EvolveModeError()
running = self._active.get(user_id)
if running is not None:
return self._runs[running]
receipt = EvolveRunReceipt(
run_id=str(uuid4()),
user_id=user_id,
agent_id=agent_id or PRODUCT_AGENT_ID,
max_batches=max_batches,
trigger=trigger,
started_at=self._now(),
)
self._remember(receipt)
self._active[user_id] = receipt.run_id
self._tasks[receipt.run_id] = asyncio.create_task(
self._run(receipt.run_id), name="one-memory-evolve"
)
return receipt
def get(self, run_id: str) -> EvolveRunReceipt | None:
return self._runs.get(run_id)
def running(self) -> tuple[EvolveRunReceipt, ...]:
return tuple(self._runs[run_id] for run_id in self._active.values())
async def wait(self, run_id: str) -> EvolveRunReceipt | None:
task = self._tasks.get(run_id)
if task is not None:
await asyncio.shield(task)
return self._runs.get(run_id)
async def shutdown(self) -> None:
tasks = tuple(self._tasks.values())
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
self._tasks.clear()
def _remember(self, receipt: EvolveRunReceipt) -> None:
self._runs[receipt.run_id] = receipt
while len(self._runs) > _KEPT_RUNS:
oldest = next(iter(self._runs))
if oldest in self._active.values():
break
del self._runs[oldest]
async def _run(self, run_id: str) -> None:
receipt = self._runs[run_id]
diagnostic_event(
level="info",
component="memory.evolve",
event="memory_evolve.started",
outcome="started",
details={"trigger": receipt.trigger, "max_batches": receipt.max_batches},
)
try:
async with (
self.coordinator.exclusive_for_evolve(receipt.user_id),
self.coordinator.use_client() as client,
):
while receipt.rounds < self.max_rounds:
result = await asyncio.to_thread(
client.evolve,
receipt.user_id,
agent_id=receipt.agent_id,
max_batches=receipt.max_batches,
)
receipt = _folded(receipt, result)
self._runs[run_id] = receipt
if receipt.state == "failed" or not receipt.should_continue:
break
else:
receipt = receipt.model_copy(update={"capped": True})
if receipt.state == "running":
receipt = receipt.model_copy(update={"state": "succeeded"})
except asyncio.CancelledError:
receipt = receipt.model_copy(
update={"state": "failed", "error_code": "memory_evolve_cancelled"}
)
raise
except Exception as error: # noqa: BLE001 - recorded as a stable code
code = safe_error_code(error)
receipt = receipt.model_copy(
update={
"state": "failed",
"error_code": _stable(code, "hy_memory_evolve_failed"),
"failure_kind": type(error).__name__,
}
)
finally:
receipt = receipt.model_copy(update={"finished_at": self._now()})
self._runs[run_id] = receipt
self._active.pop(receipt.user_id, None)
self._tasks.pop(run_id, None)
await asyncio.to_thread(self.state.record_run, receipt)
diagnostic_event(
level="info" if receipt.state == "succeeded" else "warning",
component="memory.evolve",
event="memory_evolve.finished",
outcome="succeeded" if receipt.state == "succeeded" else "failed",
details={
"trigger": receipt.trigger,
"rounds": receipt.rounds,
"tasks_processed": receipt.tasks_processed,
"batch_failures": receipt.batch_failures,
"capped": receipt.capped,
"elapsed_ms": round(receipt.elapsed_ms, 1),
"error_code": receipt.error_code,
},
)
def _folded(receipt: EvolveRunReceipt, result: object) -> EvolveRunReceipt:
"""One more evolve round's result added to the run."""
if not isinstance(result, Mapping):
return receipt.model_copy(
update={"state": "failed", "error_code": "hy_memory_evolve_shape_unknown"}
)
failures = result.get("batch_failures")
update: dict[str, object] = {
"rounds": receipt.rounds + 1,
"tasks_processed": receipt.tasks_processed + _int(result.get("tasks_processed")),
"batch_failures": receipt.batch_failures
+ (len(failures) if isinstance(failures, list | tuple) else _int(failures)),
"elapsed_ms": receipt.elapsed_ms + _float(result.get("elapsed_ms")),
"should_continue": result.get("should_continue") is True,
"cycle_status": _text(result.get("cycle_status")),
"continuation_reason": _text(result.get("continuation_reason")),
"request_id": _text(result.get("request_id")),
"results": result.get("results"),
"cross_domain_sweeper": result.get("cross_domain_sweeper"),
}
for name in _COUNTS:
value = result.get(name)
update[name] = value if isinstance(value, int) and not isinstance(value, bool) else None
if result.get("success") is not True:
code = result.get("error_code")
kind = result.get("failure_kind")
update.update(
state="failed",
should_continue=False,
error_code=(
code
if isinstance(code, str) and _STABLE_CODE.fullmatch(code)
else "hy_memory_evolve_rejected"
),
failure_kind=kind if isinstance(kind, str) and _STABLE_CODE.fullmatch(kind) else None,
)
return receipt.model_copy(update=update)
def _stable(code: str, fallback: str) -> str:
return code if _STABLE_CODE.fullmatch(code) else fallback
def _int(value: object) -> int:
return value if isinstance(value, int) and not isinstance(value, bool) else 0
def _float(value: object) -> float:
if isinstance(value, int | float) and not isinstance(value, bool):
return float(value)
return 0.0
def _text(value: object) -> str | None:
return value[:128] if isinstance(value, str) else None
def evolve_due(
now: datetime,
schedule: EvolveSchedule,
last_run_at: datetime | None,
anchor: datetime,
) -> bool:
"""Whether a user's nightly evolve is due at `now` (aware, local time).
Due once the day's slot (`schedule.time`, local) has passed and the last
run (or, for a user who never ran, the anchor) is older than that slot.
A last run more than 24 hours old is due at once as well: a machine asleep
at the slot catches up when it wakes, even before the next day's slot.
"""
if not schedule.enabled:
return False
local = now.astimezone()
hour, minute = (int(part) for part in schedule.time.split(":"))
slot = local.replace(hour=hour, minute=minute, second=0, microsecond=0)
reference = last_run_at or anchor
if local >= slot and reference < slot:
return True
return last_run_at is not None and local - last_run_at > timedelta(hours=24)
class EvolveScheduler:
"""Run evolve nightly for every user that has memories (ultra only)."""
def __init__(
self,
*,
service: EvolveService,
schedule: Callable[[], EvolveSchedule],
users_with_memories: Callable[[], Awaitable[Iterable[str]]],
now: Callable[[], datetime] = lambda: datetime.now().astimezone(),
check_seconds: float = SCHEDULE_CHECK_SECONDS,
) -> None:
self.service = service
self.schedule = schedule
self.users_with_memories = users_with_memories
self._now = now
self.check_seconds = check_seconds
self._task: asyncio.Task[None] | None = None
def start(self) -> None:
if self._task is None or self._task.done():
self._task = asyncio.create_task(self._loop(), name="one-memory-evolve-schedule")
async def tick(self) -> tuple[str, ...]:
"""Run every due user once, one after another; the users run."""
schedule = self.schedule()
if not schedule.enabled or self.service.memory_mode() != "ultra":
return ()
now = self._now()
anchor = await asyncio.to_thread(self.service.state.anchor, now)
ran: list[str] = []
for user_id in await self.users_with_memories():
last = self.service.state.user(user_id).last_run_at
if not evolve_due(now, schedule, last, anchor):
continue
try:
receipt = self.service.start(user_id, trigger="scheduled")
except EvolveModeError:
break
await self.service.wait(receipt.run_id)
ran.append(user_id)
if ran:
diagnostic_event(
level="info",
component="memory.evolve",
event="memory_evolve.schedule_ran",
outcome="succeeded",
details={"users": len(ran)},
)
return tuple(ran)
async def _loop(self) -> None:
while True:
try:
await self.tick()
except asyncio.CancelledError:
raise
except Exception as error: # noqa: BLE001 - the loop must survive
logger.warning("evolve_schedule_tick_failed error_type=%s", type(error).__name__)
await asyncio.sleep(self.check_seconds)
async def shutdown(self) -> None:
if self._task is None:
return
self._task.cancel()
await asyncio.gather(self._task, return_exceptions=True)
self._task = None
class MemoryGraphBackfill:
"""After a switch to ultra, project each user's existing facts into the graph once."""
def __init__(
self,
*,
coordinator: EvolveCoordinator,
state: EvolveStateStore,
now: Callable[[], datetime] = lambda: datetime.now(UTC),
) -> None:
self.coordinator = coordinator
self.state = state
self._now = now
self._task: asyncio.Task[None] | None = None
@property
def running(self) -> bool:
return self._task is not None and not self._task.done()
def start(self, user_ids: Iterable[str]) -> GraphBuildState:
if self.running:
return self.state.graph_build()
users = tuple(dict.fromkeys(user_ids))
state = GraphBuildState(
state="running",
started_at=self._now(),
users={user_id: GraphBuildUser() for user_id in users},
)
self.state.set_graph_build(state)
self._task = asyncio.create_task(self._run(users), name="one-memory-graph-backfill")
return state
async def wait(self) -> None:
if self._task is not None:
await asyncio.shield(self._task)
async def _run(self, users: tuple[str, ...]) -> None:
state = self.state.graph_build()
for user_id in users:
state.users[user_id] = GraphBuildUser(state="running")
await asyncio.to_thread(self.state.set_graph_build, state.model_copy(deep=True))
try:
async with (
self.coordinator.exclusive_for_evolve(user_id),
self.coordinator.use_client() as client,
):
result = await asyncio.to_thread(
client.build_memory_graph,
user_id,
agent_id=None,
limit=10000,
clear_first=False,
)
ok = isinstance(result, Mapping) and result.get("success") is True
state.users[user_id] = GraphBuildUser(
state="succeeded" if ok else "failed",
memories_scanned=_int(result.get("memories_scanned"))
if isinstance(result, Mapping)
else None,
nodes=_int(result.get("nodes")) if isinstance(result, Mapping) else None,
error_code=None if ok else "hy_memory_graph_build_rejected",
)
except asyncio.CancelledError:
raise
except Exception as error: # noqa: BLE001 - recorded as a stable code
code = safe_error_code(error)
state.users[user_id] = GraphBuildUser(
state="failed",
error_code=_stable(code, "hy_memory_graph_build_failed"),
)
failed = any(user.state == "failed" for user in state.users.values())
state.state = "failed" if failed else "succeeded"
state.finished_at = self._now()
await asyncio.to_thread(self.state.set_graph_build, state)
diagnostic_event(
level="warning" if failed else "info",
component="memory.evolve",
event="memory_graph_backfill.finished",
outcome="failed" if failed else "succeeded",
details={
"users": len(state.users),
"failed_users": sum(user.state == "failed" for user in state.users.values()),
},
)
async def shutdown(self) -> None:
if self._task is None:
return
self._task.cancel()
await asyncio.gather(self._task, return_exceptions=True)
self._task = None
@@ -0,0 +1,262 @@
"""SDK retrieval and extraction features a person may turn on or off.
MemHub hard-codes none of these. Each is one `MemoryConfig` field that the SDK
reads from its config object (never from `os.environ` at call time), so
`hy_sdk_runtime.build_sdk_config` sets it per client and a client's features
are fixed for its life, like its mode. An unset feature keeps the SDK default,
so with nothing set the SDK config is byte for byte what it was before this
setting existed.
The setting is one global document in `config.json` (`memory_features`),
beside `memory_mode`; management stays global, not per user. A change
rebuilds the SDK handles the way a mode change does (#218 use count).
What is deliberately left out, and why, is in
docs/contracts/memory-features-v1.md. In short: flags the SDK reads straight
from the process environment (the System 2 schema dedup/compaction lane, the
balanced-catalog and specific-supplement lanes, the hybrid reader's
`MEMORY_HYBRID_V2_*` knobs) cannot be scoped to one client; flags MemHub never
exercises (query rewrite needs `queries=[...]`, session narratives need
`finalize_session`); flags that are dangerous or internal (graph write, local
cross-encoder download, access write-back); and everything MemHub already
controls elsewhere (pipeline log, trace, memory operations, telemetry, mode).
No SDK import here: only `hy_sdk_runtime` and `hy_sdk_migration` import it.
`config_state` imports this module, so it imports nothing from there.
"""
from __future__ import annotations
from collections.abc import Mapping
from dataclasses import dataclass
from typing import Literal
from pydantic import BaseModel, ConfigDict, StrictBool
MemoryFeatureName = Literal[
"graph_rerank",
"same_series_timeline",
"episode_recall",
"session_episodes",
"episode_summary",
"intention",
"few_shot",
]
#: Read features change search; write features change what extraction stores.
FeatureSide = Literal["read", "write"]
#: none: no extra call. extra_query: one more store query per search.
#: llm_call: extra generation-model calls. prompt_tokens: no extra call, but
#: a longer extraction prompt.
FeatureCost = Literal["none", "extra_query", "llm_call", "prompt_tokens"]
#: Why an enabled feature does nothing right now (`active: false`).
FeatureReason = Literal["disabled", "requires_pro", "requires_ultra", "requires_graph_rerank"]
#: A client's features as a hashable key: sorted (name, value) pairs, set ones only.
type FeatureOverrides = tuple[tuple[str, bool], ...]
_MODE_RANK: dict[str, int] = {"lite": 0, "pro": 1, "ultra": 2}
@dataclass(frozen=True)
class FeatureSpec:
name: MemoryFeatureName
#: `MemoryConfig` section and field the value is written to.
section: Literal["graph_store", "recall", "llm"]
field: str
#: The SDK's own env var for the same field; only its default comes from it.
env: str
side: FeatureSide
#: Lowest mode the feature acts in.
requires_mode: Literal["pro", "ultra"]
#: Another feature that must be active for this one to act.
requires_feature: MemoryFeatureName | None
cost: FeatureCost
description: str
@property
def sdk_field(self) -> str:
return f"{self.section}.{self.field}"
FEATURES: tuple[FeatureSpec, ...] = (
FeatureSpec(
name="graph_rerank",
section="graph_store",
field="graph_rerank_enabled",
env="MEMORY_GRAPH_RERANK_ENABLED",
side="read",
requires_mode="ultra",
requires_feature=None,
cost="extra_query",
description=(
"Usage A: propagate support over knowledge-graph edges inside the "
"recall pool, so memories linked to strong hits rank higher. Needs "
"the graph store (ultra). No model call; one graph query per search."
),
),
FeatureSpec(
name="same_series_timeline",
section="recall",
field="same_series_timeline_enabled",
env="MEMORY_SAME_SERIES_TIMELINE_ENABLED",
side="read",
requires_mode="ultra",
requires_feature="graph_rerank",
cost="none",
description=(
"Keep memories joined by a same-series graph edge together, in time "
"order, in the results. Uses the edges graph rerank reads, so it "
"needs graph_rerank (ultra). No model call."
),
),
FeatureSpec(
name="episode_recall",
section="recall",
field="episode_recall_enabled",
env="MEMORY_EPISODE_RECALL_ENABLED",
side="read",
requires_mode="pro",
requires_feature=None,
cost="extra_query",
description=(
"Usage B: pull in memories from the same episode as a hit and score "
"them on tag-keyword overlap, content overlap and time affinity. No "
"graph needed (pro or ultra). No model call; one store query per "
"search. Helps only memories that carry an episode_id."
),
),
FeatureSpec(
name="session_episodes",
section="recall",
field="session_episode_enabled",
env="MEMORY_SESSION_EPISODE_ENABLED",
side="write",
requires_mode="pro",
requires_feature=None,
cost="none",
description=(
"Episode write side: memories extracted from one conversation share "
"one stable episode_id, instead of the default rule (a new episode "
"after 30 minutes idle or 50 memories). No model call."
),
),
FeatureSpec(
name="episode_summary",
section="recall",
field="episode_summary_enabled",
env="MEMORY_EPISODE_SUMMARY_ENABLED",
side="write",
requires_mode="pro",
requires_feature=None,
cost="llm_call",
description=(
"When an episode closes (the idle-gap rule), summarize it once into "
"an episode memory. One model call per closed episode of 3 or more "
"memories."
),
),
FeatureSpec(
name="intention",
section="llm",
field="enable_intention",
env="MEMORY_ENABLE_INTENTION",
side="write",
requires_mode="pro",
requires_feature=None,
cost="none",
description=(
"Extract the user's stated plans and intentions (L7) along with "
"facts. On by default; off drops them from the extraction prompt. "
"No extra call."
),
),
FeatureSpec(
name="few_shot",
section="llm",
field="few_shot_enabled",
env="MEMORY_FEW_SHOT_ENABLED",
side="write",
requires_mode="pro",
requires_feature=None,
cost="prompt_tokens",
description=(
"Append worked examples to the extraction and reconcile prompts. "
"Can help a weaker model; no extra call, but every extraction "
"sends more input tokens."
),
),
)
FEATURE_NAMES: tuple[MemoryFeatureName, ...] = tuple(spec.name for spec in FEATURES)
FEATURES_BY_NAME: dict[str, FeatureSpec] = {spec.name: spec for spec in FEATURES}
class MemoryFeatures(BaseModel):
"""`config.json` `memory_features`, and the `PUT` body.
Every key is optional; absent (or null) means the SDK default. In a `PUT`
an absent key is left as it is and null resets it. Unknown keys and
non-boolean values are refused (422).
"""
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
graph_rerank: StrictBool | None = None
same_series_timeline: StrictBool | None = None
episode_recall: StrictBool | None = None
session_episodes: StrictBool | None = None
episode_summary: StrictBool | None = None
intention: StrictBool | None = None
few_shot: StrictBool | None = None
def overrides(self) -> FeatureOverrides:
"""The features a person set, as the client key."""
return feature_overrides(
{name: value for name, value in self.model_dump().items() if value is not None}
)
def merged(self, update: MemoryFeatures) -> MemoryFeatures:
"""This document with the keys `update` names: a value sets, null resets."""
values = self.model_dump()
for name in update.model_fields_set:
values[name] = getattr(update, name)
return MemoryFeatures(**{key: value for key, value in values.items() if value is not None})
def feature_overrides(values: Mapping[str, bool]) -> FeatureOverrides:
unknown = set(values) - set(FEATURES_BY_NAME)
if unknown:
raise ValueError("unknown_memory_feature")
return tuple(sorted((name, bool(value)) for name, value in values.items()))
def effective_features(
overrides: FeatureOverrides, defaults: Mapping[str, bool]
) -> dict[str, bool]:
"""Every feature's value: what a person set, else the SDK default."""
values: dict[str, bool] = {spec.name: bool(defaults.get(spec.name, False)) for spec in FEATURES}
values.update(dict(overrides))
return values
def feature_activity(name: str, effective: Mapping[str, bool], mode: str) -> FeatureReason | None:
"""None when the feature acts in this mode; else why it does not."""
spec = FEATURES_BY_NAME[name]
if not effective.get(name, False):
return "disabled"
if _MODE_RANK.get(mode, 0) < _MODE_RANK[spec.requires_mode]:
return "requires_ultra" if spec.requires_mode == "ultra" else "requires_pro"
if spec.requires_feature is not None and (
feature_activity(spec.requires_feature, effective, mode) is not None
):
return "requires_graph_rerank"
return None
def non_default_features(
overrides: FeatureOverrides, defaults: Mapping[str, bool]
) -> dict[str, bool]:
"""The features whose value differs from the SDK default, for status."""
return {
name: value
for name, value in effective_features(overrides, defaults).items()
if value != bool(defaults.get(name, False))
}
@@ -13,9 +13,11 @@ import threading
from collections.abc import Mapping
from pathlib import Path
from tempfile import NamedTemporaryFile
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, StrictBool, field_validator
from one_memory.memory_features import MemoryFeatures
from one_memory.memory_queue import ExtractionWindowPreset
_SOURCE_ID = re.compile(r"^[a-z0-9][a-z0-9-]{0,127}$")
@@ -27,6 +29,31 @@ class SourceConfig(BaseModel):
sync_enabled: StrictBool = False
#: The SDK pipeline mode (hy-memory `HyMemoryClient(mode=...)`):
#: lite keeps raw conversations only, with no model call; pro extracts
#: memories (System 1); ultra adds the knowledge graph and evolve (System 2).
MemoryMode = Literal["lite", "pro", "ultra"]
MEMORY_MODES: tuple[MemoryMode, ...] = ("lite", "pro", "ultra")
_SCHEDULE_TIME = re.compile(r"^(?:[01][0-9]|2[0-3]):[0-5][0-9]$")
class EvolveSchedule(BaseModel):
"""When ultra runs evolve for every user: once a day, local time."""
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
enabled: StrictBool = True
#: `HH:MM`, 24-hour, the machine's local time.
time: str = "03:00"
@field_validator("time")
@classmethod
def valid_time(cls, value: str) -> str:
if _SCHEDULE_TIME.fullmatch(value) is None:
raise ValueError("invalid_evolve_schedule_time")
return value
class ConfigDocument(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
@@ -39,6 +66,13 @@ class ConfigDocument(BaseModel):
#: needs no re-entry of a key.
sedimentation_enabled: StrictBool = True
extraction_window_preset: ExtractionWindowPreset = ExtractionWindowPreset.THREE_MONTHS
#: The SDK mode every client is built with. `ONE_MEMORY_GRAPH_STORE` (dev)
#: still forces ultra over it; see `hy_graph_admin.effective_memory_mode`.
memory_mode: MemoryMode = "pro"
evolve_schedule: EvolveSchedule = Field(default_factory=EvolveSchedule)
#: SDK retrieval and extraction features a person set; an unset one keeps
#: the SDK default. See `one_memory.memory_features`.
memory_features: MemoryFeatures = Field(default_factory=MemoryFeatures)
@field_validator("sources")
@classmethod
@@ -61,6 +95,9 @@ class ConfigFileError(RuntimeError):
pass
_WRITTEN_WHEN_SET = frozenset({"memory_mode", "evolve_schedule", "memory_features"})
class ConfigStore:
"""The only writer of ``<data_dir>/config.json`` inside Core."""
@@ -107,6 +144,45 @@ class ConfigStore:
self._document = updated
return updated
def memory_mode(self) -> MemoryMode:
with self._lock:
return self._document.memory_mode
def set_memory_mode(self, mode: MemoryMode) -> ConfigDocument:
if mode not in MEMORY_MODES:
raise ValueError("invalid_memory_mode")
with self._lock:
updated = self._document.model_copy(update={"memory_mode": mode})
self._write(updated)
self._document = updated
return updated
def evolve_schedule(self) -> EvolveSchedule:
with self._lock:
return self._document.evolve_schedule
def set_evolve_schedule(self, schedule: EvolveSchedule) -> ConfigDocument:
if not isinstance(schedule, EvolveSchedule):
raise ValueError("invalid_evolve_schedule")
with self._lock:
updated = self._document.model_copy(update={"evolve_schedule": schedule})
self._write(updated)
self._document = updated
return updated
def memory_features(self) -> MemoryFeatures:
with self._lock:
return self._document.memory_features
def set_memory_features(self, features: MemoryFeatures) -> ConfigDocument:
if not isinstance(features, MemoryFeatures):
raise ValueError("invalid_memory_features")
with self._lock:
updated = self._document.model_copy(update={"memory_features": features})
self._write(updated)
self._document = updated
return updated
def migrate_extraction_window_preset(self, preset: ExtractionWindowPreset) -> ConfigDocument:
"""Adopt the legacy SQL value only when config.json has no value."""
if not isinstance(preset, ExtractionWindowPreset):
@@ -152,9 +228,16 @@ class ConfigStore:
raise ConfigFileError("config_invalid") from error
def _write(self, document: ConfigDocument) -> None:
# Settings added after 1.0.5 are written only once a person chose
# them, so an untouched install's file stays what older builds read.
unset = {name for name in _WRITTEN_WHEN_SET if name not in document.model_fields_set}
values = document.model_dump(mode="json", exclude=unset)
if "memory_features" in values:
# Only the features a person set; an absent one is the SDK default.
values["memory_features"] = document.memory_features.model_dump(exclude_none=True)
payload = (
json.dumps(
document.model_dump(mode="json"),
values,
ensure_ascii=False,
indent=2,
sort_keys=True,
@@ -75,7 +75,11 @@ def batch_id(*, internal_session_id: str, first_round: int, last_round: int) ->
INTERNAL_SESSION_ID_MAX = 128
def internal_session_id_for(conversation_id: str, source_id: str) -> str:
def internal_session_id_for(
conversation_id: str,
source_id: str,
user_id: str = "local",
) -> str:
"""Normalise a conversation identity to the durable ``internal_session_id``.
This is the single derivation shared by every Evidence-first intake path.
@@ -86,7 +90,17 @@ def internal_session_id_for(conversation_id: str, source_id: str) -> str:
per-session serialisation split). Only a value that the contract would have
rejected before it ever reached the queue is replaced by the versioned
session id, which is deterministic for a given (source, conversation).
Another user's conversation is always derived with the user in it: two
Hermes profiles can hold the same session id, and sharing a session key
would serialise and batch one person's turns behind the other's. `local`
keeps the identity it always had.
"""
if user_id != "local":
return session_id(
harness_id=source_id,
external_session_id=f"user:{user_id}|{conversation_id}",
)
if len(conversation_id) <= INTERNAL_SESSION_ID_MAX:
return conversation_id
derived = session_id(harness_id=source_id, external_session_id=conversation_id)
@@ -52,7 +52,9 @@ from one_memory.memory_tasks import (
from one_memory.memory_tasks import (
POLICY_STOPPED_CODES as _TASK_POLICY_STOPPED_CODES,
)
from one_memory.sdk_migration_log import error_chain_text
from one_memory.service_levels import ServiceLevel, service_level_context
from one_memory.user_scope import DEFAULT_USER_ID, lane_key, other_user_lane_prefix
logger = logging.getLogger(__name__)
@@ -241,14 +243,16 @@ PERMISSION_REQUIRED_CODES = frozenset(
)
#: Model-side codes the SDK write coordinator raises before it can lease a
#: client (no active profile, missing credential, secure storage down). They prove
#: nothing was sent, so the write parks instead of being lost. Transport codes
#: that can arrive after a request was in flight stay `unknown`.
#: client (no active profile, missing credential, secure storage down, or the
#: client failed to construct). They prove nothing was sent, so the write
#: parks instead of being lost. Transport codes that can arrive after a
#: request was in flight stay `unknown`.
MODEL_NOT_READY_CODES = frozenset(
{
"model_profile_not_active",
"model_secret_unavailable",
"secure_storage_unavailable",
"hy_memory_client_unavailable",
}
)
@@ -292,6 +296,23 @@ _VALID_JOB_STATES = frozenset(
#: A row in one of these states can never be moved back to a live one.
_TERMINAL_WRITE_STATES = frozenset({COMPLETED, FAILED, DEAD, CANCELLED})
#: The legacy task rows a write job can be linked to. The SDK cutover drops
#: them (`target_schema.LEGACY_TABLES_IN_DROP_ORDER`, inputs before tasks), so
#: the job-only paths must not assume any of them.
_LEGACY_TASK_LINK_TABLES = ("memory_tasks", "memory_task_inputs", "memory_write_job_evidence")
def _legacy_task_links_present(connection: sqlite3.Connection) -> bool:
present = {
str(row[0])
for row in connection.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name IN (?,?,?)",
_LEGACY_TASK_LINK_TABLES,
)
}
return present == set(_LEGACY_TASK_LINK_TABLES)
#: Columns a current schema declares but an older `memory_write_jobs` may lack.
#: `CREATE TABLE IF NOT EXISTS` is a no-op for an existing table, so without
#: this an upgraded install would keep an older queue that every claim/lease
@@ -325,6 +346,27 @@ EVENT_TIME_WINDOW = timedelta(minutes=10)
DIRECT_WRITE_SOURCES = frozenset({"one-memory-manual"})
def request_user_id(request: WriteRequest) -> str:
"""Whose write this is: its frozen input's user, `local` for id-only rows."""
users = {record.user_id for record in request.input_records}
if len(users) > 1:
raise ValueError("durable_write_multiple_users")
return next(iter(users), DEFAULT_USER_ID)
def request_source_key(request: WriteRequest) -> str:
"""The FIFO/claim key of a write: its source, per user.
`local` keeps the bare source id it always had. Another user's Hermes
lane is `hermes@user`, so one person's running job never holds the other
person's queue and the claim gate answers for that user's grant.
"""
return lane_key(
request_user_id(request),
request.source_id or request.internal_session_id,
)
@dataclass(frozen=True)
class DurableWriteRecord:
job_id: str
@@ -842,7 +884,7 @@ class DurableWriteQueue:
job_id,
queue_seq,
request.internal_session_id,
request.source_id or request.internal_session_id,
request_source_key(request),
request.batch_identity.batch_id,
key,
payload,
@@ -909,7 +951,9 @@ class DurableWriteQueue:
f"{hashlib.sha256(file_key.encode()).hexdigest()}:"
f"{hashlib.sha256(record.conversation_id.encode()).hexdigest()}"
)
session_key = internal_session_id_for(record.conversation_id, record.source_id)
session_key = internal_session_id_for(
record.conversation_id, record.source_id, record.user_id
)
written = connection.execute(
"SELECT 1 FROM memory_write_jobs AS job, json_each(job.evidence_ids_json) AS ref "
"WHERE job.session_key=? AND job.state!='assembling' AND ref.value=? LIMIT 1",
@@ -978,7 +1022,7 @@ class DurableWriteQueue:
str(uuid4()),
queue_seq,
session_key,
record.source_id,
lane_key(record.user_id, record.source_id),
assembly_key,
assembly_key,
payload_json,
@@ -1014,10 +1058,13 @@ class DurableWriteQueue:
`flush` (history at end of file) completes the last turn and writes a
short batch; without it (live polling) those stay in the row.
"""
# Two users' roots can hold a file with the same relative key (each
# Hermes profile has its own `state.db`): only this user's rows.
assembly_prefix = f"assembly:{user_id}:{source_id}:"
rows = connection.execute(
"SELECT * FROM memory_write_jobs WHERE state='assembling' "
"AND source_key=? ORDER BY queue_seq",
(source_id,),
"AND source_key=? AND substr(idempotency_key,1,?)=? ORDER BY queue_seq",
(lane_key(user_id, source_id), len(assembly_prefix), assembly_prefix),
).fetchall()
finalized: list[str] = []
for row in rows:
@@ -1140,7 +1187,7 @@ class DurableWriteQueue:
Same records, same key: a re-read returns the job already written.
"""
first = records[0]
session_id = internal_session_id_for(first.conversation_id, first.source_id)
session_id = internal_session_id_for(first.conversation_id, first.source_id, first.user_id)
material = ":".join(f"{item.evidence_id}:{item.content_hash}" for item in records)
request_id = f"turn:v2:{hashlib.sha256(material.encode()).hexdigest()}"
existing = connection.execute(
@@ -1185,7 +1232,7 @@ class DurableWriteQueue:
job_id,
queue_seq,
session_id,
first.source_id,
lane_key(first.user_id, first.source_id),
request.batch_identity.batch_id,
request_id,
json.dumps(
@@ -1982,21 +2029,24 @@ class DurableWriteQueue:
or str(row["last_error"]) in NON_RETRYABLE_POST_WRITE_CODES
):
return 0
connection.execute(
"""
UPDATE memory_tasks
SET status='pending', runner_token=NULL, attempts=0,
last_error_code='model_content_manual_retry_requested', updated_at=?
WHERE status='unknown' AND task_id IN (
SELECT inputs.task_id
FROM memory_task_inputs AS inputs, json_each(inputs.refs_json) AS ref
JOIN memory_write_job_evidence AS linked
ON linked.evidence_id=json_extract(ref.value, '$.evidence_id')
WHERE linked.job_id=?
# After the SDK cutover the legacy task tables are gone and the
# job is the only row to reset.
if not self.sdk_authority and _legacy_task_links_present(connection):
connection.execute(
"""
UPDATE memory_tasks
SET status='pending', runner_token=NULL, attempts=0,
last_error_code='model_content_manual_retry_requested', updated_at=?
WHERE status='unknown' AND task_id IN (
SELECT inputs.task_id
FROM memory_task_inputs AS inputs, json_each(inputs.refs_json) AS ref
JOIN memory_write_job_evidence AS linked
ON linked.evidence_id=json_extract(ref.value, '$.evidence_id')
WHERE linked.job_id=?
)
""",
(now, job_id),
)
""",
(now, job_id),
)
changed = connection.execute(
"""
UPDATE memory_write_jobs
@@ -2098,9 +2148,7 @@ class DurableWriteQueue:
f"WHERE state=? AND job_id IN ({placeholders})",
(PENDING, now, UNKNOWN, *job_ids),
)
if connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_tasks'"
).fetchone():
if _legacy_task_links_present(connection):
connection.execute(
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
f"last_error_code='model_content_reconciled_absent_retry', updated_at=? "
@@ -2158,9 +2206,7 @@ class DurableWriteQueue:
f"WHERE state=? AND attempts=1 AND job_id IN ({placeholders})",
(PENDING, now, UNKNOWN, *job_ids),
)
if connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_tasks'"
).fetchone():
if _legacy_task_links_present(connection):
connection.execute(
f"UPDATE memory_tasks SET status=?, runner_token=NULL, "
f"last_error_code='model_content_unknown_auto_retry', updated_at=? "
@@ -2177,6 +2223,14 @@ class DurableWriteQueue:
def resolve_unknown_from_linked_output(self, job_id: str) -> bool:
"""Close an unknown durable row when local governance output proves success."""
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
if not (
_legacy_task_links_present(connection)
and connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_task_outputs'"
).fetchone()
):
# No legacy governance output survives the SDK cutover.
return False
row = connection.execute(
"""
SELECT 1
@@ -2467,6 +2521,12 @@ async def execute_job(
source_id = request.internal_session_id
if source_ids and source_ids != {source_id}:
raise RuntimeError("durable_write_multiple_sources")
users = {getattr(record, "user_id", DEFAULT_USER_ID) for record in records}
if len(users) > 1:
raise RuntimeError("durable_write_multiple_users")
# The gates answer per user: another user's Hermes job asks about
# that user's grant, not `local`'s.
source_id = lane_key(next(iter(users), DEFAULT_USER_ID), source_id)
async def mark() -> bool:
return await in_the_background(queue.mark_invoked, job_id, owner=owner)
@@ -2503,6 +2563,8 @@ async def execute_job(
in {
"model_content_source_disabled",
"model_sedimentation_disabled",
# Evolve holds this user; the job runs once it is done.
"memory_evolve_in_progress",
}
and not hy_invoked
):
@@ -2559,9 +2621,22 @@ async def execute_job(
if code == "model_profile_not_active":
logger.debug("durable_write_job_parked job_id=%s error_code=%s", job_id, code)
elif state in PARKED_STATES:
logger.info("durable_write_job_parked job_id=%s error_code=%s", job_id, code)
logger.info(
"durable_write_job_parked job_id=%s error_code=%s chain=%s",
job_id,
code,
error_chain_text(error),
)
else:
logger.error("durable_write_job_failed job_id=%s error_code=%s", job_id, code)
# Where it failed, as `file:line:function` frames of the exception
# chain, never its message: `error_code=AttributeError` alone
# named no location.
logger.error(
"durable_write_job_failed job_id=%s error_code=%s chain=%s",
job_id,
code,
error_chain_text(error),
)
return
await in_the_background(
queue.settle,
@@ -2759,6 +2834,19 @@ class DurableWriteDrain:
source_clause = f"AND candidate.source_key IN ({placeholders}) "
source_params = tuple(sorted(enabled_sources))
direct_sources = ",".join("?" for _ in DIRECT_WRITE_SOURCES)
# Another user's hand-added memories are `one-memory-manual@user`:
# the same lane ahead of history, still serial per user.
direct_lanes = " OR ".join(
"substr(candidate.source_key,1,?)=?" for _ in DIRECT_WRITE_SOURCES
)
direct_lane_params = tuple(
value
for source in sorted(DIRECT_WRITE_SOURCES)
for value in (
len(other_user_lane_prefix(source)),
other_user_lane_prefix(source),
)
)
# Same eligibility as the window claim below (free lease, no
# running job of its source, nothing earlier in its session), but
# not bound to the window: a note must not wait for history.
@@ -2768,7 +2856,7 @@ class DurableWriteDrain:
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause
+ f"AND candidate.source_key IN ({direct_sources}) "
+ f"AND (candidate.source_key IN ({direct_sources}) OR {direct_lanes}) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
@@ -2786,6 +2874,7 @@ class DurableWriteDrain:
now.isoformat(),
*source_params,
*sorted(DIRECT_WRITE_SOURCES),
*direct_lane_params,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
@@ -19,10 +19,11 @@ does the writing. That separation is D52.
from __future__ import annotations
import hashlib
from datetime import UTC, datetime
from typing import Any, Literal, cast
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, ConfigDict, Field
from one_memory.imports.models import (
@@ -43,6 +44,24 @@ from one_memory.memory_server.durable_queue import (
)
from one_memory.memory_server.registry import WriteRegistry, WriteState
from one_memory.memory_server.submit import WriteConflictError
from one_memory.user_scope import DEFAULT_USER_ID, USER_ID_PATTERN, UserId
#: Durable keys of another user's writes start with this. A `local` request
#: id never may, so `local` cannot name another user's receipt.
_USER_KEY_PREFIX = "user:"
def durable_request_key(request_id: str, user_id: str) -> str:
"""The queue key of a caller's request id, namespaced per user.
Two users choosing the same request id must not collide: the key is the
queue's idempotency key and the SDK's `request_id`. `local` keeps its own
request id unchanged.
"""
if user_id == DEFAULT_USER_ID:
return request_id
digest = hashlib.sha256(request_id.encode()).hexdigest()
return f"{_USER_KEY_PREFIX}{user_id}:{digest}"
class MemoryWriteReceipt(BaseModel):
@@ -71,6 +90,7 @@ class SdkMemoryWriteInput(BaseModel):
messages: list[OpenAIMessage] = Field(min_length=1, max_length=200)
memory_at: datetime | None = None
priority: Literal["live", "normal"] = "normal"
user_id: UserId = DEFAULT_USER_ID
def _registry(
@@ -124,6 +144,9 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
) -> MemoryWriteReceipt:
if not any(message.role == "user" for message in payload.messages):
raise HTTPException(422, "messages_require_user_turn")
if payload.user_id == DEFAULT_USER_ID and payload.request_id.startswith(_USER_KEY_PREFIX):
raise HTTPException(422, "request_id_reserved_prefix")
request_key = durable_request_key(payload.request_id, payload.user_id)
observed_at = payload.memory_at or datetime(1970, 1, 1, tzinfo=UTC)
written_by = {
"user": WrittenBy.PERSON,
@@ -133,6 +156,7 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
records = tuple(
materialize_write_input(
EvidenceInput(
user_id=payload.user_id,
source_id=payload.source_id,
conversation_id=payload.session_id,
message_id=f"{payload.request_id}:{index}",
@@ -157,6 +181,7 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
internal_session_id = internal_session_id_for(
payload.session_id,
payload.source_id,
payload.user_id,
)
contract = WriteRequest(
internal_session_id=internal_session_id,
@@ -168,7 +193,7 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
),
evidence_ids=[record.evidence_id for record in records],
input_records=records,
idempotency_key=payload.request_id,
idempotency_key=request_key,
priority=payload.priority,
event_at=payload.memory_at,
time_quality="exact" if payload.memory_at is not None else "approximate",
@@ -178,7 +203,7 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
except (WriteConflictError, ValueError) as error:
raise HTTPException(409, str(error)) from error
return MemoryWriteReceipt(
request_id=record.request_id,
request_id=payload.request_id,
state=record.state,
created_at=record.created_at,
updated_at=record.updated_at,
@@ -186,16 +211,24 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
)
@instance.get("/{request_id}", response_model=MemoryWriteReceipt)
def status(request_id: str, request: Request) -> MemoryWriteReceipt:
def status(
request_id: str,
request: Request,
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> MemoryWriteReceipt:
# A receipt belongs to the user that submitted it. Another user's key
# is namespaced, and `local` may not name a namespaced key directly.
if user_id == DEFAULT_USER_ID and request_id.startswith(_USER_KEY_PREFIX):
raise HTTPException(404, "unknown request id")
registry = _registry(request, write=False)
try:
record = registry.get(request_id)
record = registry.get(durable_request_key(request_id, user_id))
except DurableQueueSchemaUnavailableError as error:
raise HTTPException(503, str(error)) from error
if record is None:
raise HTTPException(404, "unknown request id")
return MemoryWriteReceipt(
request_id=record.request_id,
request_id=request_id,
state=record.state,
created_at=record.created_at,
updated_at=record.updated_at,
@@ -271,7 +271,9 @@ def build_request_from_evidence(
# Keep the exact conversation id when it fits the 128-char contract;
# otherwise derive the versioned session id, so a long branch id does
# not reject the whole history import.
internal_session_id=internal_session_id_for(seed.conversation_id, seed.source_id),
internal_session_id=internal_session_id_for(
seed.conversation_id, seed.source_id, seed.user_id
),
first_round=first_round,
last_round=last_round,
evidence_ids=list(ordered),
@@ -28,6 +28,7 @@ from one_memory.memory_server.contracts import (
internal_session_id_for,
)
from one_memory.memory_server.durable_queue import DurableWriteQueue, DurableWriteRecord
from one_memory.user_scope import lane_key
WritePriority = Literal["live", "normal"]
@@ -74,7 +75,7 @@ class UnifiedWriteIntake:
if priority not in {"live", "normal"}:
raise ValueError("priority must be 'live' or 'normal'")
if self.source_writes_permitted is not None and not self.source_writes_permitted(
record.source_id
lane_key(record.user_id, record.source_id)
):
# Freeze is the product's one Harness processing decision. Refuse
# new intake so a frozen source creates no new durable write row;
@@ -106,7 +107,9 @@ class UnifiedWriteIntake:
last_round: int,
priority: WritePriority,
) -> WriteRequest:
session_id = internal_session_id_for(record.conversation_id, record.source_id)
session_id = internal_session_id_for(
record.conversation_id, record.source_id, record.user_id
)
event_at, time_quality = _job_time((record,))
request_key = (
f"entry:{record.evidence_id}"
@@ -145,14 +148,14 @@ class UnifiedWriteIntake:
):
raise ValueError("records must belong to one conversation")
if self.source_writes_permitted is not None and not self.source_writes_permitted(
first.source_id
lane_key(first.user_id, first.source_id)
):
raise SourceWriteDisabledError(first.evidence_id)
material = ":".join(
f"{record.evidence_id}:{record.revision}:{record.content_hash}" for record in records
)
request_id = f"turn:v1:{sha256(material.encode()).hexdigest()}"
session_id = internal_session_id_for(first.conversation_id, first.source_id)
session_id = internal_session_id_for(first.conversation_id, first.source_id, first.user_id)
event_at, time_quality = _job_time(records)
durable = self.queue.submit(
WriteRequest(
@@ -160,7 +160,7 @@ def pending(db: sqlite3.Connection, *, limit: int = 100) -> list[str]:
def _trigger_sql() -> list[tuple[str, str]]:
"""(table it depends on, statement) for each trigger."""
"""(table it is on, statement) for each trigger."""
return [
(
"evidence",
@@ -303,9 +303,61 @@ def install_triggers(db: sqlite3.Connection) -> None:
for table, statement in _trigger_sql():
if table not in present:
continue
if not _body_tables(statement) <= present:
# SQLite resolves a trigger body only when it fires, so a body
# naming a dropped table fails the write it rides on (#233: the
# SDK cutover drops `governed_memory_evidence`, and every grant
# write then failed). Not installed, and an old copy is removed.
cursor.execute(f"DROP TRIGGER IF EXISTS {_trigger_name(statement)}")
continue
cursor.execute(statement)
def _trigger_name(statement: str) -> str:
return statement.split("IF NOT EXISTS", 1)[1].split()[0]
def _body_tables(statement: str) -> set[str]:
"""Tables the trigger body reads or writes besides the one it is on."""
_head, _begin, body = statement.partition("BEGIN")
words = body.replace("(", " ").replace(")", " ").split()
return {
name
for keyword, name in zip(words, words[1:], strict=False)
if keyword.upper() in {"FROM", "JOIN", "INTO"}
}
def drop_stale_triggers(database_path: Path) -> tuple[str, ...]:
"""Drop this module's triggers whose body names a table that is gone.
Startup on SDK authority does not install the cache (it is read only on
legacy authority), but a home migrated by an earlier build still carries
the triggers it installed. One whose body reads a dropped table fails
every write to the table it is on, so it is dropped here. Idempotent.
"""
with main_database(database_path).write() as db, db:
cursor = db.cursor()
cursor.row_factory = sqlite3.Row
present = {
str(row["name"])
for row in cursor.execute("SELECT name FROM sqlite_master WHERE type='table'")
}
installed = {
str(row["name"])
for row in cursor.execute("SELECT name FROM sqlite_master WHERE type='trigger'")
}
dropped = tuple(
name
for _table, statement in _trigger_sql()
if (name := _trigger_name(statement)) in installed
and not _body_tables(statement) <= present
)
for name in dropped:
cursor.execute(f"DROP TRIGGER IF EXISTS {name}")
return dropped
def startup(database_path: Path) -> None:
"""Idempotent: create the derived tables and triggers. No version change.
@@ -564,7 +564,7 @@ def _embedding_request(
secret: str | None,
) -> tuple[str, str, dict[str, str], dict[str, object]]:
provider = profile.embedding_provider_kind or profile.provider_kind
base_url = (profile.embedding_base_url or profile.base_url or "").rstrip("/")
base_url = (profile.embedding_base_url or "").rstrip("/")
if provider is ProviderKind.OLLAMA:
return (
"POST",
@@ -97,7 +97,6 @@ class ModelProfileInput(BaseModel):
)
embedding_provider, resolved_embedding_base_url = resolve_embedding_endpoint(
provider_kind=self.provider_kind,
base_url=base_url,
embedding_model=embedding_model,
embedding_provider_kind=embedding_provider_kind,
embedding_base_url=embedding_base_url,
@@ -107,6 +106,12 @@ class ModelProfileInput(BaseModel):
ProviderKind.ANTHROPIC,
}:
raise ValueError("embedding provider must be OpenAI-compatible, Ollama, or local ONNX")
if (
embedding_provider is not ProviderKind.LOCAL_ONNX
and resolved_embedding_base_url is None
):
# `ModelProfileStore.create` refuses it with its own code.
return self
if (
embedding_provider is not ProviderKind.LOCAL_ONNX
and resolved_embedding_base_url is not None
@@ -308,11 +313,12 @@ class ModelProfileStore:
f"""
UPDATE {table}
SET embedding_provider_kind = provider_kind,
embedding_base_url = base_url
embedding_base_url = COALESCE(embedding_base_url, base_url)
WHERE embedding_model IS NOT NULL
AND embedding_provider_kind IS NULL
"""
)
_pin_embedding_endpoints(connection, table)
connection.execute(
"""
INSERT OR IGNORE INTO model_profile_versions (
@@ -355,11 +361,11 @@ class ModelProfileStore:
)
embedding_provider_kind, embedding_base_url = resolve_embedding_endpoint(
provider_kind=value.provider_kind,
base_url=base_url,
embedding_model=defaulted_model,
embedding_provider_kind=defaulted_provider,
embedding_base_url=defaulted_base_url,
)
require_embedding_base_url(embedding_provider_kind, embedding_base_url)
try:
with self._connect(writing=True) as connection:
connection.execute(
@@ -612,7 +618,6 @@ class ModelProfileStore:
)
embedding_provider_kind, embedding_base_url = resolve_embedding_endpoint(
provider_kind=current.provider_kind,
base_url=base_url,
embedding_model=embedding_model,
embedding_provider_kind=(
patch.embedding_provider_kind
@@ -627,6 +632,7 @@ class ModelProfileStore:
)
if embedding_provider_kind in {ProviderKind.HY_DEFAULT, ProviderKind.ANTHROPIC}:
raise ValueError("embedding provider must be OpenAI-compatible, Ollama, or local ONNX")
require_embedding_base_url(embedding_provider_kind, embedding_base_url)
if (
embedding_provider_kind is not ProviderKind.LOCAL_ONNX
and embedding_base_url is not None
@@ -1167,21 +1173,77 @@ def attach_disable_thinking(
def resolve_embedding_endpoint(
*,
provider_kind: ProviderKind,
base_url: str | None,
embedding_model: str | None,
embedding_provider_kind: ProviderKind | None,
embedding_base_url: str | None,
) -> tuple[ProviderKind | None, str | None]:
"""The embedding provider and the endpoint exactly as given.
The endpoint never follows the generation `base_url`: the collection name
hashes it, so a proxy or region edit of the generation endpoint moved
writes to a new, empty collection, and providers without embeddings were
sent embedding requests. `require_embedding_base_url` refuses a remote
embedding without one.
"""
if embedding_model is None:
return None, None
if embedding_provider_kind is ProviderKind.LOCAL_ONNX:
return ProviderKind.LOCAL_ONNX, None
if embedding_provider_kind is None:
return provider_kind, embedding_base_url or base_url
return (
embedding_provider_kind,
embedding_base_url or DEFAULT_BASE_URLS.get(embedding_provider_kind),
)
return embedding_provider_kind or provider_kind, embedding_base_url
class EmbeddingBaseUrlRequiredError(ValueError):
def __init__(self) -> None:
super().__init__("embedding_base_url_required")
def require_embedding_base_url(provider: ProviderKind | None, base_url: str | None) -> None:
"""A remote embedding must name its endpoint; the UI may offer the generation one."""
if provider not in {None, ProviderKind.LOCAL_ONNX} and not base_url:
raise EmbeddingBaseUrlRequiredError()
def pinned_embedding_base_url(
*,
provider_kind: ProviderKind | str,
base_url: str | None,
embedding_model: str | None,
embedding_provider_kind: ProviderKind | str | None,
embedding_base_url: str | None,
) -> str | None:
"""The endpoint a stored profile's embedding resolved to before it was pinned.
The Runtime used `embedding_base_url or base_url`, and the provider default
only when both were empty (no collection could have existed then). Writing
that value keeps the endpoint, and so the collection name, byte-identical.
"""
provider = ProviderKind(embedding_provider_kind or provider_kind)
if embedding_model is None or embedding_base_url or provider is ProviderKind.LOCAL_ONNX:
return embedding_base_url
return base_url or DEFAULT_BASE_URLS.get(provider)
def _pin_embedding_endpoints(connection: sqlite3.Connection, table: str) -> None:
"""Write the resolved endpoint into every remote embedding left without one."""
rows = connection.execute(
f"SELECT rowid, provider_kind, base_url, embedding_model, embedding_provider_kind "
f"FROM {table} WHERE embedding_model IS NOT NULL AND embedding_base_url IS NULL "
"AND COALESCE(embedding_provider_kind, provider_kind) != 'local_onnx'"
).fetchall()
for row in rows:
connection.execute(
f"UPDATE {table} SET embedding_base_url = ? WHERE rowid = ?",
(
pinned_embedding_base_url(
provider_kind=row["provider_kind"],
base_url=row["base_url"],
embedding_model=row["embedding_model"],
embedding_provider_kind=row["embedding_provider_kind"],
embedding_base_url=None,
),
row["rowid"],
),
)
def validate_model_base_url(value: str) -> None:
+24 -1
View File
@@ -14,6 +14,7 @@ from pydantic import BaseModel, ConfigDict, Field
from one_memory.hy_sdk_types import HySdkMemoryRecord
from one_memory.memory_governance import GovernedMemory, MemoryStatus
from one_memory.sdk_search_options import SdkSearchOptions
from one_memory.semantic_deadline import (
SemanticMemoryHit,
SemanticSearch,
@@ -22,6 +23,7 @@ from one_memory.semantic_deadline import (
normalize_semantic_hits,
search_within,
)
from one_memory.user_scope import DEFAULT_USER_ID, UserId
def recall_blend_mode() -> str:
@@ -49,7 +51,15 @@ class TruncationBudget(StrEnum):
TIMEOUT = "timeout_budget"
class RecallRequest(BaseModel):
class RecallRequest(SdkSearchOptions):
"""`POST /api/v1/recall`.
The `SdkSearchOptions` fields, `schema_limit` and `intention_limit` pass
through to the SDK memory search (and the floor and row filters to the
raw fallback). Absent, recall makes the search it always made: the
relevance floor, schema and intention lanes sized to `limit`.
"""
model_config = ConfigDict(frozen=True, extra="forbid")
query: str = Field(min_length=1, max_length=2_000)
@@ -67,6 +77,19 @@ class RecallRequest(BaseModel):
#: Wall-clock budget for candidate gather + score. Not a substitute for
#: raising the old newest-500 page; it is the third parallel budget.
timeout_ms: int = Field(default=2_000, ge=50, le=30_000)
#: Whose memories to recall. Omitted, it is `local`, the App's library.
user_id: UserId = DEFAULT_USER_ID
#: The SDK's schema (L6) and proactive intention (L7) lanes.
schema_limit: int | None = Field(default=None, ge=0, le=20)
intention_limit: int | None = Field(default=None, ge=0, le=20)
def search_overrides(self) -> Mapping[str, object]:
values = dict(super().search_overrides())
if self.schema_limit is not None:
values["schema_limit"] = self.schema_limit
if self.intention_limit is not None:
values["intention_limit"] = self.intention_limit
return values
class RecallHit(BaseModel):
@@ -6,7 +6,7 @@ import asyncio
import json
from collections.abc import AsyncIterator, Awaitable, Callable
from contextlib import aclosing
from typing import Any, Literal, Protocol
from typing import Any, Literal, Protocol, cast
import httpx
from pydantic import BaseModel, ConfigDict
@@ -14,7 +14,7 @@ from pydantic import BaseModel, ConfigDict
from one_memory.http_client import create_async_client
from one_memory.hy_runtime_config import HY_MEMORY_READER
from one_memory.hy_sdk_runtime import sdk_relevance_floor
from one_memory.hy_sdk_types import HySdkMemoryRecord, parse_sdk_search
from one_memory.hy_sdk_types import HySdkMemoryRecord, parse_sdk_lane_search, parse_sdk_search
from one_memory.memory_qa import (
MemoryAnswerError,
_DeferredSecretStore,
@@ -31,7 +31,9 @@ from one_memory.model_profiles import (
ProviderKind,
attach_disable_thinking,
)
from one_memory.sdk_search_options import SdkSearchFilters, SearchLane
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
from one_memory.user_scope import owns
_NO_GROUNDING = "现有记忆不足以回答这个问题。"
_GROUNDING_LIMIT = 6
@@ -95,29 +97,51 @@ class SdkMemoryAnswerer:
question: str,
*,
user_id: str = "local",
filters: SdkSearchFilters | None = None,
) -> tuple[SdkAskMemory, ...]:
"""The user's active memories nearest the question.
`filters` are a caller's pass-through search options; absent, the
search is the one ask always made.
"""
client = await self.client_handle.client()
overrides = filters.search_overrides() if filters is not None else {}
raw = await asyncio.to_thread(
client.search,
question,
user_ids=[user_id],
agent_ids=[],
limit=12,
min_score=sdk_relevance_floor(),
schema_min_score=sdk_relevance_floor(),
schema_limit=12,
intention_limit=12,
scene="normal",
reader=HY_MEMORY_READER,
**{
"user_ids": [user_id],
"agent_ids": [],
"limit": 12,
"min_score": sdk_relevance_floor(),
"schema_min_score": sdk_relevance_floor(),
"schema_limit": 12,
"intention_limit": 12,
"scene": "normal",
"reader": HY_MEMORY_READER,
**overrides,
},
)
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
if filters is not None and overrides:
hits = tuple(
hit
for hit in parse_sdk_lane_search(raw, error_code="hy_memory_search_shape_unknown")
if filters.passes_floor(cast(SearchLane, hit.lane), hit.score)
)
else:
hits = parse_sdk_search(raw, error_code="hy_memory_search_shape_unknown")
found: list[SdkAskMemory] = []
seen: set[str] = set()
for hit in hits:
if hit.memory_id in seen:
continue
record = await _get_memory(client, hit.memory_id)
if record is None or record.status != "active" or not record.is_latest:
if (
record is None
or record.status != "active"
or not record.is_latest
or not owns(record.user_id, user_id)
):
continue
seen.add(record.memory_id)
found.append(_wire_memory(record))
@@ -128,8 +152,9 @@ class SdkMemoryAnswerer:
question: str,
*,
user_id: str = "local",
filters: SdkSearchFilters | None = None,
) -> SdkMemoryAnswer:
candidates = await self.retrieve(question, user_id=user_id)
candidates = await self.retrieve(question, user_id=user_id, filters=filters)
if not candidates:
return SdkMemoryAnswer(answer=_NO_GROUNDING, memories=(), mode="empty")
profile = await self._active_profile()
@@ -160,8 +185,9 @@ class SdkMemoryAnswerer:
question: str,
*,
user_id: str = "local",
filters: SdkSearchFilters | None = None,
) -> AsyncIterator[dict[str, object]]:
candidates = await self.retrieve(question, user_id=user_id)
candidates = await self.retrieve(question, user_id=user_id, filters=filters)
await self._require_current(candidates)
yield {"type": "recall", "memories": candidates}
if not candidates:
@@ -5,7 +5,8 @@ from __future__ import annotations
import asyncio
import logging
import re
from contextlib import suppress
from collections.abc import AsyncIterator, Callable
from contextlib import AsyncExitStack, asynccontextmanager
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
@@ -32,16 +33,6 @@ from one_memory.semantic_deadline import SemanticMemoryHit
_STABLE_CODE = re.compile(r"[a-z][a-z0-9_]{0,95}")
logger = logging.getLogger(__name__)
RAW_RETENTION_DAYS = 3
RAW_RETENTION_CHECK_SECONDS = 6 * 60 * 60
_RAW_STATUSES = (
"active",
"superseded",
"shadow",
"negated",
"conflicted",
"archived",
)
class SdkMemoryRuntime:
@@ -55,74 +46,17 @@ class SdkMemoryRuntime:
write_queue: DurableWriteQueue,
extraction_settings: ExtractionSettingsStore,
data_dir: Path,
memory_mode: Callable[[], str] = lambda: "pro",
) -> None:
self.coordinator = coordinator
#: The SDK mode, for the graph store's path (`hy_graph_admin`).
self.memory_mode = memory_mode
self.read_handle = read_handle
self.write_queue = write_queue
self.extraction_settings = extraction_settings
self.data_dir = data_dir
self._raw_retention_task: asyncio.Task[None] | None = None
def start_raw_retention(self) -> None:
if self._raw_retention_task is None or self._raw_retention_task.done():
self._raw_retention_task = asyncio.create_task(
self._raw_retention_loop(),
name="one-memory-sdk-raw-retention",
)
async def prune_expired_raw(self, *, now: datetime | None = None) -> int:
cutoff = (now or datetime.now(UTC)) - timedelta(days=RAW_RETENTION_DAYS)
cutoff_timestamp = int(cutoff.timestamp())
client = await self.read_handle.client()
deleted = 0
while True:
page = await asyncio.to_thread(
client.list_memories,
user_id="local",
layers=["l1_raw"],
statuses=list(_RAW_STATUSES),
order="asc",
limit=500,
offset=0,
)
bucket = page.get("vdb") if isinstance(page, dict) else None
rows = bucket.get("memories") if isinstance(bucket, dict) else None
if not isinstance(rows, list) or not rows:
return deleted
expired = [
row
for row in rows
if isinstance(row, dict)
and isinstance(row.get("memory_id"), str)
and int(row.get("memory_at") or row.get("gmt_created") or 0)
< cutoff_timestamp
]
if not expired:
return deleted
removed_page = 0
for row in expired:
result = await asyncio.to_thread(client.delete, row["memory_id"])
if isinstance(result, dict) and result.get("success") is True:
count = int(result.get("deleted_count") or 0)
deleted += count
removed_page += count
if removed_page == 0:
return deleted
async def _raw_retention_loop(self) -> None:
while True:
await asyncio.sleep(RAW_RETENTION_CHECK_SECONDS)
try:
removed = await self.prune_expired_raw()
if removed:
logger.info("sdk_raw_retention_deleted count=%d", removed)
except asyncio.CancelledError:
raise
except Exception as error:
logger.warning(
"sdk_raw_retention_failed error_type=%s",
type(error).__name__,
)
# MemHub keeps every raw the SDK stores, in every mode: there is no
# retention window (owner decision; the SDK itself never deletes raw).
async def semantic_recall(
self,
@@ -153,55 +87,55 @@ class SdkMemoryRuntime:
limit: int = 12,
) -> tuple[tuple[str, float], ...]:
return tuple(
(hit.memory_id, hit.score)
for hit in await self.semantic_recall(query, limit=limit)
(hit.memory_id, hit.score) for hit in await self.semantic_recall(query, limit=limit)
)
async def graph_snapshot(self, *, limit: int = 300) -> HyGraphSnapshot:
async def graph_snapshot(self, *, limit: int = 300, user_id: str = "local") -> HyGraphSnapshot:
try:
await self.coordinator.active_profile_credentials()
except RuntimeError as error:
if str(error) == "model_profile_not_active":
return HyGraphSnapshot(consulted=False)
raise
try:
client = await self.coordinator.client()
except Exception as error:
raise HySdkError("hy_memory_graph_read_failed") from error
return await asyncio.to_thread(
read_graph_snapshot,
client,
"local",
"*",
limit,
)
async with AsyncExitStack() as stack:
try:
client = await stack.enter_async_context(self.coordinator.use_client())
except Exception as error:
raise HySdkError("hy_memory_graph_read_failed") from error
return await asyncio.to_thread(
read_graph_snapshot,
client,
user_id,
"*",
limit,
)
async def digest(self) -> dict[str, object]:
client = await self.coordinator.client()
try:
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
limit=1_000,
offset=0,
)
agent_ids = _agent_ids(raw)
digests = [
{
"agent_id": agent_id,
"schemas_recorded": 0,
**dict(
await asyncio.to_thread(
client.evolve,
user_id="local",
agent_id=agent_id,
)
),
}
for agent_id in agent_ids
]
except Exception as error:
raise HySdkError("hy_memory_digest_failed") from error
async with self.coordinator.use_client() as client:
try:
raw = await asyncio.to_thread(
client.list_memories,
user_id="local",
limit=1_000,
offset=0,
)
agent_ids = _agent_ids(raw)
digests = [
{
"agent_id": agent_id,
"schemas_recorded": 0,
**dict(
await asyncio.to_thread(
client.evolve,
user_id="local",
agent_id=agent_id,
)
),
}
for agent_id in agent_ids
]
except Exception as error:
raise HySdkError("hy_memory_digest_failed") from error
return {
"success": all(result.get("success") is not False for result in digests),
"scope_count": len(digests),
@@ -222,6 +156,7 @@ class SdkMemoryRuntime:
profile,
api_key=secret,
data_dir=self.data_dir,
mode=self.memory_mode(),
)
async def decide_graph_width(
@@ -238,38 +173,42 @@ class SdkMemoryRuntime:
api_key=secret,
data_dir=self.data_dir,
decision=decision,
mode=self.memory_mode(),
)
async def _mutation_client(self) -> Any:
@asynccontextmanager
async def _mutation_client(self) -> AsyncIterator[Any]:
"""The write client, with no-model and keychain refusals as SDK codes.
Callers record these against the edit or deletion instead of failing
the request that already committed it.
"""
try:
return await self.coordinator.client()
except RuntimeError as error:
code = str(error)
if not _STABLE_CODE.fullmatch(code):
code = "hy_memory_client_unavailable"
raise HySdkError(code) from error
async with AsyncExitStack() as stack:
try:
client = await stack.enter_async_context(self.coordinator.use_client())
except RuntimeError as error:
code = str(error)
if not _STABLE_CODE.fullmatch(code):
code = "hy_memory_client_unavailable"
raise HySdkError(code) from error
yield client
async def update_hy_memory(self, memory_id: str, content: str) -> dict[str, object]:
client = await self._mutation_client()
try:
result = await asyncio.to_thread(client.update, memory_id, content)
except Exception as error:
raise HySdkError("hy_memory_update_failed") from error
async with self._mutation_client() as client:
try:
result = await asyncio.to_thread(client.update, memory_id, content)
except Exception as error:
raise HySdkError("hy_memory_update_failed") from error
if not isinstance(result, dict) or result.get("success") is not True:
raise HySdkError("hy_memory_update_rejected")
return result
async def finish_deletion(self, memory_id: str) -> None:
client = await self._mutation_client()
try:
result = await asyncio.to_thread(client.delete, memory_id)
except Exception as error:
raise HySdkError("hy_memory_delete_failed") from error
async with self._mutation_client() as client:
try:
result = await asyncio.to_thread(client.delete, memory_id)
except Exception as error:
raise HySdkError("hy_memory_delete_failed") from error
if not isinstance(result, dict):
raise HySdkError("hy_memory_delete_rejected")
error_code = result.get("error_code")
@@ -373,12 +312,7 @@ class SdkMemoryRuntime:
return None
async def shutdown(self) -> None:
if self._raw_retention_task is None:
return
self._raw_retention_task.cancel()
with suppress(asyncio.CancelledError):
await self._raw_retention_task
self._raw_retention_task = None
return None
def _agent_ids(raw: object) -> tuple[str, ...]:
@@ -23,6 +23,7 @@ from pathlib import Path
from typing import Any
from one_memory.database_access import main_database
from one_memory.hy_sdk_runtime import builtin_retrieval_snapshot
from one_memory.imports.models import EvidenceRecord
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_server.contracts import (
@@ -35,6 +36,8 @@ from one_memory.memory_server.durable_queue import (
COMPLETED,
FAILED,
REAPED_PAYLOAD_MARKER,
RUNNING,
UNKNOWN,
DurableWriteQueue,
)
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
@@ -49,6 +52,7 @@ from one_memory.sdk_migration_bundle import inspect_sdk_migration_bundle
from one_memory.sdk_migration_verify import (
MemoryReadbackFailed,
ReadbackReport,
expected_memory_ids,
legacy_memory_ids_from,
legacy_memory_ids_from_bundle,
read_active_profile,
@@ -59,6 +63,8 @@ from one_memory.sdk_vector_reattach import (
ReattachedCollection,
legacy_vector_collections,
reattach_legacy_vectors,
stored_memory_ids,
vectors_belong_to_another_space,
)
from one_memory.target_schema import drop_legacy_tables, legacy_tables_present
@@ -257,6 +263,7 @@ def prepare_legacy_write_queue_for_cutover(database_path: Path) -> tuple[int, in
internal_session_id = internal_session_id_for(
source.conversation_id,
source.source_id,
source.user_id,
)
queue.submit(
WriteRequest(
@@ -317,11 +324,45 @@ def move_pending_dispatches(database_path: Path) -> int:
return len(conversations)
#: Why a pre-cutover write job that may have reached HY was closed.
ABANDONED_AT_CUTOVER = "abandoned_at_cutover"
def settle_legacy_unknown_write_jobs(database_path: Path) -> int:
"""Close write jobs the legacy pipeline left possibly-sent. Safe to repeat.
They ran under the legacy pipeline, so the SDK's `get_write_outcome` knows
none of them and the unknown reconcile can never settle them: a migrated
1.0.4 home showed 1000+ unknown forever. They are not retried, since they
may already have written and the SDK has no idempotency key. `cancelled`,
not `failed`: the upgrade ended them, nothing broke, and a `failed` count
would put the home in the failed state for good. `hy_invoked` is kept, so
no readmission path can pick one up again.
"""
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
if "memory_write_jobs" not in _tables(database):
return 0
now = datetime.now(UTC).isoformat()
with main_database(database_path).write(autocommit=True) as database:
# A `running` row already marked invoked would be reclaimed to
# `unknown` on the next open, so it is the same case.
return int(
database.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,"
"lease_expires_at=NULL,last_error=?,updated_at=? "
"WHERE state=? OR (state=? AND hy_invoked=1)",
(CANCELLED, ABANDONED_AT_CUTOVER, now, UNKNOWN, RUNNING),
).rowcount
)
def abandoned_legacy_work(database_path: Path) -> dict[str, int]:
"""Count legacy work that ends with the cutover instead of blocking it.
A task that may have reached the SDK is not replayed: the SDK already holds
whatever it committed. Pending user deletions are not carried over.
Settled write jobs are counted by their code, so a repeated preparation
reports the same number.
"""
abandoned: dict[str, int] = {}
with main_database(database_path).read(row_factory=sqlite3.Row) as database:
@@ -332,6 +373,11 @@ def abandoned_legacy_work(database_path: Path) -> dict[str, int]:
"memory_tasks",
"status='unknown' OR (status='running' AND hy_invocation_count > 0)",
),
(
"unknown_write_jobs",
"memory_write_jobs",
f"state='{CANCELLED}' AND last_error='{ABANDONED_AT_CUTOVER}'",
),
("deletions", "memory_deletions", "status IN ('pending','running')"),
)
for name, table, where in checks:
@@ -354,6 +400,10 @@ def prepare_legacy_work_for_cutover(
running, so a failure names it.
"""
trace = {} if trace is None else trace
# First: a possibly-sent job never runs again, so its input is not
# frozen, and input released since must not block the cutover.
trace["sub_step"] = "unknown_write_jobs"
settle_legacy_unknown_write_jobs(database_path)
trace["sub_step"] = "write_queue"
frozen, moved = prepare_legacy_write_queue_for_cutover(database_path)
trace["sub_step"] = "pending_dispatches"
@@ -411,13 +461,17 @@ def resume_legacy_cleanup(*, data_dir: Path, database_path: Path) -> tuple[str,
bundle_id=marker.backup_bundle_id,
)
from one_memory.hy_runtime_config import HY_MEMORY_NAMESPACE
from one_memory.memory_visibility import drop_stale_triggers
recover_abandoned_staging(
data_dir=data_dir,
database_path=database_path,
namespace=HY_MEMORY_NAMESPACE,
)
return drop_legacy_tables(database_path)
dropped = drop_legacy_tables(database_path)
# The visibility cache's grant triggers read a table just dropped (#233).
drop_stale_triggers(database_path)
return dropped
MIGRATION_PREFIX = "sdk-authority-v2"
@@ -654,13 +708,14 @@ def run_startup_migration(
step["authority"] = authority.value
log.migration_id = authority_migration
with log.step("preflight.profile_read") as step:
profile = read_active_profile(database_path)
step["profile_present"] = profile is not None
active = read_active_profile(database_path)
step["profile_present"] = active is not None
# No active profile means no generation model, not no embedding: the
# vectors are the built-in model's, which is also what the Runtime reads.
profile = active if active is not None else builtin_retrieval_snapshot()
if authority is MemoryAuthority.SDK:
with log.step("preflight.leftovers_scan") as step:
vectors_waiting = profile is not None and bool(
legacy_vector_collections(data_dir, profile)
)
vectors_waiting = bool(legacy_vector_collections(data_dir, profile))
leftovers = bool(legacy_tables_present(database_path)) or abandoned_staging_present(
data_dir=data_dir, namespace=namespace
)
@@ -671,7 +726,6 @@ def run_startup_migration(
reattached: tuple[ReattachedCollection, ...] = ()
readback: ReadbackReport | None = None
if vectors_waiting:
assert profile is not None
progress("vectors")
with log.step("vectors.reattach") as step:
reattached = reattach_legacy_vectors(data_dir, profile)
@@ -723,12 +777,12 @@ def run_startup_migration(
step["chat"] = len(legacy.chat)
step["productivity"] = len(legacy.productivity)
step["users"] = len(legacy.users)
if profile is None and not legacy.empty:
# Without the embedding profile the Runtime's collections cannot be
# named, so neither reattached nor read back. Legacy stays intact.
log.check("preflight.model_profile_check", "sdk_migration_model_profile_required")
raise DataPlaneMigrationError("sdk_migration_model_profile_required")
log.check("preflight.model_profile_check", None)
if active is None and not legacy.empty and vectors_belong_to_another_space(data_dir, profile):
# The memories live in the space of a profile that is no longer active,
# so the built-in model cannot read them. Legacy stays intact.
log.check("preflight.vector_space_check", "sdk_migration_vector_profile_missing")
raise DataPlaneMigrationError("sdk_migration_vector_profile_missing")
log.check("preflight.vector_space_check", None)
with capture_fence(data_dir):
# Builds through 19 left a staged SDK copy beside the original store.
# Settling it before the bundle means the bundle holds the original
@@ -784,17 +838,23 @@ def run_startup_migration(
step.update(preparation.as_document())
reattached = ()
readback = None
if profile is not None:
progress("vectors")
with log.step("vectors.reattach") as step:
reattached = reattach_legacy_vectors(data_dir, profile)
step["collections"] = len(reattached)
if not legacy.empty:
assert profile is not None
# Before reattach touches the store, and after the staging
# recovery above has settled which store is the original.
with log.step("verify.stored_ids") as step:
expected, missing = expected_memory_ids(legacy, stored_memory_ids(data_dir))
step["chat_expected"] = len(expected.chat)
step["chat_missing_before_migration"] = missing
progress("vectors")
with log.step("vectors.reattach") as step:
reattached = reattach_legacy_vectors(data_dir, profile)
step["collections"] = len(reattached)
if not expected.empty:
progress("verify")
with log.step("verify.readback") as step:
with client_for(profile) as client:
readback = verify_readback(client, legacy)
readback = verify_readback(
client, expected, chat_missing_before_migration=missing
)
step.update(readback.as_document())
if not readback.passed:
store.set_phase(migration_id, phase="readback_failed", state="blocked")
@@ -14,12 +14,12 @@ import json
import sqlite3
from collections.abc import Iterator
from contextlib import closing, contextmanager
from dataclasses import dataclass
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any
from one_memory.database_access import main_database
from one_memory.model_profiles import ModelProfileSnapshot
from one_memory.model_profiles import ModelProfileSnapshot, pinned_embedding_base_url
from one_memory.sdk_data_plane_migration import DataPlaneMigrationError
_PRODUCTIVITY_KINDS = frozenset({"coding", "working"})
@@ -47,6 +47,8 @@ class ReadbackReport:
chat_readable: int
productivity_expected: int
productivity_readable: int
#: Confirmed in the mirror, gone from the SDK before this migration began.
chat_missing_before_migration: int = 0
@property
def passed(self) -> bool:
@@ -63,6 +65,7 @@ class ReadbackReport:
"chat_readable": self.chat_readable,
"productivity_expected": self.productivity_expected,
"productivity_readable": self.productivity_readable,
"chat_missing_before_migration": self.chat_missing_before_migration,
"passed": self.passed,
}
@@ -115,6 +118,27 @@ def read_legacy_memory_ids(connection: sqlite3.Connection) -> LegacyMemoryIds:
)
def expected_memory_ids(
legacy: LegacyMemoryIds, stored: frozenset[str]
) -> tuple[LegacyMemoryIds, int]:
"""What the readback must find, and how many confirmed chats it need not.
A 1.0.4 library can hold confirmed rows whose SDK node the SDK had already
merged or deleted: no migration could read those back (658 confirmed, 657
in the store, and the whole upgrade failed). So chat is held to what the
store held before this run, and anything in there must still read back.
If none of the confirmed chats is in the store, that is not history: the
store is empty or is the wrong one. Nothing is excused then, so the
readback still fails on every id instead of checking nothing just before
the mirror is dropped.
"""
chat = tuple(memory_id for memory_id in legacy.chat if memory_id in stored)
if not chat:
return legacy, 0
return replace(legacy, chat=chat), len(legacy.chat) - len(chat)
def legacy_memory_ids_from(database_path: Path) -> LegacyMemoryIds:
with main_database(database_path).read() as connection:
return read_legacy_memory_ids(connection)
@@ -160,7 +184,14 @@ def read_active_profile(database_path: Path) -> ModelProfileSnapshot | None:
model=row["model"],
embedding_model=row["embedding_model"],
embedding_provider_kind=row["embedding_provider_kind"],
embedding_base_url=row["embedding_base_url"],
# Read-only, beside a Core that may not have pinned it yet.
embedding_base_url=pinned_embedding_base_url(
provider_kind=row["provider_kind"],
base_url=row["base_url"],
embedding_model=row["embedding_model"],
embedding_provider_kind=row["embedding_provider_kind"],
embedding_base_url=row["embedding_base_url"],
),
embedding_dimensions=row["embedding_dimensions"],
timeout_seconds=row["timeout_seconds"],
max_concurrency=row["max_concurrency"],
@@ -179,14 +210,18 @@ def runtime_sdk_client(
from one_memory.model_profiles import ModelProfileStore
from one_memory.secrets import KeyringSecretStore, SecretStoreUnavailableError
try:
secret = ModelProfileStore(database_path).resolve_snapshot_secret(
profile.profile_id, profile.revision, KeyringSecretStore()
)
except SecretStoreUnavailableError:
raise DataPlaneMigrationError("sdk_migration_secure_storage_unavailable") from None
if profile.has_api_key and secret is None:
raise DataPlaneMigrationError("sdk_migration_model_secret_unavailable")
secret: str | None = None
# Without a key there is nothing to resolve, and the built-in retrieval
# snapshot has no stored version to look one up by.
if profile.has_api_key:
try:
secret = ModelProfileStore(database_path).resolve_snapshot_secret(
profile.profile_id, profile.revision, KeyringSecretStore()
)
except SecretStoreUnavailableError:
raise DataPlaneMigrationError("sdk_migration_secure_storage_unavailable") from None
if secret is None:
raise DataPlaneMigrationError("sdk_migration_model_secret_unavailable")
lease = acquire_client(profile, api_key=secret, data_dir=data_dir)
try:
yield lease.client
@@ -194,7 +229,9 @@ def runtime_sdk_client(
lease.close()
def verify_readback(client: Any, legacy: LegacyMemoryIds) -> ReadbackReport:
def verify_readback(
client: Any, legacy: LegacyMemoryIds, *, chat_missing_before_migration: int = 0
) -> ReadbackReport:
listed: set[str] = set()
for user_id in legacy.users or ("local",):
offset = 0
@@ -228,5 +265,6 @@ def verify_readback(client: Any, legacy: LegacyMemoryIds) -> ReadbackReport:
chat_readable=sum(1 for memory_id in legacy.chat if readable(memory_id)),
productivity_expected=len(legacy.productivity),
productivity_readable=sum(1 for memory_id in legacy.productivity if readable(memory_id)),
chat_missing_before_migration=chat_missing_before_migration,
)
@@ -0,0 +1,135 @@
"""The read client, kept in the embedding space the active profile writes to.
Every read used to open the built-in 512-dim collection while writes followed
the active profile, so a custom embedding read an empty space (#242). This
handle opens the active profile's `retrieval_view` instead, and the built-in
space only when no profile is active.
A profile change builds a new client and retires the old one; a retired client
is closed once no task still uses it, as `SdkWriteCoordinator` does for writes
(#218). A use is the task that asked for the client: callers keep their
`client = await handle.client()` shape, and the use ends with that task (one
HTTP request, one retention pass).
The memory mode (lite / pro / ultra) is fixed per SDK client, so a mode change
rebuilds and retires the same way, and so do the SDK features
(`memory_features`): handles are keyed by (profile, mode, features). Neither
mode nor features names a collection; both are separate client parameters.
"""
from __future__ import annotations
import asyncio
from collections.abc import Callable
from pathlib import Path
from typing import Any
from one_memory.hy_graph_admin import effective_memory_mode
from one_memory.hy_sdk_runtime import (
BUILTIN_RETRIEVAL_PROFILE_ID,
SdkClientHandle,
builtin_retrieval_snapshot,
retrieval_view,
)
from one_memory.hy_sdk_types import HySdkRuntimeError
from one_memory.memory_features import FeatureOverrides
from one_memory.model_profiles import ModelProfileStore, ProviderKind
from one_memory.secrets import SecretStore, SecretStoreUnavailableError
_BUILTIN = (BUILTIN_RETRIEVAL_PROFILE_ID, 0)
class ActiveProfileReadHandle(SdkClientHandle):
"""An `SdkClientHandle` whose profile follows the active one."""
def __init__(
self,
*,
profiles: ModelProfileStore,
secrets: SecretStore,
data_dir: Path,
memory_mode: Callable[[], str] = effective_memory_mode,
features: Callable[[], FeatureOverrides] = lambda: (),
) -> None:
super().__init__(builtin_retrieval_snapshot(), api_key=None, data_dir=data_dir)
self.profiles = profiles
self.secrets = secrets
self.data_dir = data_dir
self.memory_mode = memory_mode
self.current_features = features
self._handles: dict[tuple[str, int, str, FeatureOverrides], SdkClientHandle] = {}
#: Tasks in flight per handle; a retired handle closes at zero.
self._uses: dict[SdkClientHandle, int] = {}
self._retired: set[SdkClientHandle] = set()
self._closing: set[asyncio.Task[None]] = set()
self._handle_lock = asyncio.Lock()
async def client(self) -> Any:
profile = await asyncio.to_thread(self.profiles.active_identity) or _BUILTIN
identity = (*profile, self.memory_mode(), self.current_features())
async with self._handle_lock:
handle = self._handles.get(identity)
if handle is None:
# Only on a change of profile: resolving a key can reach the
# system keychain, so it is not done on every read.
handle = await asyncio.to_thread(self._build, identity)
self._handles[identity] = handle
self._retired.update(value for key, value in self._handles.items() if key != identity)
self._handles = {identity: handle}
unused = {value for value in self._retired if not self._uses.get(value)}
self._retired -= unused
self._hold(handle)
await asyncio.gather(*(value.close() for value in unused))
return await handle.client()
async def close(self) -> None:
async with self._handle_lock:
handles = (*self._handles.values(), *self._retired)
self._handles, self._retired, self._uses = {}, set(), {}
await asyncio.gather(*(handle.close() for handle in handles), *self._closing)
def _hold(self, handle: SdkClientHandle) -> None:
"""Count a use of `handle` until the asking task finishes."""
task = asyncio.current_task()
if task is None: # pragma: no cover - always called from a coroutine
return
self._uses[handle] = self._uses.get(handle, 0) + 1
task.add_done_callback(lambda _task: self._release(handle))
def _release(self, handle: SdkClientHandle) -> None:
remaining = self._uses.get(handle, 0) - 1
if remaining > 0:
self._uses[handle] = remaining
return
self._uses.pop(handle, None)
if handle in self._retired:
self._retired.discard(handle)
closing = asyncio.get_running_loop().create_task(handle.close())
self._closing.add(closing)
closing.add_done_callback(self._closing.discard)
def _build(self, key: tuple[str, int, str, FeatureOverrides]) -> SdkClientHandle:
profile_id, revision, mode, features = key
identity = (profile_id, revision)
if identity == _BUILTIN:
return SdkClientHandle(
builtin_retrieval_snapshot(),
api_key=None,
data_dir=self.data_dir,
mode=mode,
features=features,
)
profile = self.profiles.snapshot(*identity)
view = retrieval_view(profile)
secret: str | None = None
# Only a remote OpenAI-compatible embedding sends the shared key.
if view.embedding_provider_kind is ProviderKind.OPENAI_COMPATIBLE:
try:
secret = self.profiles.resolve_snapshot_secret(*identity, self.secrets)
except SecretStoreUnavailableError:
raise HySdkRuntimeError("secure_storage_unavailable") from None
if secret is None:
raise HySdkRuntimeError("model_secret_unavailable")
return SdkClientHandle(
view, api_key=secret, data_dir=self.data_dir, mode=mode, features=features
)
Loaded 100 of 221 files, more files were not shown because too many files have changed in this diff. Show more