Compare commits

...
Author SHA1 Message Date
kristinheandCursor c13950a4c1 fix(memhub): clarify Lite raw conversation labels
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-05 11:29:31 +08:00
kristinhe1111 603e0843c5 style(memhub): remove Pro recommendation badge 2026-11-05 11:08:33 +08:00
kristinhe1111 80f4231e46 style(memhub): simplify responsive onboarding 2026-11-05 10:52:04 +08:00
kristinhe1111 2f9e028c50 fix(memhub): share model probe readiness 2026-11-05 03:12:34 +08:00
kristinhe1111 4cbf4f1501 feat(memhub): offer all memory modes during onboarding 2026-11-05 02:40:34 +08:00
kristinhe1111 e5959578c8 fix(memhub): let Lite onboarding complete immediately 2026-11-05 01:48:15 +08:00
kristinhe1111 df5a65797c fix(memhub): page Lite raw turns without full scans 2026-11-05 00:55:07 +08:00
kristinhe1111 2ea48e68a6 Merge pull request #326 from Tencent-Hunyuan/fix/home-recent-update-bottom-align
fix(memhub): align recent activity to card bottom
2026-11-05 00:31:55 +08:00
kristinhe1111 b8fdde7d0b Merge pull request #327 from Tencent-Hunyuan/fix/model-technical-hover
fix(web): preserve model details hover contrast
2026-11-05 00:31:36 +08:00
kristinhe1111 5134524743 Merge pull request #328 from Tencent-Hunyuan/fix/ultra-evolve-time-picker
fix(memhub): replace native evolve time input
2026-11-05 00:31:20 +08:00
kristinhe1111 e34101cb0b Merge pull request #329 from Sthreal/fix/301-model-probe-pending-state
fix(memhub): fail model probe reservations when execution never dispatches
2026-11-05 00:31:00 +08:00
kristinhe1111 995c7e6c32 Merge pull request #330 from Tencent-Hunyuan/fix/memhub-complete-log-export
fix(memhub): export complete logs from Settings
2026-11-05 00:30:38 +08:00
kristinheandCursor f1afdb03cd fix(memhub): export complete logs from settings
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 23:49:21 +08:00
Sthreal 4df4ce72bd fix(memhub): fail model probe reservations when execution never dispatches 2026-11-04 23:33:06 +08:00
kristinheandCursor a6d9894975 fix(memhub): replace native evolve time input
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 22:31:00 +08:00
kristinheandCursor 218bcf779b test: stabilize reembed merge fixture timestamps
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 22:11:46 +08:00
kristinheandCursor 7e4ee0d8ea fix(web): preserve model details hover contrast
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 21:48:08 +08:00
kristinheandCursor bce58e638d fix(memhub): align recent activity to card bottom
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 21:47:57 +08:00
feitianxiang b0e429b9ae Merge pull request #322 from Tencent-Hunyuan/docs/memhub-1.0.6-changelog
docs(changelog): complete MemHub 1.0.6 notes (build 41)
2026-11-04 20:49:02 +08:00
alvinfeiandClaude Opus 5.5 01faf28942 docs(changelog): MemHub 1.0.6 covers everything merged through build 41
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-04 20:47:19 +08:00
kristinhe1111 fb1b1373ab Merge pull request #321 from Tencent-Hunyuan/chore/memhub-build-41
chore(release): bump MemHub build to 41
2026-11-04 20:39:34 +08:00
kristinheandCursor 2823bd743d chore(release): bump MemHub build to 41
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 20:36:00 +08:00
feitianxiang 837c084be2 Merge pull request #320 from Tencent-Hunyuan/fix/memhub-db-contention
fix(memhub): bound import jobs, keep the write queue off the bulk pool, nothing blocking on the event loop
2026-11-04 20:27:32 +08:00
alvinfeiandClaude Opus 5.5 bf52724461 fix(memhub): bound import jobs, keep the write queue off the bulk pool, nothing blocking on the event loop
Under a long live-sync history, writes stalled behind "database is locked"
refusals, durable leases lapsed, and the App stopped answering for tens of
seconds at a time.

- import_jobs is bounded: finished (completed/cancelled) sync jobs are
  pruned to the newest 20 per (user, source), with their issues and skip
  counts, in bounded batches at startup and every 10 minutes. History jobs
  and every unfinished job are kept.
- Job lookups are SQL (find_jobs / latest_job) instead of reading the whole
  table, served by new indexes in schema migration 63. GET
  /api/v1/import-jobs keeps its shape and order.
- The durable write queue runs claim, settle and maintenance in its own pool
  and lease renewal in a dedicated one, so bulk work cannot starve a lease.
  The single writer and its line are unchanged.
- abandon, resume, enqueue, pause, cancel, shutdown and startup resume make
  their database calls off the loop; the file watcher resolves changed
  paths off the loop; revoke reads and writes its grant off the loop.
- The WAL cadence checkpoint is requested from the write path and run by one
  checkpointer thread instead of inline on the writing thread.
- The durable drain is supervised: an unexpected exit is reported with its
  stack and the drain restarts with capped backoff under a new lease owner.
  memory-processing and memhub status show write_drain and
  write_drain_error. Contained per-job failures are reported with stacks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-04 19:57:26 +08:00
feitianxiang 964d8610f3 Merge pull request #319 from Tencent-Hunyuan/fix/memhub-lite-raw-rag
fix(memhub): hy-memory 2.0.14; Lite search is plain RAG over raw
2026-11-04 19:42:51 +08:00
feitianxiang e1933d3307 Merge pull request #318 from Tencent-Hunyuan/release/hy-memory-2.0.14
release: hy-memory 2.0.14
2026-11-04 19:42:39 +08:00
alvinfeiandClaude Opus 5.5 5fdaf6c280 fix(memhub): Lite search is plain RAG over raw
Lite context search, recall and the Host Bridge return every raw vector
hit above the floor, ranked by score, over active and shadow raws. The
turn view shows the turns containing the query, or else the hit's
closest turn, instead of dropping semantic hits. Pro and ultra raw
search stays active-only.

A new test runs the raw routes in a child interpreter against the
installed hy-memory wheel, and the route-test fake SDK now rejects
search kwargs the pinned release does not accept.

The lite raw-turn page projects only raws newly put on top instead of
re-listing the whole raw layer on every poll.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-04 19:32:12 +08:00
alvinfeiandClaude Opus 5.5 58f2fa884a build(memhub): pin hy-memory 2.0.14
2.0.14 accepts `statuses` on explicit L1_RAW search, which Core has
passed since #316. Covers the same files as the 2.0.13 pin (#285).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-04 19:32:11 +08:00
alvinfeiandClaude Opus 5.5 c0910621c1 release: hy-memory 2.0.14
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-04 17:52:06 +08:00
kristinhe1111 dd13c551c4 Merge pull request #317 from Tencent-Hunyuan/fix/settings-field-width
fix(web): expand settings controls to content width
2026-11-04 17:41:20 +08:00
kristinheandCursor e7d82ee5e8 fix(web): expand settings controls to content width
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 16:54:55 +08:00
kristinhe1111 63e009bf6c Merge pull request #307 from Tencent-Hunyuan/fix/lite-home-saving-copy
fix(memhub): keep home status copy compact
2026-11-04 16:26:27 +08:00
kristinhe1111 d4cfd1bb74 Merge pull request #309 from Tencent-Hunyuan/fix/memhub-durable-drain-liveness
fix(memhub): keep durable write drain alive
2026-11-04 16:26:08 +08:00
kristinhe1111 2822cbf368 Merge pull request #310 from Tencent-Hunyuan/fix/memhub-inline-model-config
fix(memhub): inline model configuration panels
2026-11-04 16:25:47 +08:00
kristinhe1111 8d23347e1a Merge pull request #312 from Tencent-Hunyuan/fix/memory-mode-confirm-copy
fix(memhub): clarify memory mode switch effects
2026-11-04 16:25:22 +08:00
kristinhe1111 8ae0d1889e Merge pull request #315 from Tencent-Hunyuan/fix/ultra-topic-hover-only
fix(memhub): make Ultra topic nodes hover-only
2026-11-04 16:24:58 +08:00
kristinhe1111 2479de22eb Merge pull request #316 from Tencent-Hunyuan/fix/memhub-lite-turn-view
fix(memhub): show Lite raw conversations by turn
2026-11-04 16:24:44 +08:00
kristinheandCursor 265a22d612 fix(memhub): show lite raw conversations by turn
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 16:10:57 +08:00
kristinheandCursor 93d0c64490 style(memhub): format graph topic changes
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 16:10:30 +08:00
kristinheandCursor 1589ae31bc fix(memhub): make topic nodes hover-only
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 16:06:56 +08:00
kristinheandCursor 38ad2292d1 fix(memhub): clarify memory mode switch effects
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 15:58:03 +08:00
kristinheandCursor 413b9033f9 fix(memhub): inline model configuration panels
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 15:49:20 +08:00
kristinheandCursor 0953c5af35 fix(memhub): keep durable write drain alive
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 15:22:29 +08:00
kristinheandCursor 470ea4459a fix(memhub): keep home status copy compact
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 14:53:33 +08:00
kristinhe1111 b58e95fdae Merge pull request #305 from Tencent-Hunyuan/chore/memhub-build-40
chore(release): MemHub 1.0.6, build 40
2026-11-04 13:27:01 +08:00
kristinheandCursor 42b577f6f8 chore(release): bump MemHub build to 40
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 13:16:08 +08:00
kristinhe1111 2554fb3b96 Merge pull request #304 from Tencent-Hunyuan/feat/memhub-onboarding-mode-model
feat(memhub): 重构冷启动记忆模式与模型配置
2026-11-04 13:12:38 +08:00
kristinheandCursor e32dc987c2 feat(memhub): redesign onboarding mode setup
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 12:00:46 +08:00
kristinhe1111 598e83525e Merge pull request #298 from Tencent-Hunyuan/feat/memhub-mode-projection
feat(memhub): project home and settings by memory mode
2026-11-04 09:29:48 +08:00
kristinhe 01ace26d08 Merge remote-tracking branch 'origin/main' into feat/memhub-mode-projection
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	memhub/apps/core/src/one_memory/api.py
#	memhub/apps/core/src/one_memory/sdk_write_processor.py
#	memhub/apps/core/tests/test_memory_mode_and_evolve.py
#	memhub/apps/web/src/ApplicationsHome.tsx
#	memhub/apps/web/src/MemorySpace.tsx
2026-11-04 03:17:38 +08:00
kristinheandCursor ba2f647a30 feat(memhub): project home by memory mode
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 02:58:32 +08:00
kristinhe1111 82d9514d62 Merge pull request #297 from Tencent-Hunyuan/fix/memhub-lite-raw-conversations
fix(memhub): preserve raw conversations in lite mode
2026-11-04 01:22:04 +08:00
kristinhe1111 369351719d Merge pull request #295 from Tencent-Hunyuan/fix/memhub-model-center-main
fix(memhub): land Model Center on main
2026-11-04 00:28:38 +08:00
kristinheandCursor 1429ba2d19 fix(memhub): close model center settings gaps
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 00:22:25 +08:00
kristinheandCursor 4e7d752980 feat(memhub): add lite manual raw input
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-04 00:05:36 +08:00
kristinheandCursor fb13a6fda2 fix(memhub): keep mode-specific memory views consistent
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 23:20:41 +08:00
kristinheandCursor 6ff8b3a3b4 fix(memhub): simplify topic node details
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 22:49:47 +08:00
kristinheandCursor 17441d368c fix(memhub): harden settings state recovery
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 22:39:47 +08:00
kristinheandCursor 89d29c9c41 feat(memhub): redesign memory mode settings
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 22:27:00 +08:00
kristinheandCursor 5271bddce3 fix(memhub): preserve raw conversations in lite mode
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 22:18:12 +08:00
kristinhe eac795b2d2 Merge remote-tracking branch 'origin/main' into fix/memhub-model-center-main 2026-11-03 20:55:54 +08:00
kristinhe1111 d938f1d5b2 Merge pull request #294 from Tencent-Hunyuan/chore/memhub-build-39
chore(release): MemHub 1.0.6, build 39
2026-11-03 20:05:08 +08:00
kristinheandCursor 8cee64c813 chore(release): bump MemHub build number to 39
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 19:59:27 +08:00
kristinhe1111 1ffcc9eb57 Merge pull request #292 from Tencent-Hunyuan/feat/memhub-mode-ui
feat(memhub): complete memory mode app lifecycle
2026-11-03 19:43:30 +08:00
kristinhe 23b74f91bc Merge remote-tracking branch 'origin/feat/memhub-mode-ui' into feat/memhub-model-center 2026-11-03 19:41:41 +08:00
kristinheandCursor b36452d110 fix(memhub): reset mode state and satisfy lint
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 19:36:42 +08:00
kristinhe a8b92d21e1 Merge remote-tracking branch 'origin/feat/memhub-mode-ui' into feat/memhub-model-center 2026-11-03 19:35:22 +08:00
kristinheandCursor e58a7c0d14 fix(memhub): polish mode selector and raw transitions
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 19:32:56 +08:00
kristinheandCursor 1663c9424e fix(memhub): align model center with R6 visuals
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 19:31:00 +08:00
kristinheandCursor fba9d8cf26 fix(memhub): align lite and mode settings UI
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 19:30:43 +08:00
kristinhe f754f8bc0f Merge remote-tracking branch 'origin/feat/memhub-mode-ui' into feat/memhub-model-center
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	memhub/apps/web/src/OnboardingModelController.tsx
#	memhub/apps/web/src/R6Settings.tsx
2026-11-03 18:38:24 +08:00
kristinheandCursor 2155c4c773 feat(memhub): add safe model center
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 18:36:08 +08:00
kristinheandCursor 18e2fca34d fix(memhub): harden mode startup recovery and search retry
Close review follow-ups for graph backfill gates, onboarding resume,
settings refresh races, modal focus trapping, and raw search retry.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 18:01:18 +08:00
kristinheandCursor 88feb0adca feat(memhub): add memory mode app lifecycle
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-03 17:22:09 +08:00
feitianxiang d2e9cdaf06 Merge pull request #286 from Tencent-Hunyuan/chore/memhub-build-38
chore(release): MemHub 1.0.6, build 38
2026-11-02 21:47:36 +08:00
alvinfeiandClaude Opus 5.5 8277aaee51 chore(release): bump MemHub build number to 38
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 21:40:56 +08:00
feitianxiang 65550621e0 Merge pull request #285 from Tencent-Hunyuan/release/memhub-1.0.6-hy-2.0.13
chore(memhub): hy-memory 2.0.13; 1.0.6 changelog
2026-11-02 21:29:36 +08:00
alvinfeiandClaude Opus 5.5 1a67427bf3 Merge github/main (hy-memory 2.0.13 release) into release/memhub-1.0.6-hy-2.0.13
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 21:22:42 +08:00
feitianxiang 58f1477680 Merge pull request #282 from Tencent-Hunyuan/release/hy-memory-2.0.13
release: hy-memory 2.0.13
2026-11-02 21:20:29 +08:00
alvinfeiandClaude Opus 5.5 76c459e39a fix(memhub): a failed SDK write keeps the SDK's cause as its error code
hy-memory 2.0.13 reports a model failure as success: false with
agent_error_code / reconcile_error_code. The job's error_code now carries
that cause (hy_sdk_reconcile_empty_response, hy_sdk_llm_error, ...) instead
of a bare hy_sdk_write_failed; only upper-snake codes are kept, never a
message. Regression tests: a reconcile provider failure (401/503/timeout)
and a coding-path extractor failure wait for the provider and complete after
recovery; an empty or unparseable 200 reply is retried 3 times, then failed
with its cause.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 21:03:22 +08:00
alvinfeiandClaude Opus 5.5 b864e0bda6 chore(memhub): pin hy-memory 2.0.13; MemHub 1.0.6 changelog
Pin the Core dependency, lock, HY_MEMORY_VERSION, SBOM, notices, docs and
version guards to hy-memory 2.0.13. Move the MemHub entries from Unreleased
into the 1.0.6 section (build 37, on hy-memory 2.0.13) and update the v1.0.6
news lines.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 21:03:22 +08:00
alvinfeiandClaude Opus 5.5 6c235ef093 release: hy-memory 2.0.13
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 17:30:34 +08:00
feitianxiang 2483412475 Merge pull request #281 from Tencent-Hunyuan/fix/sdk-write-failures-not-silent
fix(sdk): a model failure during a write reports failure and stores nothing
2026-11-02 17:29:10 +08:00
feitianxiang f9ee983790 Merge pull request #278 from Tencent-Hunyuan/fix/queue-wal-and-recall-pools
fix(memhub): memory writes retry themselves, a broken model provider pauses the queue, WAL stays bounded, recall no longer starves
2026-11-02 17:28:56 +08:00
alvinfeiandClaude Opus 5.5 4746681efe fix(sdk): coding reconcile model failures fail the write, nothing half-applied
Experience reconcile no longer falls back to ADD-all, and fact reconcile no
longer inserts without superseding, when the model call fails, returns empty
content, or returns a reply that is not a JSON object with an ops list. The
write returns success=false, agent_error_code=RECONCILE_FAILED and the
cause in reconcile_error_code (LLM_UNAVAILABLE / EMPTY_RESPONSE /
JSON_PARSE_FAILED), the same as chat.

Each reconciler is split into plan() (retrieve + model decision, no store)
and apply() (store). CodingWriter plans both layers before applying either,
so a decision failure in either layer stores no experience or fact and
supersedes no older fact. reconcile() still runs plan + apply.

An experience reply of {"ops": []} is a valid "no changes" decision and
stores nothing. A non-empty ops list with no recognizable op is a parse
failure. A fact reply of {"ops": []} inserts without superseding.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 17:01:47 +08:00
alvinfeiandClaude Opus 5.5 a218a92d4e fix(memhub): memory writes retry themselves; a broken model provider pauses the queue instead of losing jobs
The SDK keeps one outcome per request id and replays it: a write that failed
under an id fails under it forever, without calling the model. Re-running a
job under its idempotency key (including the old manual retry) therefore never
reached the model again. Every re-run now gets an SDK id of its own:
`idempotency_key` on attempt 0 (unchanged for existing jobs), then
`idempotency_key#rN`, N stored on the job (`sdk_attempt`, migration 61). Before
a re-run the outcome of the previous id is checked; a write that did succeed is
completed from its receipt and not made twice.

- Provider failure (401/403, 404, 408, 429, 5xx, timeout, connection error,
  breaker refusal) during the attempt: the job goes back to `pending` as it
  was: same place in the queue, no attempt or retry spent, no cap. The breaker
  paces it. Attribution: `ModelBackoff.attempt()` charges every provider
  failure to every write in flight, because a request cannot be traced to its
  job (the SDK batches embeddings on one shared worker).
- Any other failure: retried after 30 s, 60 s, 120 s, then `failed` with its
  code; the queue moves on.
- Crash or interruption mid-write (`unknown`): re-run automatically under a
  new id after the outcome check, counted as a retry. Existing `unknown` jobs
  are handled the same way after upgrade, once. A write still running on its
  thread in this process is left until it ends.
- The manual retry (`POST /api/v1/memory-processing/retry-unknown`, the App's
  重新整理 button and its strings) is removed.
- App banner while the provider is paused, in 11 locales; `error_code` and
  `next_retry_at` on `GET /api/v2/memory-writes/{id}` and
  `GET /api/v2/memories/additions/{id}`; `memhub status` shows the pause.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:58:23 +08:00
alvinfeiandClaude Opus 5.5 6845adf083 fix(memhub): WAL stays bounded, disabled sources suspend their jobs, recall no longer starves behind stuck writes
WAL (#245): the writer sets journal_size_limit (64 MiB), Core runs a
TRUNCATE checkpoint when the main database is idle (every minute, with a
50 ms busy wait, skipped rather than raised when busy) and at shutdown, and
the size alert judges pending frames rather than the file's size.

Disabled sources (#245): jobs of a source that is off or revoked are
suspended. Rows keep their state; status counts their waiting jobs as
`suspended` and queue depths leave them out. The claim gate already skips
them, so turning the source back on resumes them.

Gate agreement: a job the execute gate hands back (release_unstarted) is
held for 5 s through its lease, so a persistent disagreement between the
claim gate (lane user) and the execute gate (Evidence user) cannot spin.

Recall (#231): SDK writes run in their own bounded pool (12), recall's SDK
reads in theirs (8). The recall route has a total budget (timeout + 3 s,
at most 12 s) and answers 504 recall_timeout. The broker maps a Core
timeout to 504 and the client maps 504/ECONNABORTED/ETIMEDOUT to
broker_timeout. retry-unknown requeues off the event loop (#162).

Model backoff: a process-wide breaker per model profile revision records
every provider exchange through the SDK transport. After a provider failure
(auth, not found, rate limit, timeout, 5xx, unreachable) the drain claims no
jobs for 5 s, then one probe; each failed probe doubles the pause up to
3 minutes. The first success, or any model profile create, save, activation
or embedding switch, resets it. memory-processing reports
llm_backoff_until and llm_backoff_error.

Fixes #245, Fixes #231, Refs #162

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:58:23 +08:00
alvinfeiandClaude Opus 5.5 76098cf060 fix(sdk): a model failure during a write reports failure and stores nothing
Reconcile (add / add_extracted): when the reconcile model call fails,
returns an empty 200 reply or an unparseable reply, or reconcile raises
before anything is stored, the write no longer falls back to adding every
extracted memory as-is with success=true. The ADD-all fallback and its
degraded flags are removed. The write returns success=false,
agent_error_code=RECONCILE_FAILED and the cause in reconcile_error_code
(LLM_UNAVAILABLE / EMPTY_RESPONSE / JSON_PARSE_FAILED /
RECONCILE_EXCEPTION). Nothing new is stored, the episode counter does not
advance, and add rolls back its L1 raw (retry_safe=true), so a re-run with
a new request_id is clean. Only a literal [] still means "no changes".
RECONCILE_ENABLED=false keeps writing directly.

Working / coding path: an extraction model failure (LLM_ERROR,
EMPTY_RESPONSE, JSON_PARSE_FAILED) and an experience or fact embedding
failure (EMBEDDING_FAILED) now return success=false instead of success
with 0 ops or a node without a vector. Embeddings are computed before the
SQLite row is written and before an older fact is superseded. A failed
write does not append its trace or record chain rounds. A model that
answers with nothing to remember is still a success.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:51:45 +08:00
feitianxiang 850c728c9d Merge pull request #280 from Tencent-Hunyuan/fix/multi-user-hardening
fix(memhub): multi-user responses echo user_id, cross-user delete answers 404, Hermes profiles isolated for any root
2026-11-02 16:44:04 +08:00
alvinfeiandClaude Opus 5.5 ce9801b449 fix(memhub): resolve a caller's Hermes root through the grant's own helper
The overlap check resolved the caller-supplied root a second time in new
code. Resolve it with the same helper the grant uses (`resolved_source_root`
in access.py); the Hermes check now compares an already-resolved root and
only resolves other users' stored grant roots itself. A root that cannot be
resolved is left to the grant that follows, which refuses it with its own
code; the `profiles` container is still refused by name before that.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:32:17 +08:00
alvinfeiandClaude Opus 5.5 576d9bf408 fix(memhub): multi-user responses echo user_id, cross-user delete answers 404, Hermes profiles isolated for any root
Hardening of the multi-user feature from #250, after review.

- Read responses carry `user_id`, the user actually read (`local` when none
  was named): memory-library, memory-library/layers, user-profile,
  home-summary, memories/graph, raw, context/search, v1/recall and
  memories/ask. ask/stream puts it on its first (`recall`) event. The
  contracts types gain it as optional, and the Host Bridge context validator
  accepts it as an optional key, so an older Core still validates.
- DELETE /api/v2/memories/{id}: 204 only when the caller's own memory is
  deleted; another user's memory, a missing id, and an SDK result with
  deleted_count 0 (a delete that lost a race) all answer the same 404
  `memory_not_found`. The App treats it as already gone; `memhub memories
  delete` prints that it was not found, exits 0, and its JSON is committed
  with `already_gone: true`.
- Hermes skips a `profiles` directory at any depth (discovery helper:
  `excluded_directory_names`), so a root above ~/.hermes never reads another
  user's profile. Connect (other-user root, local connect and authorize)
  refuses a root that is the `profiles` container (422
  `hermes_root_is_profiles_container`) and a root equal to, containing or
  inside another user's active Hermes grant, symlinks resolved (409
  `hermes_root_overlaps_other_user`); `local` on ~/.hermes with another user
  on ~/.hermes/profiles/<name> stays allowed. The CLI explains both codes.
- `sources connect --user-id` with an invalid id says the user id need not
  equal the Hermes profile name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 16:26:20 +08:00
feitianxiang 34f406e96d Merge pull request #277 from Tencent-Hunyuan/feat/memhub-hy-memory-2.0.12
feat(memhub): hy-memory 2.0.12; intention dedup switch in memory features
2026-11-02 15:16:36 +08:00
alvinfeiandClaude Opus 5.5 a653bd4065 feat(memhub): intention dedup switch in memory features
Expose hy-memory 2.0.12's per-client intention dedup in the memory-features
setting: the feature intention_dedup (recall.intention_dedup_enabled, default
on) and the parameter intention_dedup_threshold
(recall.intention_dedup_threshold, default 0.95, accepted 0.8-1.0, else 422).
Both are part of the features key, so a change rebuilds the SDK clients like
the other switches. CLI: memhub settings features set intention_dedup=false,
intention_dedup_threshold=0.9. Refs #241.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 15:04:24 +08:00
alvinfeiandClaude Opus 5.5 07b426ba7f chore(memhub): pin hy-memory 2.0.12
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 15:04:24 +08:00
feitianxiang 4b83b79b2b Merge pull request #275 from Tencent-Hunyuan/release/hy-memory-2.0.12
release: hy-memory 2.0.12
2026-11-02 14:58:42 +08:00
alvinfeiandClaude Opus 5.5 675cc30e5a release: hy-memory 2.0.12
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 14:30:15 +08:00
feitianxiang 3511153f9f Merge pull request #274 from Tencent-Hunyuan/fix/deepseek-harness-profiles
fix(memhub): DeepSeek Harness one switch grants every profile; import versioned sessions and v3 tool text
2026-11-02 14:17:44 +08:00
feitianxiang 23facf75d5 Merge pull request #273 from Tencent-Hunyuan/fix/single-runtime-quit
fix(memhub): one runtime; App quit never stops the background service, stops every runtime when it is disabled
2026-11-02 14:17:32 +08:00
alvinfeiandClaude Opus 5.5 384c7476ad fix(memhub): adopt a newer runtime silently and let quit follow the service state
The owner's decision for a runtime with a higher build than the App: startup
neither fails nor shows anything. The App adopts the runtime and never stops,
downgrades or duplicates it at startup. Quit gives it no special pass, so it
follows the same rules as any adopted runtime: preserved when the service is
enabled or its state is unknown, stopped when the service is disabled. The
now-unused "后台版本比 App 新" message is removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 14:07:57 +08:00
alvinfeiandClaude Opus 5.5 71b8f9be24 Merge github/main into fix/deepseek-harness-profiles
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 14:04:40 +08:00
feitianxiang d257188d09 Merge pull request #270 from Tencent-Hunyuan/fix/commit-validation-and-zh-language
fix(memhub): one commit format at the packaged-runtime boundary; zh-Hans UI sets the SDK language
2026-11-02 14:03:21 +08:00
feitianxiang bde0220373 Merge pull request #268 from Tencent-Hunyuan/feat/sdk-intention-dedup-switch
feat(sdk): configurable intention dedup (on by default, unchanged behaviour)
2026-11-02 14:03:04 +08:00
alvinfeiandClaude Opus 5.5 a88d227cc0 fix(memhub): one DeepSeek Harness switch for every detected profile
The app page assumed one source maps to one host. Harness has none without a
profile, so the switch was greyed out with `unsupported_host` and an installed
profile stayed `install-only` with no way to approve it (#230).

The one switch now covers every `deepseek-harness:<profile>` through the
existing install, grant, revoke and uninstall APIs. On installs and grants each
profile that is not approved, off removes and revokes each linked one. A
profile that fails is named and the others keep their result. The page lists
each profile with its state.

New profiles are connected while the switch is on. The desktop re-reads the
profile list on each host refresh, and the remembered apply gets one more
attempt per new profile. The Runtime re-reads the disk for an unknown profile
id: its list was fixed at start and it outlives the App, so one new profile
made the host list fail for every app.

Rewrite the harness doc to match, removing the sources page it described,
which is not mounted. Add the strings to all 11 locales.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 13:11:19 +08:00
alvinfeiandClaude Opus 5.5 059125a09f fix(memhub): one runtime; App quit never stops the background service, stops every runtime when it is disabled
Quit no longer trusts what startup saw. `transientRuntime` and
`preserveManagedRuntimeOnQuit` were derived from the launch path, so an
already-running transient Runtime was treated as a managed service and
outlived the App (#237, #238, #239).

- queryRuntimeServiceState asks launchd/systemd at quit time, every time.
  Enabled: a LaunchAgent declaring this data dir is installed, there is no
  `launchctl disable` override, and the enable marker ("Launch at login")
  is present. A failed or timed-out query is `unknown`.
- Quit: enabled -> never stop. Disabled -> stopAllRuntimeProcesses stops
  every Runtime of the data dir (App child, CLI-started, leftover), each
  pid verified via the receipt and the authenticated identity endpoint; an
  unverifiable lease owner is reported, never signalled. Unknown -> stop
  nothing and log it.
- Startup reconciles a running Runtime by build: the Runtime now publishes
  build_number/git_commit in runtime-service.json. Same build is adopted;
  an older service job is replaced through launchd (installUserService
  kickstart/re-bootstrap), an older transient/CLI one is stopped gracefully
  and restarted; a newer one is never downgraded or stopped; an unknown
  build is adopted and never stopped. A stray beside an enabled service is
  stopped so exactly one Runtime remains.
- The tray and the login preference re-read the real state on each refresh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:57:21 +08:00
alvinfeiandClaude Opus 5.5 0d0cefb25e fix(core): leave no cursor on a DeepSeek Harness session it refuses
An unknown session version was skipped record by record. Each skip carried a
cursor and the file ended with a cursor touch, so the refused session sat
behind a checkpoint: the release that learned the format would read nothing,
turning "not imported yet" into "never imported". The skip was also not
reported, since `report_issue` defaults to false.

Refuse the file from its header instead, with the cursorless
`ScanFailure(UNSUPPORTED_SCHEMA)` other sources already use. The parser check
stays for callers that hand it records directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:55:17 +08:00
alvinfeiandClaude Opus 5.5 be8cf028ff test(core): pin DeepSeek Harness text tool output and session name boundary
Cover the #175 tool-output half directly: a `tool/result` written as plain
`text` blocks imports next to the older `tool-result` block shape, goes
through the same credential scrubbing and clipping, and names outside
`session[.vN].jsonl[.zstd]` stay out. Correct the contract-suite comments that
still said only `session.jsonl` was accepted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:47:17 +08:00
alvinfeiandClaude Opus 5.5 d0a4fd60f2 fix(memhub): pass every UI language to the SDK, zh-Hans included
The owner's decision for #243: a zh-Hans UI now sets
MemoryConfig.language="zh-Hans" like every other UI language, so App users
never fall back to the SDK's zh/en auto-detect. Only an unset language
(CLI/API-only installs, or 'settings language system') keeps auto-detect.
status --json pipeline.sdk_language shows zh-Hans for zh-Hans users.

Desktop now syncs the language the GUI shows (snapshot.locale), not the raw
preference: an App that follows the system sends the resolved locale
(e.g. zh-Hans on a Chinese macOS) instead of null.

Refs #243 (MemHub passes the language explicitly; the SDK's System 2
auto-detect is left unchanged for direct SDK users)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:35:25 +08:00
alvinfeiandClaude Opus 5.5 aae877b880 fix(memhub): accept only a full 40-character commit at the packaged-runtime boundary
resolvePackagedCommit accepted 7-40 hex characters in any case, while every
downstream check (packaged-runtime install, local install, release download,
release scripts, after-pack) requires 40 lowercase hex characters. It now
accepts exactly 40 hex characters, case-insensitive, returns them lowercased,
and skips a short or malformed value so a bad ONE_MEMORY_COMMIT no longer
shadows the embedded commit. The tsup build fails on a malformed supplied
ONE_MEMORY_COMMIT instead of baking it.

Fixes #236, Fixes #235

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:35:14 +08:00
alvinfeiandClaude Opus 5.5 ef5ff06590 fix(sdk): intention dedup switch also gates write-batch dedup
When intention_dedup_enabled is off, no intention dedup should run, but
the writer's batch dedup still dropped near-duplicate intentions from
one extraction. Gate it on the same switch. Its threshold stays on
MEMORY_DEDUP_THRESHOLD.

Also skip the embedding backfill when dedup is off, since only dedup
uses it, and fall back to the env default on an invalid configured
threshold so a bad value cannot fail the read.

Refs #241

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:33:56 +08:00
ryanlandClaude Opus 5.5 7eb38aa6ad fix(core): import versioned DeepSeek Harness session files
DSH names a session file after its format generation
(`session.v3.jsonl.zstd`, `session.v4.jsonl.zstd`), but discovery matched only
the two unversioned names, so import jobs completed with `discovered 0` while
the sessions sat on disk.

Discover `session[.vN].jsonl[.zstd]` and gate parsing on the header `version`
field: verified generations (0/3/4) import, anything newer is skipped as
`unsupported_schema` rather than interpreted with today's field rules. v4 tool
results use plain `text` blocks, so read that shape through the same clipping
and credential-scrubbing path as the `tool-result` blocks used by v0/v3.

(cherry picked from commit 2a65e5ab2d8548cfb8d6f72a4b68b702f6b4a3cc; conflicts in CHANGELOG.md and the Core test inventory resolved against main)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:32:02 +08:00
alvinfeiandClaude Opus 5.5 2624228f6f feat(sdk): configurable intention dedup (on by default, unchanged behaviour)
Add MemoryConfig.recall.intention_dedup_enabled (default true, env
MEMORY_INTENTION_DEDUP_ENABLED) and recall.intention_dedup_threshold
(env MEMORY_INTENTION_DEDUP_THRESHOLD, falling back to the shared
MEMORY_DEDUP_THRESHOLD, then 0.95). The reader passes both from the
client config to recall_intentions. When off, read-time intention dedup
is skipped entirely: nothing is hidden and nothing is deleted.

Refs #241

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 12:25:37 +08:00
feitianxiang 670c7c6cb1 Merge pull request #267 from Tencent-Hunyuan/docs/release-1.0.6
docs: prepare MemHub v1.0.6 (news, changelog)
2026-11-02 11:50:58 +08:00
alvinfeiandClaude Opus 5.5 aa633623da docs: MemHub 1.0.6 news and changelog
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 11:46:06 +08:00
feitianxiang 37d8662b93 Merge pull request #266 from Tencent-Hunyuan/chore/build-37
chore(release): MemHub 1.0.6, build 37
2026-11-02 11:44:12 +08:00
alvinfeiandClaude Opus 5.5 9459a1107b chore(release): MemHub 1.0.6, build 37
Sets every MemHub package, the Core project and its lock entry to 1.0.6
with set-release-version.mjs, and the build number to 37. The release
facts in memhub/README.md and docs/INSTALLATION.md now name v1.0.6
(build 37).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 11:36:55 +08:00
feitianxiang 2c6f269170 Merge pull request #264 from Tencent-Hunyuan/chore/pin-hy-memory-2.0.11
chore(memhub): pin hy-memory 2.0.11 and set the SDK language directly
2026-11-02 11:34:45 +08:00
alvinfeiandClaude Opus 5.5 7a2ff2f4c4 build(memhub): pin hy-memory 2.0.11 and set the SDK language directly
Core pins and locks the published hy-memory 2.0.11 from PyPI, which adds
MemoryConfig.language and the prompt registry. apply_sdk_language now sets
the language directly; the guard that skipped it on an SDK without the
field (and its debug log) is gone, and the tests use the real field. The
version constant, README, architecture notes, SBOM, notices and their tests
follow. CHANGELOG records 2.0.11 as published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 22:22:32 +08:00
feitianxiang 35128f2c84 Merge pull request #262 from Tencent-Hunyuan/feat/memhub-eleven-languages
feat(memhub): 11 display languages; non-zh/en choices set the SDK prompt language
2026-11-01 22:17:21 +08:00
feitianxiang f12a95ce7a Merge pull request #263 from Tencent-Hunyuan/release/hy-memory-2.0.11
release: hy-memory 2.0.11
2026-11-01 22:13:44 +08:00
alvinfeiandClaude Opus 5.5 2b75adc718 release: hy-memory 2.0.11
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 21:55:18 +08:00
feitianxiang 75ac1b01f2 Merge pull request #261 from Tencent-Hunyuan/feat/sdk-multilingual-prompts
feat(sdk): prompts in 11 languages and an explicit language option
2026-11-01 21:53:38 +08:00
alvinfeiandClaude Opus 5.5 16da11f788 feat(sdk): prompts in 11 languages and an explicit language option
Move every LLM prompt that had a zh variant into one registry,
hy_memory/prompts/, with one module per language (en, zh-Hans, zh-Hant,
ja, es, de, pt, fr, ko, ru, id). Call sites look prompts up with
get_prompt(name, language), so a new language needs only a new module.
The en and zh-Hans strings are moved verbatim, a test pins their sha256,
and the old names (prompts_zh.*_ZH, extractor.EXTRACT_SYSTEM_PROMPT, ...)
still import.

Add an optional language: MemoryConfig.language (env MEMORY_LANGUAGE)
as the client default, plus a per-call language= on add / async_add,
add_extracted, finalize_session, evolve and sweep. It goes through
extract and basic profile, reconcile and search query, the coding
extractor and both coding reconcilers with the chain summary, the
episode and topic summaries, and System 2. Codes are validated, with
aliases (zh, zh-CN, zh-TW, zh-HK, pt-BR); an unknown code raises
ValueError. With an explicit language, memories are written in that
language. Without one, the zh/en auto-detection is unchanged. The
server accepts language on /add and /add_extracted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 21:29:59 +08:00
367 changed files with 64219 additions and 9718 deletions

No files matched your search

+1
View File
@@ -78,6 +78,7 @@ MEMORY_VECTOR_STORE=chroma
# ── 处理模式 ──
# MEMORY_MODE=pro # lite | pro | ultra
# MEMORY_LANGUAGE= # 提示词/记忆语言:en | zh-Hans | zh-Hant | ja | es | de | pt | fr | ko | ru | id;空=zh/en 自动检测
# ── Coding Memory(生产力 / 编码场景独立链路,默认开启)──
# 独立于 chat 记忆的一套链路:写入端对 messages 判类,coding 场景抽「任务→方案」存 sqlite;
+244
View File
@@ -4,6 +4,250 @@ Hy Memory and MemHub are versioned and released independently.
## Unreleased
## Hy Memory 2.0.14 — 2026-11-04
- `search()` / `async_search()` take an optional `statuses` for explicit
L1_RAW search (`layers=["l1_raw"]`, or any search in lite mode), e.g.
`statuses=["active", "shadow"]`, so raw turns a pro extraction already
shadowed can still be searched. Default stays active-only. Raw hits now
report their real `status`. Passing `statuses` to a non-raw search or to
`scene="all"` raises `ValueError` instead of being ignored.
## Hy Memory 2.0.13 — 2026-11-02
- A model failure during a write now reports failure and stores
nothing. **Behaviour change for direct SDK users:** writes that used to
return `success: true` in a degraded state now return `success: false`;
retry them with a new `request_id` once the model is back.
- Reconcile (pro / ultra `add` and `add_extracted`): when the reconcile model
call fails, returns HTTP 200 with empty content, or returns a reply that
cannot be parsed, or reconcile raises before anything is stored, the write
used to fall back to adding every extracted memory as-is (no dedup or
evolution) and report success. That fallback is removed, with its
`degraded` / `degraded_reason` result fields. The write now returns
`success: false`, `error_code: 502`,
`agent_error_code: "RECONCILE_FAILED"`, and the cause in the new
`reconcile_error_code` (`LLM_UNAVAILABLE`, `EMPTY_RESPONSE`,
`JSON_PARSE_FAILED`, `RECONCILE_EXCEPTION`). None of the new memories are
stored, the episode counter does not advance, and `add` rolls back its
raw turn the same way as an extraction failure
(`l1_rollback.retry_safe: true`). An empty 200 reply was previously
counted as "no changes" and reported success with 0 memories; only a
literal `[]` reply still means that. `RECONCILE_ENABLED=false` is a
configuration choice and still writes directly.
- Working / coding path: an extraction model failure used to return
success with 0 operations. It now returns `success: false` with
`agent_error_code` `LLM_ERROR`, `EMPTY_RESPONSE` or `JSON_PARSE_FAILED`.
A model that answers with nothing worth remembering is still a success.
An embedding failure used to store the experience or fact without a
vector, where search never finds it, and report success; it now returns
`EMBEDDING_FAILED` and the node is not stored. A fact embedding is checked
before any older fact is superseded. A failed write no longer appends its
turns to the recent trace and records no `chain_rounds`.
With `MEMORY_CODING_VERBATIM_FALLBACK_ENABLED` turned on, extraction
failures keep the verbatim fallback.
- Working / coding reconcile: when the experience reconcile or the fact
reconcile model call fails, returns empty content, or returns a reply that
is not a JSON object with an `ops` list, the write returns
`success: false`, `agent_error_code: "RECONCILE_FAILED"` and the cause in
`reconcile_error_code` (`LLM_UNAVAILABLE`, `EMPTY_RESPONSE`,
`JSON_PARSE_FAILED`), the same as chat. Before, experience reconcile fell
back to adding every draft, and fact reconcile inserted the new fact
without superseding older ones. Both model decisions now run before
anything is stored, so a failure in either leaves no experience or fact
from the write and supersedes no older fact. An experience reply of
`{"ops": []}` is a valid "no changes" decision: the write succeeds and no
experience is stored. A non-empty `ops` list with no recognizable op is a
parse failure. A fact reply of `{"ops": []}` inserts the new fact without
superseding anything. Fact reconcile still makes no model call when there
are no candidates.
## Hy Memory 2.0.12 — 2026-11-02
- Intention dedup is now configurable per client
(`MemoryConfig.recall.intention_dedup_enabled`, default `true`;
`recall.intention_dedup_threshold`, default `MEMORY_DEDUP_THRESHOLD` or
`0.95`). Env defaults: `MEMORY_INTENTION_DEDUP_ENABLED`,
`MEMORY_INTENTION_DEDUP_THRESHOLD`. When on, behaviour is unchanged
(duplicates found at read time are still deleted); when off, no intention
dedup runs, at read time or within a write batch, and nothing is hidden,
dropped or deleted. The threshold applies to read-time dedup only. Refs #241.
- MemHub pins `hy-memory==2.0.12` and exposes that intention dedup switch in
its memory-features setting (`GET`/`PUT /api/v1/settings/memory-features`,
`memhub settings features`): the feature `intention_dedup` (default on) and
the numeric `intention_dedup_threshold` (default 0.95, accepted 0.8–1.0,
anything else is 422 or a CLI usage error). Unset, both keep the SDK
default, so nothing changes for the App; a change rebuilds the SDK clients
like the other feature switches. The API answer gains `parameters`, and
`pipeline.features` can now carry the threshold as a number. Refs #241.
## MemHub 1.0.6 — 2026-11-04
Build 41, on `hy-memory==2.0.14`. Linux x64/arm64 Runtime and macOS arm64
App and Runtime.
- MemHub App now exposes the complete memory-mode lifecycle. New installs
explicitly choose lite or pro during onboarding; pro completes only after a
generation model is active, while ultra remains an advanced Settings choice.
Settings shows mode effects, graph backfill, the default 03:00 evolve
schedule and manual evolve. The memory page shows original conversations in
lite and advanced memories in pro/ultra. Existing installs remain pro unless
the user changes them, and embedding maintenance blocks mode/evolve changes.
- MemHub pins `hy-memory==2.0.14`. Lite search is plain RAG over raw: the
App's lite search, CLI context search, `/api/v1/recall` and the Host Bridge
(`MemHub_recall`, context search) return every vector hit above the floor,
ranked by score, over active and shadow raws, so history pro already
processed is still found after switching to lite. The turn view no longer
drops a hit whose turns do not literally contain the query; it shows the
hit's closest turn instead. Pro and ultra raw search stays active-only.
Fixes every raw search failing in packaged builds since #316, which passed
`statuses` to an SDK (2.0.13) that did not accept it. The lite raw-turn
page also stops re-listing the whole raw layer on each poll when a live
session only adds new raws.
- MemHub Core no longer stalls writes or freezes the App for tens of seconds
on a machine with a long live-sync history. Finished live-sync import jobs
are pruned to the newest 20 per source (at startup, in bounded batches, and
every 10 minutes); history imports and every unfinished job are kept. Job
lookups are filtered in SQL with new indexes (schema migration 63) instead
of reading the whole table. The durable write queue runs its claims,
settles and lease renewals on its own threads, so import work can no longer
starve a lease. Import abandon, resume and enqueue, and the file watcher's
path attribution, no longer touch the database or the disk on the event
loop, and the WAL cadence checkpoint runs on its own thread instead of on
the writer's. The write drain now restarts itself after an unexpected exit,
reports the failure with its stack, and shows `write_drain` /
`write_drain_error` in memory-processing and `memhub status`.
- Model Center (#293, #295): Settings splits into 界面语言 / 记忆模式 /
模型配置 pages, with separate 记忆整理 (generation) and 记忆搜索
(embedding) lanes. Changing one lane sends only that lane's patch. An
embedding change shows the switch plan, confirmation, progress, cancel and
rollback, and the search model only changes after the rebuilt index is
verified. Model panels open inside the lane that launched them and close
only after Core confirms the profile is active (#310).
- Lite keeps raw conversations (#297): lite raw writes are no longer stopped
by the "pause model sedimentation" switch, the memory page lists every
imported raw conversation by role (by turn since #316), and lite adds a
manual raw input (source `one-memory-manual`) that never calls a
generation model.
- The home page follows the memory mode (#298): lite counts raw
conversations and previews the latest one; pro and ultra read memories.
Pro and ultra now require an active generation model, in both the mode
preview and the switch.
- Onboarding (#304) asks for lite or pro and resumes after a restart (schema
migration 62); pro finishes only once a generation model is active.
- Smaller App fixes: compact home status copy (#307), clearer mode-switch
effects (#312), Ultra topic nodes are hover-only (#315), and settings
controls use the content width (#317).
- Interfaces in 11 languages: English, Simplified and Traditional Chinese,
Japanese, Spanish, German, Portuguese, French, Korean, Russian and
Indonesian. Every interface language, Simplified Chinese included, sets the
SDK prompt language (#270). **User-visible:** zh-Hans users now get Chinese
prompts and Chinese memory text for every conversation, English ones
included; the App sends the language it displays, also when it follows the
system. Only an unset language (CLI/API installs, `memhub settings language
system`) keeps the SDK's zh/en detection.
- Memory modes: lite (no LLM, keeps raw conversations), pro (LLM extraction)
and ultra (pro plus the knowledge graph and a nightly evolve run). Set with
`GET`/`PUT /api/v1/settings/memory-mode` or `memhub settings mode`. Raw
conversations are now kept permanently. `POST /api/v2/evolve` and
`memhub evolve` run System 2 on demand.
- Safe embedding-model switching: an embedding-space registry, a switch that
pauses writes, re-embeds every user's vectors and the graph, verifies and
commits atomically, and resumes after a crash or an App update. Every start
scans the vector store and merges data left in other spaces. When the
embedding service is unavailable, search returns `embedding_unavailable`.
- Multi-user Runtime: memory APIs and the CLI take `user_id` (default
`local`); records are isolated per user; Hermes profiles can be connected as
separate users. The App keeps using `local`. Hardening (#280): read
responses echo `user_id`; deleting another user's memory, or one that does
not exist, is 404 `memory_not_found` (the App and `memhub memories delete`
treat it as already gone); Hermes never reads a `profiles/` directory, and
a root that is the `profiles` container or overlaps another user's grant is
refused (`hermes_root_is_profiles_container`,
`hermes_root_overlaps_other_user`).
- Configurable SDK features through the CLI and API (`memhub settings
features`): graph rerank, episode recall, session episodes, episode
summaries, intention and few-shot, and the intention dedup switch with its
threshold (`intention_dedup`, `intention_dedup_threshold`, #277). Defaults
are unchanged.
- Memory writes retry themselves (#278), each time under a new SDK request
id after checking whether the previous one landed; interrupted writes
re-run automatically. A model provider failure (401/403, 404, 408, 429,
5xx, timeout, unreachable) keeps the job pending in place without spending
a retry, and a per-profile circuit breaker paces the queue (5 s doubling to
180 s, one probe job at a time). Any other failure is retried after 30, 60
and 120 s, then `failed`. With `hy-memory==2.0.13` a model failure is no
longer a silent success, and a failed job's `error_code` names the SDK's
cause (`hy_sdk_reconcile_empty_response`, `hy_sdk_llm_error`, ...) rather
than `hy_sdk_write_failed`. **User-visible:** the manual 重新整理 button,
the "整理结果不确定" display and
`POST /api/v1/memory-processing/retry-unknown` are gone; a banner
("模型配置有问题……记忆暂停处理,已有 N 条在排队") with a button to the model
settings shows while a model problem pauses processing, and clears on
recovery.
- Queue, WAL and recall (#278): the WAL stays bounded (64 MiB journal limit,
idle `TRUNCATE` checkpoints); a disabled or revoked source's jobs are
`suspended` and resume when it is turned back on; SDK writes and recall
reads use separate thread pools, and recall has a total budget (504
`recall_timeout`, `broker_timeout` in clients) instead of starving behind
stuck writes.
- One Runtime per data dir (#273). App quit asks the service manager at quit
time: an enabled background service is never stopped, a disabled one has
every Runtime of the data dir stopped (only after an authenticated identity
check), and an unknown state stops nothing. At start the App adopts a
Runtime of the same build, replaces an older one, and silently adopts a
newer one. The tray quit label follows the current state.
- DeepSeek Harness: the app page's one switch installs and grants recall for
every detected profile, and keeps new profiles granted without a restart
(#230); versioned session files (`session.v3.jsonl.zstd`,
`session.v4.jsonl.zstd`) are imported, and v4 tool output goes through the
same clipping and scrubbing (#175).
- The packaged Runtime accepts only a full 40-character commit, lowercased; a
malformed `ONE_MEMORY_COMMIT` fails the build (#235, #236).
- Search and recall pass through the SDK's search parameters with server-side
score floors; the memory library filters by layer; `/api/v2/raw` returns
data again; v2 source and import routes; profile version history.
- Upgrade fixes: migration no longer stops without a model profile (#234) or
on a pre-existing readback mismatch (#229, #240); source grants work after
the cutover (#233); reads use the active embedding (#242); the embedding
endpoint no longer follows the generation `base_url`.
- Hermes import drops gateway notices and probes and unwraps relayed
messages; `memhub logs` no longer reports valid older records as damaged.
## Hy Memory 2.0.11 — 2026-11-01
- Published
[`hy-memory==2.0.11`](https://pypi.org/project/hy-memory/2.0.11/).
- LLM prompts now ship in 11 languages: `en`, `zh-Hans`,
`zh-Hant`, `ja`, `es`, `de`, `pt`, `fr`, `ko`, `ru` and `id`. They live in
one registry, `hy_memory/prompts/`, with one module per language. Call
sites use `get_prompt(name, language)`, so a new language only needs a new
module. The registry covers chat extraction (system prompt, basic_info,
intentions, few-shot including Example 11), the no-intention variant,
reconcile and its search-query prompt, the working/coding extractor, both
coding reconcilers and the chain summary, the episode and topic summaries,
System 2 evolve, and the user name line. The English and Simplified Chinese
strings are unchanged byte for byte, and a test pins them. The old names
(`prompts_zh.*_ZH`, `extractor.EXTRACT_SYSTEM_PROMPT`, ...) still import.
The nine new languages are machine translations, and native-speaker review
is recommended.
- New optional `language` to choose the prompt language
explicitly. Set it on `MemoryConfig.language` (env `MEMORY_LANGUAGE`) for
a client default, or per call on `add` / `async_add`, `add_extracted`,
`finalize_session`, `evolve` and `sweep`. A per-call value overrides the
config. The value goes through every LLM call of the request: extract and
basic profile, reconcile, the coding path, the summaries and System 2.
Aliases: `zh`, `zh-CN` → `zh-Hans`; `zh-TW`, `zh-HK` → `zh-Hant`; `pt-BR` →
`pt`. An unknown code raises `ValueError` listing the supported codes, and
the HTTP server returns 400. With an explicit language, memories are
written in that language even when the conversation uses another one.
Without a language, the existing zh/en auto-detection runs unchanged and
memories keep the source language. `search` has no `language`: its judge
and rewrite prompts are English-only and keep the query's language.
- The HTTP server accepts `language` on `/api/v1/add` and
`/api/v1/add_extracted`.
## Hy Memory 2.0.10 — 2026-11-01
- Published
+7
View File
@@ -33,6 +33,7 @@ English | [中文](README.zh.md)
## Latest News
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** quantized **FP8** (**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**) and **GGUF** (**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**) versions of **Hy-Memory-30B-A3B** on Hugging Face!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-11]**: 📦 **Released [MemHub v1.0.6](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.6)** with interfaces in 11 languages, memory modes (lite / pro / ultra with nightly evolve), safe embedding-model switching, multi-user support for the CLI and API, memory writes that retry themselves and pause while the model provider is down, and more reliable upgrades, on **`hy-memory==2.0.14`**. See the release notes for details.
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **Released [MemHub v1.0.5](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.5)** with Chinese, English and Spanish interfaces, one-time pairing codes for more custom Harnesses, and smoother browsing, search, startup and recovery, on **`hy-memory==2.0.8`**. See the release notes for details.
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **Released [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** and **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**, with a signed macOS desktop app and verified macOS/Linux Runtimes!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.4`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.4/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
@@ -98,6 +99,12 @@ export OPENAI_API_KEY="sk-your-key"
See the [Environment Variable Reference](docs/env_reference.md) for the full list.
Prompts ship in 11 languages (`en`, `zh-Hans`, `zh-Hant`, `ja`, `es`, `de`, `pt`,
`fr`, `ko`, `ru`, `id`). By default the SDK picks Chinese or English from the
input. Set `MEMORY_LANGUAGE` (or `language=` on `add()`) to choose one
explicitly; memories are then written in that language. See
[Prompt language](docs/usage.md#提示词语言language).
### 3. Chat Memory
```python
+6
View File
@@ -33,6 +33,7 @@
## 最新动态
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **Hy-Memory-30B-A3B** 的 **FP8**(**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**)与 **GGUF**(**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**)量化版本模型!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-11]**: 📦 **正式发布 [MemHub v1.0.6](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.6)**:界面支持 11 种语言,新增记忆模式(lite / pro / ultra,ultra 每晚自动整理),可安全切换嵌入模型,CLI 和 API 支持多用户,记忆写入自动重试、模型服务异常时暂停排队,升级更可靠;底层使用 **`hy-memory==2.0.14`**。详见发布说明。
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **正式发布 [MemHub v1.0.5](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.5)**:支持中文、英文和西班牙语界面,可通过一次性配对码连接更多自定义 Harness,并优化了记忆浏览、搜索、启动和异常恢复;底层使用 **`hy-memory==2.0.8`**。详见发布说明。
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **正式发布 [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** 与 **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**,提供已签名的 macOS 桌面应用及经过验证的 macOS/Linux Runtime!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.4`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.4/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
@@ -98,6 +99,11 @@ export OPENAI_API_KEY="sk-your-key"
完整配置见 [环境变量参考](docs/env_reference.md)。
提示词提供 11 种语言(`en`、`zh-Hans`、`zh-Hant`、`ja`、`es`、`de`、`pt`、`fr`、
`ko`、`ru`、`id`)。默认按输入自动在中文与英文之间选择;设置 `MEMORY_LANGUAGE`(或在
`add()` 上传 `language=`)可显式指定,记忆随之用该语言书写。详见
[提示词语言](docs/usage.md#提示词语言language)。
### 3. Chat 记忆
```python
+21
View File
@@ -9,6 +9,7 @@
|---|---|---|
| `MEMORY_MODE` | `pro` | `lite` / `pro` / `ultra` |
| `MEMORY_DATA_DIR` | `~/.hy_memory` | 本地数据、缓存和日志根目录 |
| `MEMORY_LANGUAGE` | — | LLM 提示词与记忆正文语言:`en` / `zh-Hans` / `zh-Hant` / `ja` / `es` / `de` / `pt` / `fr` / `ko` / `ru` / `id`(别名 `zh`、`zh-CN` → `zh-Hans`,`zh-TW`、`zh-HK` → `zh-Hant`,`pt-BR` → `pt`)。未设置时按输入做 zh/en 自动检测;per-call `language=` 优先 |
- `lite`:纯 Embedding 写入,Graph 强制关闭。
- `pro`:LLM 提取与 reconcile;Graph 由 `MEMORY_ENABLE_GRAPH` 控制。
@@ -181,9 +182,29 @@ SDK 不设置 Rerank Search 并发上限。开启后模型在 `HyMemoryClient`
| `MEMORY_RECALL_STRENGTH_TAU` | `180` | 衰减时间常数,天 |
| `MEMORY_RECALL_ACCESS_TRACKING` | `true` | Strength 开启时是否回写访问信息 |
| `MEMORY_ENTITY_STORE_ENABLED` | `false` | 是否维护独立 entity collection;当前 Reader 不直接读取 |
| `MEMORY_INTENTION_DEDUP_ENABLED` | `true` | 是否对 L7 Intention 判重(读侧和写入侧批内);见下文 |
| `MEMORY_INTENTION_DEDUP_THRESHOLD` | `MEMORY_DEDUP_THRESHOLD`,再缺省为 `0.95` | Intention 读侧判重的 cosine 阈值 |
旧数据可调用当前公开方法 `client.build_entity_store(user_id)` 补建 Entity Store。
Intention 判重:开启时(默认,与以往行为一致),Search 召回的 L7 Intention 中,
同 `user_id` + 同 `agent_id` + `memory_at` 同一天 + `valid_until` 相同、且 cosine ≥ 阈值的
条目视为复本,只返回最早写入的一条,其余在后台从向量库**删除**。判重不区分内容中的项目、
对象等语义差异,内容高度相似但对象不同的意图可能被误删;写入时同一批抽取结果也会做一次
批内判重。关闭后这两处都不判重,不隐藏、不丢弃、也不删除意图。两项对应
`MemoryConfig.recall.intention_dedup_enabled` /
`recall.intention_dedup_threshold`,按 client 生效,环境变量只提供默认值:
```python
client = HyMemoryClient.from_config({
"recall": {"intention_dedup_enabled": False},
})
```
`MEMORY_DEDUP_THRESHOLD` 同时被写入侧批内意图判重和 dedup 内核使用;
`MEMORY_INTENTION_DEDUP_THRESHOLD` 只影响读侧 Intention 判重,写入侧批内判重的阈值
仍取 `MEMORY_DEDUP_THRESHOLD`。
## 日志、Trace 与历史
| 变量 | 默认值 | 说明 |
+1 -1
View File
@@ -107,5 +107,5 @@ L1_RAW 保存的内容使用上述渲染格式。历史回读时:
```bash
python -m pytest -q \
tests/test_multi_speaker_format.py \
tests/test_reconcile_fallback.py
tests/test_reconcile_failure.py
```
+17 -1
View File
@@ -150,7 +150,23 @@ working/coding 写入返回双层结果:
}
```
`scene="coding"` 还会返回 `chain_rounds`。出于历史兼容,该字段当前统计本次识别并处理
抽取 LLM 调用失败、返回空 content 或回复解析不出 JSON,以及经验 / 事实向量化失败时,
写入返回 `success=False`、`error_code=502`,`agent_error_code` 依次为 `LLM_ERROR`、
`EMPTY_RESPONSE`、`JSON_PARSE_FAILED`、`EMBEDDING_FAILED`,`error_message` 以
`[CODE]` 开头;没有向量的经验 / 事实不会落库,失败的写入也不追加对话 trace,换一个
`request_id` 重跑即可。模型正常作答但没有值得记的内容仍是 `success=True`、0 条。
显式开启 `MEMORY_CODING_VERBATIM_FALLBACK_ENABLED` 时抽取失败沿用原文兜底。
经验层 / 事实层 reconcile 的模型决策同样不再兜底:LLM 调用失败、返回空 content 或回复
不是带 `ops` 列表的 JSON 对象时,写入返回 `success=False`、
`agent_error_code="RECONCILE_FAILED"`,原因在 `reconcile_error_code`(`LLM_UNAVAILABLE` /
`EMPTY_RESPONSE` / `JSON_PARSE_FAILED`,与 chat 链路一致)。两层的模型决策都在落库之前
完成,任何一层失败时本次一条经验 / 事实都不存,旧事实也不会被覆盖。经验层回复
`{"ops": []}` 表示「无需任何操作」,写入成功、经验层不存;`ops` 非空却没有一条可识别的
op 视为解析失败。事实层回复 `{"ops": []}` 表示都不覆盖、照常插入新事实;没有候选时不
调用 LLM。
`scene="coding"` 还会返回 `chain_rounds`(写入失败时不记录演进链,也不返回该字段)。出于历史兼容,该字段当前统计本次识别并处理
的 diff hunk 数,而不保证等于最终合并后的唯一演进 round 数。`0` 表示没有识别到可记录
的代码修改,不代表双层 memory 写入失败。
+57
View File
@@ -103,6 +103,7 @@ result = client.add(
metadata=None,
meta=None,
embed_override=None,
language=None,
)
```
@@ -127,6 +128,14 @@ result = client.add(
双层 reconcile 操作、`experience_ids`、`fact_ids` 等字段,完整结构见
[scene_coding_usage.md](scene_coding_usage.md#4-返回结构)。
写入失败时 `success=False`,`error_code` 为 HTTP 风格的整数,`agent_error_code` 给出稳定
失败码。pro / ultra 的 reconcile(新事实与已有记忆判重、合并)模型调用失败、返回空
content 或回复解析不出时,`add` 与 `add_extracted` 都报 `agent_error_code="RECONCILE_FAILED"`,
具体原因在 `reconcile_error_code`(`LLM_UNAVAILABLE` / `EMPTY_RESPONSE` /
`JSON_PARSE_FAILED` / `RECONCILE_EXCEPTION`)。此时本次一条新记忆都不落库,`add` 已写入
的原始对话按抽取失败同样回滚(`l1_rollback.retry_safe=True`);模型恢复后换一个
`request_id` 重跑即可。`RECONCILE_ENABLED=false` 时不调用 reconcile,直接写入。
`mode="lite"` 不执行 LLM 抽取,写入内容直接以 `L1_RAW` 形式保存。lite 的 Search 是
严格的 raw-only 向量检索:无论调用方传入什么 `layers`、`scene` 或 `search_mode`,都只
召回 `L1_RAW`,不会进入 working/coding、Schema 或 Intention 链路,也不会调用场景判断
@@ -136,6 +145,54 @@ LLM。pro / ultra 仍遵循各自的层级与场景参数,默认不公开原
正交:前者说明“从哪个产品/入口写入”,后者仍是记忆隔离中的 agent 身份。source 会
执行 `strip().lower()` 后存储,并随 chat、working、coding 记忆一起返回。
### 提示词语言(`language`)
SDK 的 LLM 提示词提供 11 种语言,按语言码注册在 `hy_memory/prompts/`
(每种语言一个模块,`get_prompt(name, language)` 统一取用):
| 语言码 | 语言 | 别名 |
|---|---|---|
| `en` | English | `en-US`、`en-GB` |
| `zh-Hans` | 简体中文 | `zh`、`zh-CN`、`zh-SG` |
| `zh-Hant` | 繁體中文 | `zh-TW`、`zh-HK`、`zh-MO` |
| `ja` | 日本語 | |
| `es` | Español | |
| `de` | Deutsch | |
| `pt` | Português | `pt-BR`、`pt-PT` |
| `fr` | Français | |
| `ko` | 한국어 | |
| `ru` | Русский | |
| `id` | Bahasa Indonesia | |
语言码大小写不敏感,`_` 与 `-` 等价;未知语言码抛 `ValueError` 并列出支持的语言码。
```python
# client 默认值(也可用环境变量 MEMORY_LANGUAGE=ja)
client = HyMemoryClient.from_config({"mode": "pro", "language": "ja"})
# per-call 覆盖 client 默认值
client.add(messages, user_id="u1", language="de")
client.add_extracted(["Der Nutzer trinkt keinen Kaffee."], user_id="u1", language="de")
client.finalize_session(user_id="u1", session_id="s1", language="de")
client.evolve(user_id="u1", language="de") # ultra
client.sweep(user_ids=["u1"], language="de") # ultra
```
优先级:per-call `language` > `MemoryConfig.language`(`MEMORY_LANGUAGE`)> 自动检测。
- **未指定语言(默认)**:行为与以前完全一致。SDK 按输入做 zh/en 自动检测,中文输入用
简体中文提示词,其余用英文提示词;自动检测不会选到其他语言。提示词里的「输出语言与
输入语言一致」规则仍然生效,所以记忆正文保持对话的原始语言。
- **显式指定语言**:本次请求的全部 LLM 调用(抽取、基础画像、reconcile、working /
coding 链路的抽取与两个 reconciler 及演变总结、episode / topic 摘要、System 2)都使用
该语言的提示词,并在提示词末尾追加该语言的「输出语言」规则:记忆正文、摘要等自然语言
字段一律用该语言书写,即使对话是另一种语言。JSON 键、字段名、枚举值、代码、路径、
报错信息和专有名词保持原样;tags 仍是英文小写关键词。
- 只有英文版的提示词(session / dynamic episode、cross-domain sweeper)在显式指定语言时
同样追加该输出规则;搜索侧的场景判断 / query rewrite 只有英文版且保持 query 原语言,
因此 `search()` 不接受 `language`。
- 自定义 `extract_system_prompt_zh` / `extract_system_prompt_en` 只覆盖 `zh-Hans` / `en`。
### `search()` / `async_search()`
普通 chat 检索:
+38 -57
View File
@@ -43,6 +43,12 @@ from datetime import datetime
from typing import Any, Dict, List, Optional
import uuid
from ..prompts import (
get_prompt,
normalize_language,
resolve_language,
with_output_language,
)
from ..utils.source_filter import merge_sources
logger = logging.getLogger(__name__)
@@ -115,56 +121,9 @@ Messages:
"""
EPISODE_SUMMARY_PROMPT = """You are summarizing ONE episode of conversation with the user.
An episode is a single stretch of discussion — one thing that was talked about,
possibly across several sessions of writing.
## Memories captured during this episode (oldest → newest)
{memories}
---
Write a single dense summary of what happened / what was discussed in this episode.
Rules:
* 2–4 sentences, third person, starting with "The user ..." (or the named person
if the memories are about someone else).
* Tie the memories together into one coherent account — an episode usually has a
thread running through it (a plan, a problem, an outing, a decision).
* Preserve concrete details: names, places, numbers, dates.
* Do NOT invent anything that is not in the memories.
* Do NOT enumerate the memories one by one; synthesize them.
* Output language must match the language of the memories.
Return ONLY the summary text, no preamble, no JSON, no quotes."""
EPISODE_SUMMARY_PROMPT_ZH = """你在为**一段讨论**(一个 episode)做总结。
一个 episode 是一段连续的讨论 —— 围绕一件事,可能跨越好几次写入。
## 这段讨论中抽取到的记忆(时间正序)
{memories}
---
写一段关于这次讨论内容的高密度摘要。
要求:
* 2–4 句,第三人称,以"用户…"开头(若记忆讲的是别人,则以那个人的名字开头)。
* 把这些记忆串成一个连贯的叙述 —— 一段讨论通常有一根主线(一个计划、一个问题、
一次出行、一个决定)。
* 保留具体细节:人名、地点、数字、日期。
* 绝不编造记忆里没有的内容。
* 不要逐条罗列记忆,要综合成一段。
* 输出语言与记忆的语言一致。
只返回摘要正文,不要任何前言、不要 JSON、不要引号。"""
# 多语言版本见 hy_memory/prompts/<language>.py;旧名字保留为别名。
from ..prompts.en import EPISODE_SUMMARY_PROMPT # noqa: E402,F401 (re-export)
from ..prompts.zh_hans import EPISODE_SUMMARY_PROMPT as EPISODE_SUMMARY_PROMPT_ZH # noqa: E402,F401 (re-export)
SESSION_EPISODE_PROMPT = """Convert this complete conversation session into one retrieval-friendly episode narrative.
@@ -596,9 +555,13 @@ def _assign_facts_to_segments(facts: List[Any], segments: List[dict]) -> Dict[in
async def finalize_dynamic_episode_capsules(
*, vector_store, embed_service, llm_provider, user_id: str,
agent_id: Optional[str], session_id: str, summary_concurrency: int = 3,
language: Optional[str] = None,
) -> dict:
"""Split a completed session, create compact L3 capsules, and link its facts.
The capsule prompt is English-only; an explicit ``language`` appends that
language's output rule so the capsule text is written in it.
Boundary and summary generation are separate calls. Boundary failure degrades
to one whole-session capsule. A segmented-summary failure also retries the
session as one capsule so the write never loses the complete event context.
@@ -802,6 +765,7 @@ async def finalize_dynamic_episode_capsules(
start = end + 1
semaphore = asyncio.Semaphore(max(1, min(16, int(summary_concurrency or 1))))
explicit_language = normalize_language(language)
async def summarize(spec):
segment_index, segment_start, segment_end, segment = spec
@@ -809,9 +773,13 @@ async def finalize_dynamic_episode_capsules(
try:
async with semaphore:
response = await llm_provider.complete(
prompt=DYNAMIC_CAPSULE_PROMPT.format(
timestamp=timestamp_text,
messages=_render_messages(segment),
prompt=with_output_language(
DYNAMIC_CAPSULE_PROMPT.format(
timestamp=timestamp_text,
messages=_render_messages(segment),
),
explicit_language,
explicit=explicit_language is not None,
),
max_tokens=4096,
)
@@ -1128,9 +1096,13 @@ def _cosine(a: List[float], b: List[float]) -> float:
async def finalize_session_episode(
*, vector_store, embed_service, llm_provider, user_id: str,
agent_id: Optional[str], session_id: str,
language: Optional[str] = None,
) -> dict:
"""Create one first-class narrative from all raw chunks in a caller session.
The prompt is English-only; an explicit ``language`` appends that
language's output rule so the narrative is written in it.
This is deliberately explicit: the SDK cannot infer that the final write of a
session has arrived. Repeated calls upsert the same deterministic node id.
"""
@@ -1213,6 +1185,10 @@ async def finalize_session_episode(
timestamp=timestamp_text,
conversation=conversation,
)
explicit_language = normalize_language(language)
prompt = with_output_language(
prompt, explicit_language, explicit=explicit_language is not None,
)
response = await llm_provider.complete(prompt=prompt, max_tokens=3000)
narrative = (getattr(response, "content", "") or "").strip()
if not narrative:
@@ -1274,9 +1250,12 @@ async def summarize_closed_episode(
max_memories: int = 60,
temperature: Optional[float] = None,
trace=None,
language: Optional[str] = None,
) -> Optional[str]:
"""总结一个**刚刚关闭**的 episode,返回摘要节点 id(跳过则 None)。
language 为显式 prompt 语言码;None 时按记忆内容做 zh/en 自动检测。
由 writer 在检测到 episode 滚动时 fire-and-forget 调用:此时上一个 episode
不会再有新内容进来,所以这一次总结就是最终产物,不需要后续 merge / 重写。
"""
@@ -1317,6 +1296,7 @@ async def summarize_closed_episode(
_t0 = _dt.now()
summary, prompt, tokens = await _generate(
llm_provider=llm_provider, memories=contents, temperature=temperature,
language=language,
)
_elapsed = (_dt.now() - _t0).total_seconds() * 1000
if trace is not None:
@@ -1351,15 +1331,16 @@ async def summarize_closed_episode(
async def _generate(
*, llm_provider, memories: List[str], temperature: Optional[float] = None,
language: Optional[str] = None,
) -> tuple:
"""返回 (摘要文本, 实际 prompt, token 用量) —— 后两个供埋点,线上要能复盘。"""
from ..utils.lang_detect import is_chinese
body = "\n".join(f"- {c}" for c in memories if c)
if not body:
return "", "", {}
tpl = EPISODE_SUMMARY_PROMPT_ZH if is_chinese(body) else EPISODE_SUMMARY_PROMPT
prompt = tpl.format(memories=body)
explicit = normalize_language(language)
code = resolve_language(explicit, body)
prompt = get_prompt("EPISODE_SUMMARY_PROMPT", code).format(memories=body)
prompt = with_output_language(prompt, code, explicit=explicit is not None)
resp = await llm_provider.complete(
prompt=prompt,
max_tokens=400,
+69 -846
View File
@@ -20,6 +20,12 @@ import re
from .llm_provider import LLMProvider, _content_fingerprint, _log_llm_content_enabled
from ..config import LLMConfig as GlobalLLMConfig
from ..prompts import (
get_prompt,
normalize_language,
resolve_language,
with_output_language,
)
logger = logging.getLogger(__name__)
@@ -164,704 +170,13 @@ class V2ExtractResult:
# - 不再输出 time_ref 字段。时间相关信息如果是时间敏感的(schedule / past event),
# 让 LLM 把已解析的时间自然嵌入 content 即可;稳定偏好类 content 保持不含时间。
# System prompt(固定,不参与 format)
EXTRACT_SYSTEM_PROMPT = """# ROLE
You are a Memory Extractor.
Extract durable, high-value memories from conversations and convert them into structured, self-contained statements.
Only extract information that is likely to matter in future conversations.
---
# INPUTS
You will receive:
* Last k Messages
* New Messages
* Memory Date
* Current Date
Use:
* Last k Messages only for coreference resolution
* Memory Date as the ONLY temporal anchor
* Current Date only as metadata, never for resolving relative time
---
# WHAT TO EXTRACT
Extract ONLY genuinely new information introduced in the NEW user messages.
Good memories include:
* preferences
* long-term projects
* habits
* goals
* recurring activities
* opinions
* relationships
* important experiences
* stable traits
* future plans
* decisions
* constraints
* accepted recommendations
Do NOT extract:
* small talk
* acknowledgements
* generic emotions
* temporary conversational context
* assistant reminders of old memories
* information already implied by prior context
* vague personality judgments
---
# SUBJECT
Every memory must be self-contained, written in third person, with an explicit
subject. Which subject to use is decided by **who spoke**:
| Speaker | Subject to write | Example |
| --- | --- | --- |
| `role: "user"` | the user | "The user ..." / "用户..." |
| any speaker with a `name` other than the user (e.g. `name: "John"`) | that person, by name | "John ..." |
| `role: "assistant"` with NO `name` | the AI assistant | "The assistant ..." |
The two input sections express the speaker differently:
* **New Messages** is a JSON array — read `role` and the optional `name` field.
* **Last k Messages** is rendered text — the speaker is the bracket label,
`[user]:` / `[John]:` / `[assistant]:`, optionally preceded by one line
"The user's name is Tim" / "用户名叫Tim".
Rules:
* The user is the person these memories belong to. Even when the user's own name
is known (from `name` on their message, or from the leading name line), still
write "The user ...", NEVER "Tim ...".
* **`role: "assistant"` WITH a `name` is another real person, not the AI.**
Write their memories with their name as the subject: "John ...".
* Only a nameless `role: "assistant"` is the AI assistant.
* Never invent a name for the AI assistant, it has none.
---
# LANGUAGE RULES
* Output language must match input language.
* Preserve original scripts and technical terms exactly.
* Never transliterate names, brands, or equations.
---
# TWO KINDS OF MEMORY
You extract two kinds of memory: **memory** and **intentions**.
## memory
Everything durable and worth remembering about the user — and about any other
named person in the conversation — goes here:
* preferences, opinions, values, attitudes
* events, actions, experiences
* habits, behavioral patterns
* relationships, traits
* objective facts, decisions, constraints
* ongoing activities
Implicit preferences MUST be inferred when strongly suggested
(e.g. resuming an activity → positive interest; avoiding something →
negative preference; repeated voluntary effort → sustained interest).
Do NOT separate "what kind of person the user is" from "what happened".
Both go into `memory`.
## intentions
A forward-looking thing the user **plans or intends to do in the future**,
which can be acted on and which becomes irrelevant once done or past.
Examples:
* "The user has a job interview next Tuesday."
* "The user plans to move apartments at the end of the month."
* "The user intends to finish the report before Friday."
NOT intentions (these belong in `memory`):
* stable preferences / traits ("The user likes basketball")
* completed past events ("The user moved to Beijing last year")
* general goals with no actionable future step
### valid_until
For each intention, provide `valid_until`: the date after which the
intention no longer matters (the deadline / target date), resolved
against **Memory Date** (e.g. "next Tuesday" → that absolute date).
If no deadline can be determined, set `valid_until` to `null`.
---
# OWNER
Every memory and intention MUST carry an `owner` saying which memory store it
belongs to. There are exactly two values:
* `"user"` — anything belonging to the user's own memory store. This includes
facts/preferences/traits about the user themselves, AND facts about other
named real people in the conversation, because those are still things the
user's memory should hold.
* `"agent"` — information the AI assistant (a nameless `role: "assistant"`)
provides (recommendations, confirmations, things it researched), or a future
action the user wants the **assistant** to perform ("remind me in 8 hours").
A named person is NEVER `owner: "agent"` — they are a real person, not the AI,
even though their messages carry `role: "assistant"`. Their memories are
`owner: "user"` with their name as the content subject.
Decide `owner` per item. When unsure, default to `"user"`.
---
# TEMPORAL RULES
Resolve relative time using Memory Date.
Examples:
* "next week"
* "last month"
* "tomorrow"
If time cannot be resolved safely:
* remove the relative wording
* keep the memory atemporal
Do not invent chronology.
**Just-happened events are events — keep them.** When the user reports something as
just completed ("just got back", "just finished", "this morning", "yesterday"), the
completion date is the Memory Date. Emit the **event itself** with that date; do not
fold it into a background statement.
* "I just got back from a trip with my family"
→ "The user returned from a family trip on <Memory Date>."
NOT → "The user takes family trips." (the return date is the answer to "when did
they get back", and dropping it makes the question unanswerable)
Where an event has both a start and an end that the conversation gives separately,
record both rather than collapsing them into one span.
---
# QUALITY RULES
Each memory must be:
* self-contained
* contextually complete
* retrieval-friendly
* information-dense
Retain important details:
* names
* products
* locations
* numbers
* equations
* constraints
Never generalise a specific into a category. Write what was actually said.
* "Banff" → keep "Banff", NOT "the Rocky Mountains"
* "a 3-year-old border collie named Luna" → keep the breed, age and name
**Specifics supplied by the other speaker still count.** A fact is often built across
turns: the other speaker names the thing, the user confirms it. The subject rules
decide *whose* memory it is — they do not license dropping detail.
* [Sam]: "Did you take advantage of the skiing opportunities in Banff?"
[user]: "Yeah, it was great for skiing!"
→ "The user went skiing in Banff and found the snow amazing."
NOT → "The user enjoys winter sports." (both the place and the activity were
established, one by each speaker)
Avoid fragmentary memories lacking context.
Choose the smallest number of memories that still preserves distinct future
retrieval needs.
* Compression means combining, not omitting. Every explicit durable proposition
in the new messages must appear exactly once in the output.
* Combine details when they share the same subject and the same real-world event,
state, preference, plan, or decision. A coherent event may use 1–3 sentences.
* Do not emit both a broad statement and narrower restatements of the same evidence.
* Split only when the subject differs, the events happened independently, or the
memories describe different states.
* Keep a cause, recommendation, or consequence with the fact it explains when
separating them would lose the relationship.
* Never merge separate dated occurrences merely because they share a topic.
* Preserve the source's time precision. Do not turn vague phrases such as "last
weekend" or "last summer" into exact calendar dates unless the date is explicit
or uniquely determined; "the weekend before <Memory Date>" is acceptable.
---
# TAGS
Each memory must include 1–3 concise tags.
- Topical keywords stay lowercase (travel, health, work).
- Proper nouns keep capitalization (Jasper, Canada, Banff, Prius).
Examples:
* work
* travel
* Jasper
* Canada
* health
---
"""
STRUCTURE_METADATA_PROMPT_EN = """
---
# STRUCTURAL ORGANIZATION
For every `memory`, also output:
* `topic_paths`: a backward-compatible list of zero to three broad-to-specific
semantic paths.
* `event_facet`: an object with only `subject`, `participants`, `action`,
`objects`, `locations`, and `state`; state is `planned`, `ongoing`,
`completed`, or `unknown`.
* `topic_assignments`: zero to six subject-specific memberships. Each item is
`{subject, role, path}`, where `path` is a list of `{name, type}` segments and
`type` is one of `topic`, `facet`, `series`, or `episode`. Use `topic` for a
broad domain, `facet` for a concrete activity/semantic aspect, `series` only
for repeated occurrences of the same concrete activity/state line, and
`episode` only for one bounded real-world occurrence. A shared event may have
one assignment per explicit participant while keeping one canonical memory.
Types follow `topic -> facet -> series -> episode`; levels may be skipped but
types may not repeat or move backward.
This extractor sees only the current write and therefore must not emit a
`series` segment. Emit Topic/Facet/Episode here. Reconcile may later establish a
Series after comparing independent stored occurrences.
This is write-time organization. Do not infer Series/Episode from path depth,
manufacture hierarchy from a proper noun or broad tag, or assign a person only
because they co-occur in unrelated context. Stop at the most specific supported
level and use empty values when uncertain. Never encode query forms such as
"when", "how many", "who", plurality, or ordinal wording.
"""
# Few-shot examples (appended to the system prompt only when few_shot_enabled).
# Each example mirrors the real input shape (Last k Messages / New Messages /
# Memory Date) and our output schema (memory / intentions / basic_info with
# owner & tags), and ends with a one-line takeaway.
EXTRACT_FEW_SHOT_EN = """
---
# EXAMPLES
## Example 1 — Boundary: old context is for resolution only, never re-extracted
Last k Messages:
[{"role": "user", "content": "I've been looking at rescue dogs at the shelter downtown."},
{"role": "assistant", "content": "That's lovely — found one you like?"}]
New Messages:
[{"role": "user", "content": "Yeah, picked her up yesterday — a 3-year-old border collie named Luna."}]
Memory Date: 2025-04-10
Output:
```json
{
"memory": [
{"content": "The user adopted a 3-year-old border collie named Luna from a downtown shelter on 2025-04-09.", "tags": ["pets", "family"], "owner": "user"}
],
"intentions": [],
"basic_info": {}
}
```
The old messages only resolve "her" and supply context ("downtown shelter"). Do NOT emit a stale memory like "the user is considering adopting a dog" — only the genuinely new fact (the completed adoption) is extracted, with "yesterday" resolved against Memory Date.
## Example 2 — Condense a long assistant reply
New Messages:
[{"role": "user", "content": "Write me a short essay on why remote work boosts productivity."},
{"role": "assistant", "content": "Sure — opening: 'Remote work has reshaped how teams operate...' [~2000 words of body omitted] ...closing: 'In short, autonomy and fewer interruptions drive higher output.'"}]
Memory Date: 2025-03-22
Output:
```json
{
"memory": [
{"content": "The assistant drafted a short essay for the user arguing that remote work boosts productivity because autonomy and fewer interruptions drive higher output.", "tags": ["writing", "remote-work"], "owner": "agent"}
],
"intentions": [],
"basic_info": {}
}
```
The assistant's reply is long. Never store the verbatim essay — distill it to topic + thesis. owner is "agent" because the assistant produced it. Bloated verbatim content makes memory unusable.
## Example 3 — Composite extraction: basic_info + intention + condensed agent plan
New Messages:
[{"role": "user", "content": "Hi, I'm Kenji. I'm planning a trip to Japan next month — could you put together a 7-day itinerary?"},
{"role": "assistant", "content": "# 7-Day Classic Japan Golden Route (Tokyo → Kyoto → Nara → Osaka)\\nFly into Tokyo, Shinkansen south to Kyoto, day-trip Nara, depart from Osaka (Kansai).\\n\\n## Day 1: Arrival in Tokyo\\n- Land at Haneda/Narita, drop bags at hotel near Tokyo Station.\\n- Late afternoon: Sensoji Temple (Asakusa), wander Nakamise Street, try taiyaki.\\n- Evening: Shibuya Crossing + Shibuya Sky rooftop for sunset; izakaya dinner (yakitori, sake).\\n\\n## Day 2: Tokyo markets & landmarks — [omitted]\\n## Day 3: Shinkansen to Kyoto, Fushimi Inari, Gion — [omitted]\\n## Day 4: Kyoto temples (Kiyomizu-dera, Kinkaku-ji, Ginkaku-ji) — [omitted]\\n## Day 5: Nara day trip (Nara Park deer, Todai-ji) — [omitted]\\n## Day 6: Osaka (Osaka Castle, Dotonbori street food) — [omitted]\\n## Day 7: Osaka sightseeing + depart from Kansai Airport — [omitted]\\n\\nTip: buy a 7-day JR Pass before arriving. Want me to adjust anything?"}]
Memory Date: 2025-09-15
Output:
```json
{
"memory": [
{"content": "The assistant created a 7-day Japan itinerary for the user along the classic golden route: Tokyo (Asakusa, Shibuya), Shinkansen to Kyoto (Fushimi Inari, Kiyomizu-dera, Kinkaku-ji, Gion), a Nara day trip (deer park, Todai-ji), then Osaka (Osaka Castle, Dotonbori), departing from Kansai Airport.", "tags": ["travel", "japan", "itinerary"], "owner": "agent"}
],
"intentions": [
{"content": "The user plans to take a 7-day trip to Japan in October 2025.", "tags": ["travel", "japan"], "valid_until": "2025-10-31", "owner": "user"}
],
"basic_info": {"name": "Kenji"}
}
```
One turn produces three outputs in different buckets: the name goes to basic_info, the trip is a forward-looking intention with a deadline, and the assistant's long multi-section itinerary is condensed into ONE dense agent memory — the route and key stops per city, NOT every day's bullets, tips, or food lists. Storing the whole reply verbatim would bloat memory; storing the skeleton keeps it queryable.
## Example 4 — A later message confirming/adopting an earlier recommendation is itself a fact
New Messages:
[{"role": "user", "content": "My 6-year-old won't eat any vegetables. Any ideas?"},
{"role": "assistant", "content": "Try blending spinach or carrots into pasta sauce — the color and flavor mostly disappear, and many kids accept it without noticing."},
{"role": "user", "content": "Oh nice, we'll try that tonight. By the way, what's a good bedtime for that age?"},
{"role": "assistant", "content": "For a 6-year-old, aim for 7–9 PM with about 10–11 hours of sleep."}]
Memory Date: 2025-06-20
Output:
```json
{
"memory": [
{"content": "The user has a 6-year-old child who refuses to eat vegetables, and accepted the assistant's suggestion to blend spinach or carrots into pasta sauce to hide them.", "tags": ["parenting", "food"], "owner": "user"},
{"content": "The assistant advised the user that a 6-year-old should aim for a 7–9 PM bedtime with about 10–11 hours of sleep.", "tags": ["parenting", "sleep"], "owner": "agent"}
],
"intentions": [],
"basic_info": {}
}
```
In the third message the user said "we'll try that tonight" — explicitly adopting the earlier suggestion. That acceptance is valuable and must be recorded ("accepted the assistant's suggestion to ..."), not discarded as small talk. The bedtime advice was just given and not yet adopted, so it is recorded as advice (owner "agent").
## Example 5 — RAG/injected context: never store the injected text
New Messages:
[{"role": "user", "content": "[Retrieved context: The Eiffel Tower is 330m tall and was completed in 1889.] Given the above, I'm going to propose to my girlfriend at the top of the Eiffel Tower this summer."}]
Memory Date: 2025-05-02
Output:
```json
{
"memory": [],
"intentions": [
{"content": "The user plans to propose to his girlfriend at the top of the Eiffel Tower in summer 2025.", "tags": ["relationship", "travel"], "valid_until": "2025-08-31", "owner": "user"}
],
"basic_info": {}
}
```
The bracketed retrieved context (the tower's height and year) is RAG-injected reference material, NOT something the user told us about themselves — do NOT store it. Only the user's own forward-looking plan is extracted. System/RAG-injected text is context for the model, never a memory.
## Example 6 — Shared reference material: extract the content, not the sharing action
New Messages:
[{"role": "user", "content": "Remember my grandma's dumpling recipe so I don't lose it. Filling: 500g ground pork, 300g napa cabbage, 2 tbsp soy sauce, 1 tbsp sesame oil, a thumb of ginger minced. Cabbage salted 15 min then squeezed dry. Boil dumplings, add cold water twice, ready when they float."},
{"role": "assistant", "content": "Got it, saved!"}]
Memory Date: 2025-07-01
Output:
```json
{
"memory": [
{"content": "The user's grandmother's dumpling recipe: filling of 500g ground pork, 300g napa cabbage (salted 15 minutes then squeezed dry), 2 tbsp soy sauce, 1 tbsp sesame oil, and a thumb of minced ginger; boil the dumplings, add cold water twice, and they are done when they float.", "tags": ["food", "recipe", "family"], "owner": "user"}
],
"intentions": [],
"basic_info": {}
}
```
Extract the actual factual content (ingredients, quantities, the salt-and-squeeze step, the boiling method) — NOT "the user shared a recipe" or "the user asked to remember the recipe". The sharing action is worthless; the content is what's queryable.
## Example 7 — Structured numbers and attributes preserved exactly
New Messages:
[{"role": "user", "content": "Logging my home-gym PRs this cycle: Squat 140kg x 3, Bench 100kg x 5, Deadlift 180kg x 2, Overhead Press 60kg x 6. Bodyweight 78kg, 3 sets per lift, 3 min rest."},
{"role": "assistant", "content": "Logged — strong numbers!"}]
Memory Date: 2026-01-20
Output:
```json
{
"memory": [
{"content": "The user's current lifting PRs are Squat 140kg x 3, Bench 100kg x 5, Deadlift 180kg x 2, and Overhead Press 60kg x 6, at a bodyweight of 78kg.", "tags": ["fitness", "strength"], "owner": "user"},
{"content": "The user trains with 3 sets per lift and 3 minutes of rest between sets.", "tags": ["fitness", "training"], "owner": "user"}
],
"intentions": [],
"basic_info": {}
}
```
Every number and unit (kg, reps, sets, rest time) is preserved exactly — dropping them would destroy the most queryable information. Split into two clean topics (PRs vs. training protocol), but never round or omit the specifics.
## Example 8 — Long multi-topic conversation: do not stop after the first topic
New Messages:
[{"role": "user", "content": "Big week — I finally defended my PhD thesis on Tuesday!"},
{"role": "assistant", "content": "Huge congrats! How do you feel?"},
{"role": "user", "content": "Relieved! I also signed a lease on an apartment in Berlin and move next month. And I've started learning German on Duolingo — 30-day streak. My partner Sofia is staying in Madrid for now, which is hard."}]
Memory Date: 2025-11-08
Output:
```json
{
"memory": [
{"content": "The user defended their PhD thesis on 2025-11-04.", "tags": ["education", "milestone"], "owner": "user"},
{"content": "The user signed a lease on an apartment in Berlin.", "tags": ["housing"], "owner": "user"},
{"content": "The user has been learning German on Duolingo and has a 30-day streak.", "tags": ["language", "learning"], "owner": "user"},
{"content": "The user's partner Sofia is staying in Madrid for now, which the user finds hard.", "tags": ["relationship"], "owner": "user"}
],
"intentions": [
{"content": "The user plans to move to Berlin next month (around December 2025).", "tags": ["housing", "relocation"], "valid_until": "2025-12-31", "owner": "user"}
],
"basic_info": {}
}
```
Four+ topics — thesis defense, the Berlin apartment, learning German, and the partner in Madrid. Do not stop after the first topic. Note one turn split into a completed fact (lease signed) AND an intention (the upcoming move), and the emotional-but-substantive detail about Sofia is kept because it carries lasting relationship context.
## Example 9 — Two real people: `name` on an assistant message means a real person
New Messages:
[{"role": "user", "name": "Tim", "content": "Just got back from the climbing gym, third time this week. Finally sent that overhang route."},
{"role": "assistant", "name": "John", "content": "Nice! I've been swimming instead — my shoulder still hurts from last month."}]
Memory Date: 2025-08-14
Output:
```json
{
"memory": [
{"content": "The user has been climbing three times a week and recently completed a difficult overhang route.", "tags": ["fitness", "climbing"], "owner": "user"},
{"content": "John has been swimming instead of other exercise because his shoulder still hurt after an injury around July 2025.", "tags": ["fitness", "health"], "owner": "user"}
],
"intentions": [],
"basic_info": {"name": "Tim"}
}
```
John carries `role: "assistant"` but he has a `name`, so he is another real person, NOT the AI: his memory uses "John ..." as the subject and is still `owner: "user"` (it belongs to the user's memory store). The user's own memory stays "The user ..." even though we know the name is Tim — never "Tim has been climbing". `basic_info.name` comes from `name: "Tim"` on the user message, which is authoritative even though Tim never said his own name out loud.
## Example 10 — Counter-example: another person's attributes never go into basic_info
New Messages:
[{"role": "user", "name": "Tim", "content": "你最近在忙什么?"},
{"role": "assistant", "name": "John", "content": "我在柏林一家医院当护士,上的是夜班,挺累的。"}]
Memory Date: 2025-09-02
Output:
```json
{
"memory": [
{"content": "John是柏林一家医院的护士,上夜班,觉得很累。", "tags": ["work", "health"], "owner": "user"}
],
"intentions": [],
"basic_info": {"name": "Tim"}
}
```
John's occupation and location are HIS attributes, so they go into `memory` with "John" as the subject. They must NEVER land in `basic_info.occupation` / `basic_info.location` — `basic_info` is exclusively the user's own profile. Only `name` (Tim, from the user message's `name` field) belongs there.
## Example 11 — Counter-example: small talk has no basic_info
New Messages:
[{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello! The user's name is The user. How can I help?"}]
Memory Date: 2025-09-03
Output:
```json
{
"memory": [],
"intentions": [],
"basic_info": {}
}
```
The user stated nothing about themselves. The assistant's line is not a user turn, and "The user" is a placeholder, not a name. Output `{}` — never `{"name": "The user"}` or `{"name": "user"}`.
---
"""
EXTRACT_PROMPT = """## Last k Messages
{last_messages}
---
## New Messages
{content}
---
## Memory Date
{memory_at}
---
## Current Date
{current_date}
---
# Output Format
Return ONLY a JSON object inside a ```json fenced block.
```json
{{
"memory": [
{{
"content": "...",
"tags": ["..."],
"owner": "user"
}}
],
"intentions": [
{{
"content": "...",
"tags": ["..."],
"valid_until": "YYYY-MM-DD",
"owner": "user"
}}
],
"basic_info": {{}}
}}
```
Rules:
* `memory` = durable info worth remembering: preferences, opinions, attitudes,
traits, events, plans, experiences, habits, decisions — about the user or
about any other named real person, with the subject written per "# SUBJECT"
* `intentions` = forward-looking things the user (or another named person) plans /
intends to do in the future, with `valid_until` = the deadline/target date
(resolved against Memory Date) or `null` if none
* `owner` (required on every memory & intention) = `"user"` or `"agent"`
(see "# OWNER" above; default `"user"` when unsure)
* `basic_info` = the **user's own profile only**. Include stable profile
fields (e.g. name, age, location, occupation, employer) **explicitly stated**
in the NEW user messages. Never infer, never invent. If no field is explicitly
stated, output `{{}}`.
* HARD RULE: fill a field ONLY if the user stated it in a `user` turn.
Anything said only by the assistant (including the assistant quoting an
earlier memory such as "The user's name is X") does NOT count. If nothing
qualifies, output `"basic_info": {{}}`.
* NEVER write the field name itself or a placeholder as a value ("user",
"The user", "用户", "name", "N/A", "unknown", "none", "-"). Omit the field.
* Negative example: small talk such as `[user]: Hi` / `[assistant]: How can
I help?`, or `[user]: Reply with exactly: PONG` → `"basic_info": {{}}`.
* Exception for `name`: the `name` field on the user's own message, or a leading
line "The user's name is X" / "用户名叫X", is an authoritative source for
`name` — even though the user never said it out loud.
* NEVER put another real person's attributes here. If John (an assistant-role
message carrying `name`) says "I'm a nurse in Berlin", that is a `memory`
("John is a nurse in Berlin."), NOT `basic_info.occupation` / `.location`.
* No duplicate information across sections
* Tags must contain 1–3 concise lowercase keywords
* Use empty arrays `[]` or `{{}}` when nothing is extracted
* Return valid JSON only
Output the JSON now."""
# User prompt when MEMORY_ENABLE_INTENTION=false(无 intentions 桶)
EXTRACT_PROMPT_NO_INTENTION = """## Last k Messages
{last_messages}
---
## New Messages
{content}
---
## Memory Date
{memory_at}
---
## Current Date
{current_date}
---
# Output Format
Return ONLY a JSON object inside a ```json fenced block.
```json
{{
"memory": [
{{
"content": "...",
"tags": ["..."],
"owner": "user"
}}
],
"basic_info": {{}}
}}
```
Rules:
* `memory` = durable info worth remembering: preferences, opinions, attitudes,
traits, events, experiences, habits, decisions, **and future plans / upcoming
actions** — about the user or about any other named real person, with the
subject written per "# SUBJECT". Do **not** invent a separate intentions bucket.
* `owner` (required on every memory) = `"user"` or `"agent"`
(see "# OWNER" above; default `"user"` when unsure)
* `basic_info` = the **user's own profile only**. Include stable profile
fields (e.g. name, age, location, occupation, employer) **explicitly stated**
in the NEW user messages. Never infer, never invent. If no field is explicitly
stated, output `{{}}`.
* HARD RULE: fill a field ONLY if the user stated it in a `user` turn.
Anything said only by the assistant (including the assistant quoting an
earlier memory such as "The user's name is X") does NOT count. If nothing
qualifies, output `"basic_info": {{}}`.
* NEVER write the field name itself or a placeholder as a value ("user",
"The user", "用户", "name", "N/A", "unknown", "none", "-"). Omit the field.
* Negative example: small talk such as `[user]: Hi` / `[assistant]: How can
I help?`, or `[user]: Reply with exactly: PONG` → `"basic_info": {{}}`.
* Exception for `name`: the `name` field on the user's own message, or a leading
line "The user's name is X" / "用户名叫X", is an authoritative source for
`name` — even though the user never said it out loud.
* NEVER put another real person's attributes here. If John (an assistant-role
message carrying `name`) says "I'm a nurse in Berlin", that is a `memory`
("John is a nurse in Berlin."), NOT `basic_info.occupation` / `.location`.
* No duplicate information across sections
* Tags: 1–3 concise keywords; topical tags lowercase; proper nouns keep capitalization
* Use empty arrays `[]` or `{{}}` when nothing is extracted
* Return valid JSON only — **never** include an `intentions` key
Output the JSON now."""
# Prompt 文本统一放在 hy_memory/prompts/<language>.py,按语言码经 get_prompt()
# 取用。以下旧名字保留为英文版的别名(向后兼容)。
from ..prompts.en import EXTRACT_SYSTEM_PROMPT # noqa: E402,F401 (re-export)
from ..prompts.en import STRUCTURE_METADATA_PROMPT as STRUCTURE_METADATA_PROMPT_EN # noqa: E402,F401 (re-export)
from ..prompts.en import EXTRACT_FEW_SHOT as EXTRACT_FEW_SHOT_EN # noqa: E402,F401 (re-export)
from ..prompts.en import EXTRACT_PROMPT # noqa: E402,F401 (re-export)
from ..prompts.en import EXTRACT_PROMPT_NO_INTENTION # noqa: E402,F401 (re-export)
# ================================================================
@@ -870,101 +185,26 @@ Output the JSON now."""
def _system_prompt_without_intention(system_prompt: str, *, lang: str) -> str:
"""从 system prompt 去掉 intentions 分桶,改为单一 memory(含未来计划)。"""
"""从 system prompt 去掉 intentions 分桶,改为单一 memory(含未来计划)。
``lang`` 是语言码(``en`` / ``zh-Hans`` / ``ja`` ...;旧值 ``zh`` 视作 zh-Hans)。
每种语言的替换片段在 ``hy_memory/prompts/<lang>.py`` 中定义。
"""
import re
if lang == "zh":
repl = (
"# 记忆\n"
"\n"
"你只提取一类记忆:**memory(记忆)**。\n"
"\n"
"## memory(记忆)\n"
"\n"
"一切值得长期记住的信息都放这里——既包括用户本人的,也包括对话中其他具名真人的:\n"
"\n"
"* 偏好、观点、态度、价值观\n"
"* 事件、行动、经历\n"
"* 习惯、行为模式\n"
"* 关系、稳定特质\n"
"* 客观事实、决策、约束条件\n"
"* 正在进行的活动\n"
"* **未来计划 / 即将发生的安排**(若值得记住,直接写进 `memory`;"
"不要单独开 intentions 桶)\n"
"\n"
"如果用户隐式表达偏好,也应提取。\n"
"\n"
"不要把\"用户是怎样的人\"和\"发生了什么\"分开。两者都放进 `memory`。\n"
"\n"
"---\n"
"\n"
)
system_prompt = re.sub(
r"# 两类记忆.*?(?=\n# OWNER)",
repl,
system_prompt,
count=1,
flags=re.S,
)
system_prompt = system_prompt.replace(
"每条 memory 和 intention 都必须带 `owner`",
"每条 memory 都必须带 `owner`",
)
system_prompt = system_prompt.replace(
"每条 memory 和 intention 都必须带",
"每条 memory 都必须带",
)
# few-shot 里若仍出现 intentions,用硬约束盖住
system_prompt += (
"\n\n---\n\n# INTENTION 关闭\n"
"当前配置关闭了 intention 抽取。即使示例里出现 `intentions`,"
"也**不要**输出该字段;未来计划写入 `memory`。\n"
)
else:
repl = (
"# MEMORY\n"
"\n"
"You extract one kind of memory: **memory**.\n"
"\n"
"## memory\n"
"\n"
"Everything durable and worth remembering about the user — and about any other\n"
"named person in the conversation — goes here:\n"
"\n"
"* preferences, opinions, values, attitudes\n"
"* events, actions, experiences\n"
"* habits, behavioral patterns\n"
"* relationships, traits\n"
"* objective facts, decisions, constraints\n"
"* ongoing activities\n"
"* **future plans / upcoming actions** that are worth remembering\n"
" (put them in `memory`; do NOT invent a separate intentions bucket)\n"
"\n"
"Implicit preferences MUST be inferred when strongly suggested.\n"
"\n"
"Do NOT separate \"what kind of person the user is\" from \"what happened\".\n"
"Both go into `memory`.\n"
"\n"
"---\n"
"\n"
)
system_prompt = re.sub(
r"# TWO KINDS OF MEMORY.*?(?=\n# OWNER\n)",
repl,
system_prompt,
count=1,
flags=re.S,
)
system_prompt = system_prompt.replace(
"Every memory and intention MUST carry an `owner`",
"Every memory MUST carry an `owner`",
)
system_prompt += (
"\n\n---\n\n# INTENTION DISABLED\n"
"Intention extraction is disabled. Even if few-shot examples show "
"`intentions`, do **not** emit that key; put future plans in `memory`.\n"
)
return system_prompt
language = "zh-Hans" if lang == "zh" else lang
repl = get_prompt("NO_INTENTION_MEMORY_SECTION", language)
system_prompt = re.sub(
get_prompt("NO_INTENTION_SECTION_PATTERN", language),
lambda _m: repl,
system_prompt,
count=1,
flags=re.S,
)
for old, new in get_prompt("NO_INTENTION_REPLACEMENTS", language):
system_prompt = system_prompt.replace(old, new)
# few-shot 里若仍出现 intentions,用硬约束盖住
return system_prompt + get_prompt("INTENTION_DISABLED_NOTE", language)
class Extractor:
@@ -1001,6 +241,7 @@ class Extractor:
tool_context: Optional[Dict[str, Any]] = None,
history_context: str = "",
basic_profile_fields: Optional[Dict[str, str]] = None,
language: Optional[str] = None,
) -> ExtractResult:
"""
统一提取:identity / facts / basic_info
@@ -1022,6 +263,9 @@ class Extractor:
**DEPRECATED**: 曾用于渲染 EXTRACT prompt 的基础画像 schema 段落,
现已移除该段落(basic_info 由内置 few-shot / 指令引导)。保留参数
仅为向后兼容,传入不再生效。
language: 显式 prompt 语言码(见 hy_memory.prompts.SUPPORTED_LANGUAGES,
接受别名)。None 时按 content 做 zh/en 自动检测。显式指定时
追加 OUTPUT_LANGUAGE_DIRECTIVE,记忆正文用该语言书写。
"""
MAX_TOOL_ROUNDS = 5
@@ -1048,73 +292,52 @@ class Extractor:
else:
_memory_at = _current_date
# Select prompt based on input language
from ..utils.lang_detect import is_chinese
# 选 prompt 语言:显式 language(per-call / MemoryConfig.language)优先;
# 未指定时沿用 zh/en 自动检测(行为与以前完全一致)。
_explicit_language = normalize_language(language)
_language = resolve_language(_explicit_language, content)
_enable_intention = bool(
getattr(self._llm_config, "enable_intention", True)
)
if is_chinese(content):
from .prompts_zh import (
EXTRACT_PROMPT_ZH,
EXTRACT_PROMPT_ZH_NO_INTENTION,
EXTRACT_SYSTEM_PROMPT_ZH,
EXTRACT_FEW_SHOT_ZH,
)
# client 自定义 system prompt 仅覆盖对应语言(zh → zh-Hans, en → en);
# 直接用,不追加内置 few-shot
_custom_sys = None
if _language == "zh-Hans":
_custom_sys = getattr(
self._llm_config, "extract_system_prompt_zh", None
)
if _custom_sys:
# client 自定义 system prompt:直接用,不追加内置 few-shot
system_prompt = _custom_sys
else:
system_prompt = EXTRACT_SYSTEM_PROMPT_ZH
if getattr(self._llm_config, "few_shot_enabled", False):
system_prompt = system_prompt + EXTRACT_FEW_SHOT_ZH
if not _enable_intention and not _custom_sys:
system_prompt = _system_prompt_without_intention(
system_prompt, lang="zh"
)
if getattr(self._llm_config, "structure_metadata_enabled", False):
from .prompts_zh import STRUCTURE_METADATA_PROMPT_ZH
system_prompt += STRUCTURE_METADATA_PROMPT_ZH
_user_tmpl = (
EXTRACT_PROMPT_ZH
if _enable_intention
else EXTRACT_PROMPT_ZH_NO_INTENTION
)
prompt = _user_tmpl.format(
content=content,
current_date=_current_date,
memory_at=_memory_at,
last_messages=_last_messages,
)
else:
elif _language == "en":
_custom_sys = getattr(
self._llm_config, "extract_system_prompt_en", None
)
if _custom_sys:
system_prompt = _custom_sys
else:
system_prompt = EXTRACT_SYSTEM_PROMPT
if getattr(self._llm_config, "few_shot_enabled", False):
system_prompt = system_prompt + EXTRACT_FEW_SHOT_EN
if not _enable_intention and not _custom_sys:
system_prompt = _system_prompt_without_intention(
system_prompt, lang="en"
if _custom_sys:
system_prompt = _custom_sys
else:
system_prompt = get_prompt("EXTRACT_SYSTEM_PROMPT", _language)
if getattr(self._llm_config, "few_shot_enabled", False):
system_prompt = system_prompt + get_prompt(
"EXTRACT_FEW_SHOT", _language
)
if getattr(self._llm_config, "structure_metadata_enabled", False):
system_prompt += STRUCTURE_METADATA_PROMPT_EN
_user_tmpl = (
EXTRACT_PROMPT if _enable_intention else EXTRACT_PROMPT_NO_INTENTION
)
prompt = _user_tmpl.format(
content=content,
current_date=_current_date,
memory_at=_memory_at,
last_messages=_last_messages,
if not _enable_intention and not _custom_sys:
system_prompt = _system_prompt_without_intention(
system_prompt, lang=_language
)
if getattr(self._llm_config, "structure_metadata_enabled", False):
system_prompt += get_prompt("STRUCTURE_METADATA_PROMPT", _language)
system_prompt = with_output_language(
system_prompt, _language, explicit=_explicit_language is not None,
)
_user_tmpl = get_prompt(
"EXTRACT_PROMPT" if _enable_intention else "EXTRACT_PROMPT_NO_INTENTION",
_language,
)
prompt = _user_tmpl.format(
content=content,
current_date=_current_date,
memory_at=_memory_at,
last_messages=_last_messages,
)
# 累计 token 统计
if _log_llm_content_enabled():
+3
View File
@@ -144,6 +144,7 @@ class MemAgent:
tool_context: Optional[Dict[str, Any]] = None,
history_context: str = "",
basic_profile_fields: Optional[Dict[str, str]] = None,
language: Optional[str] = None,
) -> AgentResult:
"""
处理添加记忆请求
@@ -161,6 +162,7 @@ class MemAgent:
basic_profile_fields:
{field_name: description} 字段表,由调用方从 MemoryConfig.basic_profile.fields
透传过来,extractor 会渲染到 prompt 中要求 LLM 在 JSON basic_info 字段返回。
language: 显式 prompt 语言码,透传给 extractor;None 时 zh/en 自动检测。
Returns:
处理结果
@@ -187,6 +189,7 @@ class MemAgent:
tool_context=tool_context,
history_context=history_context,
basic_profile_fields=basic_profile_fields,
language=language,
)
_t_parallel = datetime.now()
File diff suppressed because it is too large. Load diff
+78 -533
View File
@@ -136,10 +136,10 @@ class ReconcileResult:
ops: List[ReconcileOp] = field(default_factory=list)
success: bool = True
error: Optional[str] = None
# True = reconcile 失败后走了「全部 ADD」兜底(见 _fallback_add_ops)。
# 结果仍然可用(success=True),但没有做过去重/演化判定,可能引入重复。
degraded: bool = False
degraded_reason: Optional[str] = None
# success=False 时的稳定失败码:LLM_UNAVAILABLE / EMPTY_RESPONSE /
# JSON_PARSE_FAILED / RECONCILE_EXCEPTION。失败时 ops 恒为空:writer 什么都
# 不落库,整次写入报失败,调用方换个 request_id 重跑即可(reconcile 会判重)。
error_code: Optional[str] = None
# reconcile 链路 LLM token 消耗(search-query + reconcile 两次 LLM 调用之和)
prompt_tokens: int = 0
completion_tokens: int = 0
@@ -151,7 +151,7 @@ class ReconcileResult:
# 用 content 作 key 而不是 node_id:新记忆此刻还没入库、没有 id。
candidate_scores: Dict[str, List[Tuple[str, float]]] = field(default_factory=dict)
# reconcile 候选检索已经为每条新 memory 算过 embedding。把原始文本对应向量交回
# writer,ADD-all / degraded ADD 可直接落库,避免同一事实再次请求 embedding。
# writer,原文不变的 ADD 可直接落库,避免同一事实再次请求 embedding。
# 仅按完全相同 content 复用;LLM 改写后的 UPDATE/SUPERSEDE 会重新计算,防止错配。
new_memory_embeddings: Dict[str, List[float]] = field(default_factory=dict)
@@ -160,299 +160,18 @@ class ReconcileResult:
# Prompts
# ================================================================
SEARCH_QUERY_PROMPT = """You are a search query generator. Given a list of newly extracted memories, generate a set of short search queries that can be used to find related existing memories in a vector database.
The goal is to maximize recall — find existing memories that are semantically related to the new memories, even if the wording is very different.
## New memories:
{new_memories}
## Instructions
Generate search queries that cover:
- Key topics, entities, and themes mentioned in the memories
- Rephrased or abstracted versions of the core concepts
- Related concepts that might exist in the user's memory store
Output a JSON array of query strings (5-15 queries, short and focused):
["query1", "query2", "query3", ...]
Output JSON array only, no other text."""
RECONCILE_PROMPT = """You are a memory management system.
Your task is to merge new memories into an existing memory base while keeping it:
* lossless
* non-contradictory
* compact
* highly retrievable
---
# Operations
## ADD
Use when the information is entirely new.
```json
{{
"op": "ADD",
"content": "...",
"owner": "user",
"tags": ["..."]
}}
```
---
## UPDATE
Use when:
* the new memory is compatible with an existing one
* they describe the same topic
* merging improves completeness or retrievability
Rules:
* merge losslessly
* preserve all useful information
* do NOT create duplicates
* if the two disagree on a value, keep the value from the memory with the later
`memory_at`; never let an older memory overwrite a newer value
### Same occurrence vs. another occurrence
Use **UPDATE** only when the new memory adds detail to the same real-world
occurrence or durable fact already recorded.
If the same person does the same kind of activity/event again at another time
(another trip, game, injury, purchase, health scare, etc.), keep the occurrence
as a separate **ADD**. Preserve its own date and details. When an existing memory
is clearly another occurrence in that same concrete series, link it with
`related: [{{"memory_id":"...","relation":"same_series"}}]`.
Do not merge separate dated occurrences into one long count/timeline memory.
The retrieval layer builds timelines from the linked L2 memories.
```json
{{
"op": "UPDATE",
"memory_id": "...",
"content": "...",
"owner": "user",
"tags": ["..."]
}}
```
---
## SUPERSEDE
Use ONLY when:
* the new memory contradicts an existing memory
* both cannot be true at the same time
* the old memory would now give the wrong current-state answer
* the new memory's `memory_at` is the same as or LATER than the existing memory's `memory_at`
Examples:
* city changed
* employer changed
* relationship status changed
Do NOT use for:
* added detail
* compatible preferences
* wording differences
* semantic equivalence
* a new memory whose `memory_at` is EARLIER than the existing memory's — it is an
older state and must never replace a newer one (see "Time order")
Rules:
* content must contain ONLY the new state
* do NOT copy unchanged information
* the system keeps the old node and links it onto the evolution chain automatically; you don't need to handle history
```json
{{
"op": "SUPERSEDE",
"memory_id": "...",
"content": "...",
"supersede_reason": "...",
"owner": "user",
"tags": ["..."]
}}
```
### Folding several existing memories into one chain
If TWO OR MORE existing memories track the SAME dimension and the new memory is
the latest change on that dimension — its `memory_at` is later than every folded
memory's `memory_at` — you may fold them into a single evolution
chain. Use `memory_ids` (an ordered list, OLDEST → NEWEST) instead of a single
`memory_id`:
```json
{{
"op": "SUPERSEDE",
"memory_ids": ["<oldest_id>", "<newer_id>"],
"content": "<only the newest state>",
"supersede_reason": "...",
"owner": "user",
"tags": ["..."]
}}
```
The listed memories are chained oldest → newest and the new memory becomes the
chain head. `content` still describes ONLY the newest state — never copy the
older memories' text into it; they remain queryable in the chain history.
---
# Rules
* Preserve all meaningful information.
* Skip exact duplicates.
* Prefer merging related fragments.
* Separate dated occurrences stay separate ADDs; use `same_series` only for the
same subject and same concrete activity/event type.
* Keep each memory focused on one coherent topic.
* Follow "Time order" below.
* **NEVER merge or supersede across different subjects.** The subject is whoever
the content is about: "The user ..." vs "John ..." vs "Mary ...". Memories about
different people are independent facts even when the sentence is otherwise
identical — "The user likes coffee" and "John likes coffee" are two separate
memories, never an UPDATE or a SUPERSEDE of each other.
* `owner` (`user`/`agent`): copy it from the memory being written. NEVER merge or supersede across different owners — a user's fact and the assistant's fact are distinct even on the same topic.
* A SUPERSEDE either targets ONE memory (`memory_id`) or folds several memories on the same dimension into one chain (`memory_ids`, oldest→newest). An UPDATE targets exactly ONE memory.
* Never touch the same existing memory from more than one op.
* Output language matches input memory language.
---
# Time order
Every memory, new or existing, has `memory_at`: when it happened. Memories are
NOT always written in the order they happened — history can be imported late,
and a memory can be added while older history is still being processed. A new
memory can therefore be OLDER than an existing one.
* Decide which memory is newer ONLY by `memory_at`. "New memories" means "being
written now", not "more recent".
* An older memory must never replace a newer one: no SUPERSEDE of a newer
memory, and no UPDATE that overwrites a newer value with an older one.
* If a new memory is older than an existing memory it conflicts with, keep the
existing memory as the current state. ADD the new one as a past state, worded
in the past with its time (for example "In 2023 the user lived in Shanghai"),
or output nothing if it adds no information.
* `memory_at: null` means unknown time. Unknown is NOT newer: never SUPERSEDE a
memory that has a known `memory_at` with one whose `memory_at` is null.
* Do not invent chronology.
---
# Optional: `related`
On any op you MAY add a `related` array recording how the memory being written
relates to an EXISTING memory. This does not change what gets stored — it only
records a link.
```json
{{
"op": "ADD",
"content": "...",
"owner": "user",
"tags": ["..."],
"related": [{{"memory_id": "<existing_id>", "relation": "causes"}}]
}}
```
Only these four relations are allowed:
* `causes` — the two memories are in a cause/effect relationship
(e.g. "started running daily" → "knee started hurting")
* `elaborates` — adds detail to, or is a later development of, the existing memory
* `contrasts` — they are in tension but BOTH still true (if one makes the other
false, that is a SUPERSEDE, not a `contrasts` link)
* `same_series` — separate dated occurrences of the same concrete activity or
event type for the same subject (for example, two different skiing trips)
Rules:
* Only link to memories that appear in "Existing memories" — never invent an id.
* Only emit a link when the relationship is **explicit and specific**. Two memories
merely being about the same broad topic is NOT a relation — leave `related` out.
* `same_series` requires the same subject and a concrete repeated action/event;
shared broad tags such as health, travel, sport, food, or work are insufficient.
* Omit the field entirely when there is nothing to link. Zero links is the normal case.
* Never use `related` as a substitute for UPDATE / SUPERSEDE.
---
{few_shot_section}
###################
Input Data
###################
Current date: {current_date}
## Existing memories
{existing_memories}
## New memories
{new_memories}
---
# Output
Return ONLY a JSON array inside a fenced ```json code block.
If no changes are needed:
```json
[]
```
Now produce the JSON array."""
RECONCILE_STRUCTURE_METADATA_PROMPT_EN = """
Additional structural output contract:
For every ADD, UPDATE, or SUPERSEDE result, preserve or refine `topic_paths`,
`event_facet`, and `topic_assignments`. `event_facet` contains only `subject`,
`participants`, `action`, `objects`, `locations`, and `state`.
`topic_assignments` contains subject-specific `{subject, role, path}` entries;
each path segment is `{name, type}` and type is `topic|facet|series|episode`.
Types follow `topic -> facet -> series -> episode`; levels may be skipped but
types may not repeat or move backward.
Use Series only for repeated occurrences of one concrete activity/state line and
Episode only for one bounded real-world occurrence. Never infer these types from
depth, a broad shared tag, or unrelated co-occurrence. When an Existing memory
already has a semantically equivalent subject-specific path, reuse its exact node
names and types instead of creating a synonym fork. To request exact reuse, emit
`structure_references: [{"memory_id":"<existing_id>", "subject":"...",
"through":"topic|facet|series|episode"}]`. The referenced ID must appear in
Existing memories. `through` means copy that existing path prefix through the
named structural type; an explicitly authored deeper suffix remains new. Reuse
is forbidden when the subject, concrete activity/state line, or Episode boundary
differs. These fields describe the memory, never a future query form such as
"when", "how many", or "who".
Series is a reconcile-time relation: establish it only when the new memory and
at least one independent Existing memory are distinct occurrences of the same
concrete activity/state line. Multiple propositions from the current write do
not prove repetition.
"""
# Prompt 文本统一放在 hy_memory/prompts/<language>.py(get_prompt 按语言码取);
# 以下旧名字保留为英文版别名(向后兼容)。
from ..prompts import ( # noqa: E402
get_prompt,
normalize_language,
resolve_language,
with_output_language,
)
from ..prompts.en import SEARCH_QUERY_PROMPT # noqa: E402,F401 (re-export)
from ..prompts.en import RECONCILE_PROMPT # noqa: E402,F401 (re-export)
from ..prompts.en import RECONCILE_STRUCTURE_METADATA_PROMPT as RECONCILE_STRUCTURE_METADATA_PROMPT_EN # noqa: E402,F401 (re-export)
from ..prompts.en import RECONCILE_FEW_SHOT as RECONCILE_FEW_SHOT_EN # noqa: E402,F401 (re-export)
def append_reconcile_structure_prompt(
@@ -461,10 +180,8 @@ def append_reconcile_structure_prompt(
"""Append the structure contract only for an explicit experiment config."""
if not enabled:
return prompt
if language == "zh":
from .prompts_zh import RECONCILE_STRUCTURE_METADATA_PROMPT_ZH
return prompt + RECONCILE_STRUCTURE_METADATA_PROMPT_ZH
return prompt + RECONCILE_STRUCTURE_METADATA_PROMPT_EN
code = "zh-Hans" if language == "zh" else language
return prompt + get_prompt("RECONCILE_STRUCTURE_METADATA_PROMPT", code)
def _as_aware(value: Any) -> Optional[datetime]:
@@ -550,147 +267,42 @@ def build_reconcile_prompt(
current_date: str,
few_shot_enabled: bool = False,
structure_metadata_enabled: bool = False,
language: Optional[str] = None,
) -> str:
"""Build the language-matched reconcile prompt used by the write path."""
from ..utils.lang_detect import is_chinese
"""Build the reconcile prompt used by the write path.
if is_chinese("\n".join(new_memories)):
from .prompts_zh import RECONCILE_FEW_SHOT_ZH, RECONCILE_PROMPT_ZH
few_shot = ("\n" + RECONCILE_FEW_SHOT_ZH + "\n") if few_shot_enabled else ""
prompt = RECONCILE_PROMPT_ZH.format(
current_date=current_date,
new_memories=new_memories_block,
existing_memories=existing_memories_block,
few_shot_section=few_shot,
)
return append_reconcile_structure_prompt(
prompt,
enabled=structure_metadata_enabled,
language="zh",
)
few_shot = ("\n" + RECONCILE_FEW_SHOT_EN + "\n") if few_shot_enabled else ""
prompt = RECONCILE_PROMPT.format(
``language`` set → that language's prompt plus the output-language rule;
None → zh/en auto-detection over the new memories (unchanged behaviour).
"""
explicit = normalize_language(language)
code = resolve_language(explicit, "\n".join(new_memories))
few_shot = (
("\n" + get_prompt("RECONCILE_FEW_SHOT", code) + "\n")
if few_shot_enabled else ""
)
prompt = get_prompt("RECONCILE_PROMPT", code).format(
current_date=current_date,
new_memories=new_memories_block,
existing_memories=existing_memories_block,
few_shot_section=few_shot,
)
return append_reconcile_structure_prompt(
prompt = append_reconcile_structure_prompt(
prompt,
enabled=structure_metadata_enabled,
language="en",
language=code,
)
return with_output_language(prompt, code, explicit=explicit is not None)
# Reconcile few-shot examples (injected into {few_shot_section} only when
# few_shot_enabled). Each shows the existing/new input and the expected ops,
# with a one-line takeaway. IDs are shortened for readability.
RECONCILE_FEW_SHOT_EN = """
###################
Examples
###################
## Example 1 — One new memory: part conflicts with an existing chain, part is unrelated
Existing memories:
[
{"memory_id": "m1", "content": "The user works as a product manager at Stripe.", "owner": "user", "history_versions": [{"content": "The user works as a product manager at a fintech startup."}]}
]
New memories:
[
{"content": "The user has left Stripe and is now a product lead at Notion, and has recently taken up pottery on weekends.", "owner": "user"}
]
Output:
```json
[
{"op": "SUPERSEDE", "memory_id": "m1", "content": "The user works as a product lead at Notion.", "supersede_reason": "employer changed from Stripe to Notion", "owner": "user", "tags": ["work"]},
{"op": "ADD", "content": "The user has recently taken up pottery on weekends.", "owner": "user", "tags": ["hobby"]}
]
```
A SINGLE new memory carries two things: the job change (same dimension as the existing employment chain m1, which already has history → SUPERSEDE, becomes the new chain head) and an unrelated detail (pottery → ADD). Split the one memory into the right op per part — the new head describes ONLY the employment state, never the pottery.
## Example 2 — Fold two unlinked existing memories + the new change into one 3-node chain
Existing memories:
[
{"memory_id": "a1", "content": "The user is learning to play the guitar.", "owner": "user", "memory_at": "2024-02-10 09:00"},
{"memory_id": "a2", "content": "The user can now play a few full songs on the guitar.", "owner": "user", "memory_at": "2024-09-05 09:00"}
]
New memories:
[
{"content": "The user now performs guitar at local open-mic nights.", "owner": "user"}
]
Output:
```json
[
{"op": "SUPERSEDE", "memory_ids": ["a1", "a2"], "content": "The user performs guitar at local open-mic nights.", "supersede_reason": "guitar skill progressed: learning → playing songs → performing live", "owner": "user", "tags": ["music", "hobby"]}
]
```
a1 and a2 were never linked, but they track the same dimension (guitar progress) and the new memory is the latest step. Fold them with `memory_ids` ordered oldest→newest; they become a 3-node chain (a1 ← a2 ← new head). The head `content` describes ONLY the current state — it does not restate "learning" or "a few songs".
## Example 3 — New head must NOT copy the old memory's content
Existing memories:
[
{"memory_id": "c1", "content": "The user lives in a rented one-bedroom apartment in Shibuya, Tokyo, pays ¥180,000/month, loves the 5-minute walk to the station, but complains the place is noisy at night and has no parking.", "owner": "user"}
]
New memories:
[
{"content": "The user just bought and moved into a three-bedroom house in the suburbs of Yokohama with a garden and a two-car garage.", "owner": "user"}
]
Output:
```json
[
{"op": "SUPERSEDE", "memory_id": "c1", "content": "The user bought and moved into a three-bedroom house in suburban Yokohama, with a garden and a two-car garage.", "supersede_reason": "residence changed from a rented Shibuya apartment to an owned Yokohama house", "owner": "user", "tags": ["housing"]}
]
```
The new memory SUPERSEDEs c1 and becomes the chain head. The head `content` describes ONLY the new Yokohama house. Do NOT drag the old apartment's distinctive details (Shibuya, ¥180,000 rent, the station walk, the noise/parking complaints) into the new node — those clearly belong to the old residence, not the new one. They stay in chain history and remain retrievable; copying them would make the head self-contradictory (renting in Shibuya AND owning in Yokohama at once).
## Example 4 — Different subjects are never merged, however similar the wording
Existing memories:
[
{"memory_id": "d1", "content": "The user drinks black coffee every morning.", "owner": "user"}
]
New memories:
[
{"content": "John drinks black coffee every morning.", "owner": "user"},
{"content": "The user now drinks black coffee twice a day, morning and after lunch.", "owner": "user"}
]
Output:
```json
[
{"op": "ADD", "content": "John drinks black coffee every morning.", "owner": "user", "tags": ["food"]},
{"op": "UPDATE", "memory_id": "d1", "content": "The user drinks black coffee twice a day, in the morning and after lunch.", "owner": "user", "tags": ["food"]}
]
```
The two new memories are near-identical in wording but have DIFFERENT subjects. "John ..." is about another person, so it is a plain ADD — never an UPDATE or SUPERSEDE of d1, even though d1 says almost the same thing about the user. Only the memory whose subject is also "The user" may merge with d1.
## Example 5 — Recurring event: bump count and keep a timeline (UPDATE, not ADD)
Existing memories:
[
{"memory_id": "i1", "content": "John injured his ankle last season and needed time off plus physical therapy.", "owner": "user", "memory_at": "2023-05-21 19:48"}
]
New memories:
[
{"content": "John twisted his ankle again during a recent game.", "owner": "user", "memory_at": "2023-10-13 13:50"}
]
Output:
```json
[
{"op": "UPDATE", "memory_id": "i1", "content": "John has injured his ankle two times: first last season (around the 2022–2023 season), needing time off and physical therapy; second during a game around 2023-10-13.", "owner": "user", "tags": ["health", "basketball", "injury"]}
]
```
Same event type happening again → UPDATE the old memory. State the **count** and list occurrences with times when known. Do NOT ADD a second "John injured his ankle" fact, and do NOT SUPERSEDE (the first injury did not stop being true).
"""
def build_search_query_prompt(
new_memories: List[str], *, language: Optional[str] = None,
) -> str:
"""Search-query generation prompt (same language rule as reconcile)."""
explicit = normalize_language(language)
code = resolve_language(explicit, "\n".join(new_memories))
mem_lines = "\n".join(f"{i+1}. {m}" for i, m in enumerate(new_memories))
prompt = get_prompt("SEARCH_QUERY_PROMPT", code).format(new_memories=mem_lines)
return with_output_language(prompt, code, explicit=explicit is not None)
# ================================================================
@@ -745,7 +357,9 @@ class MemoryReconciler:
self._llm = LLMProvider(self.config)
return self._llm
async def _generate_search_queries(self, new_memories: List[str]) -> tuple:
async def _generate_search_queries(
self, new_memories: List[str], language: Optional[str] = None,
) -> tuple:
"""
Pre-process: 从全部新 memory 中提取 search queries,
用于补充向量搜索的召回能力。
@@ -753,17 +367,7 @@ class MemoryReconciler:
Returns:
(queries: List[str], llm_response: Optional[LLMResponse])
"""
mem_lines = "\n".join(f"{i+1}. {m}" for i, m in enumerate(new_memories))
# Select prompt based on input language
from ..utils.lang_detect import is_chinese
if is_chinese("\n".join(new_memories)):
from .prompts_zh import SEARCH_QUERY_PROMPT_ZH
prompt = SEARCH_QUERY_PROMPT_ZH.format(new_memories=mem_lines)
else:
prompt = SEARCH_QUERY_PROMPT.format(new_memories=mem_lines)
prompt = build_search_query_prompt(new_memories, language=language)
try:
llm = self._get_llm()
@@ -790,49 +394,6 @@ class MemoryReconciler:
return [], None
@staticmethod
def _fallback_add_ops(
new_memories: List[str],
new_memories_with_meta: Optional[List[Dict]] = None,
) -> List[ReconcileOp]:
"""reconcile 彻底失败时的兜底:把抽取结果**原样全部 ADD**。
为什么必须有这个兜底:reconcile 只是决定「新事实和旧记忆怎么合」。它挂了(网关 500、
JSON 解析不出)不代表抽取结果没价值 —— 此前的行为是把一整批完好的事实**全部丢弃**,
线上实测因此丢掉过整块内容(含评测题的正确答案)。宁可留重复也不要丢内容:
重复还能被后续 dedup / reconcile 收拾,丢了就永远找不回来。
"""
metas = new_memories_with_meta or []
by_content = {
(m.get("content") or ""): m for m in metas if isinstance(m, dict)
}
ops: List[ReconcileOp] = []
for text in new_memories:
if not text:
continue
meta = by_content.get(text) or {}
ops.append(ReconcileOp(
op="ADD",
content=text,
layer=meta.get("layer"),
tags=meta.get("tags") or [],
owner=meta.get("owner"),
topic_paths=meta.get("topic_paths") or [],
event_facet=meta.get("event_facet") or {},
topic_assignments=meta.get("topic_assignments") or [],
topic_paths_provided=bool(
meta.get("topic_paths_provided", "topic_paths" in meta)
),
event_facet_provided=bool(
meta.get("event_facet_provided", "event_facet" in meta)
),
topic_assignments_provided=bool(meta.get(
"topic_assignments_provided", "topic_assignments" in meta,
)),
reason="reconcile unavailable → stored as-is (degraded)",
))
return ops
async def reconcile(
self,
new_memories: List[str],
@@ -849,6 +410,7 @@ class MemoryReconciler:
graph_store=None, # Optional GraphStore:Entity MENTIONS boost
new_memory_embeddings: Optional[Dict[str, List[float]]] = None,
source: str = "default",
language: Optional[str] = None,
) -> ReconcileResult:
"""
对 new_memories 做 reconcile。
@@ -856,6 +418,7 @@ class MemoryReconciler:
Args:
new_memories: 新记忆内容列表(content only,用于 embed 搜索)
new_memories_with_meta: 可选,含 tags 的完整 meta
language: 显式 prompt 语言码;None 时按新记忆做 zh/en 自动检测
"""
if not new_memories:
return ReconcileResult(ops=[], success=True)
@@ -876,7 +439,7 @@ class MemoryReconciler:
if self._enable_search_query:
search_queries, sq_response = await self._generate_search_queries(
new_memories
new_memories, language=language,
)
sq_elapsed = (_time.perf_counter() - t_sq) * 1000
@@ -887,18 +450,10 @@ class MemoryReconciler:
# 写 SEARCH_QUERY pipeline log
if cache and request_id:
mem_lines = "\n".join(f"{i+1}. {m}" for i, m in enumerate(new_memories))
if self._enable_search_query:
from ..utils.lang_detect import is_chinese as _is_zh
if _is_zh("\n".join(new_memories)):
from .prompts_zh import SEARCH_QUERY_PROMPT_ZH
sq_prompt = SEARCH_QUERY_PROMPT_ZH.format(
new_memories=mem_lines
)
else:
sq_prompt = SEARCH_QUERY_PROMPT.format(new_memories=mem_lines)
sq_prompt = build_search_query_prompt(
new_memories, language=language,
)
else:
sq_prompt = "[search query disabled — hybrid retrieval per new memory]"
try:
@@ -1447,6 +1002,7 @@ class MemoryReconciler:
getattr(self.config.llm, "few_shot_enabled", False)
),
structure_metadata_enabled=_structure_metadata_on,
language=language,
)
# ── Step 5: LLM 调用 + 重试 ──
@@ -1489,8 +1045,8 @@ class MemoryReconciler:
)
except Exception as e:
# LLM 彻底不可用(provider 已按 max_retries 重试过,网关 500 /
# 连接错误等)。**不能让它冒泡** —— 冒泡会让整批抽取结果连带丢弃。
# 交给下面的 ADD-all 兜底。
# 连接错误等)。不冒泡:下面统一记 RECONCILE 失败日志,再返回
# success=False —— writer 不落任何新记忆,整次写入报失败。
_llm_error = f"{type(e).__name__}: {e}"
logger.warning(f"[reconciler] RECONCILE LLM unavailable: {_llm_error}")
break
@@ -1498,7 +1054,7 @@ class MemoryReconciler:
_tok_pt += response.prompt_tokens
_tok_ct += response.completion_tokens
_tok_total += response.tokens_used
raw_response = response.content
raw_response = response.content or ""
ops = self._parse_ops(raw_response)
ops = enforce_time_order(ops, current_time, candidate_map)
ops = self._inherit_exact_input_structure(
@@ -1507,7 +1063,9 @@ class MemoryReconciler:
if _structure_metadata_on:
ops = self._resolve_structure_references(ops, candidate_map)
_is_empty_array = self._strip_code_fence(raw_response) in ("[]", "")
# 只有字面量 "[]" 才是「模型判定无需任何操作」。HTTP 200 但 content
# 为空不是决策,是失败:按 EMPTY_RESPONSE 处理,不能当成 0 条成功。
_is_empty_array = self._strip_code_fence(raw_response) == "[]"
if ops or _is_empty_array:
_parse_ok = True
break
@@ -1537,7 +1095,7 @@ class MemoryReconciler:
error_msg = (
f"RECONCILE LLM returned empty response after {_n_attempts} attempts"
)
error_code = "EMPTY_RESPONSE"
error_code = "EMPTY_RESPONSE"
logger.warning(f"[reconciler] {error_msg}")
if cache and request_id:
@@ -1554,14 +1112,13 @@ class MemoryReconciler:
{
"error": error_code,
"message": error_msg,
"fallback": "ADD_ALL",
"fallback_count": len(new_memories or []),
"stored": 0,
},
ensure_ascii=False,
),
elapsed_ms=recon_elapsed,
# LLM 彻底不可用时 response 是 None —— 这行本身会抛
# AttributeError,而它正好在兜底路径上,会把兜底也一起废掉
# LLM 彻底不可用时 response 是 None —— 直接取属性会抛
# AttributeError,把失败日志本身也废掉
prompt_tokens=getattr(response, "prompt_tokens", 0) or 0,
completion_tokens=getattr(response, "completion_tokens", 0) or 0,
total_tokens=getattr(response, "tokens_used", 0) or 0,
@@ -1571,22 +1128,16 @@ class MemoryReconciler:
f"[reconciler] store failed RECONCILE log failed: {e}"
)
# 兜底:JSON 解析不出来不等于抽取结果没价值 —— 全部 ADD,别丢内容。
_fb = self._fallback_add_ops(new_memories, new_memories_with_meta)
logger.warning(
f"[reconciler] falling back to ADD-all: {len(_fb)} memories "
f"({error_code})"
)
# 不做 ADD-all 兜底:没经过判重 / 演化的整批 ADD 会在重跑时产生重复,
# 而且调用方看到 success 就不会重试。这里报失败、不落库,由调用方重跑。
return ReconcileResult(
ops=_fb,
success=bool(_fb),
error=None if _fb else error_msg,
degraded=bool(_fb),
degraded_reason=error_code,
ops=[],
success=False,
error=error_msg,
error_code=error_code,
prompt_tokens=_tok_pt,
completion_tokens=_tok_ct,
total_tokens=_tok_total,
new_memory_embeddings=new_memory_embeddings,
)
# 写 RECONCILE pipeline log
@@ -1651,20 +1202,14 @@ class MemoryReconciler:
except Exception as e:
logger.error(f"[reconciler] failed: {e}", exc_info=True)
# 兜底同样适用于「reconcile 链路任意环节炸了」(候选检索、embed、
# prompt 组装…)。此前这里返回 ops=[] → writer 直接丢弃整批抽取结果。
_fb = self._fallback_add_ops(new_memories, new_memories_with_meta)
if _fb:
logger.warning(
f"[reconciler] falling back to ADD-all after unexpected error: "
f"{len(_fb)} memories"
)
return ReconcileResult(
ops=_fb, success=True,
degraded=True, degraded_reason=f"EXCEPTION:{type(e).__name__}",
new_memory_embeddings=dict(new_memory_embeddings or {}),
)
return ReconcileResult(ops=[], success=False, error=str(e))
# reconcile 链路任意环节炸了(候选检索、embed、prompt 组装…):此时还
# 没有任何新记忆落库,报失败让整次写入可以干净地重跑。
return ReconcileResult(
ops=[],
success=False,
error=f"{type(e).__name__}: {e}",
error_code="RECONCILE_EXCEPTION",
)
@staticmethod
def _inherit_exact_input_structure(
+10
View File
@@ -225,6 +225,16 @@ _PLACEHOLDER_VALUES = frozenset({
"not provided", "not specified", "not mentioned",
"未提供", "未知", "不详", "未说明", "未提及", "无",
"-", "?",
# 其他提示词语言里的「用户」/「未知」(显式 language 时模型可能照抄)
"使用者", "該使用者", "用戶", "該用戶", "不詳", "無",
"ユーザー", "ユーザ", "不明", "なし",
"사용자", "유저", "알 수 없음", "없음",
"el usuario", "usuario", "la usuaria", "desconocido", "ninguno",
"der nutzer", "nutzer", "der benutzer", "benutzer", "unbekannt", "keine",
"o usuário", "usuário", "o utilizador", "utilizador", "desconhecido", "nenhum",
"l'utilisateur", "utilisateur", "inconnu", "aucun",
"пользователь", "неизвестно", "нет",
"pengguna", "tidak diketahui", "tidak ada",
})
# 内置字段的中文标签:弱模型也会把字段说明的中文词当值。
+94 -10
View File
@@ -134,6 +134,7 @@ _WRITE_DIAGNOSTIC_KEYS = (
"agent_status",
"agent_error_code",
"agent_error",
"reconcile_error_code",
"l1_rollback",
"l1_rollback_failed",
"partial_agent_persist",
@@ -909,6 +910,19 @@ class HyMemoryClient:
"""
return "coding" if str(scene or "").strip().lower() == "coding" else "working"
def _resolve_language(self, language: Optional[str]) -> Optional[str]:
"""per-call language > MemoryConfig.language; None = zh/en 自动检测。
返回 canonical 码;未知码抛 ValueError(列出支持的码与别名)。
"""
from .prompts import normalize_language
code = normalize_language(language)
if code is not None:
return code
config = getattr(self, "_config", None)
return normalize_language(getattr(config, "language", None))
@staticmethod
def _parse_input(data: AddInput):
"""
@@ -1723,6 +1737,7 @@ class HyMemoryClient:
request_id: Optional[str] = None,
meta: Optional[Dict[str, Any]] = None,
embed_override: Optional[Dict[str, Any]] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""
写入记忆。
@@ -1773,6 +1788,13 @@ class HyMemoryClient:
meta 独立持久化,只做业务透传。
语义为「该次 write 时刻的上下文」,故存为 list——同一节点被
多次写入合并时会追加而非覆盖,详见 get() 返回的 meta 字段。
language: 本次请求的 LLM prompt 语言码(en / zh-Hans / zh-Hant / ja / es /
de / pt / fr / ko / ru / id;别名 zh、zh-CN → zh-Hans,zh-TW、
zh-HK → zh-Hant,pt-BR → pt)。优先于 MemoryConfig.language
(env MEMORY_LANGUAGE);两者都未设时沿用 zh/en 自动检测。
显式指定时 extract / reconcile / coding 链路 / 基础画像等本次
请求的全部 LLM 调用都用该语言的 prompt,记忆正文也用该语言书写。
未知码抛 ValueError(列出支持的码)。
Returns:
chat 链路:{"success": True, "memory_id": "...", "request_id": "...", "elapsed_ms": ...}
@@ -1788,6 +1810,7 @@ class HyMemoryClient:
workspace_id=workspace_id, branch=branch,
request_id=request_id, meta=meta,
embed_override=embed_override,
language=language,
),
include_runtime_diag=True,
)
@@ -1798,15 +1821,20 @@ class HyMemoryClient:
user_id: str,
agent_id: str = "default_agent",
session_id: str,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""Finalize one caller session into a first-class episode narrative.
The feature is opt-in through MEMORY_SESSION_EPISODE_ENABLED. Calls are
idempotent: the same user/agent/session tuple upserts one deterministic node.
``language`` (or MemoryConfig.language) makes the narrative be written in
that language; see add(). The episode prompts themselves are English-only.
"""
return self._loop_thread.run(
self.async_finalize_session(
user_id=user_id, agent_id=agent_id, session_id=session_id,
language=language,
),
include_runtime_diag=True,
)
@@ -1836,6 +1864,7 @@ class HyMemoryClient:
branch: Optional[str] = None,
request_id: Optional[str] = None,
layers: Optional[List[Any]] = None,
statuses: Optional[List[Any]] = None,
embed_override: Optional[Dict[str, Any]] = None,
query_embedding: Optional[List[float]] = None,
rerank: bool = True,
@@ -1928,6 +1957,8 @@ class HyMemoryClient:
layers: 可选。pro/ultra 下只召回指定层级(字符串或
MemoryLayer),在召回阶段生效。lite 下忽略该参数并
强制只召回 L1_RAW。
statuses: 仅用于 L1_RAW 定向搜索,可显式包含 active / shadow。
非 raw 搜索或 scene="all" 传入时拒绝,避免静默忽略。
rerank: 默认 True(原行为)。False 时本次调用跳过全部重排阶段:
chat 路的本地 cross-encoder 重排(recall.local_rerank_enabled)
与记忆关系图重排(graph_rerank / episode 扩展),working/
@@ -1971,7 +2002,7 @@ class HyMemoryClient:
intention_limit=intention_limit, created_after=created_after,
reader=reader,
workspace_id=workspace_id, branch=branch,
request_id=request_id, layers=layers,
request_id=request_id, layers=layers, statuses=statuses,
embed_override=embed_override,
query_embedding=query_embedding,
rerank=rerank,
@@ -2263,6 +2294,7 @@ class HyMemoryClient:
user_id: str,
agent_id: str = "default_agent",
max_batches: Optional[int] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""
手动触发 System 2 认知加工(ultra 模式专用,同步阻塞直到完成)。
@@ -2285,6 +2317,8 @@ class HyMemoryClient:
Args:
user_id: 用户 ID
agent_id: Agent ID
language: System 2 / sweeper prompt 语言码,语义同 add()(per-call >
MemoryConfig.language > 按 materials 做 zh/en 检测)。
Returns:
{"success": True, "request_id": "...", "tasks_processed": 1, "results": {...},
@@ -2298,6 +2332,7 @@ class HyMemoryClient:
return self._loop_thread.run(
self.async_evolve(
user_id=user_id, agent_id=agent_id, max_batches=max_batches,
language=language,
)
)
@@ -2307,8 +2342,10 @@ class HyMemoryClient:
user_id: str,
agent_id: str = "default_agent",
max_batches: Optional[int] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""手动触发 System 2 认知加工(异步)"""
language = self._resolve_language(language)
t0 = __import__("time").perf_counter()
# 获取 System2Writer
@@ -2331,6 +2368,7 @@ class HyMemoryClient:
result = await writer.evolve(
user_id=user_id, agent_id=agent_id, max_batches=max_batches,
language=language,
)
result["elapsed_ms"] = round((__import__("time").perf_counter() - t0) * 1000, 2)
return result
@@ -2339,6 +2377,7 @@ class HyMemoryClient:
self,
user_ids: List[str],
agent_ids: Optional[List[str]] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""
手动触发 cross-domain sweeper(同步)。
@@ -2352,6 +2391,8 @@ class HyMemoryClient:
Args:
user_ids: SDK user_id 列表
agent_ids: Agent ID 列表(默认 ["default_agent"])
language: 输出语言码,语义同 add()。sweeper prompt 只有英文版,显式
指定时追加该语言的输出规则。
Returns:
{"success": True, "results": [{...}, ...]}
@@ -2361,7 +2402,9 @@ class HyMemoryClient:
f"sweep() requires mode='ultra', but this client is mode='{self._mode}'."
)
return self._loop_thread.run(
self.async_sweep(user_ids=user_ids, agent_ids=agent_ids)
self.async_sweep(
user_ids=user_ids, agent_ids=agent_ids, language=language,
)
)
@_ensure_internal_loop
@@ -2369,11 +2412,13 @@ class HyMemoryClient:
self,
user_ids: List[str],
agent_ids: Optional[List[str]] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""手动触发 cross-domain sweeper(异步)
对每个 user_id × agent_id 组合执行一次 sweeper。
"""
language = self._resolve_language(language)
t0 = __import__("time").perf_counter()
try:
@@ -2385,7 +2430,9 @@ class HyMemoryClient:
results = []
for uid in user_ids:
for aid in effective_agents:
r = await writer.run_sweeper(user_id=uid, agent_id=aid)
r = await writer.run_sweeper(
user_id=uid, agent_id=aid, language=language,
)
results.append({"user_id": uid, "agent_id": aid, **r})
elapsed_ms = round((__import__("time").perf_counter() - t0) * 1000, 2)
@@ -2740,13 +2787,17 @@ class HyMemoryClient:
request_id: Optional[str] = None,
meta: Optional[Dict[str, Any]] = None,
embed_override: Optional[Dict[str, Any]] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""写入记忆(异步)
embed_override: 本次写入的 embedding 端点覆盖(不改 model/dims,只换接入点)。
dict,可含 base_url / api_key / extra_headers / timeout;None = 用 client 配置。
可用于按请求切换任意 OpenAI-compatible embedding endpoint。
language: 本次请求的 prompt 语言码,详见 add()。
"""
# 先校验语言码:非法值直接 ValueError,不占用 request_id 的幂等槽位
language = self._resolve_language(language)
request_id = request_id or str(uuid.uuid4())
replay = await self._claim_write_outcome(
request_id=request_id,
@@ -2770,7 +2821,7 @@ class HyMemoryClient:
source=source,
metadata=metadata, memory_at=memory_at,
scene=scene, workspace_id=workspace_id, branch=branch,
request_id=request_id, meta=meta,
request_id=request_id, meta=meta, language=language,
)
else:
result = await self._async_add_impl(
@@ -2778,7 +2829,7 @@ class HyMemoryClient:
source=source,
metadata=metadata, memory_at=memory_at,
scene=scene, workspace_id=workspace_id, branch=branch,
request_id=request_id, meta=meta,
request_id=request_id, meta=meta, language=language,
)
except BaseException:
await self._mark_write_outcome_unknown(request_id)
@@ -2797,8 +2848,10 @@ class HyMemoryClient:
user_id: str,
agent_id: str = "default_agent",
session_id: str,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""Async counterpart of :meth:`finalize_session`."""
language = self._resolve_language(language)
series = bool(getattr(self._config.recall, "series_episode_enabled", False))
dynamic = bool(
getattr(self._config.recall, "dynamic_episode_enabled", False) or series
@@ -2828,6 +2881,7 @@ class HyMemoryClient:
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
language=language,
)
if finalize is finalize_dynamic_episode_capsules:
finalize_kwargs["summary_concurrency"] = getattr(
@@ -2867,6 +2921,7 @@ class HyMemoryClient:
branch: Optional[str] = None,
request_id: Optional[str] = None,
meta: Optional[Dict[str, Any]] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""写入记忆(异步)
@@ -2875,6 +2930,7 @@ class HyMemoryClient:
None = 自动判定。详见 add() docstring。
workspace_id / branch: Coding memory 路径专用,详见 add() docstring。
meta: 调用方自定义业务 meta,透传存储,详见 add() docstring。
language: 已解析的 canonical 语言码(None = zh/en 自动检测)。
"""
from .utils.source_filter import normalize_source
source = normalize_source(source)
@@ -2973,13 +3029,16 @@ class HyMemoryClient:
request_id=request_id,
source=source,
memory_at=memory_at,
language=language,
)
# Public receipt always identifies the productivity namespace.
resp["variant"] = scene_variant
# 版本演进链层(scene="coding" 独有)。放在 writer.write 之后、返回
# 之前,因为它只依赖对话原文,与向量层写入结果无关。
# best-effort:这层挂不上不影响本次 add 的向量层产出。
if scene_variant == "coding":
# 向量层写入失败(success=False)时跳过:调用方会重跑整段,失败的
# 那次不该先把改动记进演进链。
if scene_variant == "coding" and resp.get("success"):
resp["chain_rounds"] = self._observe_chain(
chat_messages, user_id=uid,
workspace_id=workspace_id, branch=branch,
@@ -3038,6 +3097,7 @@ class HyMemoryClient:
# 落库 memory_ops / WRITE_SUMMARY / pipeline_logs 的单一真相源。
request_id=request_id,
client_meta=meta,
language=language,
)
# mode → writer:
@@ -3126,6 +3186,7 @@ class HyMemoryClient:
metadata: Optional[Dict[str, Any]] = None,
memory_at: Optional[datetime] = None,
request_id: Optional[str] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""写入「调用方已抽取好的」记忆:跳过 extract,直接进 reconcile。
@@ -3140,6 +3201,7 @@ class HyMemoryClient:
Args:
contents: 单条事实字符串,或多条事实字符串列表。
language: reconcile prompt 的语言码,语义同 add()。
其余参数语义同 add()。
Returns:
@@ -3151,6 +3213,7 @@ class HyMemoryClient:
contents, user_id=user_id, agent_id=agent_id,
session_id=session_id, source=source, metadata=metadata,
memory_at=memory_at, request_id=request_id,
language=language,
),
include_runtime_diag=True,
)
@@ -3167,8 +3230,10 @@ class HyMemoryClient:
metadata: Optional[Dict[str, Any]] = None,
memory_at: Optional[datetime] = None,
request_id: Optional[str] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""Idempotent async wrapper for extracted-memory writes."""
language = self._resolve_language(language)
request_id = request_id or str(uuid.uuid4())
replay = await self._claim_write_outcome(
request_id=request_id,
@@ -3189,6 +3254,7 @@ class HyMemoryClient:
metadata=metadata,
memory_at=memory_at,
request_id=request_id,
language=language,
)
except BaseException:
await self._mark_write_outcome_unknown(request_id)
@@ -3211,6 +3277,7 @@ class HyMemoryClient:
metadata: Optional[Dict[str, Any]] = None,
memory_at: Optional[datetime] = None,
request_id: Optional[str] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""写入已抽取好的记忆(异步);跳过 extract,直接 reconcile。详见 add_extracted()。"""
request_id = request_id or str(uuid.uuid4())
@@ -3251,6 +3318,7 @@ class HyMemoryClient:
extra=extra,
memory_at=memory_at,
request_id=request_id,
language=language,
)
# lite/pro → MemoryWriter;ultra → System2Writer(内部复用 MemoryWriter)
@@ -3322,12 +3390,14 @@ class HyMemoryClient:
request_id: str,
t0: float,
query_embedding: Optional[List[float]],
statuses: Optional[List[Any]],
) -> Dict[str, Any]:
"""Implement `search(..., layers=['l1_raw'])` without a second API."""
from .models.memory import MemoryLayer, MemoryStatus
embedding = query_embedding or await self._embed_service.embed(query)
fetch = min(max(limit * (10 if session_ids else 2), limit), 10_000)
status_filter = self._normalize_statuses(statuses) or [MemoryStatus.ACTIVE]
hits = await self._vector_store.search(
query_embedding=embedding,
user_ids=user_ids,
@@ -3337,7 +3407,7 @@ class HyMemoryClient:
layers=[MemoryLayer.L1_RAW],
limit=fetch,
score_threshold=min_score,
status_filter=[MemoryStatus.ACTIVE],
status_filter=status_filter,
only_latest=False,
created_after=created_after,
)
@@ -3355,7 +3425,7 @@ class HyMemoryClient:
"memory_id": row.pop("raw_id"),
**row,
"layer": MemoryLayer.L1_RAW.value,
"status": MemoryStatus.ACTIVE.value,
"status": node.status.value,
"score": round(float(hit.get("score", 0.0)), 4),
}
)
@@ -3799,6 +3869,7 @@ class HyMemoryClient:
branch: Optional[str] = None,
request_id: Optional[str] = None,
layers: Optional[List[Any]] = None,
statuses: Optional[List[Any]] = None,
embed_override: Optional[Dict[str, Any]] = None,
query_embedding: Optional[List[float]] = None,
rerank: bool = True,
@@ -3828,7 +3899,7 @@ class HyMemoryClient:
schema_min_score=schema_min_score, schema_limit=schema_limit,
intention_limit=intention_limit, created_after=created_after,
reader=reader, workspace_id=workspace_id, branch=branch,
request_id=request_id, layers=layers,
request_id=request_id, layers=layers, statuses=statuses,
query_embedding=query_embedding,
rerank=rerank,
)
@@ -3843,7 +3914,7 @@ class HyMemoryClient:
schema_min_score=schema_min_score, schema_limit=schema_limit,
intention_limit=intention_limit, created_after=created_after,
reader=reader, workspace_id=workspace_id, branch=branch,
request_id=request_id, layers=layers,
request_id=request_id, layers=layers, statuses=statuses,
query_embedding=query_embedding,
rerank=rerank,
)
@@ -3873,6 +3944,7 @@ class HyMemoryClient:
branch: Optional[str] = None,
request_id: Optional[str] = None,
layers: Optional[List[Any]] = None,
statuses: Optional[List[Any]] = None,
query_embedding: Optional[List[float]] = None,
rerank: bool = True,
) -> Dict[str, Any]:
@@ -3958,6 +4030,17 @@ class HyMemoryClient:
"Use single agent_id with multiple session_ids, or multiple agent_ids without session_ids."
)
if statuses is not None:
from .models.memory import MemoryLayer
requested_layers = self._normalize_layers(layers)
if scene_all or (
not lite_mode and requested_layers != [MemoryLayer.L1_RAW]
):
raise ValueError(
"statuses is supported only for explicit L1_RAW search"
)
if scene_all:
return await self._search_all_scenes(
query, queries=queries, search_mode=search_mode,
@@ -4293,6 +4376,7 @@ class HyMemoryClient:
request_id=request_id,
t0=t0,
query_embedding=query_embedding,
statuses=statuses,
)
read_req = ReadRequest(
+2
View File
@@ -12,6 +12,7 @@ HY Memory - Coding Memory 模块
from .types import (
CodingMemory,
CodingWriteError,
CodingMemoryDraft,
ReconcileOp,
WorkingFact,
@@ -26,6 +27,7 @@ from .types import (
__all__ = [
"CodingMemory",
"CodingWriteError",
"CodingMemoryDraft",
"ReconcileOp",
"WorkingFact",
+96 -703
View File
@@ -21,14 +21,19 @@ from typing import List, Optional, Dict, Any, Tuple
from ..agent.llm_provider import LLMProvider
from ..pipelines.base import ChatMessage
from ..utils.lang_detect import is_chinese
from ..prompts import (
get_prompt,
normalize_language,
resolve_language,
with_output_language,
)
from .preproc import (
extract_files,
keep_last_k_user_turns,
prepare_extractor_messages,
)
from .judge import _store_coding_log
from .types import CodingMemoryDraft, WorkingFactDraft
from .types import CodingMemoryDraft, CodingWriteError, WorkingFactDraft
@@ -65,332 +70,7 @@ _VERBATIM_STATEMENT_CHARS = 600
# Prompt
# ================================================================
EXTRACT_PROMPT = """\
You extract durable WORKING memories from a task-execution session segment.
A "working" session is any hands-on task where the user operates a computer /
system through tools — this covers BOTH software/code work (writing, debugging,
refactoring, building, deploying, API/library usage) AND system & operations
work (running shell/PowerShell commands, disk/storage analysis or cleanup, file
& folder operations, installing/configuring software, environment/device setup,
hardware/disk health inspection, backup, scheduling/automation, data processing).
You produce TWO layers from the SAME segment:
EXPERIENCES (procedural, "how to do it") — reusable method, extracted only
when it clears the VALUE BAR below. A (task, search_keys, solution,
boundary_envs) bundle.
FACT (episodic, "what happened this time") — EXACTLY ONE record of THIS
session: what the user wanted, how it was gone about, how it came out. No
value bar, but there is never more than one.
The two layers answer different questions and are stored separately. A method
that generalizes goes in EXPERIENCES; the story of this particular session —
only true for this system / this project / right now — goes in the FACT.
═══════════════════════════════════════════════════════════════════
VALUE BAR (EXPERIENCES ONLY) — DO NOT cross this lightly.
═══════════════════════════════════════════════════════════════════
A memory MUST satisfy ALL of:
(a) Solves a problem likely to recur for the same user
(b) Solution is non-trivial: combines multiple pieces of info / steps, OR
carries a decision rationale, OR captures a hard-won workaround, OR is a
reusable procedure/recipe for an operational task (e.g. how to analyze &
clean up disk usage, how to batch-organize files, how to inspect hardware
health, how to set up a scheduled job)
(c) A future agent / user would prefer looking it up over re-discovering it
IMPORTANT — analysis / diagnostic / investigation tasks COUNT:
A task does NOT have to make a change or "fix" something to be worth remembering.
Read-only / analytical / troubleshooting work is valuable when its PROCEDURE or
CONCLUSION is reusable. Explicitly extract:
- Analysis / diagnostic recipes: the reusable procedure + what it revealed
(e.g. "how to analyze which folders eat C-drive space" — the scan commands,
the gotchas hit, and the finding). "It only produced a report, no change"
is NOT a reason to skip — the reusable METHOD is the memory.
- Investigation outcomes, INCLUDING negative / not-found results
(e.g. "searched folder X across C: and D: with rg / fs_search / SQL index —
not found anywhere"). A recorded dead-end saves the next agent from repeating
the same fruitless search, so it clears the bar.
Do NOT reject a memory merely because "the assistant could regenerate the
commands" or "nothing was actually modified". If the procedure or the conclusion
would save future effort, keep it.
DO NOT extract:
- Truly trivial single-step ops with no reusable procedure ("rename one file",
"git status", "run ls once")
- One-off issues already resolved with no transferable lesson
- Information that is already in the repo/system (don't duplicate package.json)
- Per-PR / per-session throwaway tasks unless they encode a reusable pattern
- The act of ASKING THE USER for missing information. Requesting a parameter,
listing what is unknown, or eliciting required fields is conversational
behaviour, not a method: every agent already knows how to ask a question, so
there is nothing to look up later. Never emit an experience whose task or
search_keys are shaped like "how to ask for X", "gather the missing Y",
"eliciting required fields", "what to clarify before starting". Note this
stays out of EXPERIENCES only — that the session ended waiting on the user
is real state and belongs in the FACT.
If the segment made no substantive progress because it stalled waiting on the
user — the assistant only asked for parameters, or was blocked before doing any
real work — emit "experiences": [] and write the FACT alone. Nothing was
learned yet, so there is no method to store. This matters more than it looks:
an experience distilled from a stall reads as a legitimate skill on retrieval,
and a later segment of the same task will match it and ask the user again
instead of proceeding, so the stall reproduces itself for the rest of the
session. Judge substance by what the tools actually did, not by whether the
turn produced fluent text.
Heuristic: if the user (or a future agent doing the same kind of task) would
want this written down instead of figuring it out again, keep it. A multi-step
operational procedure (even a system/ops or read-only analysis/search one, not
just code) usually qualifies. Only drop it if it is a single command any user
would find via `--help` in 30 seconds AND carries no decision/gotcha worth
remembering.
═══════════════════════════════════════════════════════════════════
EXPERIENCE FIELDS
═══════════════════════════════════════════════════════════════════
task — Query-shaped name; phrase as how a user would later ASK.
"publish a release to the xx platform" not
"publish_release_internal"
Self-contained (no "this", "the file", "we").
search_keys — 0 to 5 alternative phrasings capturing important aspects of
solution that `task` alone wouldn't match. Phrase as questions
or topics future users would ASK ABOUT. Distinct from task.
Omit if task already covers everything.
solution — Complete, self-contained content the user actually needs:
commands, credential locations (NOT credential VALUES),
file paths, ordered steps, gotchas, decision rationale.
Aggregate multi-turn end-to-end work into ONE solution if it
forms a coherent task; split into multiple memories only when
the user did unrelated tasks.
boundary_envs — Concrete pinning that solution depends on, multi-line "key: value":
runtime / SDK / library + version
API endpoint or method signature
external service / middleware version
env var NAME (not value)
config file path
host / cluster / region identifier
If solution is environment-agnostic, leave as "" (empty).
Purpose: when the underlying SDK/middleware upgrades, future
consumers can inspect this to decide if solution still applies.
confidence — 0.0 to 1.0
1.0 user explicitly stated this as a rule / instruction
0.7 inferred with good evidence from tool actions
0.5 weakly inferred; one observation; uncertain
═══════════════════════════════════════════════════════════════════
FACT FIELD — EXACTLY ONE per segment, no value bar
═══════════════════════════════════════════════════════════════════
You write ONE fact for the whole segment — a recoverable state record for this
session, not a polished retrospective and never a list of separate findings. A
future agent must be able to resume from it without mistaking a suggestion,
inference, or partial search for something verified.
subject — WHO or WHAT this session is about: the single entity the work was
performed on or for. One short noun phrase, no sentence, no verb.
Pick the most specific entity that the session is *about*, not the
tool it used:
a person's name when the session serves one named person
("Dylan", "Matthew")
the file / module / symbol when the work targets code
("auth/middleware.ts", "parse_plan_text")
the service / host / device / drive for ops work
("D:\\WXWork", "redis-cache-prod", "ci.yml")
This is a HARD KEY, not prose: two sessions about different people
or different files MUST get different subjects, and two sessions
about the same one MUST get the same string, spelled identically.
Prefer the exact identifier as it appears in the trace over a
description of it. If the segment genuinely has no single subject,
output "".
Why it matters: subject is the gate that decides whether a later
session is allowed to overwrite this one as "the same task,
continued". Get it wrong and one person's state silently replaces
another's.
statement — A compact record of THIS session. Two parts, and the second is the
one that gets lost.
Preserve task-scoped givens and concrete constraints instead of
replacing them with a generic evaluation of the outcome. State
each rule once: repeated instruction text competes with the data
the memory must retain.
PART 1 — what happened, about 300 latin characters (~150 CJK).
Start with the specific goal and object ("reclaim disk space taken
by D:\\WXWork", "locate the new_file_search folder"), not a generic
"fix a bug" / "find a file". Then roughly what was inspected, tried,
or changed. Then how it ended, with the semantics of exactly one of:
completed, partially completed, failed, blocked, or
still awaiting verification. Include negative results and dead ends
without overstating them.
PART 2 — the TASK-SCOPED GIVENS: everything the WHOLE task runs on,
even when this segment did not establish it and even when it feels
too elementary to state. Bindings (what each name/symbol/label
stands for), fixed values the task supplies rather than derives,
conventions adopted (indexing base, units, timezone, defaults), and
this segment's own result WITH its value ("x = 7", not "the value
was computed").
Write PART 2 as data, one item per line, values as given:
retry budget = 5; timeout 30s; index starts at 0
step 3 input: n=64, tolerance 1e-6
slot B: type=private, cost 688.7-721.3 per unit
A line should be almost entirely values. Keep the trace's notation —
symbols, units, numbers, spellings — and drop its sentences: strip
the phrases that announce a given rather than being one ("I would
prefer", "it should be somewhere between … and …"). They repeat once
per item, so a 15-item list pays for them 15 times.
Carry them on EVERY fact; never write "as above" or "same as the
previous session" — retrieval returns a handful of facts by
similarity and the referent may not be among them.
Aim under 1200 characters total. If PART 2 does not fit, fix the
notation, never the coverage: statement is the retrieval vector AND
the only field a later segment can read (details is neither embedded
nor injected), so anything cut from it is gone. Compress the wording
around a given; never the given.
The test: if the next segment cannot see the original material, can
it continue from this record alone? Only a yes counts. A missing
binding does not fail loudly — the name still looks meaningful, so
it gets read with its general-usage meaning and the answer that
follows is fluent and about a different question.
LANGUAGE — same language as the session trace: statement is the
retrieval vector and queries arrive in the user's working language,
so another language pays a cross-lingual penalty on every lookup.
EVIDENCE FIDELITY — record only actions actually executed and
evidence explicitly present in the trace. Preserve material
commands, flags, arguments, errors, and numbers verbatim.
NEVER complete a command with customary flags, and never turn
advice or plans into actions:
"ran Get-ChildItem -Force" → "ran Get-ChildItem"
(the trace did not contain `-Force`; do not invent it)
Scope every negative finding to what was actually searched, and
enumerate EVERY explicit coverage limit that could invalidate it,
especially permission failures and depth limits: keep the roots,
filters, and depth that were in effect ("not found under C:\\ and
D:\\ at depth 3; some directories were not covered due to Access
denied"), never widen it to "does not exist". Do not omit one limit
to make the record prettier.
Never promote a partial search to global absence.
A blocked attempt keeps its blocker.
Preserve the distinction among attempted, observed, verified,
inferred, and unresolved — express it naturally ("tried … but was
blocked by Access denied", "suspected …, not yet verified"), not as
labels. Without tool output or explicit confirmation, NEVER claim a
fix succeeded, a cause was established, or a path does not exist.
Do NOT record greetings or pure chatter — that is the only case
getting "fact": null. A segment that merely stated requirements, or
stalled before doing work, still established something.
details — The session's concrete OUTPUT, copied over verbatim. `statement` is
a summary and a summary necessarily drops the individual values; a
later lookup almost always wants one specific value, not the gist.
So whatever addressable result this session produced, reproduce it
here in its original form and structure:
a plan / schedule / itinerary → keep every line, every day, every
slot, and every price exactly as written
a config or code block → keep the block
a command's material output → keep the lines that carry the
findings (sizes, counts, versions, error text)
a structured result (table, list of records) → keep the rows
Preserve the original layout — line breaks, `Day 1:` / key: value
prefixes, units, currency symbols, ids. Do not re-summarize, do not
reorder, do not round numbers, do not translate.
Copy ONLY what is present in the trace. Never fill a gap: if a slot
was left unfilled, leave it unfilled or say so — an invented value
is worse than a missing one, because it will be retrieved and used.
Cap around 4000 characters; if the output is longer, keep the parts
carrying concrete values and drop decorative prose, noting what was
dropped.
"" when the session produced no such artifact (pure investigation
with no findings yet, or a stall waiting on the user).
This field is NOT part of the retrieval vector — `statement` and
`subject` do the finding, `details` supplies the value once found.
So do not write retrieval bait here, and do not duplicate the
summary; write the raw thing.
files — 0..N absolute or repo-relative FILE OR DIRECTORY paths actually
operated on or investigated during this session. Keep investigated
roots such as `C:\\` and `D:\\`. Only include paths that were
actually accessed/searched, not paths merely suggested or planned;
[] if none apply.
═══════════════════════════════════════════════════════════════════
HOST CONTEXT (given by SDK)
═══════════════════════════════════════════════════════════════════
- workspace_id present: {workspace_present}
- branch present: {branch_present}
When no workspace_id is present, the solution cannot rely on unstated
project-local context — spell out any repo / path / service it needs inside
`solution` or `boundary_envs` so the memory stays usable on its own.
═══════════════════════════════════════════════════════════════════
CONVERSATION
═══════════════════════════════════════════════════════════════════
{conversation_block}
═══════════════════════════════════════════════════════════════════
OUTPUT (strict JSON only, no markdown)
═══════════════════════════════════════════════════════════════════
{{
"experiences": [
{{
"task": "...",
"search_keys": ["...", "..."],
"solution": "...",
"boundary_envs": "...",
"confidence": 0.9
}}
],
"fact": {{
"subject": "...",
"statement": "...",
"files": ["..."],
"details": "..."
}}
}}
DEFAULT TO EXTRACTING. When the segment did real work, you MUST output at least
one experience — even if the task looks simple or was a one-shot success, still
capture the reusable essence of what was done (the method / command / how-to).
When the session is thin, the single most valuable experience is enough.
The ONE exception is the stall case above: if the segment did no real work
because it was blocked asking the user for missing information, output
"experiences": [] and write only the fact. Do not manufacture an experience to
satisfy this instruction — a fabricated one ("how to ask for the departure
city") is worse than none, because it is later retrieved and followed. Empty is
allowed ONLY for that reason, never as a shortcut when work was actually done.
"fact" is a SINGLE object, never a list — one record per session.
ALWAYS write the fact. The bar for the fact is not "was this valuable" but "did
anything at all happen", and a segment reaching you has already cleared that.
This includes segments that look like setup rather than work: a single
user message stating requirements, parameters, or a starting state, with no
tools run, is exactly the material a later session cannot reconstruct on its own
— 把需求、参数、起始状态连同情形一起写进 `statement`(紧凑记法,不要照抄对话原句),
不要推给 `details`(`details` 不进检索向量,读取侧也不再注入它)。Use
"fact": null ONLY for greetings and pure chatter with no content whatsoever.
When in doubt, write the fact: an unneeded record costs one row, while a dropped
one costs every later session that needed it and has no way to know it existed.
"""
from ..prompts.en import CODING_EXTRACT_PROMPT as EXTRACT_PROMPT # noqa: E402,F401 (re-export)
# ================================================================
@@ -421,350 +101,7 @@ one costs every later session that needed it and has no way to know it existed.
#
# 修改 prompt 时应使用代表性 coding traces 做回归,确认坐标、决策与验证状态仍保留。
EXTRACT_PROMPT_CODING = """\
╔═══════════════════════════════════════════════════════════════════════════╗
║ HARD CONSTRAINT — read this before anything else, it overrides any ║
║ phrasing later in this prompt that sounds like it wants a narrative. ║
║ ║
║ You are NOT writing a report of what a person did. You are writing what a ║
║ FUTURE AGENT needs in order to act. That agent already has its own task; ║
║ it does not care what this user wanted. It needs to know: what was decided ║
║ here, at which code coordinates, what was tried and rejected, and whether ║
║ it was verified. ║
║ ║
║ Concretely, in every `statement` and every `solution`: ║
║ · The grammatical subject is the CODE or THE DECISION — never "the user" ║
║ and never "the agent". Do not write "The user attempted to…", ║
║ "The user wanted…", "The agent then checked…". ║
║ · Never close with a bookkeeping phrase like "the session was completed", ║
║ "the error was noted", "no solution was found yet". Close with the ║
║ decision and its verification status. If nothing was decided, close ║
║ with what was RULED OUT and the next unexplored lead — that is the ║
║ reusable part. ║
║ · Keep the concrete symptom (error string, failing command, wrong ║
║ output) — that is what a future search will match on. ║
║ ║
║ bad: "The user attempted to write a QTable to RST with header_rows and ║
║ hit a TypeError. The session was completed with the error noted." ║
║ good: "ascii.rst's RST writer rejects header_rows: `tbl.write(..., \ ║
║ format='ascii.rst', header_rows=['name','unit'])` raises TypeError ║
║ because RST.__init__ does not forward it, unlike ascii.fixed_width ║
║ which accepts it. No fix written — the writer needs the same ║
║ header_rows plumbing as fixed_width." ║
╚═══════════════════════════════════════════════════════════════════════════╝
You extract durable WORKING memories from a task-execution session segment.
A "working" session is any hands-on task where the user operates a computer /
system through tools — this covers BOTH software/code work (writing, debugging,
refactoring, building, deploying, API/library usage) AND system & operations
work (running shell/PowerShell commands, disk/storage analysis or cleanup, file
& folder operations, installing/configuring software, environment/device setup,
hardware/disk health inspection, backup, scheduling/automation, data processing).
You produce TWO layers from the SAME segment:
EXPERIENCES (procedural, "how to do it") — reusable method, extracted only
when it clears the VALUE BAR below. A (task, search_keys, solution,
boundary_envs) bundle.
FACT (episodic, "what was established this time") — EXACTLY ONE record of THIS
session. It is NOT a narrative of what the user did. It records, in this order:
the task and its concrete symptom (this is what makes it findable later), the
code coordinates, WHAT WAS SETTLED ON and what was rejected, and how far it was
verified. No value bar, but there is never more than one.
The two layers answer different questions and are stored separately. A method
that generalizes goes in EXPERIENCES; the story of this particular session —
only true for this system / this project / right now — goes in the FACT.
═══════════════════════════════════════════════════════════════════
VALUE BAR (EXPERIENCES ONLY) — DO NOT cross this lightly.
═══════════════════════════════════════════════════════════════════
A memory MUST satisfy ALL of:
(a) Solves a problem likely to recur for the same user
(b) Solution is non-trivial: combines multiple pieces of info / steps, OR
carries a decision rationale, OR captures a hard-won workaround, OR is a
reusable procedure/recipe for an operational task (e.g. how to analyze &
clean up disk usage, how to batch-organize files, how to inspect hardware
health, how to set up a scheduled job)
(c) A future agent / user would prefer looking it up over re-discovering it
IMPORTANT — analysis / diagnostic / investigation tasks COUNT:
A task does NOT have to make a change or "fix" something to be worth remembering.
Read-only / analytical / troubleshooting work is valuable when its PROCEDURE or
CONCLUSION is reusable. Explicitly extract:
- Analysis / diagnostic recipes: the reusable procedure + what it revealed
(e.g. "how to analyze which folders eat C-drive space" — the scan commands,
the gotchas hit, and the finding). "It only produced a report, no change"
is NOT a reason to skip — the reusable METHOD is the memory.
- Investigation outcomes, INCLUDING negative / not-found results
(e.g. "searched folder X across C: and D: with rg / fs_search / SQL index —
not found anywhere"). A recorded dead-end saves the next agent from repeating
the same fruitless search, so it clears the bar.
Do NOT reject a memory merely because "the assistant could regenerate the
commands" or "nothing was actually modified". If the procedure or the conclusion
would save future effort, keep it.
DO NOT extract:
- Truly trivial single-step ops with no reusable procedure ("rename one file",
"git status", "run ls once")
- One-off issues already resolved with no transferable lesson
- Information that is already in the repo/system (don't duplicate package.json)
- Per-PR / per-session throwaway tasks unless they encode a reusable pattern
Heuristic: if the user (or a future agent doing the same kind of task) would
want this written down instead of figuring it out again, keep it. A multi-step
operational procedure (even a system/ops or read-only analysis/search one, not
just code) usually qualifies. Only drop it if it is a single command any user
would find via `--help` in 30 seconds AND carries no decision/gotcha worth
remembering.
═══════════════════════════════════════════════════════════════════
CODE WORK — what a future agent cannot re-derive, so it MUST be captured
═══════════════════════════════════════════════════════════════════
When the segment is software work on a repository, the reusable method is only
half of it. The other half is WHERE the work lives and WHICH VERSION won. Those
cannot be recovered by reading the code later — the code shows the current state,
not which branch it sits on, nor what was already rejected.
This section does NOT ask for a longer record. It asks you to SPEND THE SAME
BUDGET on higher-value tokens: drop restated goals, narration of what the
assistant "then decided to", and pasted code you are only echoing; spend those
characters on the four items below instead. A record that omits the branch but
quotes 30 lines of diff is a bad trade.
1. BRANCH / WORKING LOCATION — verbatim, in the fact `statement`. If the segment
mentions or checks out `feat/x` / `fix/y`, that token MUST appear. An agent
that resumes on the wrong branch produces changes that are silently thrown
away, so this one token outranks every other detail.
2. WHICH VERSION WON — when the user changed their mind, name the winner and mark
the loser in one clause: "settled on A; the interim B was dropped". Not both in
full — one clause. Keeping only the latest is wrong: an agent that meets the
rejected code must recognize it, or it will "fix" things back to the discarded
variant. Phrases like `scratch that / go back to / actually, let's / no, not that`
mark this: what follows is of record, what precedes is superseded.
3. EXACT COORDINATES, NOT PASTED CODE — repo-relative path
(`qutebrowser/utils/debug.py`, never bare `debug.py`) plus the function /
class touched. If a signature changed, give the new signature. If a constant,
regex or literal was decided, quote THAT verbatim — but quote only it, not the
surrounding block. One accurate line beats a paragraph of approximation.
4. VERIFIED vs MERELY WRITTEN — four words is enough ("tests run and passing" /
"confirmed by git diff only, tests not run"). "the edit is in" and "the edit is in
and tested" lead a future agent to do very different things next.
Anti-pattern — summarizing the goal and dropping the coordinates:
bad: "refactored the colour parsing logic to be cleaner."
good: "on feat/q4, in qutebrowser/config/configtypes.py QtColor._parse_value:
settled on `mult = 360.0 if kind == 'h' else 255.0`; the interim version that
used 255.0 for everything was dropped. Tests not run."
(the good one is SHORTER than most bad ones — it just spends every character
on something that cannot be re-derived)
═══════════════════════════════════════════════════════════════════
EXPERIENCE FIELDS
═══════════════════════════════════════════════════════════════════
task — Query-shaped name; phrase as how a user would later ASK.
"publish a release to the xx platform", not "publish_release_internal"
Self-contained (no "this", "the file", "we").
search_keys — 0 to 5 alternative phrasings capturing important aspects of
solution that `task` alone wouldn't match. Phrase as questions
or topics future users would ASK ABOUT. Distinct from task.
Omit if task already covers everything.
solution — Complete, self-contained content the user actually needs:
commands, credential locations (NOT credential VALUES),
file paths, ordered steps, gotchas, decision rationale.
Aggregate multi-turn end-to-end work into ONE solution if it
forms a coherent task; split into multiple memories only when
the user did unrelated tasks.
boundary_envs — Concrete pinning that solution depends on, multi-line "key: value":
runtime / SDK / library + version
API endpoint or method signature
external service / middleware version
env var NAME (not value)
config file path
host / cluster / region identifier
If solution is environment-agnostic, leave as "" (empty).
Purpose: when the underlying SDK/middleware upgrades, future
consumers can inspect this to decide if solution still applies.
confidence — 0.0 to 1.0
1.0 user explicitly stated this as a rule / instruction
0.7 inferred with good evidence from tool actions
0.5 weakly inferred; one observation; uncertain
═══════════════════════════════════════════════════════════════════
FACT FIELD — EXACTLY ONE per segment, no value bar
═══════════════════════════════════════════════════════════════════
You write ONE fact for the whole segment — a recoverable state record for this
session, not a polished retrospective and never a list of separate findings. A
future agent must be able to resume from it without mistaking a suggestion,
inference, or partial search for something verified.
statement — A compact summary of THIS session. It has two jobs at once, and it
must do BOTH: it is the retrieval vector (a future agent finds it by
searching with a fresh issue report, phrased the way a user phrases a
problem), and it is the payload that agent then acts on. So:
OPEN with the task and the symptom, in the words a user would use.
This is what makes it findable — keep the surface form close to how
the problem would be reported. Name the specific goal and object
("reclaim disk space under D:\\WXWork", "adjust the hue multiplier
in QtColor._parse_value on feat/q4", "locate the new_file_search
folder"), never a generic "fix a bug" / "look for a file". Include
the concrete symptom: the error string, the failing command, the
wrong output.
THEN give the coordinates and the decision.
repo-relative path, symbol, and branch for code work. State what was
SETTLED ON, and in the same clause mark what was dropped
("settled on X; the interim Y was dropped"). This is the part that
cannot be re-derived by reading the code — the code only shows the
winner, never that Y was tried and rejected. An agent that meets the
rejected variant later must be able to recognize it, or it will
"fix" the code back to the discarded version.
CLOSE with the evidence status.
Exactly one of: completed, partially completed, failed, blocked, or
still awaiting verification — plus how it was checked ("tests run and
passing" / "confirmed by git diff only, tests not run"). Include
negative results and dead ends without overstating them.
⚠️ Never stop at the narrative. "The user tried X and hit error Y; no
solution found yet" is a wasted record: it retrieves fine but gives a
future agent nothing to act on. If the session genuinely reached no
decision, say what was ruled out and what the next unexplored lead is —
that is the reusable part.
The subject of the sentences should be the code and the decision, not
"the user". 2 to 4 sentences, flowing prose, not labelled sections.
Hard ceiling ~400 latin characters (~200 CJK characters).
EVIDENCE FIDELITY — mandatory:
- Record only actions actually executed and evidence explicitly
present in the trace. Preserve material commands, flags,
arguments, errors, and numbers verbatim. NEVER complete a
command with customary flags or turn advice/plans into actions.
- Preserve the distinction among attempted, observed, verified,
inferred, and unresolved. Do not add those labels to the prose;
express their meaning naturally, e.g. "tried … but was blocked by
Access denied" or "suspect … , not yet verified".
- Without tool output or explicit confirmation, NEVER claim a fix
succeeded, a cause was established, a file/folder does not
exist, or tests passed.
- Scope every negative finding to what was actually searched and
enumerate EVERY explicit coverage limit that could invalidate
it, especially permission failures and maximum search depth:
"not found within the searched parts of C:\\ and D:\\; some
directories were skipped on Access denied, and the search was also
depth-limited". Do not omit one limit to make the
summary prettier. Never promote a partial search to global
absence.
- State causality as certain only when the trace directly verifies
it. Otherwise retain the hedge ("suspected" / "inferred" / "not yet
verified"); do not rewrite an
incorrect test assumption as the system's root cause.
CRITICAL — this text IS the retrieval vector, and it is the only
thing a future query gets matched against. A generic summary is
worthless, because EVERY session looks like a generic summary: it
would match every query and answer none of them. So you MUST name
the concrete anchors that make THIS session findable — real file
paths, function / symbol names, branch names, commands, error
strings, versions, config keys and values, host / service names.
If someone later searches for any one of those specifics, this
summary has to be the thing that comes back.
good: "fixing a failing `npm run build` in CI. Read
.github/workflows/ci.yml and package.json; traced it to webpack
5.74 hitting ERR_OSSL_EVP_UNSUPPORTED under node 18. Bumping the
workflow's node to 20 made the build pass; node 18 still
reproduces locally."
good: "on branch feat/q0, in qutebrowser/utils/debug.py signal_name:
PyQt<5.11 only exposes the name via repr(sig), and repr comes in
both a bracketed and a parenthesised form. Settled on two patterns
tried in order with fullmatch, breaking on the first hit; the
single-regex version was dropped. Confirmed by git diff, tests not
run."
bad: "user wanted a build bug fixed, the agent looked at some config
files, changed some code, and fixed it."
(no path, no command, no error string, no version — nothing
to retrieve on; every session reads exactly like this)
bad: "fixed the signal-name parsing compatibility issue on a feature
branch."
(which branch? which file? which function? which of the two
competing versions won? nothing here survives being
retrieved a week later)
Fidelity examples (bad → good):
"ran Get-ChildItem -Force" → "ran Get-ChildItem"
(the trace did not contain `-Force`; do not invent it)
"confirmed new_file_search does not exist" →
"new_file_search was not found within the searched parts of C:\\ and
D:\\; some directories were skipped on Access denied or by the depth
limit, so the investigation is only partly done"
"identical payloads cause a spurious lock error" →
"the verification script assumed the last writer is always a fixed
rank; that assumption does not hold, so whether there is a lock bug
is still unverified"
Do NOT record greetings or pure chatter. If the segment established
nothing at all, output "fact": null.
files — 0..N absolute or repo-relative FILE OR DIRECTORY paths actually
operated on or investigated during this session. Keep investigated
roots such as `C:\\` and `D:\\`. Only include paths that were
actually accessed/searched, not paths merely suggested or planned;
[] if none apply.
═══════════════════════════════════════════════════════════════════
HOST CONTEXT (given by SDK)
═══════════════════════════════════════════════════════════════════
- workspace_id present: {workspace_present}
- branch present: {branch_present}
When no workspace_id is present, the solution cannot rely on unstated
project-local context — spell out any repo / path / service it needs inside
`solution` or `boundary_envs` so the memory stays usable on its own.
═══════════════════════════════════════════════════════════════════
CONVERSATION
═══════════════════════════════════════════════════════════════════
{conversation_block}
═══════════════════════════════════════════════════════════════════
OUTPUT (strict JSON only, no markdown)
═══════════════════════════════════════════════════════════════════
{{
"experiences": [
{{
"task": "...",
"search_keys": ["...", "..."],
"solution": "...",
"boundary_envs": "...",
"confidence": 0.9
}}
],
"fact": {{
"statement": "...",
"files": ["..."]
}}
}}
MANDATORY: You MUST output at least ONE experience. Do NOT return an empty list.
Even if the task looks simple or is a one-shot success, still capture the
reusable essence of what was done (the method / command / how-to). Only the
single most valuable experience is required when the session is thin — but never
output an empty "experiences" array.
"fact" is a SINGLE object, never a list — one record per session. Use
"fact": null only when the segment is pure chatter that established nothing.
"""
from ..prompts.en import CODING_EXTRACT_PROMPT_CODING as EXTRACT_PROMPT_CODING # noqa: E402,F401 (re-export)
# ================================================================
@@ -839,15 +176,22 @@ class CodingMemoryExtractor:
if str(message.content or "").strip()
)
def _select_prompt_template(self, messages: List[ChatMessage]) -> Tuple[str, str]:
"""Return (template, language) for this write's new messages."""
if not is_chinese(self._language_text(messages)):
return self._prompt_tmpl, "en"
def _select_prompt_template(
self, messages: List[ChatMessage], language: Optional[str] = None,
) -> Tuple[str, str]:
"""Return (template, language code) for this write's new messages.
from .prompts_zh import EXTRACT_PROMPT_CODING_ZH, EXTRACT_PROMPT_ZH
prompt = EXTRACT_PROMPT_CODING_ZH if self.scene == "coding" else EXTRACT_PROMPT_ZH
return prompt, "zh"
Explicit ``language`` wins; otherwise zh/en auto-detection over the new
user text (unchanged behaviour).
"""
code = resolve_language(language, self._language_text(messages))
if code == "en":
return self._prompt_tmpl, code
name = (
"CODING_EXTRACT_PROMPT_CODING" if self.scene == "coding"
else "CODING_EXTRACT_PROMPT"
)
return get_prompt(name, code), code
async def extract(
self,
@@ -861,16 +205,22 @@ class CodingMemoryExtractor:
branch: Optional[str] = None,
session_id: Optional[str] = None,
request_id: str = "",
language: Optional[str] = None,
) -> Tuple[List[CodingMemoryDraft], List[WorkingFactDraft]]:
"""
Args:
messages: 本次 write 的完整新消息;只做单条限长,不裁 turn
history_messages: 之前 write 的历史消息;仅取最近 5 个 user-turn 作消解上下文
user_id / agent_id / workspace_id / branch / session_id: SDK 注入
language: 显式 prompt 语言码;None 时按新 user 文本做 zh/en 自动检测
Returns:
(经验草稿, 事实草稿)。两层都可能为空。只有显式启用
verbatim_fallback_enabled 时,模型没产出 fact 才调用 _verbatim_fact。
Raises:
CodingWriteError: 未开启原文兜底时,LLM 调用失败(LLM_ERROR)、返回空
content(EMPTY_RESPONSE)或回复解析不出 JSON 对象(JSON_PARSE_FAILED)。
"""
if not messages:
return [], []
@@ -892,27 +242,26 @@ class CodingMemoryExtractor:
history_block = self._format_conversation(history_window).strip()
new_block = self._format_conversation(new_messages).strip()
prompt_tmpl, prompt_language = self._select_prompt_template(new_messages)
if prompt_language == "zh":
conversation_block = (
"最近 5 个用户轮次(仅作上下文,不要重复抽取):\n"
+ (history_block or "(无)")
+ "\n\n本次新增消息(只从这次完整写入中抽取):\n"
+ (new_block or "(无)")
)
else:
conversation_block = (
"LAST 5 USER TURNS (CONTEXT ONLY; do not extract them again):\n"
+ (history_block or "(none)")
+ "\n\nNEW MESSAGES (extract only from this complete write):\n"
+ (new_block or "(none)")
)
explicit_language = normalize_language(language)
prompt_tmpl, prompt_language = self._select_prompt_template(
new_messages, explicit_language,
)
empty_block = get_prompt("CODING_EMPTY_BLOCK", prompt_language)
conversation_block = get_prompt(
"CODING_CONVERSATION_BLOCK", prompt_language,
).format(
history=history_block or empty_block,
new=new_block or empty_block,
)
prompt = prompt_tmpl.format(
workspace_present=str(bool(workspace_id)).lower(),
branch_present=str(bool(branch)).lower(),
conversation_block=conversation_block,
)
prompt = with_output_language(
prompt, prompt_language, explicit=explicit_language is not None,
)
_t0 = time.perf_counter()
try:
@@ -923,8 +272,9 @@ class CodingMemoryExtractor:
)
except Exception as e:
logger.warning(f"[coding-extract] LLM call failed: {e}")
# 经验层不兜底。事实层原文兜底也默认关闭,避免 assistant/tool 轨迹被
# 当作 statement 直接展示;仅在配置显式开启时保留旧行为。
# 经验层不兜底。事实层原文兜底默认关闭,避免 assistant/tool 轨迹被
# 当作 statement 直接展示;关闭时 LLM 失败抛 CodingWriteError(LLM_ERROR),
# 整次写入报失败,而不是「成功、0 条」。仅在配置显式开启时保留旧行为。
verbatim = (
self._verbatim_fact(messages)
if self.verbatim_fallback_enabled else None
@@ -951,9 +301,33 @@ class CodingMemoryExtractor:
),
elapsed_ms=round((time.perf_counter() - _t0) * 1000, 2),
)
if not self.verbatim_fallback_enabled:
raise CodingWriteError(
"LLM_ERROR",
f"coding extract LLM call failed: {type(e).__name__}: {e}",
) from e
return [], ([verbatim] if verbatim else [])
_elapsed = round((time.perf_counter() - _t0) * 1000, 2)
# 模型答了且没有可记的内容(如 {"experiences": [], "fact": null})是正常的空
# 抽取;200 但 content 为空、或回复解析不出 JSON 对象,是模型失败,不能当成
# 「成功、0 条」。原文兜底显式开启时沿用旧行为(交给下面的 verbatim 兜底)。
failure = self._output_failure(resp.content)
if failure is not None and not self.verbatim_fallback_enabled:
code, message = failure
await _store_coding_log(
self.cache, request_id=request_id, user_id=user_id, agent_id=agent_id,
session_id=session_id or "", step="CODING_EXTRACT", prompt=prompt,
response=resp.content or "",
parsed=json.dumps(
{"drafts": 0, "error": code, "message": message},
ensure_ascii=False,
),
elapsed_ms=_elapsed, prompt_tokens=resp.prompt_tokens,
completion_tokens=resp.completion_tokens, total_tokens=resp.tokens_used,
)
raise CodingWriteError(code, message)
raw_experiences, raw_facts = self._parse_output(resp.content)
# 经验层的 files 走规则抽取(整段轨迹碰过的文件);事实层的 files 由 LLM
@@ -1133,12 +507,22 @@ class CodingMemoryExtractor:
return "\n".join(rows)
@classmethod
def _parse_output(
cls, text: str
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
"""宽松解析 LLM 输出 JSON,返回 (experiences, facts) 两个原始列表。"""
def _output_failure(cls, text: Optional[str]) -> Optional[Tuple[str, str]]:
"""回复本身是失败时返回 (error_code, message);正常(含空抽取)返回 None。"""
if not (text or "").strip():
return "EMPTY_RESPONSE", "coding extract LLM returned empty response"
if cls._load_output_object(text) is None:
return (
"JSON_PARSE_FAILED",
f"coding extract output is not a JSON object ({len(text)} chars)",
)
return None
@staticmethod
def _load_output_object(text: Optional[str]) -> Optional[Dict[str, Any]]:
"""宽松解析 LLM 输出,返回顶层 JSON 对象;解析不出返回 None。"""
if not text:
return [], []
return None
s = text.strip()
# 去 markdown fence
if s.startswith("```"):
@@ -1164,8 +548,17 @@ class CodingMemoryExtractor:
obj = json.loads(s[start: end + 1])
except Exception:
obj = None
return obj if isinstance(obj, dict) else None
if not isinstance(obj, dict):
@classmethod
def _parse_output(
cls, text: str
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
"""宽松解析 LLM 输出 JSON,返回 (experiences, facts) 两个原始列表。"""
if not text:
return [], []
obj = cls._load_output_object(text)
if obj is None:
logger.warning(
f"[coding-extract] failed to parse output ({len(text or '')} chars)"
)
+161 -182
View File
@@ -18,17 +18,24 @@ import json
import logging
import os
import uuid
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
from ..agent.llm_provider import LLMProvider
from ..core.embed_service import EmbedService
from ..utils.lang_detect import is_chinese
from ..prompts import (
get_prompt,
normalize_language,
resolve_language,
with_output_language,
)
from .store import CodingMemoryStore
from .types import (
MAX_FACT_CHAIN_ENTRIES,
CodingWriteError,
FactReconcileOp,
WorkingFact,
WorkingFactDraft,
@@ -206,143 +213,10 @@ def _subject_conflict(new_subject: str, old_subject: str) -> bool:
# General subject-isolation rules belong in the prompt; lookalike examples
# encourage pattern matching instead of teaching the rule.
FACT_RECONCILE_PROMPT = """\
You maintain session-level WORKING FACTS. Each statement summarizes one whole
trace/session: the user's task, the rough approach, and the result. It is not an
atomic claim.
Every new session fact below WILL be stored; that is not your decision. Your only
job is to detect task continuation: is the new session continuing to advance the
SAME user task as an existing session fact? If so, SUPERSEDE the older fact so
the links form a cross-session progress chain for that task.
Mark an existing fact as superseded ONLY when there is strong evidence that:
- The subjects match: both records are about the SAME entity — the same
person, the same file/module, the same host/service. A shared subject is
necessary but not sufficient; AND
- The user goal is materially the SAME, not merely in the same topic/project;
AND
- The new trace continues, retries, extends, verifies, or finishes the work
recorded by the existing trace.
Do NOT supersede when:
- The two records name different subjects. Two people in one group, two files
in one module, two hosts in one cluster are PARALLEL, not successive: one
person's state is not a newer version of another's. Where a subject is shown
as (unspecified), fall back to judging by the statement alone.
- The sessions merely mention the same file, symbol, component, repository,
error, or technology but pursue different user goals.
- One task only supplies context or a technique for another task.
- The relationship is uncertain. Be conservative: INSERT is the safe default;
merging unrelated sessions destroys their separate task history.
Treat each statement as evidence-bounded text. Do not reinterpret an inference
inside it as an established cause or use a shared conjecture/error as proof that
the user goals are the same. Your decision is only about task continuation; do
not "correct", complete, or embellish either session record.
Examples:
existing: "The user needed to fix CI's npm run build; traced it to
ERR_OSSL_EVP_UNSUPPORTED under Node 18, workflow not yet changed."
new: "The user continued fixing the same CI build; raised Node in
.github/workflows/ci.yml to 20 and npm run build passed."
→ SUPERSEDE
existing: "The user needed to fix JWT expiry handling in
auth/middleware.ts; refresh logic was added."
new: "The user needed performance logging added to
auth/middleware.ts; a latency metric was introduced."
→ keep both
(same file/project, different user goal)
═══════════════════════════════════════════════════════════════════
NEW SESSION FACTS
═══════════════════════════════════════════════════════════════════
{drafts_block}
═══════════════════════════════════════════════════════════════════
EXISTING SESSION FACTS (per new fact, retrieved by statement similarity)
═══════════════════════════════════════════════════════════════════
{candidates_block}
═══════════════════════════════════════════════════════════════════
OUTPUT (strict JSON only, no markdown)
═══════════════════════════════════════════════════════════════════
{{
"ops": [
{{"draft_idx": 0, "supersedes": ["fact_xxx"], "reason": "continues the same CI fix task"}},
{{"draft_idx": 1, "supersedes": []}}
]
}}
Only use fact ids that appear in the candidate list for that same draft_idx.
If nothing is superseded anywhere, output {{"ops": []}}.
"""
from ..prompts.en import CODING_FACT_RECONCILE_PROMPT as FACT_RECONCILE_PROMPT # noqa: E402,F401 (re-export)
CHAIN_SUMMARY_PROMPT = """\
A user task in a working-memory store has progressed across sessions. Below are
the session summaries in that task's progress chain, oldest first, ending with
the current session.
Summarize how the SAME task progressed across sessions to its current state.
Cover every important stage in chronological order. For each stage, preserve
where available: the actual action or change, its direct result, and any
unresolved issue or limiting condition. End with an explicit current status.
Rules:
- This is compression only. Report ONLY what the entries say; never add or
complete a command, cause, motive, action, test result, success state, or
global negative conclusion.
- Preserve evidence status exactly: attempted is not completed, observed is
not verified, inferred is not established, and unresolved stays unresolved.
Keep uncertainty words and search/permission/coverage limits.
- State the current ending status faithfully (completed, partially completed,
failed, blocked, or still awaiting verification). Never upgrade it based on
a plausible causal story or on a later entry that did not verify it.
- When entries mention a suspected cause and a later result, connect them
causally only if an entry directly says that cause was verified.
- Preserve concrete retrieval anchors: file paths, symbols, commands, error
strings, versions, config keys/values, and important numeric values that
matter to the progress.
- For chains of 2-3 generations, do not over-compress the progression into one
sentence. Prefer 2-4 compact paragraphs when useful, or readable prose with
a clear task goal, stage-by-stage progression, and current status.
- Do not mechanically copy the full history. Omit repetition and low-value
detail while retaining important actions, direct results, and leftovers.
- Do not invent stage headings or dates. Mention a date only when the entries
make it meaningful; bare timestamps are noise.
- If the entries say older revisions were omitted, explicitly state that
earlier history was omitted; do not imply the supplied entries are complete.
- Write in the same language the entries are written in.
- No preamble or enclosing quotes.
Example
entries:
(2026-07-20) The user needed to fix CI's npm run build; traced it to
ERR_OSSL_EVP_UNSUPPORTED under Node 18, workflow not yet
changed.
(2026-07-24) The user continued that CI fix; raised Node in
.github/workflows/ci.yml to 20, but the dependency cache
still reproduced the old error.
(now) The user continued verifying the same fix; after clearing the
npm cache, npm run build passed.
output:
The task goal is fixing CI's npm run build. It was first traced to
ERR_OSSL_EVP_UNSUPPORTED under Node 18, with the workflow not yet changed;
then .github/workflows/ci.yml was raised to Node 20, but the dependency
cache kept the old error present. Finally, after clearing the npm cache,
npm run build passed, so this build fix is currently verified as complete.
═══════════════════════════════════════════════════════════════════
ENTRIES
═══════════════════════════════════════════════════════════════════
{chain_block}
═══════════════════════════════════════════════════════════════════
OUTPUT (the summary itself, nothing else)
═══════════════════════════════════════════════════════════════════
"""
from ..prompts.en import CODING_CHAIN_SUMMARY_PROMPT as CHAIN_SUMMARY_PROMPT # noqa: E402,F401 (re-export)
def _fact_drafts_language_text(drafts: List[WorkingFactDraft]) -> str:
@@ -357,37 +231,44 @@ def _fact_drafts_language_text(drafts: List[WorkingFactDraft]) -> str:
def build_fact_reconcile_prompt(
drafts: List[WorkingFactDraft],
candidates: List[List[WorkingFact]],
language: Optional[str] = None,
) -> Tuple[str, str]:
"""Return the language-matched fact reconcile prompt and language."""
if is_chinese(_fact_drafts_language_text(drafts)):
from .prompts_zh import FACT_RECONCILE_PROMPT_ZH
"""Return the fact reconcile prompt and its language code.
template = FACT_RECONCILE_PROMPT_ZH
language = "zh"
else:
template = FACT_RECONCILE_PROMPT
language = "en"
return template.format(
Explicit ``language`` wins; otherwise zh/en auto-detection over the drafts.
"""
explicit = normalize_language(language)
code = resolve_language(explicit, _fact_drafts_language_text(drafts))
prompt = get_prompt("CODING_FACT_RECONCILE_PROMPT", code).format(
drafts_block=WorkingFactReconciler._format_drafts(drafts),
candidates_block=WorkingFactReconciler._format_candidates(drafts, candidates),
), language
)
return with_output_language(prompt, code, explicit=explicit is not None), code
def build_chain_summary_prompt(
chain_block: str,
*,
current_statement: str,
language: Optional[str] = None,
) -> Tuple[str, str]:
"""Return the language-matched cross-session summary prompt and language."""
if is_chinese(current_statement or chain_block):
from .prompts_zh import CHAIN_SUMMARY_PROMPT_ZH
"""Return the cross-session summary prompt and its language code."""
explicit = normalize_language(language)
code = resolve_language(explicit, current_statement or chain_block)
prompt = get_prompt("CODING_CHAIN_SUMMARY_PROMPT", code).format(
chain_block=chain_block,
)
return with_output_language(prompt, code, explicit=explicit is not None), code
template = CHAIN_SUMMARY_PROMPT_ZH
language = "zh"
else:
template = CHAIN_SUMMARY_PROMPT
language = "en"
return template.format(chain_block=chain_block), language
@dataclass
class FactReconcilePlan:
"""plan() 的产物:覆盖判定已完成、尚未落库。apply() 才执行 store 操作。"""
drafts: List[WorkingFactDraft]
candidates: List[List[WorkingFact]] = field(default_factory=list)
supersedes: Dict[int, List[str]] = field(default_factory=dict)
context: Dict[str, Any] = field(default_factory=dict)
# ================================================================
@@ -455,16 +336,66 @@ class WorkingFactReconciler:
branch: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
language: Optional[str] = None,
) -> Tuple[List[WorkingFact], List[FactReconcileOp]]:
"""
执行 reconcile + 持久化。memory_at 是这段对话的时间,记到新事实上。
执行 reconcile + 持久化(plan() 后紧接 apply())。memory_at 是这段对话的
时间,记到新事实上。language 为显式 prompt 语言码(None → zh/en 自动检测),
同时用于演变总结。
Returns:
(真正落库的新事实, 执行过的 ops)。LLM 挂掉时退化为全 INSERT 不覆盖,
宁可多留一条过期事实,也不因为一次调用失败丢结论。
(真正落库的新事实, 执行过的 ops)。
Raises:
CodingWriteError(RECONCILE_FAILED):覆盖判定的 LLM 失败 / 空回复 /
解析不出。此时既不插入新事实,也不覆盖任何旧事实。
"""
plan = await self.plan(
drafts,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
language=language,
)
return await self.apply(plan)
async def plan(
self,
drafts: List[WorkingFactDraft],
*,
user_id: str,
agent_id: str = "default_agent",
session_id: Optional[str] = None,
workspace_id: Optional[str] = None,
branch: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
language: Optional[str] = None,
) -> FactReconcilePlan:
"""检索 candidates + 覆盖判定,不落库。
没有任何候选时不调 LLM(全部 INSERT 是确定的结论,不是兜底)。调了 LLM 时,
有效回复 = JSON 对象且 "ops" 是列表(空列表 = 都不覆盖、只插入)。LLM 调用
失败(LLM_UNAVAILABLE)、200 但 content 为空(EMPTY_RESPONSE)、回复不是
上述形状(JSON_PARSE_FAILED)时抛 CodingWriteError(RECONCILE_FAILED)。
此前 LLM 失败会退化为全 INSERT 不覆盖,留下本该被接续的旧事实。
"""
context = dict(
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
language=language,
)
if not drafts:
return [], []
return FactReconcilePlan(drafts=[], context=context)
candidates = await self._retrieve_candidates(
drafts,
@@ -477,27 +408,38 @@ class WorkingFactReconciler:
)
try:
supersedes = await self._llm_decide(drafts, candidates)
supersedes = await self._llm_decide(drafts, candidates, language=language)
except CodingWriteError:
raise
except Exception as e:
logger.warning(
f"[fact-reconcile] LLM decide failed, insert without supersede: {e}"
)
supersedes = {}
logger.warning(f"[fact-reconcile] LLM decide failed: {e}")
raise CodingWriteError(
"RECONCILE_FAILED",
f"working fact reconcile LLM call failed: {type(e).__name__}: {e}",
cause="LLM_UNAVAILABLE",
) from e
return FactReconcilePlan(
drafts=list(drafts),
candidates=candidates,
supersedes=supersedes,
context=context,
)
async def apply(
self, plan: FactReconcilePlan,
) -> Tuple[List[WorkingFact], List[FactReconcileOp]]:
"""执行 plan() 的决策(覆盖旧事实 + 插入新事实)。"""
if not plan.drafts:
return [], []
facts, ops = await self._execute(
drafts,
candidates,
supersedes,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
plan.drafts,
plan.candidates,
plan.supersedes,
**plan.context,
)
logger.info(
f"[fact-reconcile] drafts={len(drafts)} inserted={len(facts)} "
f"[fact-reconcile] drafts={len(plan.drafts)} inserted={len(facts)} "
f"superseded={sum(1 for o in ops if o.action == 'SUPERSEDE')}"
)
return facts, ops
@@ -588,18 +530,24 @@ class WorkingFactReconciler:
self,
drafts: List[WorkingFactDraft],
candidates: List[List[WorkingFact]],
language: Optional[str] = None,
) -> Dict[int, List[str]]:
"""返回 {draft_idx: [被它覆盖的旧 fact_id, ...]}。"""
# 一条候选都没有就没得判,省一次 LLM 调用
if not any(candidates):
return {}
prompt, _ = build_fact_reconcile_prompt(drafts, candidates)
prompt, _ = build_fact_reconcile_prompt(drafts, candidates, language=language)
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=self.max_tokens,
audit_stage="coding_fact_reconcile",
)
failure = self._reply_failure(resp.content)
if failure is not None:
code, message = failure
logger.warning(f"[fact-reconcile] {code}: {message}")
raise CodingWriteError("RECONCILE_FAILED", message, cause=code)
return self._parse_ops(resp.content)
# ------------------------------------------------------------
@@ -638,7 +586,8 @@ class WorkingFactReconciler:
return ordered[-MAX_FACT_CHAIN_ENTRIES:], truncated
async def _summarize_chain(
self, history: List[WorkingFact], current: str, *, truncated: bool = False
self, history: List[WorkingFact], current: str, *, truncated: bool = False,
language: Optional[str] = None,
) -> str:
"""让 LLM 把同一任务的跨 session 进展写成简短摘要。
@@ -659,6 +608,7 @@ class WorkingFactReconciler:
prompt, _ = build_chain_summary_prompt(
"\n".join(rows),
current_statement=current,
language=language,
)
try:
@@ -724,11 +674,24 @@ class WorkingFactReconciler:
rows.append(f"for draft[{i}]:\n" + "\n".join(cand_rows))
return "\n\n".join(rows) if rows else "(none)"
@classmethod
def _reply_failure(cls, text: Optional[str]) -> Optional[Tuple[str, str]]:
"""回复本身是失败时返回 (cause, message);有效回复(含 {"ops": []})返回 None。"""
if not (text or "").strip():
return "EMPTY_RESPONSE", "working fact reconcile LLM returned empty response"
obj = cls._load_object(text)
if not isinstance(obj, dict) or not isinstance(obj.get("ops"), list):
return (
"JSON_PARSE_FAILED",
f"working fact reconcile output has no ops list ({len(text)} chars)",
)
return None
@staticmethod
def _parse_ops(text: str) -> Dict[int, List[str]]:
"""宽松解析 ops,返回 {draft_idx: [fact_id...]}。"""
def _load_object(text: Optional[str]) -> Optional[Dict[str, Any]]:
"""宽松解析出顶层 JSON 对象;解析不出返回 None。"""
if not text:
return {}
return None
s = text.strip()
# 去 markdown fence
if s.startswith("```"):
@@ -746,7 +709,14 @@ class WorkingFactReconciler:
obj = json.loads(s[start: end + 1])
except Exception:
obj = None
return obj if isinstance(obj, dict) else None
@classmethod
def _parse_ops(cls, text: str) -> Dict[int, List[str]]:
"""宽松解析 ops,返回 {draft_idx: [fact_id...]}。"""
if not text:
return {}
obj = cls._load_object(text)
if not isinstance(obj, dict):
logger.warning(
f"[fact-reconcile] failed to parse ops output ({len(text or '')} chars)"
@@ -790,6 +760,7 @@ class WorkingFactReconciler:
branch: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
language: Optional[str] = None,
) -> Tuple[List[WorkingFact], List[FactReconcileOp]]:
inserted: List[WorkingFact] = []
ops: List[FactReconcileOp] = []
@@ -813,6 +784,11 @@ class WorkingFactReconciler:
memory_at=memory_at,
)
# 先确认新事实能向量化,再去覆盖旧事实:否则 embedding 挂掉时旧事实
# 已被软删、新事实又落不了库,这条结论就凭空没了。失败抛
# CodingWriteError(EMBEDDING_FAILED),整次写入报失败。
new_embedding = await self.store.embed_fact(new_fact)
allowed = {c.fact_id for c in (candidates[idx] if idx < len(candidates) else [])}
targets = [
fid for fid in supersedes.get(idx, [])
@@ -873,7 +849,8 @@ class WorkingFactReconciler:
retired, user_id=user_id
)
new_fact.fact_chain_summary = await self._summarize_chain(
history, new_fact.statement, truncated=truncated
history, new_fact.statement, truncated=truncated,
language=language,
)
# 摘要之外再把历代自己的 details 原样过继一份。摘要保叙事,过继保数值,
# 两者不重复:details 本来就不进摘要(_collect_history 只读 statement)。
@@ -892,7 +869,9 @@ class WorkingFactReconciler:
)
try:
await self.store.insert_fact(new_fact)
await self.store.insert_fact(new_fact, embedding=new_embedding)
except CodingWriteError:
raise
except Exception as e:
logger.warning(f"[fact-reconcile] INSERT failed: {e}")
continue
+9 -741
View File
@@ -1,745 +1,13 @@
# -*- coding: utf-8 -*-
"""HY Memory working/coding 链路使用的中文提示词模板。"""
"""HY Memory working/coding 链路使用的中文提示词模板(兼容入口)。
EXTRACT_PROMPT_ZH = """\
你需要从一段任务执行会话中抽取可长期保留的 WORKING 记忆。
“working”会话是指用户通过工具操作计算机或系统的任何实操任务——既包括软件/代码
工作(编写、调试、重构、构建、部署、API/library 使用),也包括系统与运维工作
(运行 shell/PowerShell 命令、磁盘/存储分析或清理、文件与文件夹操作、安装/配置
软件、环境/设备设置、硬件/磁盘健康检查、备份、调度/自动化、数据处理)。
你需要从同一段会话中产出两个层次:
EXPERIENCES(过程性,“如何做”)——可复用的方法;只有达到下述 VALUE BAR 时才
抽取。每条是一个 (task, search_keys, solution, boundary_envs) 组合。
FACT(情节性,“这次发生了什么”)——关于本次会话的恰好一条记录:用户想做什么、
大致如何推进、最终结果如何。它不受价值门槛限制,但绝不能超过一条。
这两个层次回答不同问题,并分别存储。可泛化的方法放入 EXPERIENCES;只对这个系统、
这个项目、此时此刻成立的本次会话经历放入 FACT。
═══════════════════════════════════════════════════════════════════
VALUE BAR(仅适用于 EXPERIENCES)——不要轻易越过此门槛。
═══════════════════════════════════════════════════════════════════
一条记忆必须同时满足以下全部条件:
(a) 解决了同一用户以后很可能再次遇到的问题
(b) 解决方案并非琐碎操作:它组合了多项信息/多个步骤,或者包含决策理由,或者记录了
来之不易的 workaround,或者是某项运维任务的可复用流程/配方(例如如何分析并
清理磁盘占用、如何批量整理文件、如何检查硬件健康、如何设置 scheduled job)
(c) 未来的 agent / 用户会更愿意查阅它,而不是重新摸索
重要——分析、诊断、调查任务也算:
一项任务不必实际修改或“修复”任何东西才值得记忆。只读、分析、故障排查工作,只要其
流程或结论可复用,就有价值。明确抽取:
- 分析/诊断配方:可复用的流程以及它揭示的结果(例如“如何分析哪些文件夹占用了
C-drive 空间”——扫描命令、遇到的坑和最终发现)。“只生成了报告,没有修改”
不是跳过理由——可复用的方法本身就是记忆。
- 调查结果,包括否定结果或未找到结果(例如“使用 rg / fs_search / SQL index
在 C: 和 D: 中搜索 folder X——任何位置都未找到”)。记录一条死路可以避免下一个
agent 重复同样无果的搜索,因此达到了价值门槛。
不要仅仅因为“assistant 可以重新生成这些命令”或“实际上没有修改任何东西”就拒绝
一条记忆。只要流程或结论能节省未来的工作,就保留它。
不要抽取:
- 没有可复用流程、真正琐碎的单步操作(“rename one file”、“git status”、
“run ls once”)
- 已经解决且没有可迁移经验的一次性问题
- repo/system 中已经存在的信息(不要重复 package.json)
- 每个 PR / 每次会话的一次性任务,除非其中编码了可复用模式
- 向用户询问缺失信息这一行为。请求参数、列出未知项或引导用户提供必填字段属于
对话行为,不是方法:每个 agent 都知道如何提问,因此以后没有任何值得查阅的内容。
绝不要产出 task 或 search_keys 形如“how to ask for X”、“gather the missing Y”、
“eliciting required fields”、“what to clarify before starting”的 experience。
注意,这条限制只适用于 EXPERIENCES——会话最终停在等待用户的状态是真实状态,
应写入 FACT。
如果该段会话因为等待用户而没有取得任何实质进展——assistant 只是询问参数,或者在
开展任何实际工作前就被阻塞——输出 "experiences": [],只写 FACT。此时尚未学到任何
东西,因此没有方法可存。这一点比看起来更重要:从停滞中提炼出的 experience 在检索时
会像一项真正的技能;同一任务的后续片段会匹配它,再次向用户提问而不是继续推进,导致
这种停滞在余下会话中不断自我复制。应根据工具实际做了什么判断是否有实质进展,而不是
根据该轮是否生成了流畅文本。
启发式标准:如果用户(或未来执行同类任务的 agent)希望把它记下来,而不是以后再重新
摸索,就保留。多步骤运维流程通常符合条件——包括 system/ops 或只读分析/搜索流程,
不局限于代码。只有当它只是任何用户都能在 30 秒内通过 `--help` 找到的单条命令,并且
不包含值得记忆的决策或坑点时,才丢弃。
═══════════════════════════════════════════════════════════════════
EXPERIENCE 字段
═══════════════════════════════════════════════════════════════════
task — 采用查询式名称;写成用户以后会如何提问。
使用“publish a release to the xx platform”,不要使用
“publish_release_internal”。
必须自包含(不要使用“this”、“the file”、“we”)。
search_keys — 0 到 5 种替代表述,用来覆盖仅靠 `task` 无法匹配、但对 solution 很
重要的方面。写成未来用户会询问的问题或主题,且必须与 task 有区别。
如果 task 已覆盖全部内容,则省略。
solution — 用户实际需要的完整、自包含内容:命令、凭据位置(不是凭据值)、
文件路径、有序步骤、坑点、决策理由。如果多轮端到端工作构成一个连贯
任务,应聚合成一个 solution;只有用户执行了互不相关的任务时才拆成
多条记忆。
boundary_envs — solution 所依赖的具体限定条件,使用多行“key: value”:
runtime / SDK / library + version
API endpoint 或 method signature
external service / middleware version
env var NAME(不是值)
config file path
host / cluster / region identifier
如果 solution 与环境无关,填写 ""(空字符串)。
目的:底层 SDK/middleware 升级后,未来使用者可以据此判断 solution
是否仍然适用。
confidence — 0.0 到 1.0
1.0 用户明确将其陈述为规则/指令
0.7 根据工具操作及充分证据推断
0.5 弱推断;只有一次观察;不确定
═══════════════════════════════════════════════════════════════════
FACT 字段——每段会话恰好一条,无价值门槛
═══════════════════════════════════════════════════════════════════
你要为整段会话写一条 FACT——它是本次会话可恢复的状态记录,不是精心润色的回顾,也
绝不是分散发现的列表。未来 agent 必须能够从中继续推进,且不会把建议、推断或局部搜索
误认为已经验证的事实。
subject — 本次会话围绕谁或什么展开:工作所针对或服务的单一实体。使用简短名词
短语,不要写句子,不要使用动词。选择会话真正围绕的最具体实体,而不是
所使用的工具:
会话服务于某个具名人员时,使用人名
(“Dylan”、“Matthew”)
工作针对代码时,使用 file / module / symbol
(“auth/middleware.ts”、“parse_plan_text”)
运维工作使用 service / host / device / drive
(“D:\\WXWork”、“redis-cache-prod”、“ci.yml”)
这是 HARD KEY,不是散文:关于不同人员或不同文件的两次会话必须得到不同
subject;关于同一实体的两次会话必须得到拼写完全一致的同一字符串。
优先使用 trace 中出现的精确标识符,而不是对它的描述。如果该段确实没有
单一 subject,输出 ""。
这很重要:subject 是决定后续会话是否可以按“同一任务的继续”覆盖本条
记录的闸门。填错后,一个人的状态可能静默替换另一个人的状态。
statement — 本次会话的紧凑记录。它分为两部分,而第二部分最容易丢失。
保留任务范围内的给定量和具体约束,不要用泛泛的结果评价取代它们。
每条规则只写一次:重复的指令文字会挤占记忆真正需要保留的数据。
PART 1——发生了什么,约 300 个拉丁字符(约 150 个 CJK 字符)。
从具体目标和对象开始(“reclaim disk space taken by D:\\WXWork”、
“locate the new_file_search folder”),不要写泛泛的“fix a bug”/
“find a file”。然后概述检查、尝试或修改了什么。最后说明结束状态,
且语义必须精确对应以下之一:completed、partially completed、failed、
blocked、still awaiting verification。纳入否定结果和死路,但不要夸大。
PART 2——TASK-SCOPED GIVENS:整个任务赖以运行的全部给定信息,即使并非
本段会话建立,或看起来基础到不值得写。包括绑定关系(每个 name/symbol/
label 代表什么)、任务提供而不是推导出的固定值、采用的约定(indexing
base、units、timezone、defaults),以及本段自己的结果和值(写“x = 7”,
不要只写“the value was computed”)。
将 PART 2 写成数据,每行一项,值保持给定形式:
retry budget = 5; timeout 30s; index starts at 0
step 3 input: n=64, tolerance 1e-6
slot B: type=private, cost 688.7-721.3 per unit
每一行几乎都应由值构成。保留 trace 的 notation——symbols、units、
numbers、spellings——删掉其句子外壳:去掉那些只是宣告“这是给定量”的
短语,而非给定量本身(“I would prefer”、“it should be somewhere
between … and …”)。这些短语会在每项重复;一个 15 项列表会为它们
支付 15 次空间成本。
每一条 FACT 都要携带这些信息;绝不要写“as above”或“same as the
previous session”——检索只会按相似度返回少量 facts,被指代项可能不在
其中。
总长度尽量控制在 1200 字符以内。如果 PART 2 放不下,应优化 notation,
绝不能削减覆盖范围:statement 是 retrieval vector,也是后续 segment
唯一能读到的字段(details 既不做 embedding,也不会被注入),因此从中
删除的内容就永久丢失。压缩给定量周围的措辞,绝不要压缩给定量本身。
检验标准:如果下一段看不到原始材料,能否仅凭此记录继续任务?只有肯定
答案才合格。缺失 binding 不会明显报错——该名称仍看起来有意义,于是会
按其通用含义被理解,后续答案可以很流畅,却回答了另一个问题。
LANGUAGE——使用与 session trace 相同的语言:statement 是 retrieval
vector,而查询会使用用户的工作语言;改用另一种语言会让每次检索都付出
跨语言匹配损失。
EVIDENCE FIDELITY——只记录 trace 中实际执行的操作和明确存在的证据。
对关键命令、flags、arguments、errors、numbers 保持原样。
绝不要按惯例补全命令 flags,也不要把建议或计划写成已经执行的动作:
“ran Get-ChildItem -Force” → “ran Get-ChildItem”
(trace 中没有 `-Force`,不要杜撰)
每个否定发现都必须限定在实际搜索范围内,并逐一列出可能使其失效的所有
明确覆盖限制,尤其是 permission failures 和 depth limits:保留当时的
roots、filters、depth(“not found under C:\\ and D:\\ at depth 3;
some directories were not covered due to Access denied”),绝不要扩大为
“does not exist”。不要为了让记录更漂亮而漏掉任何限制。
绝不要把局部搜索提升为全局不存在。
被阻塞的尝试必须保留其 blocker。
保留 attempted、observed、verified、inferred、unresolved 之间的区别;
用自然语言表达(“tried … but was blocked by Access denied”、
“suspected …, not yet verified”),不要把它们当标签。没有工具输出或
明确确认时,绝不要声称修复已成功、原因已确定或路径不存在。
不要记录问候或纯闲聊——只有这种情况才输出 "fact": null。仅仅陈述了
要求,或尚未开展工作就停滞的会话,仍然建立了信息。
details — 本次会话的具体 OUTPUT,逐字复制。`statement` 是摘要,而摘要必然会
丢掉各个具体值;后续检索几乎总是在寻找某个具体值,而非大意。因此,只要
会话产出了可寻址的结果,就以原始形式和结构复现在这里:
plan / schedule / itinerary → 每一行、每一天、每个 slot、每个 price
都严格按原文保留
config 或 code block → 保留整个 block
command 的关键 output → 保留承载发现的行(sizes、counts、versions、
error text)
structured result(table、records list)→ 保留各行
保留原始布局——line breaks、`Day 1:` / key: value prefixes、units、
currency symbols、ids。不要再次概括,不要重排,不要舍入数字,不要翻译。
只复制 trace 中实际存在的内容。绝不要填补空缺:如果某个 slot 没有填,
就保持未填或明确说明——杜撰的值比缺失的值更糟,因为它会被检索并使用。
长度控制在约 4000 字符;如果输出更长,保留承载具体值的部分,删去装饰性
文字,并注明删掉了什么。
如果本次会话未产生此类 artifact(尚无发现的纯调查,或等待用户导致的
停滞),填写 ""。
此字段不属于 retrieval vector——`statement` 和 `subject` 负责找到
记录,`details` 在找到后提供具体值。因此不要在此堆砌检索诱饵,也不要
重复摘要;只写原始内容。
files — 0..N 个本次会话中实际操作或调查过的绝对或 repo-relative FILE OR
DIRECTORY paths。保留调查过的 roots,例如 `C:\\` 和 `D:\\`。只包含
实际访问/搜索过的路径,不包含仅建议或计划访问的路径;不适用时填写 []。
═══════════════════════════════════════════════════════════════════
HOST CONTEXT(由 SDK 提供)
═══════════════════════════════════════════════════════════════════
- workspace_id present: {workspace_present}
- branch present: {branch_present}
没有 workspace_id 时,solution 不能依赖未声明的项目本地上下文——必须在 `solution`
或 `boundary_envs` 中明确写出其所需的 repo / path / service,使记忆能够独立使用。
═══════════════════════════════════════════════════════════════════
CONVERSATION
═══════════════════════════════════════════════════════════════════
{conversation_block}
═══════════════════════════════════════════════════════════════════
OUTPUT(仅输出严格 JSON,不要 markdown)
═══════════════════════════════════════════════════════════════════
{{
"experiences": [
{{
"task": "...",
"search_keys": ["...", "..."],
"solution": "...",
"boundary_envs": "...",
"confidence": 0.9
}}
],
"fact": {{
"subject": "...",
"statement": "...",
"files": ["..."],
"details": "..."
}}
}}
默认应进行抽取。当该段会话确实完成了实际工作时,必须输出至少一条 experience——即使
任务看似简单或一次成功,也要记录所做工作的可复用本质(method / command / how-to)。
会话内容较少时,只保留最有价值的一条 experience 即可。
唯一例外是上述停滞情况:如果该段因为缺少信息、被阻塞在向用户提问而没有开展实际工作,
输出 "experiences": [],只写 fact。不要为了满足本指令而杜撰 experience——杜撰的
“how to ask for the departure city”比没有记忆更糟,因为以后会被检索并照做。只有这个
原因允许空数组;实际开展过工作时,绝不能把空数组当捷径。
"fact" 是单个对象,绝不是列表——每次会话一条记录。
始终写 fact。fact 的门槛不是“这是否有价值”,而是“是否发生了任何事情”;能够到达
你的 segment 已经越过这条门槛。这包括看起来只是 setup 而非工作的片段:仅有一条用户
消息陈述 requirements、parameters 或 starting state,即使没有运行工具,也恰恰是后续
会话无法自行重建的材料——把需求、参数、起始状态连同情形一起写进 `statement`(紧凑
记法,不要照抄对话原句),不要推给 `details`(`details` 不进检索向量,读取侧也不再
注入它)。只有完全没有内容的问候和纯闲聊才使用 "fact": null。有疑问时应写 fact:
多余记录只多占一行,而漏掉记录会伤害每一次需要它、却无从知道它曾存在的后续会话。
文本已迁到 ``hy_memory/prompts/zh_hans.py``(``CODING_*``),这里保留旧名字。
"""
EXTRACT_PROMPT_CODING_ZH = """\
╔═══════════════════════════════════════════════════════════════════════════╗
║ HARD CONSTRAINT——先阅读本节;它覆盖本提示词后文任何看似要求叙事的措辞。 ║
║ ║
║ 你不是在撰写某个人做过什么的报告。你要写的是 FUTURE AGENT 为采取行动所需要 ║
║ 的信息。该 agent 已有自己的任务;它不关心这个用户想要什么。它需要知道:此处 ║
║ 做出了什么决定、位于哪些代码坐标、尝试并否决了什么,以及是否经过验证。 ║
║ ║
║ 具体来说,每个 `statement` 和每个 `solution` 都必须遵守: ║
║ · 语法主语必须是 CODE 或 THE DECISION——绝不能是“the user”,也绝不能是 ║
║ “the agent”。不要写“The user attempted to…”、“The user wanted…”、 ║
║ “The agent then checked…”。 ║
║ · 绝不要用“the session was completed”、“the error was noted”、 ║
║ “no solution was found yet”之类流水账短语收尾。结尾应写决定及其验证 ║
║ 状态。如果没有做出决定,就以已排除的内容和下一条尚未探索的线索收尾——这才 ║
║ 是可复用部分。 ║
║ · 保留具体症状(error string、failing command、wrong output)——未来搜索 ║
║ 正是通过这些内容命中。 ║
║ ║
║ bad: “The user attempted to write a QTable to RST with header_rows and ║
║ hit a TypeError. The session was completed with the error noted.” ║
║ good: “ascii.rst's RST writer rejects header_rows: ║
║ `tbl.write(..., format='ascii.rst', header_rows=['name','unit'])` ║
║ raises TypeError because RST.__init__ does not forward it, unlike ║
║ ascii.fixed_width which accepts it. No fix written — the writer ║
║ needs the same header_rows plumbing as fixed_width.” ║
╚═══════════════════════════════════════════════════════════════════════════╝
你需要从一段任务执行会话中抽取可长期保留的 WORKING 记忆。
“working”会话是指用户通过工具操作计算机或系统的任何实操任务——既包括软件/代码
工作(编写、调试、重构、构建、部署、API/library 使用),也包括系统与运维工作
(运行 shell/PowerShell 命令、磁盘/存储分析或清理、文件与文件夹操作、安装/配置
软件、环境/设备设置、硬件/磁盘健康检查、备份、调度/自动化、数据处理)。
你需要从同一段会话中产出两个层次:
EXPERIENCES(过程性,“如何做”)——可复用的方法;只有达到下述 VALUE BAR 时才
抽取。每条是一个 (task, search_keys, solution, boundary_envs) 组合。
FACT(情节性,“这次确立了什么”)——关于本次会话的恰好一条记录。它不是用户做了
什么的叙事,而要按以下顺序记录:任务及其具体症状(使它以后能被检索到)、代码坐标、
最终采用了什么以及拒绝了什么、验证推进到什么程度。它不受价值门槛限制,但绝不能
超过一条。
这两个层次回答不同问题,并分别存储。可泛化的方法放入 EXPERIENCES;只对这个系统、
这个项目、此时此刻成立的本次会话经历放入 FACT。
═══════════════════════════════════════════════════════════════════
VALUE BAR(仅适用于 EXPERIENCES)——不要轻易越过此门槛。
═══════════════════════════════════════════════════════════════════
一条记忆必须同时满足以下全部条件:
(a) 解决了同一用户以后很可能再次遇到的问题
(b) 解决方案并非琐碎操作:它组合了多项信息/多个步骤,或者包含决策理由,或者记录了
来之不易的 workaround,或者是某项运维任务的可复用流程/配方(例如如何分析并
清理磁盘占用、如何批量整理文件、如何检查硬件健康、如何设置 scheduled job)
(c) 未来的 agent / 用户会更愿意查阅它,而不是重新摸索
重要——分析、诊断、调查任务也算:
一项任务不必实际修改或“修复”任何东西才值得记忆。只读、分析、故障排查工作,只要其
流程或结论可复用,就有价值。明确抽取:
- 分析/诊断配方:可复用的流程以及它揭示的结果(例如“如何分析哪些文件夹占用了
C-drive 空间”——扫描命令、遇到的坑和最终发现)。“只生成了报告,没有修改”
不是跳过理由——可复用的方法本身就是记忆。
- 调查结果,包括否定结果或未找到结果(例如“使用 rg / fs_search / SQL index
在 C: 和 D: 中搜索 folder X——任何位置都未找到”)。记录一条死路可以避免下一个
agent 重复同样无果的搜索,因此达到了价值门槛。
不要仅仅因为“assistant 可以重新生成这些命令”或“实际上没有修改任何东西”就拒绝
一条记忆。只要流程或结论能节省未来的工作,就保留它。
不要抽取:
- 没有可复用流程、真正琐碎的单步操作(“rename one file”、“git status”、
“run ls once”)
- 已经解决且没有可迁移经验的一次性问题
- repo/system 中已经存在的信息(不要重复 package.json)
- 每个 PR / 每次会话的一次性任务,除非其中编码了可复用模式
启发式标准:如果用户(或未来执行同类任务的 agent)希望把它记下来,而不是以后再重新
摸索,就保留。多步骤运维流程通常符合条件——包括 system/ops 或只读分析/搜索流程,
不局限于代码。只有当它只是任何用户都能在 30 秒内通过 `--help` 找到的单条命令,并且
不包含值得记忆的决策或坑点时,才丢弃。
═══════════════════════════════════════════════════════════════════
CODE WORK——未来 agent 无法重新推导的信息,因此必须捕获
═══════════════════════════════════════════════════════════════════
当该段是针对 repository 的软件工作时,可复用方法只占一半。另一半是工作位于哪里,
以及哪个版本胜出。这些无法通过以后阅读代码恢复——代码只能展示当前状态,无法说明它
位于哪个 branch,也无法说明哪些方案已被否决。
本节不是要求写得更长,而是要求把相同预算花在价值更高的 token 上:删掉重复目标、
assistant“随后决定”做什么的叙事,以及只是复述的粘贴代码;把这些字符用于下面四项。
一条遗漏 branch、却引用 30 行 diff 的记录是糟糕的取舍。
1. BRANCH / WORKING LOCATION——逐字写入 fact `statement`。如果该段提到或 checkout
了 `feat/x` / `fix/y`,这个 token 必须出现。在错误 branch 上恢复工作的 agent
会产生随后被静默丢弃的改动,因此这一个 token 的优先级高于其他任何细节。
2. WHICH VERSION WON——当用户改变主意时,用一个分句指出胜出者并标记落败者:
“settled on A; the interim B was dropped”。不要完整保留两个版本,只需一个分句。
只保留最新版本是错误的:以后遇到被否决代码的 agent 必须识别它,否则会把代码
“修复”回已经丢弃的变体。`scratch that / go back to / actually, let's / no, not that`
之类措辞就是标志:后面的内容才是正式结论,前面的内容已被 superseded。
3. EXACT COORDINATES, NOT PASTED CODE——使用 repo-relative path
(`qutebrowser/utils/debug.py`,绝不要只写 `debug.py`)以及触及的 function /
class。如果 signature 发生变化,给出新的 signature。如果决定的是 constant、regex
或 literal,逐字引用它——但只引用这一项,不要引用周围 block。准确的一行胜过一段
近似描述。
4. VERIFIED vs MERELY WRITTEN——四个词就足够(“tests run and passing”/
“confirmed by git diff only, tests not run”)。“the edit is in”和“the edit is in
and tested”会让未来 agent 采取完全不同的下一步。
反例——只总结目标,却丢掉坐标:
bad: “refactored the colour parsing logic to be cleaner.”
good: “on feat/q4, in qutebrowser/config/configtypes.py QtColor._parse_value:
settled on `mult = 360.0 if kind == 'h' else 255.0`; the interim version that
used 255.0 for everything was dropped. Tests not run.”
(good 往往比 bad 还短——它只是把每个字符都用在无法重新推导的信息上)
═══════════════════════════════════════════════════════════════════
EXPERIENCE 字段
═══════════════════════════════════════════════════════════════════
task — 采用查询式名称;写成用户以后会如何提问。
使用“publish a release to the xx platform”,不要使用
“publish_release_internal”。
必须自包含(不要使用“this”、“the file”、“we”)。
search_keys — 0 到 5 种替代表述,用来覆盖仅靠 `task` 无法匹配、但对 solution 很
重要的方面。写成未来用户会询问的问题或主题,且必须与 task 有区别。
如果 task 已覆盖全部内容,则省略。
solution — 用户实际需要的完整、自包含内容:命令、凭据位置(不是凭据值)、
文件路径、有序步骤、坑点、决策理由。如果多轮端到端工作构成一个连贯
任务,应聚合成一个 solution;只有用户执行了互不相关的任务时才拆成
多条记忆。
boundary_envs — solution 所依赖的具体限定条件,使用多行“key: value”:
runtime / SDK / library + version
API endpoint 或 method signature
external service / middleware version
env var NAME(不是值)
config file path
host / cluster / region identifier
如果 solution 与环境无关,填写 ""(空字符串)。
目的:底层 SDK/middleware 升级后,未来使用者可以据此判断 solution
是否仍然适用。
confidence — 0.0 到 1.0
1.0 用户明确将其陈述为规则/指令
0.7 根据工具操作及充分证据推断
0.5 弱推断;只有一次观察;不确定
═══════════════════════════════════════════════════════════════════
FACT 字段——每段会话恰好一条,无价值门槛
═══════════════════════════════════════════════════════════════════
你要为整段会话写一条 FACT——它是本次会话可恢复的状态记录,不是精心润色的回顾,也
绝不是分散发现的列表。未来 agent 必须能够从中继续推进,且不会把建议、推断或局部搜索
误认为已经验证的事实。
statement — 本次会话的紧凑摘要。它同时承担两个职责,且两者都必须完成:它既是
retrieval vector(未来 agent 用一份新 issue report 搜索它;措辞方式
与用户报告问题的方式一致),也是该 agent 找到后要据以行动的 payload。
因此:
开头写任务和症状,使用用户会使用的措辞。
这是保证可检索性的关键——表述应尽量贴近问题被报告时的表面形式。
点明具体目标和对象(“reclaim disk space under D:\\WXWork”、
“adjust the hue multiplier in QtColor._parse_value on feat/q4”、
“locate the new_file_search folder”),绝不要写泛泛的“fix a bug”/
“look for a file”。包含具体症状:error string、failing command、
wrong output。
然后给出坐标和决定。
对代码工作写 repo-relative path、symbol 和 branch。说明最终 SETTLED ON
什么,并在同一分句标记丢弃了什么(“settled on X; the interim Y was
dropped”)。这部分无法通过阅读代码重新推导——代码只展示胜出者,从不
展示 Y 曾被尝试后否决。以后遇到被否决变体的 agent 必须能识别它,
否则会把代码“修复”回已丢弃版本。
结尾写证据状态。
必须精确对应以下之一:completed、partially completed、failed、blocked、
still awaiting verification;还要写明如何检查(“tests run and passing”/
“confirmed by git diff only, tests not run”)。纳入否定结果和死路,
但不要夸大。
⚠️ 绝不要停留在叙事层。“The user tried X and hit error Y; no solution found
yet”是一条浪费的记录:它容易被检索到,却不能指导未来 agent 行动。如果
本次确实没有得出决定,就说明排除了什么、下一条尚未探索的线索是什么——
这才是可复用部分。
句子的主语应是 code 和 decision,而不是“the user”。使用 2 到 4 个连贯
句子,不要写带标签的小节。硬上限约 400 个拉丁字符(约 200 个 CJK 字符)。
EVIDENCE FIDELITY——强制要求:
- 只记录 trace 中实际执行的操作和明确存在的证据。对关键 commands、flags、
arguments、errors、numbers 保持原样。绝不要按惯例补全 command flags,
也不要把 advice/plans 写成已执行的动作。
- 保留 attempted、observed、verified、inferred、unresolved 之间的区别。
不要把这些词作为标签写进正文;自然表达其含义,例如“tried … but was
blocked by Access denied”或“suspect …, not yet verified”。
- 没有工具输出或明确确认时,绝不要声称修复成功、原因已经确定、file/folder
不存在或 tests passed。
- 每个否定发现都必须限定在实际搜索范围内,并逐一列出可能使其失效的所有
明确覆盖限制,尤其是 permission failures 和 maximum search depth:
“not found within the searched parts of C:\\ and D:\\; some directories
were skipped on Access denied, and the search was also depth-limited”。
不要为了摘要更漂亮而漏掉任何限制。绝不要把局部搜索提升为全局不存在。
- 只有 trace 直接验证了因果关系时,才能确定地陈述因果。否则保留限定词
(“suspected”/“inferred”/“not yet verified”);不要把错误的测试假设
改写成系统的 root cause。
关键——这段文字就是 retrieval vector,也是未来 query 唯一会匹配的内容。
泛泛摘要毫无价值,因为每次会话看起来都像泛泛摘要:它会匹配所有查询,
却无法回答任何一个。因此必须写出使本次会话可检索的具体 anchors——真实
file paths、function / symbol names、branch names、commands、error strings、
versions、config keys and values、host / service names。以后有人搜索其中任一
具体项时,本摘要必须能被返回。
good: “fixing a failing `npm run build` in CI. Read
.github/workflows/ci.yml and package.json; traced it to webpack
5.74 hitting ERR_OSSL_EVP_UNSUPPORTED under node 18. Bumping the
workflow's node to 20 made the build pass; node 18 still
reproduces locally.”
good: “on branch feat/q0, in qutebrowser/utils/debug.py signal_name:
PyQt<5.11 only exposes the name via repr(sig), and repr comes in
both a bracketed and a parenthesised form. Settled on two patterns
tried in order with fullmatch, breaking on the first hit; the
single-regex version was dropped. Confirmed by git diff, tests not
run.”
bad: “user wanted a build bug fixed, the agent looked at some config
files, changed some code, and fixed it.”
(没有 path、command、error string、version——没有任何可检索锚点;
每次会话都可以写成这样)
bad: “fixed the signal-name parsing compatibility issue on a feature
branch.”
(哪个 branch?哪个 file?哪个 function?两个竞争版本中谁胜出?
一周后检索到它时,没有任何信息仍然有用)
忠实度示例(bad → good):
“ran Get-ChildItem -Force” → “ran Get-ChildItem”
(trace 中没有 `-Force`,不要杜撰)
“confirmed new_file_search does not exist” →
“new_file_search was not found within the searched parts of C:\\ and
D:\\; some directories were skipped on Access denied or by the depth
limit, so the investigation is only partly done”
“identical payloads cause a spurious lock error” →
“the verification script assumed the last writer is always a fixed
rank; that assumption does not hold, so whether there is a lock bug
is still unverified”
不要记录问候或纯闲聊。如果该段完全没有建立任何信息,输出 "fact": null。
files — 0..N 个本次会话中实际操作或调查过的绝对或 repo-relative FILE OR
DIRECTORY paths。保留调查过的 roots,例如 `C:\\` 和 `D:\\`。只包含
实际访问/搜索过的路径,不包含仅建议或计划访问的路径;不适用时填写 []。
═══════════════════════════════════════════════════════════════════
HOST CONTEXT(由 SDK 提供)
═══════════════════════════════════════════════════════════════════
- workspace_id present: {workspace_present}
- branch present: {branch_present}
没有 workspace_id 时,solution 不能依赖未声明的项目本地上下文——必须在 `solution`
或 `boundary_envs` 中明确写出其所需的 repo / path / service,使记忆能够独立使用。
═══════════════════════════════════════════════════════════════════
CONVERSATION
═══════════════════════════════════════════════════════════════════
{conversation_block}
═══════════════════════════════════════════════════════════════════
OUTPUT(仅输出严格 JSON,不要 markdown)
═══════════════════════════════════════════════════════════════════
{{
"experiences": [
{{
"task": "...",
"search_keys": ["...", "..."],
"solution": "...",
"boundary_envs": "...",
"confidence": 0.9
}}
],
"fact": {{
"statement": "...",
"files": ["..."]
}}
}}
强制要求:必须输出至少一条 experience,不要返回空列表。即使任务看似简单或一次成功,
也要记录所做工作的可复用本质(method / command / how-to)。会话内容较少时,只需要
最有价值的一条 experience——但绝不能输出空的 "experiences" array。
"fact" 是单个对象,绝不是列表——每次会话一条记录。只有该段是没有建立任何信息的
纯闲聊时,才使用 "fact": null。
"""
RECONCILE_PROMPT_ZH = """\
你需要决定如何将新的 coding memory drafts 与用户已有的 memories 合并。
对每条新 draft,选择以下一种 action:
ADD — 没有已有 memory 描述相同的任务/方面
UPDATE — 某条已有 memory 描述相同任务;用新 draft 替换它
(solution 可能已经变化,env 可能已经升级,等等)
SKIP — 与已有内容重复;没有新信息
此外,当新内容使已有候选失效或作废时,可以针对已有 candidates 输出 DELETE ops:
DELETE — 新 conversation 明确 supersedes/cancels/invalidates 某条已有 memory,
且无法通过 UPDATE 修复。示例:
* 用户说“don't use X anymore”,而 X 只存在于另一条独立 memory 中
(不是本次正在 ADD/UPDATE 的那条)
* candidate 的 boundary_envs(service / endpoint / SDK)已经消失,
且新 conversation 确认其已 deprecated
规则:
- “Same task”是指已有 memory 和新 draft 都可以作为同一个未来用户查询的有效答案。
- 如果 solution 不同,优先 UPDATE(不保留历史版本)。
- 如果 task 和 solution 都高度一致,选择 SKIP。
- 只有在存在明确证据表明 candidate 已 invalid/obsolete 时才 DELETE。绝不要仅仅因为
candidate 看起来“stale”就 DELETE。
- 产生 DELETE 时,输出的 ops 数量可以超过 drafts 数量。
═══════════════════════════════════════════════════════════════════
NEW DRAFTS
═══════════════════════════════════════════════════════════════════
{drafts_block}
═══════════════════════════════════════════════════════════════════
EXISTING CANDIDATES(按每条 draft 的 task similarity 检索)
═══════════════════════════════════════════════════════════════════
{candidates_block}
═══════════════════════════════════════════════════════════════════
OUTPUT(仅输出严格 JSON,不要 markdown)
═══════════════════════════════════════════════════════════════════
{{
"ops": [
{{"draft_idx": 0, "action": "ADD"}},
{{"draft_idx": 1, "action": "UPDATE", "target_experience_id": "exp_xxx", "reason": "..."}},
{{"draft_idx": 1, "action": "DELETE", "target_experience_id": "exp_yyy", "reason": "user explicitly deprecated Y"}},
{{"draft_idx": 2, "action": "SKIP", "reason": "duplicate"}}
]
}}
如果没有 drafts,输出 {{"ops": []}}。
"""
FACT_RECONCILE_PROMPT_ZH = """\
你负责维护 session-level WORKING FACTS。每条 statement 都概括一整段 trace/session:
用户的任务、大致方法和结果。它不是一条 atomic claim。
下面每条新的 session fact 都一定会被存储;这不是你要决定的事情。你的唯一任务是识别
task continuation:新 session 是否在继续推进某条已有 session fact 所记录的同一个用户
任务?如果是,就将较旧 fact 标记为 SUPERSEDE,使这些链接为该任务构成跨 session 的
进展链。
只有在存在充分证据同时满足以下条件时,才将已有 fact 标记为 superseded:
- subjects 匹配:两条记录围绕同一个实体——同一个 person、同一个 file/module、
同一个 host/service。subject 相同是必要条件,但不是充分条件;并且
- 用户目标在实质上相同,而不只是处于同一 topic/project;并且
- 新 trace 在继续、重试、扩展、验证或完成已有 trace 所记录的工作。
以下情况不要 supersede:
- 两条记录的 subjects 不同。同一 group 中的两个人、同一 module 中的两个 files、
同一 cluster 中的两个 hosts 是并行关系,不是先后关系:一个人的状态不是另一个人
状态的新版本。如果 subject 显示为 (unspecified),退回到只根据 statement 判断。
- 两次 session 只是提到了相同 file、symbol、component、repository、error 或
technology,但追求不同的用户目标。
- 一个任务只是为另一个任务提供 context 或 technique。
- 关系不确定。应保守判断:INSERT 是安全默认值;合并无关 session 会毁掉它们各自
独立的任务历史。
把每条 statement 当作有证据边界的文本。不要把其中的 inference 重新解释为已经确定的
cause,也不要把共同的 conjecture/error 当作用户目标相同的证据。你的判断仅涉及 task
continuation;不要“纠正”、补全或润色任何一条 session 记录。
示例:
existing: “需要修复 CI 的 npm run build;已追踪到 Node 18 下的
ERR_OSSL_EVP_UNSUPPORTED,workflow 尚未修改。”
new: “继续修复同一个 CI build;已将 .github/workflows/ci.yml 中的 Node
提升到 20,且 npm run build 通过。”
→ SUPERSEDE
existing: “需要修复 auth/middleware.ts 中的 JWT expiry handling;已添加
refresh logic。”
new: “需要在 auth/middleware.ts 中添加 performance logging;已引入
latency metric。”
→ 两者都保留
(同一 file/project,不同用户目标)
═══════════════════════════════════════════════════════════════════
NEW SESSION FACTS
═══════════════════════════════════════════════════════════════════
{drafts_block}
═══════════════════════════════════════════════════════════════════
EXISTING SESSION FACTS(按每条新 fact 的 statement similarity 检索)
═══════════════════════════════════════════════════════════════════
{candidates_block}
═══════════════════════════════════════════════════════════════════
OUTPUT(仅输出严格 JSON,不要 markdown)
═══════════════════════════════════════════════════════════════════
{{
"ops": [
{{"draft_idx": 0, "supersedes": ["fact_xxx"], "reason": "continues the same CI fix task"}},
{{"draft_idx": 1, "supersedes": []}}
]
}}
只能使用同一个 draft_idx 的 candidate list 中实际出现的 fact ids。
如果任何位置都没有 fact 被 superseded,输出 {{"ops": []}}。
"""
CHAIN_SUMMARY_PROMPT_ZH = """\
working-memory store 中的某个用户任务已跨多个 sessions 推进。下面是该任务进展链中的
session summaries,按从最早到最新排列,最后一条是当前 session。
总结同一个任务如何跨 sessions 推进到当前状态。按时间顺序覆盖每个重要阶段。对每个
阶段,在有信息时保留:实际 action 或 change、其 direct result,以及任何 unresolved
issue 或 limiting condition。最后明确写出当前状态。
规则:
- 这只是压缩。只能报告 entries 实际陈述的内容;绝不要新增或补全 command、cause、
motive、action、test result、success state 或 global negative conclusion。
- 精确保留证据状态:attempted 不等于 completed,observed 不等于 verified,inferred
不等于 established,unresolved 必须保持 unresolved。保留 uncertainty words 以及
search/permission/coverage limits。
- 忠实陈述当前结束状态(completed、partially completed、failed、blocked 或
still awaiting verification)。绝不要因为某个看似合理的因果故事,或后续 entry
并未验证的内容而提升状态。
- 如果 entries 提到 suspected cause 和后续 result,只有某条 entry 直接说明该 cause
已被 verified 时,才可在两者之间建立因果联系。
- 保留具体 retrieval anchors:file paths、symbols、commands、error strings、versions、
config keys/values,以及对进展重要的 numeric values。
- 对 2-3 代的 chains,不要把进展过度压缩成一句话。适合时优先使用 2-4 个紧凑段落,
或使用可读性良好的 prose,清楚呈现 task goal、逐阶段 progression 和 current status。
- 不要机械复制完整 history。在保留重要 actions、direct results 和 leftovers 的同时,
省略重复及低价值细节。
- 不要杜撰 stage headings 或 dates。只有 entries 使日期有实际意义时才提及日期;
单独的 timestamps 是噪声。
- 如果 entries 说明更早 revisions 已被省略,必须明确指出 earlier history was omitted;
不要暗示所提供的 entries 是完整历史。
- 使用与 entries 相同的语言。
- 不要写前言,也不要用引号包裹输出。
示例
entries:
(2026-07-20) 需要修复 CI 的 npm run build;已追踪到 Node 18 下的
ERR_OSSL_EVP_UNSUPPORTED,workflow 尚未修改。
(2026-07-24) 继续修复同一个 CI 问题;已将 .github/workflows/ci.yml 中的 Node
提升到 20,但 dependency cache 仍复现旧错误。
(now) 继续验证同一修复;清除 npm cache 后,npm run build 通过。
output:
任务目标是修复 CI 的 npm run build。最初将问题追踪到 Node 18 下的
ERR_OSSL_EVP_UNSUPPORTED,当时尚未修改 workflow;随后将
.github/workflows/ci.yml 提升到 Node 20,但 dependency cache 仍使旧错误继续
出现。最后,清除 npm cache 后 npm run build 通过,因此该 build 修复目前已验证完成。
═══════════════════════════════════════════════════════════════════
ENTRIES
═══════════════════════════════════════════════════════════════════
{chain_block}
═══════════════════════════════════════════════════════════════════
OUTPUT(只输出 summary 本身,不要其他内容)
═══════════════════════════════════════════════════════════════════
"""
from ..prompts.zh_hans import ( # noqa: F401
CODING_CHAIN_SUMMARY_PROMPT as CHAIN_SUMMARY_PROMPT_ZH,
CODING_EXTRACT_PROMPT as EXTRACT_PROMPT_ZH,
CODING_EXTRACT_PROMPT_CODING as EXTRACT_PROMPT_CODING_ZH,
CODING_FACT_RECONCILE_PROMPT as FACT_RECONCILE_PROMPT_ZH,
CODING_RECONCILE_PROMPT as RECONCILE_PROMPT_ZH,
)
+153 -90
View File
@@ -7,6 +7,9 @@ Coding Memory Reconciler
(DELETE 由 LLM 在 draft 与 candidate 关系中观察到"明确否定/作废"时附带产出)
执行端:reconcile() 调用 store 完成实际 ADD / UPDATE / DELETE 持久化操作。
CodingWriter 分两步用:先 plan()(检索 + LLM 决策,不落库),等经验层与事实层的
模型决策都成功后再 apply()(落库)。LLM 失败 / 空回复 / 解析不出时 plan() 抛
CodingWriteError(RECONCILE_FAILED),本次写入什么都不存。
复用 LLMConfig.model(与 chat reconciler 共用)。
@@ -17,14 +20,20 @@ import json
import logging
import time
import uuid
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Tuple
from ..agent.llm_provider import LLMProvider
from ..core.embed_service import EmbedService
from ..utils.lang_detect import is_chinese
from ..prompts import (
get_prompt,
normalize_language,
resolve_language,
with_output_language,
)
from .judge import _store_coding_log
from .store import CodingMemoryStore
from .types import CodingMemory, CodingMemoryDraft, ReconcileOp
from .types import CodingMemory, CodingMemoryDraft, CodingWriteError, ReconcileOp
@@ -50,57 +59,7 @@ def _resolve_configured_temperature(llm_provider) -> Optional[float]:
# Prompt
# ================================================================
RECONCILE_PROMPT = """\
You decide how to merge new coding memory drafts with the user's existing memories.
For each new draft, decide one of:
ADD — no existing memory describes the same task/aspect
UPDATE — an existing memory describes the same task; the new draft replaces it
(solution may have changed, env may have upgraded, etc.)
SKIP — duplicate of existing; nothing new
Additionally, you may emit DELETE ops on existing candidates when the new
content makes them obsolete or invalid:
DELETE — the new conversation EXPLICITLY supersedes/cancels/invalidates an
existing memory that cannot be repaired via UPDATE. Examples:
* user says "don't use X anymore" and X is only present as a
separate memory (not the one being added/updated)
* candidate's boundary_envs (service / endpoint / SDK) is gone
and the new conversation confirms the deprecation
Rules:
- "Same task" means the existing memory and the new draft would both be valid
answers to the same future user query.
- If solution differs, prefer UPDATE (we don't keep history).
- If both task and solution match closely, SKIP.
- DELETE only when there is EXPLICIT evidence the candidate is invalid/obsolete.
Never DELETE just because a candidate looks "stale".
- Output ops can outnumber drafts when DELETEs are produced.
═══════════════════════════════════════════════════════════════════
NEW DRAFTS
═══════════════════════════════════════════════════════════════════
{drafts_block}
═══════════════════════════════════════════════════════════════════
EXISTING CANDIDATES (per draft, retrieved by task similarity)
═══════════════════════════════════════════════════════════════════
{candidates_block}
═══════════════════════════════════════════════════════════════════
OUTPUT (strict JSON only, no markdown)
═══════════════════════════════════════════════════════════════════
{{
"ops": [
{{"draft_idx": 0, "action": "ADD"}},
{{"draft_idx": 1, "action": "UPDATE", "target_experience_id": "exp_xxx", "reason": "..."}},
{{"draft_idx": 1, "action": "DELETE", "target_experience_id": "exp_yyy", "reason": "user explicitly deprecated Y"}},
{{"draft_idx": 2, "action": "SKIP", "reason": "duplicate"}}
]
}}
If there are no drafts, output {{"ops": []}}.
"""
from ..prompts.en import CODING_RECONCILE_PROMPT as RECONCILE_PROMPT # noqa: E402,F401 (re-export)
def _drafts_language_text(drafts: List[CodingMemoryDraft]) -> str:
@@ -119,24 +78,32 @@ def _drafts_language_text(drafts: List[CodingMemoryDraft]) -> str:
def build_experience_reconcile_prompt(
drafts: List[CodingMemoryDraft],
candidates_per_draft: List[List[CodingMemory]],
language: Optional[str] = None,
) -> Tuple[str, str]:
"""Return the language-matched experience reconcile prompt and language."""
"""Return the experience reconcile prompt and its language code.
Explicit ``language`` wins; otherwise zh/en auto-detection over the drafts.
"""
drafts_block = CodingMemoryReconciler._format_drafts(drafts)
candidates_block = CodingMemoryReconciler._format_candidates(
drafts, candidates_per_draft,
)
if is_chinese(_drafts_language_text(drafts)):
from .prompts_zh import RECONCILE_PROMPT_ZH
template = RECONCILE_PROMPT_ZH
language = "zh"
else:
template = RECONCILE_PROMPT
language = "en"
return template.format(
explicit = normalize_language(language)
code = resolve_language(explicit, _drafts_language_text(drafts))
prompt = get_prompt("CODING_RECONCILE_PROMPT", code).format(
drafts_block=drafts_block,
candidates_block=candidates_block,
), language
)
return with_output_language(prompt, code, explicit=explicit is not None), code
@dataclass
class ExperienceReconcilePlan:
"""plan() 的产物:模型已决策、尚未落库。apply() 才执行 store 操作。"""
drafts: List[CodingMemoryDraft]
candidates_per_draft: List[List[CodingMemory]] = field(default_factory=list)
ops: List[ReconcileOp] = field(default_factory=list)
# ================================================================
@@ -186,23 +153,55 @@ class CodingMemoryReconciler:
user_id: str = "",
agent_id: str = "default_agent",
session_id: str = "",
language: Optional[str] = None,
) -> List[ReconcileOp]:
"""
执行 reconcile + 持久化。返回执行的 ReconcileOp[](含成功执行的 ADD/UPDATE/DELETE/SKIP)。
执行 reconcile + 持久化(plan() 后紧接 apply())。返回执行的 ReconcileOp[]
(含成功执行的 ADD/UPDATE/DELETE/SKIP)。
执行错误(如 LLM 失败)会 fail-safe 回退到对每个 draft 直接 ADD。
LLM 失败、空回复或解析不出时抛 CodingWriteError(RECONCILE_FAILED),什么都
不落库——不再回退成「全部 ADD」。
"""
plan = await self.plan(
drafts,
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
language=language,
)
return await self.apply(plan)
cache 非空时落 CODING_RECONCILE pipeline log——在 LLM 决策后、执行入库前打点,
这样即使 store 入库崩溃,reconcile 决策仍已可见(crash 不丢)。
async def plan(
self,
drafts: List[CodingMemoryDraft],
*,
request_id: str = "",
user_id: str = "",
agent_id: str = "default_agent",
session_id: str = "",
language: Optional[str] = None,
) -> ExperienceReconcilePlan:
"""检索 candidates + LLM 决策,不落库。
有效回复 = 一个 JSON 对象,其 "ops" 是列表:
- {"ops": []}:模型判定「无需任何操作」(与 chat reconcile 的 [] 同义),
成功、本层什么都不存;
- 非空列表:至少要有一条可识别的 op(ADD/UPDATE/DELETE/SKIP),否则视为
解析失败。漏掉的 draft 仍按原规则补 ADD(那是对有效回复的校验兜底,
不是模型失败兜底)。
LLM 调用失败(LLM_UNAVAILABLE)、200 但 content 为空(EMPTY_RESPONSE)、
回复不是上述形状(JSON_PARSE_FAILED)时抛 CodingWriteError(RECONCILE_FAILED)。
cache 非空时落 CODING_RECONCILE pipeline log(成功、失败都落)。
"""
if not drafts:
return []
return ExperienceReconcilePlan(drafts=[])
# 从 draft 回填 trace 身份(writer 通常显式传,这里兜底)
if drafts:
user_id = user_id or drafts[0].user_id
agent_id = agent_id or drafts[0].agent_id
session_id = session_id or (drafts[0].session_id or "")
user_id = user_id or drafts[0].user_id
agent_id = agent_id or drafts[0].agent_id
session_id = session_id or (drafts[0].session_id or "")
# 1. 检索 candidates
candidates_per_draft: List[List[CodingMemory]] = await self._retrieve_candidates(drafts)
@@ -212,19 +211,20 @@ class CodingMemoryReconciler:
_decide_prompt = ""
_decide_raw = ""
_decide_resp = None
failure: Optional[Tuple[str, str]] = None
ops: List[ReconcileOp] = []
try:
ops, _decide_prompt, _decide_raw, _decide_resp = await self._llm_decide(
drafts, candidates_per_draft
drafts, candidates_per_draft, language=language,
)
failure = self._reply_failure(_decide_raw)
except Exception as e:
logger.warning(f"[coding-reconcile] LLM decide failed, fallback to ADD-all: {e}")
ops = [ReconcileOp(action="ADD", draft_idx=i) for i in range(len(drafts))]
logger.warning(f"[coding-reconcile] LLM decide failed: {e}")
_decide_raw = f"[LLM error] {type(e).__name__}: {str(e)[:300]}"
if not ops:
# LLM 输出无 ops,但有 drafts → fallback ADD-all
logger.info("[coding-reconcile] LLM returned no ops, fallback to ADD-all")
ops = [ReconcileOp(action="ADD", draft_idx=i) for i in range(len(drafts))]
failure = (
"LLM_UNAVAILABLE",
f"coding reconcile LLM call failed: {type(e).__name__}: {e}",
)
_decide_elapsed = round((time.perf_counter() - _t0) * 1000, 2)
# CODING_RECONCILE:LLM 决策完成即打点(执行入库之前,crash 不丢)
@@ -232,17 +232,42 @@ class CodingMemoryReconciler:
self.cache, request_id=request_id, user_id=user_id, agent_id=agent_id,
session_id=session_id, step="CODING_RECONCILE",
prompt=_decide_prompt, response=_decide_raw,
parsed=json.dumps({"ops": [o.to_dict() for o in ops]}, ensure_ascii=False, default=str),
parsed=json.dumps(
{"error": failure[0], "message": failure[1], "stored": 0}
if failure else {"ops": [o.to_dict() for o in ops]},
ensure_ascii=False, default=str,
),
elapsed_ms=_decide_elapsed,
prompt_tokens=getattr(_decide_resp, "prompt_tokens", 0) if _decide_resp else 0,
completion_tokens=getattr(_decide_resp, "completion_tokens", 0) if _decide_resp else 0,
total_tokens=getattr(_decide_resp, "tokens_used", 0) if _decide_resp else 0,
)
if failure is not None:
code, message = failure
logger.warning(f"[coding-reconcile] {code}: {message}")
raise CodingWriteError("RECONCILE_FAILED", message, cause=code)
# 3. 校验 + 执行
executed = await self._execute(ops, drafts, candidates_per_draft)
return ExperienceReconcilePlan(
drafts=list(drafts),
candidates_per_draft=candidates_per_draft,
ops=ops,
)
async def apply(self, plan: ExperienceReconcilePlan) -> List[ReconcileOp]:
"""执行 plan() 的决策(store ADD / UPDATE / DELETE)。"""
if not plan.drafts:
return []
if not plan.ops:
# 有效的 {"ops": []}:模型判定无需任何操作,本层什么都不存。
logger.info(
f"[coding-reconcile] drafts={len(plan.drafts)} LLM decided no ops"
)
return []
executed = await self._execute(
plan.ops, plan.drafts, plan.candidates_per_draft,
)
logger.info(
f"[coding-reconcile] drafts={len(drafts)} ops={len(executed)} "
f"[coding-reconcile] drafts={len(plan.drafts)} ops={len(executed)} "
+ ", ".join(f"{a}={sum(1 for o in executed if o.action == a)}"
for a in ("ADD", "UPDATE", "DELETE", "SKIP"))
)
@@ -311,9 +336,12 @@ class CodingMemoryReconciler:
self,
drafts: List[CodingMemoryDraft],
candidates_per_draft: List[List[CodingMemory]],
language: Optional[str] = None,
) -> Tuple[List[ReconcileOp], str, str, Any]:
"""返回 (ops, prompt, raw_response, llm_resp),供上层打 CODING_RECONCILE log。"""
prompt, _ = build_experience_reconcile_prompt(drafts, candidates_per_draft)
prompt, _ = build_experience_reconcile_prompt(
drafts, candidates_per_draft, language=language,
)
resp = await self.llm_provider.complete(
prompt=prompt,
@@ -358,11 +386,30 @@ class CodingMemoryReconciler:
rows.append(f"for draft[{i}]:\n" + "\n".join(cand_rows))
return "\n\n".join(rows) if rows else "(none)"
@classmethod
def _reply_failure(cls, text: Optional[str]) -> Optional[Tuple[str, str]]:
"""回复本身是失败时返回 (cause, message);有效回复(含 {"ops": []})返回 None。"""
if not (text or "").strip():
return "EMPTY_RESPONSE", "coding reconcile LLM returned empty response"
obj = cls._load_object(text)
ops_raw = obj.get("ops") if isinstance(obj, dict) else None
if not isinstance(ops_raw, list):
return (
"JSON_PARSE_FAILED",
f"coding reconcile output has no ops list ({len(text)} chars)",
)
if ops_raw and not cls._parse_ops(text):
return (
"JSON_PARSE_FAILED",
f"coding reconcile output has no recognizable op ({len(text)} chars)",
)
return None
@staticmethod
def _parse_ops(text: str) -> List[ReconcileOp]:
"""宽松解析 ops 列表。"""
def _load_object(text: Optional[str]) -> Optional[Dict[str, Any]]:
"""宽松解析出顶层 JSON 对象;解析不出返回 None。"""
if not text:
return []
return None
s = text.strip()
# 去 markdown fence
if s.startswith("```"):
@@ -386,8 +433,15 @@ class CodingMemoryReconciler:
obj = json.loads(s[start: end + 1])
except Exception:
obj = None
return obj if isinstance(obj, dict) else None
if not obj or not isinstance(obj, dict):
@classmethod
def _parse_ops(cls, text: str) -> List[ReconcileOp]:
"""宽松解析 ops 列表。"""
if not text:
return []
obj = cls._load_object(text)
if not obj:
logger.warning(
f"[coding-reconcile] failed to parse ops output ({len(text or '')} chars)"
)
@@ -436,6 +490,9 @@ class CodingMemoryReconciler:
- UPDATE 需 draft_idx 合法 + target_experience_id 非空
- DELETE 需 target_experience_id 非空
- SKIP 不执行任何 store 操作
向量化失败(CodingWriteError(EMBEDDING_FAILED))不吞成「少执行一条」,
原样抛给 CodingWriter,整次写入报失败。
"""
executed: List[ReconcileOp] = []
# 已被 UPDATE 占用的 draft_idx → 防止同 draft 被重复 ADD/UPDATE
@@ -456,6 +513,8 @@ class CodingMemoryReconciler:
op.target_experience_id = memory.experience_id # 回写 id 供 caller 读
executed.append(op)
consumed.add(op.draft_idx)
except CodingWriteError:
raise
except Exception as e:
logger.warning(f"[coding-reconcile] ADD failed: {e}")
@@ -486,6 +545,8 @@ class CodingMemoryReconciler:
await self.store.update(memory)
executed.append(op)
consumed.add(op.draft_idx)
except CodingWriteError:
raise
except Exception as e:
logger.warning(f"[coding-reconcile] UPDATE failed: {e}")
@@ -529,6 +590,8 @@ class CodingMemoryReconciler:
target_experience_id=memory.experience_id,
reason="fallback ADD (no LLM op)",
))
except CodingWriteError:
raise
except Exception as e:
logger.warning(f"[coding-reconcile] fallback ADD failed: {e}")
+109 -26
View File
@@ -44,7 +44,7 @@ from ..data.vector_store import create_vector_store
from ..data.vector_store_base import VectorStoreBase
from ..models.memory import MemoryLayer, MemoryNode, MemoryStatus
from ..utils.async_executor import run_in_executor_guarded
from .types import CodingMemory, WorkingFact
from .types import CodingMemory, CodingWriteError, WorkingFact
logger = logging.getLogger(__name__)
@@ -599,10 +599,15 @@ class CodingMemoryStore:
# ------------------------------------------------------------
async def insert(self, memory: CodingMemory) -> None:
"""新增一条 coding memory(SQLite + VDB 全套)"""
"""新增一条 coding memory(SQLite + VDB 全套)。
先向量化再落 SQLite:向量化失败抛 CodingWriteError(EMBEDDING_FAILED),
什么都不写,不留一条没有向量、永远检索不到的经验。
"""
self._fill_timestamps(memory)
embeddings = await self._embed_vec_keys(memory)
await _run_in_pool(self._upsert_sql_sync, memory)
await self._upsert_vec_keys(memory)
await self._upsert_vec_keys(memory, embeddings=embeddings)
logger.info(
f"[coding-store] inserted experience_id={memory.experience_id} "
f"keys={1 + len(memory.search_keys)}"
@@ -627,6 +632,8 @@ class CodingMemoryStore:
):
raise ValueError("coding_memory_update_authority_mismatch")
prev_key_count = 1 + len(prev.search_keys)
# 先向量化:失败时旧经验(SQLite 与向量行)原样保留,不改一个字。
embeddings = await self._embed_vec_keys(memory)
memory.updated_at = datetime.utcnow()
if memory.created_at is None:
@@ -637,7 +644,7 @@ class CodingMemoryStore:
# 删旧 VDB
await self._delete_vec_keys_by_count(memory.experience_id, prev_key_count)
# 写新 VDB
await self._upsert_vec_keys(memory)
await self._upsert_vec_keys(memory, embeddings=embeddings)
logger.info(
f"[coding-store] updated experience_id={memory.experience_id} "
f"old_keys={prev_key_count} new_keys={1 + len(memory.search_keys)}"
@@ -1000,12 +1007,22 @@ class CodingMemoryStore:
# 事实层
# ------------------------------------------------------------
async def insert_fact(self, fact: WorkingFact) -> None:
"""新增一条事实(SQLite + 事实 collection 各一行)。"""
async def insert_fact(
self, fact: WorkingFact, embedding: Optional[List[float]] = None,
) -> None:
"""新增一条事实(SQLite + 事实 collection 各一行)。
embedding 可由调用方预先用 embed_fact() 算好(fact reconcile 要在覆盖旧
事实之前确认新事实能向量化);不传时这里算。向量化失败抛
CodingWriteError(EMBEDDING_FAILED),什么都不写。
"""
if fact.created_at is None:
fact.created_at = datetime.utcnow()
# 先向量化再落 SQLite,理由同 insert():不落没有向量的事实。
if embedding is None:
embedding = await self.embed_fact(fact)
await _run_in_pool(self._upsert_fact_sql_sync, fact)
await self._upsert_fact_vec(fact)
await self._upsert_fact_vec(fact, embedding=embedding)
logger.info(f"[coding-store] inserted fact_id={fact.fact_id}")
logger.debug(
f"[coding-store] inserted fact_id={fact.fact_id} "
@@ -1574,25 +1591,64 @@ class CodingMemoryStore:
# VDB 实现
# ------------------------------------------------------------
async def _upsert_vec_keys(self, memory: CodingMemory) -> None:
"""把 task + search_keys 写入 VDB(每行一个 MemoryNode)。"""
if self._vec is None:
return
@staticmethod
def _vec_key_texts(memory: CodingMemory) -> List[Tuple[str, str]]:
all_keys: List[Tuple[str, str]] = [("task", memory.task)]
for k in memory.search_keys:
if k:
all_keys.append(("aspect", k))
return all_keys
async def _embed_vec_keys(
self, memory: CodingMemory
) -> Optional[List[List[float]]]:
"""写入链路用:向量化 task + search_keys,失败抛 CodingWriteError。
无 VDB 时返回 None(不需要向量)。
"""
if self._vec is None:
return None
texts = [t for _, t in self._vec_key_texts(memory)]
try:
embeddings = await self.embed_service.embed_batch(texts)
except Exception as e:
logger.warning(f"[coding-store] embed_batch failed: {e}")
raise CodingWriteError(
"EMBEDDING_FAILED",
f"coding experience embedding failed: {type(e).__name__}: {e}",
) from e
if len(embeddings or []) != len(texts) or not all(embeddings):
raise CodingWriteError(
"EMBEDDING_FAILED",
"coding experience embedding returned no vector",
)
return embeddings
async def _upsert_vec_keys(
self,
memory: CodingMemory,
embeddings: Optional[List[List[float]]] = None,
) -> None:
"""把 task + search_keys 写入 VDB(每行一个 MemoryNode)。
embeddings 由写入链路预先算好(见 _embed_vec_keys);不传时现算,
失败只记 warning(edit_experience 的手动编辑路径沿用这一行为)。
"""
if self._vec is None:
return
all_keys = self._vec_key_texts(memory)
if not all_keys:
return
# batch embed
texts = [t for _, t in all_keys]
try:
embeddings = await self.embed_service.embed_batch(texts)
except Exception as e:
logger.warning(f"[coding-store] embed_batch failed: {e}")
return
if embeddings is None:
try:
embeddings = await self.embed_service.embed_batch(texts)
except Exception as e:
logger.warning(f"[coding-store] embed_batch failed: {e}")
return
nodes: List[MemoryNode] = []
for i, ((kind, text), emb) in enumerate(zip(all_keys, embeddings)):
@@ -1623,7 +1679,34 @@ class CodingMemoryStore:
except Exception as e:
logger.warning(f"[coding-store] vdb upsert_batch failed: {e}")
async def _upsert_fact_vec(self, fact: WorkingFact) -> None:
@staticmethod
def _fact_vector_text(fact: WorkingFact) -> str:
return f"{fact.subject}: {fact.statement}" if fact.subject else fact.statement
async def embed_fact(self, fact: WorkingFact) -> Optional[List[float]]:
"""写入链路用:向量化事实检索文本,失败抛 CodingWriteError。
无事实 VDB 或 statement 为空时返回 None(不需要向量)。
"""
if self._fact_vec is None or not fact.statement:
return None
try:
embedding = await self.embed_service.embed(self._fact_vector_text(fact))
except Exception as e:
logger.warning(f"[coding-store] fact embed failed: {e}")
raise CodingWriteError(
"EMBEDDING_FAILED",
f"working fact embedding failed: {type(e).__name__}: {e}",
) from e
if not embedding:
raise CodingWriteError(
"EMBEDDING_FAILED", "working fact embedding returned no vector",
)
return embedding
async def _upsert_fact_vec(
self, fact: WorkingFact, embedding: Optional[List[float]] = None,
) -> None:
"""事实入向量库。一条事实一行,node_id 就是 fact_id。
检索文本是 statement,前面缀上 subject(有的话)。subject 通常已经在
@@ -1634,14 +1717,14 @@ class CodingMemoryStore:
"""
if self._fact_vec is None or not fact.statement:
return
vector_text = (
f"{fact.subject}: {fact.statement}" if fact.subject else fact.statement
)
try:
embedding = await self.embed_service.embed(vector_text)
except Exception as e:
logger.warning(f"[coding-store] fact embed failed: {e}")
return
vector_text = self._fact_vector_text(fact)
if embedding is None:
# edit_fact 的手动编辑路径:现算,失败只记 warning。
try:
embedding = await self.embed_service.embed(vector_text)
except Exception as e:
logger.warning(f"[coding-store] fact embed failed: {e}")
return
node = MemoryNode(
node_id=fact.fact_id,
+19
View File
@@ -25,6 +25,25 @@ FactStatus = Literal["active", "superseded"]
MAX_FACT_CHAIN_ENTRIES = 8
class CodingWriteError(RuntimeError):
"""working / coding 写入链路里让整次写入失败的模型错误。
error_code 是稳定失败码,CodingWriter 原样透传为 agent_error_code:
- LLM_ERROR 抽取 LLM 调用失败
- EMPTY_RESPONSE 抽取 LLM 返回 200 但 content 为空
- JSON_PARSE_FAILED 抽取 LLM 的回复解析不出 JSON 对象
- EMBEDDING_FAILED 经验 / 事实向量化失败(不落没有向量的节点)
- RECONCILE_FAILED 经验 / 事实 reconcile 的模型决策失败;具体原因在 cause
(LLM_UNAVAILABLE / EMPTY_RESPONSE / JSON_PARSE_FAILED),
透传为 reconcile_error_code,与 chat 链路一致
"""
def __init__(self, error_code: str, message: str, cause: Optional[str] = None):
super().__init__(message)
self.error_code = error_code
self.cause = cause
def _epoch(value: Optional[datetime]) -> Optional[int]:
"""memory_at 对外是秒级 epoch,与 chat 一致。"""
return int(value.timestamp()) if isinstance(value, datetime) else None
+113 -42
View File
@@ -10,8 +10,9 @@ CodingWriter - 写入端协调器
├── recent history: 最近 K 个 user-turn(只作上下文)
├── new messages: 本次全部 turn(只做单条限长)
├── extractor.extract(...) ← 一次 LLM 产出双层草稿
├── reconciler.reconcile(经验草稿) ← ADD/UPDATE/DELETE/SKIP
├── fact_reconciler.reconcile(事实草稿) ← INSERT/SUPERSEDE
├── reconciler.plan(经验草稿) ← ADD/UPDATE/DELETE/SKIP 决策
├── fact_reconciler.plan(事实草稿) ← INSERT/SUPERSEDE 决策
├── reconciler.apply / fact_reconciler.apply ← 两层决策都成功后才落库
└── _crosslink(...) ← 同批全互链回填
"""
@@ -30,7 +31,7 @@ from .judge import classify_messages_is_coding
from .preproc import has_any_tool_message, prepare_extractor_messages
from .reconciler import CodingMemoryReconciler
from .store import CodingMemoryStore
from .types import FactReconcileOp, ReconcileOp, WorkingFact
from .types import CodingWriteError, FactReconcileOp, ReconcileOp, WorkingFact
logger = logging.getLogger(__name__)
@@ -81,10 +82,13 @@ class CodingWriter:
request_id: Optional[str] = None,
source: str = "default",
memory_at: Optional[datetime] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""
working/coding 段写入端到端。memory_at 是对话时间,记到这次写出的经验和
事实上;不传则用收到请求的时间(与 chat 链路一致)。返回响应 dict(兼容 client.add 返回结构):
事实上;不传则用收到请求的时间(与 chat 链路一致)。language 是显式 prompt
语言码,透传给抽取、经验 reconcile、事实 reconcile 与演变总结;None 时各步
沿用 zh/en 自动检测。返回响应 dict(兼容 client.add 返回结构):
{
"success": True,
"scene": "productivity",
@@ -101,6 +105,13 @@ class CodingWriter:
],
"fact_ids": ["..."], # 这趟真正落库的 fact_id
}
模型失败(抽取 LLM 调用失败 / 空回复 / 解析不出)或向量化失败时返回
success=False,agent_error_code 为 LLM_ERROR / EMPTY_RESPONSE /
JSON_PARSE_FAILED / EMBEDDING_FAILED,error_code=502。经验层 / 事实层
reconcile 的模型决策失败时 agent_error_code=RECONCILE_FAILED,原因在
reconcile_error_code(LLM_UNAVAILABLE / EMPTY_RESPONSE / JSON_PARSE_FAILED),
本次什么都不落库。模型答了但没有值得记的内容仍是 success=True、0 条。
"""
request_id = request_id or str(uuid.uuid4())
t0 = time.perf_counter()
@@ -140,6 +151,10 @@ class CodingWriter:
# 跟库里已有的什么关系是 reconcile 的事。把已有 task 摆到它面前,它会把
# 一个其实不同的任务往清单里某个措辞上靠,reconciler 随后判成 UPDATE 合并
# 掉,那个区别就没了——措辞对不上顶多多存一条,措辞被带偏是直接写错。
#
# 模型失败(LLM 调用失败 / 空回复 / 解析不出)抛 CodingWriteError:整次写入
# 报失败、什么都不存,连 trace 也不追加——调用方换个 request_id 重跑时,
# 历史窗口里不会多出这段本该是「新消息」的对话。
try:
drafts, fact_drafts = await self.extractor.extract(
messages,
@@ -151,27 +166,28 @@ class CodingWriter:
branch=branch,
session_id=session_id,
request_id=request_id,
language=language,
)
finally:
# 历史服务于下一次 write;即使本次没跨过价值门槛,也不能丢掉上下文。
try:
await self.store.append_trace(
current_messages,
request_id=request_id,
user_id=user_id,
agent_id=agent_id or "default_agent",
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
source=source,
)
except Exception as e:
logger.warning(f"[coding-write] append trace failed: {e}")
except CodingWriteError as e:
return self._failure_response(request_id, t0, e)
trace_kwargs = dict(
request_id=request_id,
user_id=user_id,
agent_id=agent_id or "default_agent",
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
source=source,
)
if not drafts and not fact_drafts:
# 模型答了、只是没有值得记的:正常的空抽取,仍算成功。历史服务于下一次
# write;即使本次没跨过价值门槛,也不能丢掉上下文。
logger.info(
f"[coding-write] no drafts produced (value bar); request_id={request_id}"
)
await self._append_trace(current_messages, **trace_kwargs)
return self._empty_response(request_id, t0)
# extractor 只负责内容;调用来源由 SDK 入口注入,不能交给 LLM 生成。
@@ -179,32 +195,51 @@ class CodingWriter:
draft.source = source
draft.memory_at = memory_at
# 3. 经验层 reconcile + 持久化
ops: List[ReconcileOp] = (
await self.reconciler.reconcile(
drafts,
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id or "",
)
if drafts else []
)
# 4. 事实层 reconcile + 持久化(独立链路,不受经验层结果影响)
# 3. 两层 reconcile 的模型决策(只检索 + 调 LLM,不落库)
# 4. 两层决策都成功后才落库:经验层 apply → 事实层 apply
#
# 先把所有模型决策做完再写库,任何一层的决策失败(LLM 失败 / 空回复 /
# 解析不出 → CodingWriteError(RECONCILE_FAILED))都发生在落库之前,本次
# 写入一条经验 / 事实都没存、也没覆盖旧事实,换个 request_id 重跑即干净。
# 两层决策互不依赖(各自检索各自的候选),提前做不改变结果。
#
# 向量化失败抛 CodingWriteError(EMBEDDING_FAILED):不落没有向量的节点,
# 整次写入报失败。
facts: List[WorkingFact] = []
fact_ops: List[FactReconcileOp] = []
if fact_drafts:
facts, fact_ops = await self.fact_reconciler.reconcile(
fact_drafts,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
ops: List[ReconcileOp] = []
try:
experience_plan = (
await self.reconciler.plan(
drafts,
request_id=request_id,
user_id=user_id,
agent_id=agent_id,
session_id=session_id or "",
language=language,
)
if drafts else None
)
fact_plan = (
await self.fact_reconciler.plan(
fact_drafts,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
source=source,
memory_at=memory_at,
language=language,
)
if fact_drafts else None
)
if experience_plan is not None:
ops = await self.reconciler.apply(experience_plan)
if fact_plan is not None:
facts, fact_ops = await self.fact_reconciler.apply(fact_plan)
except CodingWriteError as e:
return self._failure_response(request_id, t0, e)
experience_ids = [op.target_experience_id for op in ops
if op.action in ("ADD", "UPDATE") and op.target_experience_id]
@@ -212,6 +247,7 @@ class CodingWriter:
# 5. 同批全互链
await self._crosslink(experience_ids, facts, user_id=user_id)
await self._append_trace(current_messages, **trace_kwargs)
add_cnt = sum(1 for op in ops if op.action == "ADD")
update_cnt = sum(1 for op in ops if op.action == "UPDATE")
@@ -292,6 +328,41 @@ class CodingWriter:
f"[coding-write] crosslink fact {f.fact_id} failed: {e}"
)
async def _append_trace(self, current_messages: List[ChatMessage], **kwargs) -> None:
"""把本次对话追加进 recent trace(下一次 write 的历史上下文),best-effort。"""
try:
await self.store.append_trace(current_messages, **kwargs)
except Exception as e:
logger.warning(f"[coding-write] append trace failed: {e}")
def _failure_response(
self, request_id: str, t0: float, error: CodingWriteError,
) -> Dict[str, Any]:
"""模型 / 向量化失败:success=False,稳定失败码在 agent_error_code。
形状与 chat 链路的失败回执一致(error_code=502,error_message 带
「[CODE] 」前缀),调用方据 success 判定并重跑。
"""
code = error.error_code
cause = getattr(error, "cause", None)
message = f"{cause}: {error}" if cause else str(error)
logger.warning(
f"[coding-write] failed: code={code} request_id={request_id}: {message}"
)
resp = self._empty_response(request_id, t0)
resp.update({
"success": False,
"error_code": 502,
"error_message": f"[{code}] {message}",
"agent_status": "failed",
"agent_error_code": code,
"agent_error": message,
})
if cause:
# 与 chat 链路一致:reconcile 失败的具体原因
resp["reconcile_error_code"] = cause
return resp
def _empty_response(self, request_id: str, t0: float) -> Dict[str, Any]:
return {
"success": True,
+35
View File
@@ -432,6 +432,14 @@ class RecallConfig:
retrieval_clauses_require_keyword: bool = None # env: MEMORY_RETRIEVAL_CLAUSES_REQUIRE_KEYWORD
retrieval_clauses_min_keyword_score: float = None # normalized BM25 floor
supplemental_propositions_enabled: bool = None # env: MEMORY_SUPPLEMENTAL_PROPOSITIONS_ENABLED
# ── Intention(L7)判重 ──
# 开启(默认):召回时同 user+agent+同天 memory_at+同 valid_until 且
# cosine≥阈值 的意图视为复本,只返回最早一条,其余异步从 VDB 删除(历史行为)。
# 关闭:读侧与写入侧批内都不判重——不隐藏、不丢弃、不删除任何意图。
intention_dedup_enabled: bool = None # env: MEMORY_INTENTION_DEDUP_ENABLED
# 读侧意图判重 cosine 阈值。env: MEMORY_INTENTION_DEDUP_THRESHOLD,
# 未设置时回退到共享的 MEMORY_DEDUP_THRESHOLD(兼容旧部署),默认 0.95。
intention_dedup_threshold: float = None
# 最小分数阈值
min_score_threshold: float = 0.0
@@ -750,6 +758,15 @@ class RecallConfig:
self.supplemental_propositions_enabled = _get_env_bool(
"MEMORY_SUPPLEMENTAL_PROPOSITIONS_ENABLED", False
)
if self.intention_dedup_enabled is None:
self.intention_dedup_enabled = _get_env_bool(
"MEMORY_INTENTION_DEDUP_ENABLED", True
)
if self.intention_dedup_threshold is None:
self.intention_dedup_threshold = _get_env_float(
"MEMORY_INTENTION_DEDUP_THRESHOLD",
_get_env_float("MEMORY_DEDUP_THRESHOLD", 0.95),
)
@dataclass
@@ -1264,6 +1281,12 @@ class MemoryConfig:
enable_agent: bool = False
debug: bool = False
metrics_enabled: bool = True
# LLM prompt 语言(client 默认值;env: MEMORY_LANGUAGE)。canonical 码见
# hy_memory.prompts.SUPPORTED_LANGUAGES(en / zh-Hans / zh-Hant / ja / es / de /
# pt / fr / ko / ru / id),也接受别名 zh、zh-CN → zh-Hans,zh-TW、zh-HK →
# zh-Hant,pt-BR → pt。add() 等的 per-call language 优先于它。None(默认)=
# 沿用 zh/en 自动检测。显式指定时记忆正文用该语言书写。未知码抛 ValueError。
language: Optional[str] = None
# Runtime-only HTTP transport factory. MemHub uses this for request-cost
# budgeting without restricting the embedded SDK's local store access.
http_transport_factory: Optional[Callable[[], Any]] = field(
@@ -1278,6 +1301,11 @@ class MemoryConfig:
self.enable_agent = _get_env_bool("MEMORY_ENABLE_AGENT", False)
self.debug = _get_env_bool("MEMORY_DEBUG", False)
self.metrics_enabled = _get_env_bool("MEMORY_METRICS", True)
if self.language is None:
self.language = os.getenv("MEMORY_LANGUAGE", "").strip() or None
from .prompts import normalize_language
self.language = normalize_language(self.language)
@classmethod
def from_env(cls) -> "MemoryConfig":
@@ -1328,6 +1356,10 @@ class MemoryConfig:
config.metrics_enabled = config_dict.get(
"metrics_enabled", config.metrics_enabled
)
if config_dict.get("language") is not None:
from .prompts import normalize_language
config.language = normalize_language(config_dict["language"])
return config
@@ -1369,6 +1401,8 @@ class MemoryConfig:
"supplemental_propositions_enabled": (
self.recall.supplemental_propositions_enabled
),
"intention_dedup_enabled": self.recall.intention_dedup_enabled,
"intention_dedup_threshold": self.recall.intention_dedup_threshold,
},
"pipeline": {
"default_version": self.pipeline.default_version,
@@ -1386,6 +1420,7 @@ class MemoryConfig:
"enable_graph": self.enable_graph,
"enable_agent": self.enable_agent,
"metrics_enabled": self.metrics_enabled,
"language": self.language,
}
def print_config(self) -> None:
+21 -57
View File
@@ -64,59 +64,15 @@ def _topic_lock(graph_store, isolation_key: str, topic_id: str) -> asyncio.Lock:
return lock
TOPIC_SUMMARY_PROMPT = """You are summarizing everything known about ONE topic.
## Topic
{topic_name}
## Memories under this topic (oldest → newest)
{memories}
---
Write a single dense summary of what is known about the user with respect to this
topic.
Rules:
* 2–4 sentences, third person, starting with "The user ..." (or the named person
if the memories are about someone else).
* Cover the topic as a whole — state the current situation, and mention how it
developed only when the change matters.
* Preserve concrete details: names, places, numbers, dates.
* Do NOT invent anything that is not in the memories.
* Do NOT enumerate the memories one by one; synthesize them.
* Output language must match the language of the memories.
Return ONLY the summary text, no preamble, no JSON, no quotes."""
TOPIC_SUMMARY_PROMPT_ZH = """你在为**一个主题**总结已知的全部信息。
## 主题
{topic_name}
## 该主题下的记忆(时间正序)
{memories}
---
写一段关于用户在这个主题上的高密度摘要。
要求:
* 2–4 句,第三人称,以"用户…"开头(若记忆讲的是别人,则以那个人的名字开头)。
* 覆盖整个主题 —— 说明当前状况;只在变化本身重要时才提它是怎么演变过来的。
* 保留具体细节:人名、地点、数字、日期。
* 绝不编造记忆里没有的内容。
* 不要逐条罗列记忆,要综合成一段。
* 输出语言与记忆的语言一致。
只返回摘要正文,不要任何前言、不要 JSON、不要引号。"""
# 多语言版本见 hy_memory/prompts/<language>.py;旧名字保留为别名。
from ...prompts import ( # noqa: E402
get_prompt,
normalize_language,
resolve_language,
with_output_language,
)
from ...prompts.en import TOPIC_SUMMARY_PROMPT # noqa: E402,F401 (re-export)
from ...prompts.zh_hans import TOPIC_SUMMARY_PROMPT as TOPIC_SUMMARY_PROMPT_ZH # noqa: E402,F401 (re-export)
async def find_latest_topic_summary(
@@ -193,9 +149,13 @@ async def maybe_summarize_topics(
max_concurrency: int = 3,
temperature: Optional[float] = None,
trace=None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""对本次写入触达的 Topic,按需有界并发生成 / 刷新 summary。
language 为显式 prompt 语言码(随触发本次写入的请求);None 时按记忆内容
做 zh/en 自动检测。
触发条件:Topic 下 active 记忆数 ≥ `min_memories`,且比上次总结时多了 ≥ `step` 条。
这样既不会每写一条就重算(贵),也不会一直不更新(陈旧)。
@@ -254,6 +214,7 @@ async def maybe_summarize_topics(
topic_name=topic_names.get(topic_id) or topic_id,
memories=[m.get("content") or "" for m in mems],
temperature=temperature,
language=language,
)
if trace is not None:
await trace(
@@ -317,15 +278,18 @@ async def maybe_summarize_topics(
async def _generate(
*, llm_provider, topic_name: str, memories: List[str],
temperature: Optional[float] = None,
language: Optional[str] = None,
) -> tuple:
"""返回 (摘要文本, 实际 prompt, token 用量) —— 后两个供埋点,线上要能复盘。"""
from ...utils.lang_detect import is_chinese
body = "\n".join(f"- {c}" for c in memories if c)
if not body:
return "", "", {}
tpl = TOPIC_SUMMARY_PROMPT_ZH if is_chinese(body) else TOPIC_SUMMARY_PROMPT
prompt = tpl.format(topic_name=topic_name, memories=body)
explicit = normalize_language(language)
code = resolve_language(explicit, body)
prompt = get_prompt("TOPIC_SUMMARY_PROMPT", code).format(
topic_name=topic_name, memories=body,
)
prompt = with_output_language(prompt, code, explicit=explicit is not None)
resp = await llm_provider.complete(
prompt=prompt,
max_tokens=400,
+31 -10
View File
@@ -15,6 +15,9 @@ HY Memory - Intention (L7) 召回 + 惰性过期转换 + 复本判定
→ 视为同一个意图的复本(extractor 复读机 / 多 session 重复表达)。
保留组里 `gmt_created` 最早的(老的先来先得,稳定 id);
- **异步删库**:其余复本 fire-and-forget 从 VDB 删除,不影响本次调用 RT。
- **开关**:判重受 `MemoryConfig.recall.intention_dedup_enabled`(默认开)与
`recall.intention_dedup_threshold` 控制,由 reader 按 client 配置传入。
关闭时完全跳过判重:不隐藏、不删除任何意图(写入侧批内判重同样跳过)。
Q: 为什么要看 memory_at 和 valid_until?
A: 意图是 point-in-time 信号。同一句"我周五要开会",如果一条 memory_at 在
@@ -38,8 +41,14 @@ from ...models.memory import MemoryLayer, MemoryStatus
logger = logging.getLogger(__name__)
def _get_dedup_threshold() -> float:
"""去重 cosine 阈值:沿用 dedup 内核的 MEMORY_DEDUP_THRESHOLD,默认 0.95。"""
def _get_dedup_threshold(configured: Optional[float] = None) -> float:
"""去重 cosine 阈值:优先用 client 配置;否则(或配置非法)沿用 dedup 内核的
MEMORY_DEDUP_THRESHOLD,默认 0.95。"""
try:
if configured is not None:
return float(configured)
except (TypeError, ValueError):
logger.warning(f"[intention] invalid dedup threshold: {configured!r}")
try:
return float(os.environ.get("MEMORY_DEDUP_THRESHOLD", 0.95))
except (TypeError, ValueError):
@@ -244,6 +253,8 @@ async def recall_intentions(
stats_out: Optional[Dict[str, Any]] = None,
include_sources: Optional[List[str]] = None,
exclude_sources: Optional[List[str]] = None,
dedup_enabled: bool = True,
dedup_threshold: Optional[float] = None,
) -> List[Dict[str, Any]]:
"""
召回未过期的 L7_INTENTION 节点(VDB),惰性过期转 L2_FACT,判重并异步删复本。
@@ -260,6 +271,11 @@ async def recall_intentions(
now: 当前时间(默认 datetime.now(),测试可注入)
stats_out: 可选 dict,本函数把分段耗时与数量**原地回填**进去,供调用方落
READ_INTENTION pipeline log。传 None 则完全不统计(零开销)。
dedup_enabled: 是否执行读侧判重(及复本异步删库)。False 时跳过判重,
所有未过期意图原样返回,不删除任何节点。默认 True(历史行为)。
dedup_threshold: 判重 cosine 阈值。None 时回退到 env
MEMORY_DEDUP_THRESHOLD(默认 0.95),兼容直接调用本函数的旧代码;
reader 总是传入 `recall.intention_dedup_threshold`。
Returns:
[{"node_id", "score", "node", "layer": "l7_intention", "source": "vdb_intention"}, ...]
@@ -272,7 +288,7 @@ async def recall_intentions(
埋点字段(回填进 stats_out):
耗时(ms): vdb_search_ms / expire_ms / backfill_embed_ms / dedup_ms / total_ms
数量: fetched / expired / survivors / dedup_dropped / returned
配置: fetch_limit / out_limit / dedup_threshold
配置: fetch_limit / out_limit / dedup_enabled / dedup_threshold
说明: expire_ms、backfill_embed_ms 只有真的发生了对应操作才非 0,
借此可判断慢在 VDB 检索、逐条过期转换、补拉向量还是判重计算。
"""
@@ -358,7 +374,7 @@ async def recall_intentions(
_t = time.perf_counter()
_backfilled = 0
_missing_cnt = 0
if survivors:
if survivors and dedup_enabled: # 补拉只为判重;关闭时无需这次 VDB 读
missing_ids = [
h.get("node_id") for h in survivors
if h.get("node_id") and not getattr(h.get("node"), "embedding", None)
@@ -391,18 +407,23 @@ async def recall_intentions(
)
# 2. 判重:同 user + 同天 memory_at + valid_until 同日 + cosine≥threshold
# dedup_enabled=False → 完全跳过:不隐藏、不删除。
_t = time.perf_counter()
threshold = _get_dedup_threshold()
try:
kept, delete_ids = _partition_duplicates(survivors, threshold)
except Exception as e:
# 任何 dedup 异常 → 降级为不去重(返回全部 survivors 截断 limit)
logger.warning(f"[intention] dedup partition failed, degrade to raw: {e}")
threshold = _get_dedup_threshold(dedup_threshold)
if dedup_enabled:
try:
kept, delete_ids = _partition_duplicates(survivors, threshold)
except Exception as e:
# 任何 dedup 异常 → 降级为不去重(返回全部 survivors 截断 limit)
logger.warning(f"[intention] dedup partition failed, degrade to raw: {e}")
kept, delete_ids = survivors, []
else:
kept, delete_ids = survivors, []
# 纯 CPU(cosine 两两比较);survivors 多时可能显著
_stat(
dedup_ms=round((time.perf_counter() - _t) * 1000, 2),
dedup_dropped=len(delete_ids),
dedup_enabled=bool(dedup_enabled),
dedup_threshold=threshold,
)
+54 -15
View File
@@ -95,21 +95,48 @@ class ToolCall:
# [John]: 我也喜欢茶
# 名字单独占一行、整段只出现一次;[user] 标签**不换成名字**,视角信息留在标签里。
# extract 用这一行给 L0 basic_info 取 name(用户不必在对话里自报姓名)。
# 各语言的模板在 hy_memory/prompts/<language>.py 的 USER_NAME_LINE;这两个旧名字保留。
_USER_NAME_LINE_ZH = "用户名叫{name}"
_USER_NAME_LINE_EN = "The user's name is {name}"
_USER_NAME_LINE_RE = re.compile(
r"^(?:用户名叫|The user's name is)\s*(.+?)\s*$"
)
_user_name_line_res: Optional[List["re.Pattern"]] = None
def render_user_name_line(name: str, is_zh: bool) -> str:
"""渲染文首的用户名字行;name 为空返回空串。"""
def _user_name_line_patterns() -> List["re.Pattern"]:
"""每种语言一个匹配名字行的正则(L1_RAW 里可能存着任意语言的名字行)。"""
global _user_name_line_res
if _user_name_line_res is None:
from ..prompts import SUPPORTED_LANGUAGES, get_prompt
patterns = []
for code in SUPPORTED_LANGUAGES:
try:
tpl = get_prompt("USER_NAME_LINE", code)
except (ImportError, AttributeError):
continue
prefix, _, suffix = tpl.partition("{name}")
patterns.append(re.compile(
r"^" + re.escape(prefix.strip()) + r"\s*(.+?)\s*"
+ re.escape(suffix.strip()) + r"$"
))
_user_name_line_res = patterns
return _user_name_line_res
def render_user_name_line(
name: str, is_zh: bool = False, *, language: Optional[str] = None,
) -> str:
"""渲染文首的用户名字行;name 为空返回空串。
language(语言码)优先;未给时按 is_zh 在 zh-Hans / en 间选(旧行为)。
"""
name = (name or "").strip()
if not name:
return ""
tpl = _USER_NAME_LINE_ZH if is_zh else _USER_NAME_LINE_EN
return tpl.format(name=name)
from ..prompts import get_prompt
code = language or ("zh-Hans" if is_zh else "en")
return get_prompt("USER_NAME_LINE", code).format(name=name)
def parse_user_name_line(line: str) -> Optional[str]:
@@ -118,8 +145,12 @@ def parse_user_name_line(line: str) -> Optional[str]:
L1_RAW 存的就是 get_flat_content() 的产物,回读历史时要靠这个把名字行识别出来,
否则它会被当成一条无前缀的裸消息(见 MemoryWriter._get_recent_history)。
"""
m = _USER_NAME_LINE_RE.match((line or "").strip())
return m.group(1) if m else None
text = (line or "").strip()
for pattern in _user_name_line_patterns():
m = pattern.match(text)
if m:
return m.group(1)
return None
@dataclass
@@ -257,6 +288,11 @@ class WriteRequest:
branch: Optional[str] = None
# 分支名。同 workspace_id。
# 显式 prompt 语言码(canonical,见 hy_memory.prompts.SUPPORTED_LANGUAGES)。
# 由 client 入口按「per-call > MemoryConfig.language」解析后注入;None 时
# 抽取 / reconcile / 画像等各步沿用 zh/en 自动检测。
language: Optional[str] = None
# ---- 便捷方法 ----
def has_messages(self) -> bool:
@@ -286,8 +322,8 @@ class WriteRequest:
[John]: 我也喜欢茶
无 name 时与历史行为完全一致(`[user]: ...` / `[assistant]: ...`)。
名字行只在 role=user 带 name 时出现,且整段只出现一次;语言按消息内容判定
(与 extractor 选 zh/en prompt 同一套判据)。
名字行只在 role=user 带 name 时出现,且整段只出现一次;语言取显式
language,未指定时按消息内容判定(与 extractor 选 zh/en prompt 同一套判据)。
"""
if not self.messages:
return self.content or ""
@@ -296,11 +332,14 @@ class WriteRequest:
name = self.user_name()
if name:
from ..utils.lang_detect import is_chinese
if self.language:
name_line = render_user_name_line(name, language=self.language)
else:
from ..utils.lang_detect import is_chinese
name_line = render_user_name_line(
name, is_chinese(messages=self.messages)
)
name_line = render_user_name_line(
name, is_chinese(messages=self.messages)
)
if name_line:
lines.insert(0, name_line)
+6
View File
@@ -1856,6 +1856,12 @@ class BaseReadPipeline(ReadPipeline):
stats_out=_intent_stats,
include_sources=request.include_sources,
exclude_sources=request.exclude_sources,
dedup_enabled=bool(getattr(
self.config.recall, "intention_dedup_enabled", True
)),
dedup_threshold=getattr(
self.config.recall, "intention_dedup_threshold", None
),
)
await trace_log.log_intention(
stats=_intent_stats,
+80 -368
View File
@@ -979,171 +979,16 @@ def s2_agent_skip_reason(materials: Dict[str, Any]) -> Optional[str]:
return None
# ====================================================================
# System Prompt (dual language)
# System Prompt (multilingual: hy_memory/prompts/<language>.py)
# ====================================================================
SYSTEM2_AGENT_PROMPT_ZH = """你是一个认知加工 Agent,负责从用户的记忆数据中演化高层认知结构。
## 记忆分层架构
用户的记忆存储在两个系统中:
### VDB(向量数据库)— 你只能读取,不能修改
| 层级 | 说明 |
|------|------|
| L0 BASIC_INFO | 用户基础属性(姓名、年龄、职业等)|
| L1 RAW | 原始对话记录 |
| L2 FACT | 原子事实(从对话中提取的客观事实)|
| L3 SUMMARY | 会话摘要 |
| L4 IDENTITY | 身份画像(性格特征、偏好、习惯)|
| L5 KNOWLEDGE | 领域知识 |
### Graph(知识图谱)— 你是唯一的写入者
| 层级 | 说明 |
|------|------|
| L6 SCHEMA | 行为模式 — 域内、原子化、创建后不可变 |
## L6 Schema 是什么?
Schema 是从多次独立经历归纳出的长期行为模式,包含以下三个要素:
- **Circumstance(场景)**:该模式发生的领域/话题/场景。Schema 必须限定在其场景内,不要跨域泛化。
- **Pattern(模式)**:用户在该场景下的惯常行为、思维方式或行动倾向。
- **Insight(洞察)**:底层心理驱动力或心智模型。
### 内容格式
行为模式用一句话组合三要素:
"当[场景]时,用户[模式]——反映了[洞察]。"
对与情绪、治疗、压力、哀伤或创伤有关的长期模式,不要把具体人生背景压缩成
空泛人格词。优先保留这三部分:
"Trigger/background(持久触发背景) → emotional response(反复情绪反应) → observed coping resource(用户已实际使用的应对资源)。"
只能写入 user-owned 证据中出现的触发背景、反应和应对方式;助手建议不是用户已观察到的应对资源。
### 规则
- **原子化**:一个 Schema 只包含一个模式。两个不同模式 → 两个 Schema。
- **不可变**:Schema 创建后内容永远不变。不要修改已有 Schema 的内容。
- **累积证据**:新事实只是在支持已有表述 → 调 `add_evidence`。如果新证据会让旧画像的当前状态产生实质遗漏、错误或误导,使用 `revise_schema` 创建新版本,让旧版本保留审计但退出 active 检索。
- **域内约束**:不要跨域。跨域抽象由系统的其他流程处理。
- **事实与模式分层**:单条明确身份、健康状态、设备使用、隶属关系、项目或偏好属于 L2 状态,不创建 L6。L6 必须由至少两个独立 raw/session 中的 user-owned 证据支持。
- **主体安全**:同一原始写入可能包含他人资料或示例。只有材料明确归因于当前用户时才创建用户 Schema;否则对该材料不执行任何操作。
- **候选覆盖**:source-context 审查时,每条候选事实必须二选一:用于 `create_schema`/`add_evidence` 的 evidence,或用 `skip_fact` 明确跳过。原始上下文只用于确认主体或补回该事实遗漏的细节,不得据此创建无关 Schema。
- **证据门槛**:创建 L6 必须有 user-owned evidence,且这些证据合计代表至少两个独立 raw/session。
一个被 S1 合并的 L2 事实可以代表多次独立写入,应检查它的 `source_raw_memory_ids`,不要只数节点。
助手建议不能作为用户模式的独立证据。
## 工作流程
输入材料分两部分(仅当已形成聚类时才会调用你):
- **聚类结果(Cluster Results)**:主输入,按语义相似度归组的事实
- **未聚类事实(补充)**:未进入任一聚类的事实,优先级低于聚类结果,酌情审视即可
对于每组事实(无论聚类还是未聚类):
1. **搜索已有 Schema**:如果已有 Schema 已覆盖该模式,使用 `add_evidence`。只有加入新证据后旧表述会产生实质遗漏、错误或误导时才使用 `revise_schema`。
2. **创建新 Schema**:仅当没有已有 Schema 覆盖时。一句话,三要素。
3. **建立关系**:两个 Schema 主题相关 → 用 `add_edge`(RELATED_TO)。
## 原则
- 所有 Graph 操作必须通过 tools 执行
- Schema 内容创建后不可变——绝不重新创建已存在的 Schema
- 优先 `add_evidence` 而非创建重复节点
- 宁可不创建,也不要创建低质量节点
- 一个 Schema = 一个域内原子模式
- 打标签时优先使用已有标签列表中的标签"""
SYSTEM2_AGENT_PROMPT_EN = """You are a cognitive processing Agent. Your job is to evolve higher-order cognitive structures from a user's raw memory data.
## Memory Architecture
### VDB (Vector Database) — read-only
| Layer | Description |
|-------|-------------|
| L0 BASIC_INFO | User attributes (name, age, occupation) |
| L1 RAW | Original conversation transcripts |
| L2 FACT | Atomic facts extracted from conversations |
| L3 SUMMARY | Session summaries |
| L4 IDENTITY | Identity traits, preferences, habits |
| L5 KNOWLEDGE | Domain knowledge |
### Graph (Knowledge Graph) — you are the sole writer
| Layer | Description |
|-------|-------------|
| L6 SCHEMA | Behavioral patterns — domain-bound, atomic, immutable once created |
## What is an L6 Schema?
A Schema is one cross-event behavioral pattern in the long-term user model and
contains the following three components:
- **Circumstance**: The domain/topic/situation where this pattern is observed. A Schema MUST stay within its circumstance — do NOT generalize across domains. Cross-domain abstraction is handled separately by the system.
- **Pattern**: The user's habitual behavior, thinking style, or action tendency in this circumstance.
- **Insight**: The underlying psychological driver or mental model.
### Content format
Use a single sentence combining all three:
"When [circumstance], the user [pattern] — reflecting [insight]."
For durable emotional, therapy, stress, grief, or trauma-related patterns, do
not erase the specific autobiographical background into a generic personality
label. Prefer an anchored structure that preserves:
"Trigger/background -> recurring emotional response -> observed coping resource."
Only include triggers, responses, and coping resources present in user-owned
evidence. Assistant advice is not an observed user coping resource.
### Rules
- **Atomic**: One pattern per Schema. Two distinct patterns → two Schemas.
- **Immutable**: Once created, content NEVER changes.
- **Evidence only**: New facts that merely support existing wording → call
`add_evidence`. If new evidence materially changes, corrects, or narrows the
current durable state, use `revise_schema` so the old version is retained for
audit but removed from active retrieval.
- **Domain-bound**: Stay within the observed domain. Cross-domain abstraction is handled by a separate system process.
- **Separate facts from patterns**: A single explicit identity, health state,
equipment use, affiliation, project, or preference belongs in L2 state memory,
not L6. L6 requires user-owned evidence representing at least two independent
raw writes or sessions. One reconciled L2 fact may represent multiple writes;
inspect its source_raw_memory_ids rather than counting fact nodes.
- **Subject safety**: Facts sharing a source chunk may describe other people or
examples. Create a user Schema only when the material explicitly attributes
the pattern to the user. Otherwise output no operation for that material.
- **Candidate coverage**: In a source-context review, account for every listed
candidate fact: either use it as evidence in `create_schema`/`add_evidence`,
or emit `skip_fact`. Raw context may confirm its subject or restore omitted
detail, but must not seed unrelated Schemas.
- **Evidence threshold**: Creating L6 requires one or more user-owned evidence
facts representing at least two independent raw writes or sessions.
Assistant advice does not establish a user pattern by itself.
### Good vs Bad examples
✅ "When cooking, the user strictly follows recipe steps and precisely measures ingredients — reflecting a need for external structure to manage uncertainty."
✅ "When gaming, the user always checks walkthroughs to achieve 100% completion — driven by low tolerance for incompleteness."
✅ "After an unexpected mentor loss, mortality reminders can bring grief back suddenly; the user pauses to reflect and takes quiet walks to regain steadiness."
❌ "The user approaches music as a medium for creative synthesis and cultural bridge-building, consistently blending diverse musical traditions..." (multiple patterns merged, too broad)
❌ "The user is passionate about many things and values quality." (no circumstance, too vague)
❌ "The user is meaning-driven and resilient." (erases the durable trigger and observed coping evidence)
## Your Workflow
Input materials (you are only invoked when at least one cluster exists):
- **Cluster Results**: Primary input — facts grouped by semantic similarity
- **Unclustered Facts (supplemental)**: Facts not in any cluster; lower priority than cluster results
For each group of facts (clustered or unclustered):
1. **Search existing Schemas**: If an existing Schema already captures this pattern, call `add_evidence`. Use `revise_schema` only when the old wording would become materially incomplete or misleading after adding the new evidence.
2. **Create new Schema**: Only if no existing Schema covers this pattern. One sentence, three components.
3. **Build relationships**: If two Schemas are thematically related, use `add_edge` (RELATED_TO).
## Principles
- All Graph operations MUST go through tools
- Schema content is IMMUTABLE — never recreate what already exists
- Prefer `add_evidence` over creating duplicates
- Prefer not creating over creating low-quality nodes
- One Schema = one atomic pattern in one domain
- When tagging, prefer reusing tags from the existing tags list"""
from ..prompts import ( # noqa: E402
get_prompt,
normalize_language,
with_output_language,
)
from ..prompts.zh_hans import SYSTEM2_AGENT_PROMPT as SYSTEM2_AGENT_PROMPT_ZH # noqa: E402,F401 (re-export)
from ..prompts.en import SYSTEM2_AGENT_PROMPT as SYSTEM2_AGENT_PROMPT_EN # noqa: E402,F401 (re-export)
# backward compat alias
SYSTEM2_AGENT_PROMPT = SYSTEM2_AGENT_PROMPT_ZH
@@ -2095,10 +1940,14 @@ async def run_system2_agent(
tool_executor: System2ToolExecutor,
config: MemoryConfig,
max_iterations: int = 10, # kept for backward compat, unused in single-call mode
language: Optional[str] = None,
) -> Dict[str, Any]:
"""
Phase 2: System 2 Agent — Single LLM call + JSON output + post-process execution.
``language``: explicit prompt language code (overrides config.language);
None and no config default → zh/en detection over the materials.
Instead of multi-turn tool-calling loop, this version:
1. Puts all materials (facts + existing graph) into the prompt
2. Asks LLM to output a JSON array of operations
@@ -2120,9 +1969,17 @@ async def run_system2_agent(
# Build LLMProvider from config
llm_provider = LLMProvider(config)
# Detect language from materials content
lang = _detect_language(materials)
system_prompt = _build_single_call_system_prompt(lang)
# Prompt language: explicit language (evolve(language=...) or
# MemoryConfig.language) wins; otherwise detect zh/en from the materials.
explicit_language = normalize_language(language) or normalize_language(
getattr(config, "language", None)
)
lang = explicit_language or _s2_language_code(_detect_language(materials))
system_prompt = with_output_language(
_build_single_call_system_prompt(lang),
lang,
explicit=explicit_language is not None,
)
user_msg = _build_materials_message(materials, lang=lang)
messages = [
@@ -2393,101 +2250,30 @@ async def run_system2_agent(
}
def _build_single_call_system_prompt(lang: str) -> str:
"""Build system prompt for single-call mode (includes output format instructions)."""
base = SYSTEM2_AGENT_PROMPT_ZH if lang == "zh" else SYSTEM2_AGENT_PROMPT_EN
def _s2_language_code(lang: str) -> str:
"""旧的 "zh" / "en" 标记映射到语言码;其余值按语言码处理。"""
if lang == "zh":
output_instructions = """
return "zh-Hans"
return lang or "en"
## 输出格式
不要调用 tools。直接输出一个 JSON 数组,包含你决定执行的操作。用 ```json 代码块包裹。
def _build_single_call_system_prompt(lang: str) -> str:
"""Build system prompt for single-call mode (includes output format instructions).
可用操作:
1. **create_schema** — 创建新的 L6 行为 Schema
```{"op": "create_schema", "schema_kind": "behavior_pattern", "content": "供用户阅读的简洁画像", "retrieval_text": "用于检索的行为与适用场景", "evidence_list": ["fact_node_id_1", "fact_node_id_2"], "tags": ["tag1"]}```
2. **add_evidence** — 给已有 Schema 追加证据(不修改内容)
```{"op": "add_evidence", "node_id": "existing_schema_node_id", "evidence_list": ["new_fact_id_1"]}```
3. **revise_schema** — 新证据使已有 Schema 的当前状态发生实质变化时,创建新版本并让旧版本退出默认检索。不要用于可共存的细节补充。
```{"op": "revise_schema", "node_id": "old_schema_id", "schema_kind": "explicit_state", "content": "更新后的当前画像", "retrieval_text": "更新后的检索文本", "evidence_list": ["new_fact_id_1"], "tags": ["tag1"], "reason": "明确说明旧画像为何会误导"}```
4. **add_edge** — 建立两个 Schema 之间的关系
```{"op": "add_edge", "source_id": "schema_id_1", "target_id": "schema_id_2", "reason": "关系原因"}```
5. **skip_fact** — 明确标记某条候选事实已审查但不创建 Schema。仅用于他人事实、临时事件、不能证明用户特征的助手建议,或已被现有 Schema 完整覆盖的事实。
```{"op": "skip_fact", "node_id": "candidate_fact_id", "reason": "不是稳定的用户模式"}```
## 输出约定
1. 输出必须是 JSON 数组,用 ```json 代码块包裹。
2. 如果数据不足以得出可靠结论,输出 `[]`。在 source-context review 中,
`[]` 表示已审查该组全部候选,且都不足以支持稳定 Schema。
3. 代码块外不要有任何其他文字。
4. evidence_list 中的 id 必须来自材料中提供的 fact node_id。
5. add_evidence / revise_schema 的 node_id 必须来自已有 Graph 节点的 id。
6. 对 source-context 审查,必须让每条候选事实出现在 evidence_list 或 skip_fact 中。"""
else:
output_instructions = """
## Output Format
Do NOT call tools. Output a JSON array of operations directly, wrapped in a ```json code block.
Available operations:
1. **create_schema** — Create a new L6 behavioral Schema
```{"op": "create_schema", "schema_kind": "behavior_pattern", "content": "concise user-facing pattern", "retrieval_text": "behavior plus relevant situations", "evidence_list": ["fact_node_id_1", "fact_node_id_2"], "tags": ["tag1"]}```
2. **add_evidence** — Add evidence to an existing Schema (content stays immutable)
```{"op": "add_evidence", "node_id": "existing_schema_node_id", "evidence_list": ["new_fact_id_1"]}```
3. **revise_schema** — Create a new version when new evidence materially changes the current state and retire the old version from active retrieval. Do not use it for compatible detail.
```{"op": "revise_schema", "node_id": "old_schema_id", "schema_kind": "explicit_state", "content": "updated current profile", "retrieval_text": "updated retrieval text", "evidence_list": ["new_fact_id_1"], "tags": ["tag1"], "reason": "why the old wording would mislead"}```
4. **add_edge** — Create a relationship between two Schemas
```{"op": "add_edge", "source_id": "schema_id_1", "target_id": "schema_id_2", "reason": "relationship reason"}```
5. **skip_fact** — Explicitly mark a candidate fact as reviewed without creating
a Schema. Use only for facts about another person, transient events, assistant
advice that does not establish a user trait, or facts already fully covered.
```{"op": "skip_fact", "node_id": "candidate_fact_id", "reason": "not a durable user pattern"}```
## Output Contract
1. Output MUST be a JSON array wrapped in a ```json code block.
2. If data is insufficient for reliable conclusions, output `[]`. In a
source-context review, `[]` means every candidate in that review was examined
and none supports a durable Schema.
3. NO text outside the code block.
4. evidence_list IDs must come from the fact node_ids provided in the materials.
5. add_evidence / revise_schema node_id must come from existing Graph node IDs.
6. For a source-context review, every candidate fact must appear in an evidence_list or skip_fact."""
``lang`` is a language code (``en`` / ``zh-Hans`` / ``ja`` ...; legacy ``zh``
means zh-Hans). Every fragment comes from hy_memory/prompts/<language>.py.
"""
code = _s2_language_code(lang)
base = get_prompt("SYSTEM2_AGENT_PROMPT", code)
output_instructions = get_prompt("SYSTEM2_OUTPUT_INSTRUCTIONS", code)
if _S2_SCHEMA_EVIDENCE_GATE_ENABLED:
if lang == "zh":
output_instructions += """
output_instructions += get_prompt("SYSTEM2_SCHEMA_EVIDENCE_GATE", code)
7. L6 只表示跨事件行为模式。create_schema 必须使用 user-owned 证据,且证据合计代表至少两个独立 raw/session。当 `source_raw_memory_ids` 包含多个独立写入时,一个合并后的 L2 事实也可以满足门槛。
8. 单条明确身份、健康状态、设备使用、隶属关系、项目或偏好不要创建 L6;这些由 canonical L2 状态通道维护。"""
else:
output_instructions += """
7. L6 represents cross-event behavioral patterns only. create_schema must use user-owned evidence representing at least two independent raw writes or sessions. A single reconciled fact can qualify when source_raw_memory_ids contains multiple independent writes.
8. Do not create L6 for a single explicit identity, health state, equipment use, affiliation, project, or preference. Canonical L2 state projection owns those facts."""
# Remove the old "All Graph operations MUST go through tools" principle
base = base.replace("- 所有 Graph 操作必须通过 tools 执行\n", "")
base = base.replace("- All Graph operations MUST go through tools\n", "")
# Remove old tool-calling instructions from workflow
base = base.replace("调 `add_evidence` 添加证据", "使用 add_evidence 操作")
base = base.replace("call `add_evidence`", "use add_evidence operation")
base = base.replace("调 `add_evidence`", "使用 add_evidence")
base = base.replace("用 `add_edge`(RELATED_TO)", "使用 add_edge 操作")
base = base.replace("use `add_edge` (RELATED_TO)", "use add_edge operation")
# Remove the old "All Graph operations MUST go through tools" principle and
# the tool-calling wording from the workflow (single-call mode has no tools).
for old, new in get_prompt("SYSTEM2_SINGLE_CALL_REPLACEMENTS", code):
base = base.replace(old, new)
return base + output_instructions
@@ -2554,10 +2340,14 @@ def _detect_language(materials: Dict[str, Any]) -> str:
def _build_materials_message(materials: Dict[str, Any], lang: str = "zh") -> str:
"""将预处理好的 materials 格式化为 Agent 能理解的 user message"""
is_zh = (lang == "zh")
"""将预处理好的 materials 格式化为 Agent 能理解的 user message
parts = [("## 预处理材料\n" if is_zh else "## Preprocessed Materials\n")]
``lang`` 为语言码(旧值 "zh" 视作 zh-Hans);文案见 prompts/<language>.py 的
SYSTEM2_MATERIALS。
"""
text = get_prompt("SYSTEM2_MATERIALS", _s2_language_code(lang))
parts = [text["title"]]
stats = materials.get("stats", {})
fresh = stats.get("fresh_facts", 0)
@@ -2566,55 +2356,35 @@ def _build_materials_message(materials: Dict[str, Any], lang: str = "zh") -> str
graph_total = stats.get("graph_total_schemas", 0)
recalled_schemas = stats.get("graph_schemas_recalled", 0)
if is_zh:
parts.append(f"### 统计\n- VDB 事实总数: {stats.get('total_facts_all', 0)}")
parts.append(f"- 新鲜事实(待处理): {fresh}(已被引用排除={evidenced})")
parts.append(f"- 聚类数: {stats.get('clusters_found', 0)}")
parts.append(
f"- 聚类内事实: {stats.get('clustered_facts_count', 0)},"
f"未聚类补充: {stats.get('unclustered_facts_count', 0)}"
)
parts.append(f"- Graph Schema 总数: {stats.get('graph_total_schemas', 0)}(本次召回={recalled_schemas})")
if graph_total == 0:
parts.append("- Graph 状态: **空**(首次认知加工)\n")
else:
parts.append("")
parts.append(text["stats"].format(
total=stats.get('total_facts_all', 0),
fresh=fresh,
evidenced=evidenced,
clusters=stats.get('clusters_found', 0),
clustered=stats.get('clustered_facts_count', 0),
unclustered=stats.get('unclustered_facts_count', 0),
graph_total=stats.get('graph_total_schemas', 0),
recalled=recalled_schemas,
))
if graph_total == 0:
parts.append(text["graph_status_empty"])
else:
parts.append(f"### Statistics\n- Total VDB facts: {stats.get('total_facts_all', 0)}")
parts.append(f"- Fresh facts (to process): {fresh} (evidenced_excluded={evidenced})")
parts.append(f"- Clusters found: {stats.get('clusters_found', 0)}")
parts.append(
f"- Facts in clusters: {stats.get('clustered_facts_count', 0)}, "
f"unclustered supplement: {stats.get('unclustered_facts_count', 0)}"
)
parts.append(f"- Graph Schemas total: {stats.get('graph_total_schemas', 0)} (recalled={recalled_schemas})")
if graph_total == 0:
parts.append("- Graph status: **EMPTY** (first cognitive processing run)\n")
else:
parts.append("")
parts.append("")
# Clusters
clusters = materials.get("clusters", [])
clustered_ids = set()
if clusters:
parts.append(("### 聚类结果\n" if is_zh else "### Cluster Results\n"))
parts.append(text["clusters_header"])
for i, c in enumerate(clusters):
n_facts = len(c['facts'])
group_type = c.get("group_type", "semantic_cluster")
if group_type == "holistic_therapy_review":
label = (
f"{n_facts} 条跨会话用户事实;只归纳有证据的长期情绪/治疗模式"
if is_zh else
f"{n_facts} cross-session user facts; only evidence-backed durable emotional/therapy patterns"
)
label = text["label_holistic"].format(n=n_facts)
elif group_type == "source_context_review":
label = (
f"{n_facts} 条候选事实;原始写入仅用于主体与细节核验"
if is_zh else
f"{n_facts} candidate facts; raw write is only for subject/detail validation"
)
label = text["label_source_context"].format(n=n_facts)
else:
label = f"{n_facts} 条,主题" if is_zh else f"{n_facts} facts, topic"
label = text["label_topic"].format(n=n_facts)
parts.append(f"**Cluster {i}** ({label}: {c['centroid_text']})")
for f in c["facts"]:
source_ids = list(f.get("source_raw_memory_ids") or [])
@@ -2628,8 +2398,7 @@ def _build_materials_message(materials: Dict[str, Any], lang: str = "zh") -> str
)
clustered_ids.add(f['node_id'])
if group_type == "source_context_review" and c.get("raw_contexts"):
raw_label = "候选事实对应的原始片段" if is_zh else "Relevant raw excerpt per candidate"
parts.append(f"\n**{raw_label}:**")
parts.append(f"\n**{text['raw_label']}:**")
for item in c["raw_contexts"]:
parts.append(
f"- candidate_id={item.get('node_id')}, "
@@ -2638,30 +2407,12 @@ def _build_materials_message(materials: Dict[str, Any], lang: str = "zh") -> str
)
parts.append("")
if materials.get("holistic_therapy_review"):
parts.append(
(
"这是一次有边界的夜间全局整理。最多创建 "
f"{_S2_HOLISTIC_THERAPY_MAX_SCHEMAS} 条 anchored therapy Schema。"
"优先联合不同 raw/session 中的持久 trigger/background、"
"反复 emotional response 和用户已实际使用的 coping resource/decision constraint。"
"证据不足时输出 []。\n"
if is_zh else
"This is one bounded nightly holistic consolidation pass. Create at most "
f"{_S2_HOLISTIC_THERAPY_MAX_SCHEMAS} anchored therapy Schemas. "
"Prefer joining durable trigger/background, recurring emotional response, "
"and observed coping resource or decision constraint across different raw writes/sessions. "
"Before create_schema, union raw_sources for its evidence IDs and require at least two "
"distinct raw source IDs; two facts from one raw write are one event. "
"Output [] when evidence is insufficient.\n"
)
)
parts.append(text["holistic_note"].format(
max_schemas=_S2_HOLISTIC_THERAPY_MAX_SCHEMAS,
))
companion_hints = materials.get("independent_source_companions") or {}
if companion_hints:
parts.append(
"**独立来源 companion 提示(只是候选,不是结论):**"
if is_zh else
"**Independent-source companion hints (candidates, not conclusions):**"
)
parts.append(text["companion_header"])
for node_id, companions in companion_hints.items():
parts.append(
f"- {node_id} -> "
@@ -2677,14 +2428,8 @@ def _build_materials_message(materials: Dict[str, Any], lang: str = "zh") -> str
all_facts = materials.get("unprocessed_facts", [])
unclustered = [f for f in all_facts if f["node_id"] not in clustered_ids]
if clusters and unclustered:
header = "### 未聚类事实(补充,次要)\n" if is_zh else "### Unclustered Facts (supplemental)\n"
hint = (
"以下事实未进入上述任一聚类,可在完成聚类主题加工后酌情审视;优先级低于聚类结果。\n"
if is_zh
else "Facts below were not grouped into any cluster above. Review after cluster themes; lower priority than cluster results.\n"
)
parts.append(header)
parts.append(hint)
parts.append(text["unclustered_header"])
parts.append(text["unclustered_hint"])
for f in unclustered:
parts.append(f" - [{f['layer']}] {f['content']} (id={f['node_id']})")
parts.append("")
@@ -2692,10 +2437,7 @@ def _build_materials_message(materials: Dict[str, Any], lang: str = "zh") -> str
# Existing graph nodes
forward = materials.get("graph_forward", [])
if forward:
if is_zh:
parts.append(f"### 相关 Graph 节点(通过 cluster centroid 向量召回,共 {len(forward)} 条)\n")
else:
parts.append(f"### Related Graph Nodes (retrieved via cluster centroid search, {len(forward)} hits)\n")
parts.append(text["forward_header"].format(count=len(forward)))
for g in forward:
sim = g.get("similarity", "")
sim_str = f", sim={sim}" if sim else ""
@@ -2705,30 +2447,17 @@ def _build_materials_message(materials: Dict[str, Any], lang: str = "zh") -> str
)
parts.append("")
elif graph_total == 0:
if is_zh:
parts.append("### 已有 Graph 节点\n当前 Graph 为空,这是首次认知加工。无需搜索 Graph,直接根据聚类结果创建 Schema。\n")
else:
parts.append("### Existing Graph Nodes\nThe Graph is currently EMPTY — this is the first cognitive processing run. Do NOT search the Graph (it will return nothing). Proceed directly to create Schemas from the cluster results.\n")
parts.append(text["graph_empty"])
else:
# Graph 有节点但 centroid search 没召回(可能 threshold 太高或语义距离远)
if is_zh:
parts.append(
f"### 已有 Graph 节点\n"
f"Graph 中共有 {stats.get('graph_total_schemas', 0)} 个 Schema,"
f"但 centroid 向量召回未命中相关节点。请根据聚类结果判断是否需要创建新 Schema。\n"
)
else:
parts.append(
f"### Existing Graph Nodes\n"
f"The Graph has {stats.get('graph_total_schemas', 0)} Schemas, "
f"but centroid-based vector recall found no matches. Decide based on cluster results whether new Schemas are needed.\n"
)
parts.append(text["graph_no_recall"].format(
graph_total=stats.get('graph_total_schemas', 0),
))
# Reverse hits
reverse = materials.get("graph_reverse", [])
if reverse:
parts.append(("### Graph 反向引用(已有 Schema 引用的 VDB 事实)\n" if is_zh
else "### Graph Reverse References (VDB facts referenced by existing Schemas)\n"))
parts.append(text["reverse_header"])
for r in reverse:
parts.append(
f" - [{r['layer']}] {r['content']} "
@@ -2739,26 +2468,9 @@ def _build_materials_message(materials: Dict[str, Any], lang: str = "zh") -> str
# Existing tags(指导 Agent 复用已有 tag,减少发散)
existing_tags = materials.get("existing_tags", [])
if existing_tags:
if is_zh:
parts.append("### 已有标签(请优先从中选择,避免创建同义标签)\n")
parts.append(f" {', '.join(existing_tags)}\n")
else:
parts.append("### Existing Tags (prefer reusing these over creating synonyms)\n")
parts.append(f" {', '.join(existing_tags)}\n")
parts.append(text["tags_header"])
parts.append(f" {', '.join(existing_tags)}\n")
if is_zh:
parts.append(
"请分析以上材料,输出操作 JSON 数组。"
"决定是否需要创建新的 Schema,或为已有 Schema 追加 evidence。"
"如果数据不足以得出可靠结论,输出空数组 `[]`。"
"打标签时请优先使用已有标签列表中的标签。"
)
else:
parts.append(
"Analyze the materials above and output a JSON array of operations. "
"Decide whether to create new Schemas, or add evidence to existing ones. "
"If the data is insufficient for reliable conclusions, output an empty array `[]`. "
"When tagging, prefer reusing tags from the existing tags list above."
)
parts.append(text["closing"])
return "\n".join(parts)
+25 -1
View File
@@ -243,6 +243,7 @@ class System2Writer(WritePipeline):
materials=materials,
tool_executor=tool_executor,
config=self.config,
language=payload.get("language"),
)
# 计算 timing
@@ -508,6 +509,7 @@ class System2Writer(WritePipeline):
materials=batch,
tool_executor=tool_executor,
config=self.config,
language=payload.get("language"),
)
batch_record = {
"batch_index": batch_index,
@@ -632,6 +634,7 @@ class System2Writer(WritePipeline):
materials=holistic_materials,
tool_executor=tool_executor,
config=self.config,
language=payload.get("language"),
)
if holistic_result.get("success"):
await self._mark_holistic_therapy_reviewed(
@@ -1044,11 +1047,26 @@ class System2Writer(WritePipeline):
agent_id: str,
llm_call,
request_id: str = "",
language: Optional[str] = None,
) -> Dict[str, Any]:
"""在 System2 Agent 之后运行跨域突破性归纳"""
"""在 System2 Agent 之后运行跨域突破性归纳
sweeper 的 prompt 只有英文版(要求与输入 schema 同语言)。显式 language
时在每个 prompt 末尾追加该语言的输出规则,让产出用该语言书写。
"""
if not _SWEEPER_ENABLED:
return {"skipped": True, "reason": "SWEEPER_ENABLED=false"}
if language:
from ..prompts import with_output_language
_base_call = llm_call
async def llm_call(prompt: str) -> str:
return await _base_call(
with_output_language(prompt, language, explicit=True)
)
try:
from .cross_domain_sweeper import CrossDomainSweeper
@@ -1119,6 +1137,7 @@ class System2Writer(WritePipeline):
self,
user_id: str,
agent_id: str = "default_agent",
language: Optional[str] = None,
) -> Dict[str, Any]:
"""
手动触发 cross-domain sweeper(同步等待完成)。
@@ -1145,6 +1164,7 @@ class System2Writer(WritePipeline):
agent_id=agent_id,
llm_call=self._get_llm_call(),
request_id=request_id,
language=language,
)
result["request_id"] = request_id
return result
@@ -1154,6 +1174,7 @@ class System2Writer(WritePipeline):
user_id: str,
agent_id: str = "default_agent",
max_batches: Optional[int] = None,
language: Optional[str] = None,
) -> Dict[str, Any]:
"""
手动触发 System 2 认知加工(由调用方/App 层控制触发时机)。
@@ -1186,6 +1207,8 @@ class System2Writer(WritePipeline):
"content": "",
"created_at": datetime.now().isoformat(),
"max_batches": max_batches,
# 显式 prompt 语言码(None → System 2 按 materials 做 zh/en 检测)
"language": language,
}
results = await self._run_system2_workers(payload)
@@ -1204,6 +1227,7 @@ class System2Writer(WritePipeline):
agent_id=agent_id,
llm_call=self._get_llm_call(),
request_id=request_id,
language=language,
)
agent_success = bool((results.get("system2_agent") or {}).get("success", True))
+91 -34
View File
@@ -73,6 +73,19 @@ def _write_apply_verify_enabled() -> bool:
_RECONCILE_LOOP_USER_LOCKS = weakref.WeakKeyDictionary()
class ReconcileFailure(str):
"""reconcile 失败时 `_reconcile_and_store` 返回的 error:仍是错误信息字符串,
另带稳定的失败码 `code`(LLM_UNAVAILABLE / EMPTY_RESPONSE / JSON_PARSE_FAILED /
RECONCILE_EXCEPTION),供 writer 透传给调用方。"""
code: str = "RECONCILE_FAILED"
def __new__(cls, message: str, code: str = "RECONCILE_FAILED"):
obj = super().__new__(cls, message)
obj.code = code
return obj
def _reconcile_user_lock(user_id: str) -> asyncio.Lock:
loop = asyncio.get_running_loop()
locks = _RECONCILE_LOOP_USER_LOCKS.get(loop)
@@ -635,6 +648,7 @@ class MemoryWriter(WritePipeline):
不查 VDB、不做"新写入 vs 库里已有"的对比,避免给 VDB 增加读压力。
库内存量污染由 recall 侧异步删除逐步清理。best-effort:任何异常都
回退到"不判重、全写入",绝不影响 upsert 主流程。
`recall.intention_dedup_enabled=False` 时跳过批内判重,全部写入。
"""
if not intentions:
return []
@@ -660,7 +674,11 @@ class MemoryWriter(WritePipeline):
# ============ 批内 dedup(不查 VDB,只对本批 intentions 互斥判重) ============
keep_mask: List[bool] = [True] * len(intentions)
try:
dropped = self._intention_batch_dedup(intentions, embeddings)
dropped = (
self._intention_batch_dedup(intentions, embeddings)
if getattr(self.config.recall, "intention_dedup_enabled", True)
else []
)
for i in dropped:
if 0 <= i < len(keep_mask):
keep_mask[i] = False
@@ -997,6 +1015,7 @@ class MemoryWriter(WritePipeline):
# 这是一次 LLM 调用:prompt / 产出必须能在线上追到,
# 否则 badcase 只能看到"摘要内容不对"却拿不到输入
trace=self._make_trace_sink(request, "EPISODE_SUMMARY"),
language=request.language,
)
if nid:
logger.info(
@@ -1210,6 +1229,7 @@ class MemoryWriter(WritePipeline):
self._make_trace_sink(request, "TOPIC_SUMMARY")
if request is not None else None
),
language=getattr(request, "language", None),
)
except Exception as e:
logger.warning(f"[topic-summary] failed (non-fatal): {e}")
@@ -1255,15 +1275,6 @@ class MemoryWriter(WritePipeline):
recon_tokens 含 prompt/completion/total(reconcile 链路 LLM 消耗)
"""
stored_ids: List[str] = []
# Episode:本次写入归属的内部会话 id(跨 write 把"一件事"归到一起)。
# 只数 user 侧记忆(与 rolling summary 只数 pending_user_count 一致)——
# agent 侧记忆若参与计数会把 episode 提前撑到上限、切得过碎,白付摘要的 token。
episode_id = await self._resolve_episode_id(
request, _count_user_owned(new_memories_meta)
)
session_episode_id = self._session_episode_id(request)
if session_episode_id:
episode_id = session_episode_id
# 关系图用:收集本次落库的 (node, op),供 graph 投影建边。
# 收 op 是为了拿 op.related(LLM 标注的 typed 关系)。
graph_payload: List[Tuple[Any, Any]] = []
@@ -1286,10 +1297,36 @@ class MemoryWriter(WritePipeline):
graph_store=getattr(self, "_graph_store", None),
new_memory_embeddings=precomputed_embeddings,
source=request.source,
language=request.language,
)
if not recon_result.success:
return [], recon_result.error, {"add": 0, "supersede": 0, "update": 0, "total": 0}, {"prompt": 0, "completion": 0, "total": 0}
# reconcile 失败:本次一条新记忆都不落库(不做 ADD-all 兜底),由调用方重跑。
# episode 状态也还没推进,重跑不会把同一批记忆重复计数。
error = ReconcileFailure(
recon_result.error or "reconcile failed",
code=recon_result.error_code or "RECONCILE_FAILED",
)
logger.warning(
f"TRACE_PERF [{req_id}] S1_RECONCILE_FAILED "
f"code={error.code} candidates={len(new_memory_texts)} stored=0"
)
return [], error, {"add": 0, "supersede": 0, "update": 0, "total": 0}, {
"prompt": recon_result.prompt_tokens,
"completion": recon_result.completion_tokens,
"total": recon_result.total_tokens,
}
# Episode:本次写入归属的内部会话 id(跨 write 把"一件事"归到一起)。
# 只数 user 侧记忆(与 rolling summary 只数 pending_user_count 一致)——
# agent 侧记忆若参与计数会把 episode 提前撑到上限、切得过碎,白付摘要的 token。
# 放在 reconcile 成功之后:失败的写入不能推进 episode 计数。
episode_id = await self._resolve_episode_id(
request, _count_user_owned(new_memories_meta)
)
session_episode_id = self._session_episode_id(request)
if session_episode_id:
episode_id = session_episode_id
# 分类统计
_op_counts = {}
@@ -1300,16 +1337,7 @@ class MemoryWriter(WritePipeline):
logger.info(
f"TRACE_PERF [{req_id}] S1_RECONCILE_DONE "
f"ops={len(recon_result.ops)} candidates={len(new_memory_texts)} | {_op_counts_str}"
+ (f" | DEGRADED({recon_result.degraded_reason})"
if getattr(recon_result, "degraded", False) else "")
)
if getattr(recon_result, "degraded", False):
# 走了 ADD-all 兜底:内容保住了,但没做去重/演化判定。必须可观测,
# 否则线上只会看到"记忆变多了"而不知道为什么。
logger.warning(
f"[write] reconcile degraded → ADD-all "
f"({recon_result.degraded_reason}), {len(recon_result.ops)} memories stored as-is"
)
# 逐条 op 详情由 reconciler 的 "[reconciler] ops detail" 单条 JSON list 输出,
# 这里不再逐条打印(避免重复刷屏)。
@@ -2499,12 +2527,10 @@ class MemoryWriter(WritePipeline):
response.extra["extract_tokens"] = {"prompt": 0, "completion": 0, "total": 0}
if recon_error is not None:
response.success = False
response.error_code = 502
response.error_message = f"[RECONCILE_FAILED] {recon_error}"
response.extra["agent_status"] = "failed"
response.extra["agent_error"] = recon_error
response.extra["agent_nodes"] = 0
# 与 write() 一致:reconcile 失败不落任何记忆,报失败供调用方重跑。
self._mark_reconcile_failure(response, recon_error)
response.extra["agent_mode"] = "extracted"
logger.warning(f"[write_extracted] reconcile failed: {recon_error}")
return response
response.success = True
@@ -2714,6 +2740,7 @@ class MemoryWriter(WritePipeline):
vector_store, request.user_id, request.agent_id,
exclude_memory_id=memory_id,
max_turns=_hist_turns * 2, # k 轮对话 = k*2 条 message
language=request.language,
)
_recent_history_ms = (_t_recent_history and (datetime.now() - _t_recent_history).total_seconds() * 1000) or 0.0
_recent_history_count = len([line for line in (_history_context or "").splitlines() if line.strip()])
@@ -2794,6 +2821,7 @@ class MemoryWriter(WritePipeline):
user_id=request.user_id,
agent_id=request.agent_id,
session_id=request.session_id or "default_session",
language=request.language,
)
if finalize is finalize_dynamic_episode_capsules:
finalize_kwargs["summary_concurrency"] = getattr(
@@ -2954,9 +2982,11 @@ class MemoryWriter(WritePipeline):
exclude_memory_id: str = "",
max_turns: int = 20,
max_chars_assistant: Optional[int] = None,
language: Optional[str] = None,
) -> str:
"""
获取该用户最近的 L1_RAW 对话记录,按轮次拆分后拼成 history context。
language:显式 prompt 语言码,用于渲染名字行;None 时按内容判 zh/en。
每条 L1_RAW 的 content 是多轮对话([user]: ...\n[assistant]: ...)。
raw 在写入时已按 utils/raw_trim.py 的规则裁好(custom[RAW_TRIMMED_KEY]),
@@ -3060,9 +3090,13 @@ class MemoryWriter(WritePipeline):
from .base import render_user_name_line
from ..utils.lang_detect import is_chinese
_name_line = render_user_name_line(
user_name,
is_chinese("\n".join(b for _, b, _ in recent)),
_name_line = (
render_user_name_line(user_name, language=language)
if language else
render_user_name_line(
user_name,
is_chinese("\n".join(b for _, b, _ in recent)),
)
)
if _name_line:
all_messages.append(_name_line)
@@ -3305,6 +3339,7 @@ class MemoryWriter(WritePipeline):
existing_tags=existing_tags,
history_context=history_context,
basic_profile_fields=_basic_profile_fields,
language=request.language,
)
extract_timing["extract_total_ms"] = round(
(datetime.now() - _t_extract_total).total_seconds() * 1000, 2
@@ -3514,12 +3549,11 @@ class MemoryWriter(WritePipeline):
response.extra["reconcile_ops"] = ops_counts
response.extra["reconcile_tokens"] = recon_tokens
if recon_error is not None:
# stored_ids 为空:reconcile 失败时一条新记忆都不落库,write()
# 随后回滚 L1_RAW(retry_safe=True),换个 request_id 重跑即可。
response.extra["agent_stored_ids"] = stored_ids
self._mark_agent_failure(
response,
error_code="RECONCILE_FAILED",
error_message=recon_error,
agent_nodes=len(stored_ids),
self._mark_reconcile_failure(
response, recon_error, agent_nodes=len(stored_ids),
)
logger.warning(f"[write] reconcile failed: {recon_error}")
return
@@ -3642,6 +3676,29 @@ class MemoryWriter(WritePipeline):
response.error_code = 502
response.error_message = f"[{code}] {message}"
@classmethod
def _mark_reconcile_failure(
cls,
response: WriteResponse,
recon_error: str,
*,
agent_nodes: int = 0,
) -> None:
"""reconcile 失败:agent_error_code 恒为 RECONCILE_FAILED,具体原因
(LLM_UNAVAILABLE / EMPTY_RESPONSE / JSON_PARSE_FAILED / RECONCILE_EXCEPTION)
放在 reconcile_error_code。"""
reason = getattr(recon_error, "code", "") or "RECONCILE_FAILED"
response.extra["reconcile_error_code"] = reason
cls._mark_agent_failure(
response,
error_code="RECONCILE_FAILED",
error_message=(
f"{reason}: {recon_error}"
if reason != "RECONCILE_FAILED" else str(recon_error)
),
agent_nodes=agent_nodes,
)
async def _handle_agent_failure(self, request, response, agent_result, tracer_span, extract_timing: Optional[Dict[str, Any]] = None) -> None:
"""处理 agent extract 失败的情况。"""
if agent_result is None:
+242
View File
@@ -0,0 +1,242 @@
"""Prompt registry: every multilingual LLM prompt, keyed by language code.
One module per language (``en.py``, ``zh_hans.py``, ``ja.py`` ...) defines the
same set of names (``PROMPT_NAMES``). Call sites look prompts up with
``get_prompt(name, language)`` instead of importing a language module, so a
new language only needs a new module and an entry in ``_MODULES``.
Language selection (``resolve_language``):
* explicit ``language`` (per call, else ``MemoryConfig.language`` /
``MEMORY_LANGUAGE``) wins and is validated by ``normalize_language``;
* otherwise the existing zh/en auto-detection runs unchanged
(``utils.lang_detect.is_chinese`` → ``zh-Hans`` or ``en``). Auto-detection
never picks any of the other languages.
When the language is explicit, ``with_output_language`` appends
``OUTPUT_LANGUAGE_DIRECTIVE`` so memories are written in that language even
if the conversation uses another one. Auto-detected requests keep the
"same language as the input" rule and their prompts are byte-identical to
earlier releases.
"""
from __future__ import annotations
import importlib
from types import ModuleType
from typing import Any, Dict, Optional, Tuple
# Canonical codes, in documentation order.
SUPPORTED_LANGUAGES: Tuple[str, ...] = (
"en",
"zh-Hans",
"zh-Hant",
"ja",
"es",
"de",
"pt",
"fr",
"ko",
"ru",
"id",
)
# Accepted aliases (matched case-insensitively, "_" treated as "-").
LANGUAGE_ALIASES: Dict[str, str] = {
"zh": "zh-Hans",
"zh-cn": "zh-Hans",
"zh-sg": "zh-Hans",
"zh-hans-cn": "zh-Hans",
"zh-tw": "zh-Hant",
"zh-hk": "zh-Hant",
"zh-mo": "zh-Hant",
"zh-hant-tw": "zh-Hant",
"zh-hant-hk": "zh-Hant",
"pt-br": "pt",
"pt-pt": "pt",
"en-us": "en",
"en-gb": "en",
}
_MODULES: Dict[str, str] = {
"en": "en",
"zh-Hans": "zh_hans",
"zh-Hant": "zh_hant",
"ja": "ja",
"es": "es",
"de": "de",
"pt": "pt",
"fr": "fr",
"ko": "ko",
"ru": "ru",
"id": "id",
}
# Every language module must define all of these.
PROMPT_NAMES: Tuple[str, ...] = (
# chat extraction
"EXTRACT_SYSTEM_PROMPT",
"STRUCTURE_METADATA_PROMPT",
"EXTRACT_FEW_SHOT",
"EXTRACT_PROMPT",
"EXTRACT_PROMPT_NO_INTENTION",
"NO_INTENTION_SECTION_PATTERN",
"NO_INTENTION_MEMORY_SECTION",
"NO_INTENTION_REPLACEMENTS",
"INTENTION_DISABLED_NOTE",
"USER_NAME_LINE",
# chat reconcile
"SEARCH_QUERY_PROMPT",
"RECONCILE_PROMPT",
"RECONCILE_STRUCTURE_METADATA_PROMPT",
"RECONCILE_FEW_SHOT",
# working / coding memory
"CODING_EXTRACT_PROMPT",
"CODING_EXTRACT_PROMPT_CODING",
"CODING_CONVERSATION_BLOCK",
"CODING_EMPTY_BLOCK",
"CODING_RECONCILE_PROMPT",
"CODING_FACT_RECONCILE_PROMPT",
"CODING_CHAIN_SUMMARY_PROMPT",
# summaries
"EPISODE_SUMMARY_PROMPT",
"TOPIC_SUMMARY_PROMPT",
# System 2 evolve
"SYSTEM2_AGENT_PROMPT",
"SYSTEM2_OUTPUT_INSTRUCTIONS",
"SYSTEM2_SCHEMA_EVIDENCE_GATE",
"SYSTEM2_SINGLE_CALL_REPLACEMENTS",
"SYSTEM2_MATERIALS",
# explicit-language output rule
"OUTPUT_LANGUAGE_DIRECTIVE",
"LANGUAGE_NAME",
)
# Prompts rendered with str.format(); the value lists the expected fields.
# The rest are appended verbatim (their braces are literal JSON).
FORMAT_FIELDS: Dict[str, Tuple[str, ...]] = {
"EXTRACT_PROMPT": ("content", "current_date", "last_messages", "memory_at"),
"EXTRACT_PROMPT_NO_INTENTION": ("content", "current_date", "last_messages", "memory_at"),
"USER_NAME_LINE": ("name",),
"SEARCH_QUERY_PROMPT": ("new_memories",),
"RECONCILE_PROMPT": ("current_date", "existing_memories", "few_shot_section", "new_memories"),
"CODING_EXTRACT_PROMPT": ("branch_present", "conversation_block", "workspace_present"),
"CODING_EXTRACT_PROMPT_CODING": ("branch_present", "conversation_block", "workspace_present"),
"CODING_CONVERSATION_BLOCK": ("history", "new"),
"CODING_RECONCILE_PROMPT": ("candidates_block", "drafts_block"),
"CODING_FACT_RECONCILE_PROMPT": ("candidates_block", "drafts_block"),
"CODING_CHAIN_SUMMARY_PROMPT": ("chain_block",),
"EPISODE_SUMMARY_PROMPT": ("memories",),
"TOPIC_SUMMARY_PROMPT": ("memories", "topic_name"),
}
# Fields of each SYSTEM2_MATERIALS entry (all other entries have none).
SYSTEM2_MATERIALS_FIELDS: Dict[str, Tuple[str, ...]] = {
"title": (),
"stats": ("clustered", "clusters", "evidenced", "fresh", "graph_total", "recalled", "total", "unclustered"),
"graph_status_empty": (),
"clusters_header": (),
"label_holistic": ("n",),
"label_source_context": ("n",),
"label_topic": ("n",),
"raw_label": (),
"holistic_note": ("max_schemas",),
"companion_header": (),
"unclustered_header": (),
"unclustered_hint": (),
"forward_header": ("count",),
"graph_empty": (),
"graph_no_recall": ("graph_total",),
"reverse_header": (),
"tags_header": (),
"closing": (),
}
_loaded: Dict[str, ModuleType] = {}
def _supported_text() -> str:
aliases = ", ".join(f"{k}->{v}" for k, v in LANGUAGE_ALIASES.items())
return f"supported: {', '.join(SUPPORTED_LANGUAGES)} (aliases: {aliases})"
def normalize_language(language: Optional[str]) -> Optional[str]:
"""Return the canonical code for ``language``; None/"" stays None.
Raises ValueError for an unknown code, listing the supported codes.
"""
if language is None:
return None
if not isinstance(language, str):
raise ValueError(
f"language must be a string, got {type(language).__name__}; "
+ _supported_text()
)
raw = language.strip()
if not raw:
return None
key = raw.replace("_", "-").lower()
for code in SUPPORTED_LANGUAGES:
if code.lower() == key:
return code
if key in LANGUAGE_ALIASES:
return LANGUAGE_ALIASES[key]
raise ValueError(f"unsupported language {language!r}; " + _supported_text())
def detect_language_code(content: str = "", messages=None) -> str:
"""The legacy automatic choice: ``zh-Hans`` if Chinese, else ``en``."""
from ..utils.lang_detect import is_chinese
return "zh-Hans" if is_chinese(content, messages=messages) else "en"
def resolve_language(
language: Optional[str] = None,
content: str = "",
messages=None,
) -> str:
"""Explicit language if given (validated), else zh/en auto-detection."""
code = normalize_language(language)
if code is not None:
return code
return detect_language_code(content, messages=messages)
def _module(language: str) -> ModuleType:
code = normalize_language(language) or "en"
mod = _loaded.get(code)
if mod is None:
mod = importlib.import_module(f"{__name__}.{_MODULES[code]}")
_loaded[code] = mod
return mod
def get_prompt(name: str, language: Optional[str] = "en") -> Any:
"""Look up prompt ``name`` for ``language`` (canonical code or alias)."""
if name not in PROMPT_NAMES:
raise KeyError(f"unknown prompt {name!r}")
return getattr(_module(language or "en"), name)
def with_output_language(
prompt: str, language: Optional[str], explicit: bool,
) -> str:
"""Append the output-language rule when the language was set explicitly."""
if not explicit or not language:
return prompt
return prompt + get_prompt("OUTPUT_LANGUAGE_DIRECTIVE", language)
__all__ = [
"SUPPORTED_LANGUAGES",
"LANGUAGE_ALIASES",
"PROMPT_NAMES",
"FORMAT_FIELDS",
"SYSTEM2_MATERIALS_FIELDS",
"normalize_language",
"detect_language_code",
"resolve_language",
"get_prompt",
"with_output_language",
]
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+24
View File
@@ -21,6 +21,11 @@ API Endpoints:
GET /healthz — Health check (static, process alive)
GET /health — Liveness probe (pings internal event loop; 503 if loop blocked)
GET /info — Server info
/add and /add_extracted accept an optional "language" (en, zh-Hans, zh-Hant, ja,
es, de, pt, fr, ko, ru, id, or an alias such as zh-CN / zh-TW / pt-BR). An
unknown code returns 400. /search has no language option: the search-time
judge / rewrite prompts are English-only and keep the query's own language.
"""
import asyncio
@@ -100,6 +105,20 @@ def _get_client():
return _client
def _language_arg(handler: BaseHTTPRequestHandler, body: Dict, kwargs: Dict) -> bool:
"""Validate optional ``language`` and add it to kwargs; False after a 400 reply."""
if body.get("language") in (None, ""):
return True
from .prompts import normalize_language
try:
kwargs["language"] = normalize_language(body["language"])
except ValueError as e:
_json_response(handler, 400, {"error": str(e)})
return False
return True
def _json_response(handler: BaseHTTPRequestHandler, status: int, data: Any):
"""Send a JSON response."""
body = json.dumps(data, ensure_ascii=False, default=str).encode("utf-8")
@@ -436,6 +455,9 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
kwargs["memory_at"] = datetime.fromisoformat(body["memory_at"])
except Exception:
pass
# language: explicit prompt / memory language (see HyMemoryClient.add)
if not _language_arg(self, body, kwargs):
return
result = client.add(data, **kwargs)
_json_response(self, 200, result)
@@ -462,6 +484,8 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
kwargs["memory_at"] = datetime.fromisoformat(body["memory_at"])
except Exception:
pass
if not _language_arg(self, body, kwargs):
return
result = client.add_extracted(contents, **kwargs)
_json_response(self, 200, result)
+9
View File
@@ -137,6 +137,15 @@ memhub profile
`user-profile` / `memhub profile` 调用 SDK `get_user_profile()`,返回组装后的
用户长期画像,不通过普通 Search 拼装。
读接口都回显 `user_id`(实际读的用户,未指定时为 `local`):
`GET /api/v2/memory-library`、`/memory-library/layers`、`/user-profile`、
`/user-profile/versions`、`/home-summary`、`/memories/graph`、`/raw`,
`POST /api/v2/context/search`、`/api/v1/recall`、`/api/v2/memories/ask`;
`/memories/ask/stream` 写在第一个 `recall` 事件上。
`DELETE /api/v2/memories/{id}` 只在删掉调用者自己的记忆时回 204;另一个用户的
记忆和不存在的 ID 一律回 404 `memory_not_found`(不暴露是否存在),调用方按
「已不存在」处理。
返回值:
```python
+4 -4
View File
@@ -18,13 +18,13 @@ If you run into a problem, please open an Issue with the version, platform,repro
## 发行与平台合同
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.10`。MemHub 源码版本为 `1.0.5`
(build 36),GitHub 仓库为
源码当前固定使用公开 PyPI 正式版 `hy-memory==2.0.14`。MemHub 源码版本为 `1.0.6`
(build 41),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
`v1.0.5` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64 App 和 Runtime。
`v1.0.6` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64 App 和 Runtime。
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
@@ -118,7 +118,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
## Hy Memory
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.10`。SDK 配置、权限边界及 search
SDK 依赖固定为公开 PyPI 正式版 `hy-memory==2.0.14`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+1 -1
View File
@@ -2352,7 +2352,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.10",
"versionInfo": "2.0.14",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+1 -1
View File
@@ -9,7 +9,7 @@ Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.10
## hy-memory 2.0.14
License: Apache-2.0
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/cli",
"version": "1.0.5",
"version": "1.0.6",
"description": "MemHub bootstrap CLI for installing and managing the local-first shared memory runtime",
"license": "Apache-2.0",
"repository": {
+29 -13
View File
@@ -1938,7 +1938,7 @@ async function runCommand(
return reject(
descriptor,
asJson,
"--user-id 和 --root 只能用于接入单个 hermes 来源(Hermes profile),用户 ID 只能是 1–64 位字母、数字、点、下划线或连字符。",
"--user-id 和 --root 只能用于接入单个 hermes 来源(Hermes profile),用户 ID 只能是 1–64 位字母、数字、点、下划线或连字符。用户 ID 不必等于 Hermes profile 名,例如 --user-id wife-work --root ~/.hermes/profiles/wife:work。",
);
}
if (!parsed) return usage();
@@ -2328,29 +2328,41 @@ async function runCommand(
...userIdParameters(userId),
}),
);
// A delete answered 404 `memory_not_found` is already gone (deleted
// before, or never this user's): what was asked for holds, so it exits 0
// as committed and says so, instead of failing a retry or a double run.
const alreadyGone =
parsed.kind === "delete" &&
outcome.state === "rejected" &&
outcome.code === "memory_not_found";
const state = alreadyGone ? "committed" : outcome.state;
const document = {
schema_version: 2,
product: PRODUCT,
command: `memories ${parsed.kind}`,
state: outcome.state,
state,
operation: parsed.kind,
memory_id: parsed.memoryId,
...(outcome.state === "committed"
? parsed.kind === "update"
? { result: outcome.data }
: {}
: { code: outcome.code }),
...(alreadyGone
? { already_gone: true }
: outcome.state === "committed"
? parsed.kind === "update"
? { result: outcome.data }
: {}
: { code: outcome.code }),
};
emit(
asJson
? `${JSON.stringify(document)}\n`
: outcome.state === "committed"
? `已${parsed.kind === "update" ? "更新" : "删除"}记忆 ${parsed.memoryId}\n`
: outcome.code === "memory_id_ambiguous"
? "这个 ID 同时存在于多个场景;请加 --kind(以及 --variant)指明是哪一条。\n"
: `操作失败:${outcome.code}\n`,
: alreadyGone
? `未找到记忆 ${parsed.memoryId}(可能已删除),无需再删。\n`
: outcome.state === "committed"
? `已${parsed.kind === "update" ? "更新" : "删除"}记忆 ${parsed.memoryId}\n`
: outcome.code === "memory_id_ambiguous"
? "这个 ID 同时存在于多个场景;请加 --kind(以及 --variant)指明是哪一条。\n"
: `操作失败:${outcome.code}\n`,
);
return outcome.state === "committed" ? EXIT_OK : EXIT_FAILED;
return state === "committed" ? EXIT_OK : EXIT_FAILED;
}
if (
named === "settings" &&
@@ -4371,6 +4383,10 @@ function renderWriteReceiptText(receipt: MemoryWriteReceipt): string {
...(receipt.memory_ids && receipt.memory_ids.length > 0
? [`memory_ids: ${receipt.memory_ids.join(",")}`]
: []),
...(receipt.error_code ? [`error_code: ${receipt.error_code}`] : []),
...(receipt.next_retry_at
? [`next_retry_at: ${receipt.next_retry_at}`]
: []),
"",
].join("\n");
}
+6 -5
View File
@@ -258,7 +258,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "sources",
summary:
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。Hermes 的每个 profile 是一个独立用户:用 --user-id 和 --root 接入,授权并导入历史(会自动排除 ~/.hermes/profiles/**,各用户互不读取)。",
"列出来源,直接接入来源,或按 request id 查接入回执。connect 不加 --import 只授权(处理保持冻结:不导入历史、不扫描新会话);加 --import 同时导入历史并开启持续同步,返回 request id。目录来自环境变量覆盖的来源必须加 --confirm-overrides。Hermes 的每个 profile 是一个独立用户:用 --user-id 和 --root 接入,授权并导入历史(任何层级的 profiles/ 目录都会被排除,各用户互不读取)。--root 不能是 profiles 目录本身,也不能与另一个用户的 Hermes 目录相同或互相包含(~/.hermes 与 ~/.hermes/profiles/<name> 除外)。",
usage:
"memhub sources list [--json]\n memhub sources connect <id> [<id> ...] [--import] [--confirm-overrides] [--request-id <uuid>] [--json]\n memhub sources connect hermes --user-id <id> --root <path> [--import] [--request-id <uuid>] [--json]\n memhub sources connect --all-detected [--import] [--confirm-overrides] [--json]\n memhub sources receipt <request-id> [--json]",
options: [
@@ -270,7 +270,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
flag: "--user-id <id>",
summary:
"只用于 hermes:这条接入属于哪个用户(默认 local)。和 --root 一起接入一个 Hermes profile,例如 --user-id wife --root ~/.hermes/profiles/wife;其他来源只能是 local。",
"只用于 hermes:这条接入属于哪个用户(默认 local)。和 --root 一起接入一个 Hermes profile,例如 --user-id wife --root ~/.hermes/profiles/wife;用户 ID 不必等于 profile 名,例如 --user-id wife-work --root ~/.hermes/profiles/wife:work。其他来源只能是 local。",
},
{
flag: "--root <path>",
@@ -397,7 +397,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "memories",
summary:
"列出、搜索、查看、添加、修改或物理删除记忆(对话、工作、编码三个场景)。add 和 App 的「添加记忆」是同一个入口;delete 不可逆,需 --yes。",
"列出、搜索、查看、添加、修改或物理删除记忆(对话、工作、编码三个场景)。add 和 App 的「添加记忆」是同一个入口;delete 不可逆,需 --yes;要删的记忆已不存在(或不属于该用户)时提示未找到,按已删除处理并退出 0。",
usage:
"memhub memories list [--scene <all|chat|working|coding>] [--source <id>] [--query <text>] [--status <state>] [--offset <n>] [--limit <n>] [--user-id <id>] [--json]\n memhub memories show <memory-id> [--kind <kind>] [--variant <variant>] [--user-id <id>] [--json]\n memhub memories add <text> [--request-id <uuid>] [--user-id <id>] [--json]\n memhub memories add --content-file <path> [--request-id <uuid>] [--user-id <id>] [--json]\n memhub memories add-status <request-id> [--user-id <id>] [--json]\n memhub memories update <memory-id> --content-file <path> [--kind <kind>] [--variant <variant>] [--task <text>] [--user-id <id>] [--json]\n memhub memories delete <memory-id> [--kind <kind>] [--variant <variant>] [--yes] [--user-id <id>] [--json]",
options: [
@@ -461,11 +461,12 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
name: "settings",
summary:
"查看或切换记忆模式(lite / pro / ultra)。lite 不调用模型,只在原始对话里检索;pro 由模型提取记忆(System 1);ultra 在 pro 之上每晚自动进化(System 2,使用知识图谱)。任何模式都保留全部原始对话。切换不迁移数据:已处理的不会重做,排队中的和新的按新模式处理;需 --yes,不加时只显示计划。环境变量 ONE_MEMORY_GRAPH_STORE 会强制 ultra。settings features 查看或开关 SDK 的可选检索与提取功能(全局设置,默认全部保持 SDK 默认值):graph_rerank 图谱重排(仅 ultra)、same_series_timeline 同系列时间线(需 graph_rerank)、episode_recall 片段召回(pro / ultra,需先开 session_episodes 写入片段)、session_episodes 片段写入、episode_summary 片段摘要(每个片段一次模型调用)、intention 意图提取(默认开)、few_shot 提取示例;set 需 --yes,不加时只显示计划;reset 恢复默认值。settings language 查看或设置界面语言(App 切换语言时会自动设置):简体中文(zh-Hans)和 system 不指定 SDK 语言,SDK 照旧自动识别中文或英文;其余语言(en、zh-Hant、ja、es、de、pt、fr、ko、ru、id)会让提取等 SDK 提示使用该语言,立即生效。",
"查看或切换记忆模式(lite / pro / ultra)。lite 不调用模型,只在原始对话里检索;pro 由模型提取记忆(System 1);ultra 在 pro 之上每晚自动进化(System 2,使用知识图谱)。任何模式都保留全部原始对话。切换不迁移数据:已处理的不会重做,排队中的和新的按新模式处理;需 --yes,不加时只显示计划。环境变量 ONE_MEMORY_GRAPH_STORE 会强制 ultra。settings features 查看或开关 SDK 的可选检索与提取功能(全局设置,默认全部保持 SDK 默认值):graph_rerank 图谱重排(仅 ultra)、same_series_timeline 同系列时间线(需 graph_rerank)、episode_recall 片段召回(pro / ultra,需先开 session_episodes 写入片段)、session_episodes 片段写入、episode_summary 片段摘要(每个片段一次模型调用)、intention 意图提取(默认开)、intention_dedup 意图判重(默认开;关闭则检索与写入都不判重、不删除意图)、few_shot 提取示例,以及数值 intention_dedup_threshold 意图判重阈值(默认 0.95,可设 0.8–1.0);set 需 --yes,不加时只显示计划;reset 恢复默认值。settings language 查看或设置界面语言(App 切换语言时会自动设置):简体中文(zh-Hans)和 system 不指定 SDK 语言,SDK 照旧自动识别中文或英文;其余语言(en、zh-Hant、ja、es、de、pt、fr、ko、ru、id)会让提取等 SDK 提示使用该语言,立即生效。",
usage:
"memhub settings mode [lite|pro|ultra] [--yes] [--json]\n" +
" memhub settings features [--json]\n" +
" memhub settings features set <name>=<true|false> [...] [--yes] [--json]\n" +
" memhub settings features set intention_dedup_threshold=<0.8-1.0> [--yes] [--json]\n" +
" memhub settings features reset [<name> ...] [--json]\n" +
" memhub settings language [<code>|system] [--json]",
options: [
@@ -476,7 +477,7 @@ export const COMMANDS: readonly CommandDescriptor[] = [
{
flag: "features set <name>=<true|false>",
summary:
"开关一个或多个 SDK 功能;功能在当前模式下不生效时会注明原因(例如 graph_rerank 需要 ultra)。",
"开关一个或多个 SDK 功能;功能在当前模式下不生效时会注明原因(例如 graph_rerank 需要 ultra)。intention_dedup_threshold 取 0.8–1.0 的数值。",
},
{
flag: "features reset [<name> ...]",
+8
View File
@@ -17,11 +17,14 @@ const READ_ONLY_PATHS = [
"/api/v1/model-profiles",
"/api/v1/source-grants",
"/api/v1/sources",
"/api/v2/home-projection",
"/api/v2/home-summary",
"/api/v2/raw",
"/api/v2/memory-library",
"/api/v2/user-profile",
"/api/v2/user-profile/versions",
"/api/v1/settings/memory-mode",
"/api/v1/settings/memory-mode/plan",
"/api/v1/settings/memory-features",
"/api/v1/settings/ui-language",
"/api/v2/evolve",
@@ -268,6 +271,11 @@ export interface MemoryWriteReceipt {
detail?: string;
raw_ids?: string[];
memory_ids?: string[];
/** Stable code of the last failure while the write still waits; null when
* none. Optional so older Runtimes stay readable. */
error_code?: string | null;
/** When Core will try this write again (ISO); null when not scheduled. */
next_retry_at?: string | null;
}
export interface OpenAIMessage {
+175
View File
@@ -1,4 +1,5 @@
import type {
HomeProjectionV1,
ImportIssue,
ImportJob,
LiveSyncHealth,
@@ -26,6 +27,9 @@ export const MEMORY_STATUSES = [
const ID_MAX = 128;
const LABEL_MAX = 256;
const LIST_MAX = 500;
const USER_ID = /^[A-Za-z0-9._-]{1,64}$/;
const ISO_TIME =
/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})$/;
const LIVE_SYNC_STATES = [
"idle",
"running",
@@ -406,6 +410,153 @@ export function validateHomeSummary(value: unknown): SdkHomeSummary | null {
return summary as unknown as SdkHomeSummary;
}
/** `GET /api/v2/home-projection`: the mode-aware home content plane. */
export function validateHomeProjection(
value: unknown,
): HomeProjectionV1 | null {
const projection = exactRecord(value, [
"projection_version",
"user_id",
"effective_mode",
"capabilities",
"visible_count",
"display_state",
"latest",
"observed_at",
]);
if (!projection) return null;
const mode = projection.effective_mode;
const capabilities = exactRecord(projection.capabilities, [
"content_plane",
"generation_required",
"graph_enabled",
"evolve_available",
]);
if (
projection.projection_version !== 1 ||
typeof projection.user_id !== "string" ||
!USER_ID.test(projection.user_id) ||
!oneOf(mode, ["lite", "pro", "ultra"] as const) ||
!capabilities ||
capabilities.content_plane !== (mode === "lite" ? "raw" : "memory") ||
capabilities.generation_required !== (mode !== "lite") ||
capabilities.graph_enabled !== (mode === "ultra") ||
capabilities.evolve_available !== (mode === "ultra") ||
count(projection.visible_count) === null ||
!oneOf(projection.display_state, [
"empty",
"excerpt",
"no_user_excerpt",
"no_displayable_memory",
] as const) ||
!isoTime(projection.observed_at) ||
!validProjectionLatest(projection.latest, mode)
) {
return null;
}
const visible = projection.visible_count as number;
const latest = projection.latest as Record<string, unknown> | null;
if (
(visible === 0 &&
(latest !== null || projection.display_state !== "empty")) ||
(visible > 0 && projection.display_state === "empty") ||
(latest === null &&
["excerpt", "no_user_excerpt"].includes(
projection.display_state as string,
)) ||
(latest?.type === "raw" &&
projection.display_state !== latest.display_state) ||
(latest?.type === "memory" && projection.display_state !== "excerpt")
) {
return null;
}
return projection as unknown as HomeProjectionV1;
}
function validProjectionLatest(
value: unknown,
mode: "lite" | "pro" | "ultra",
): boolean {
if (value === null) return true;
const base = [
"type",
"time",
"time_basis",
"excerpt",
"truncated",
"source_id",
];
const candidate = record(value);
if (!candidate) return false;
const keys =
candidate.type === "raw"
? [...base, "raw_id", "session_id", "display_state"]
: [...base, "memory_id", "scene", "kind", "variant"];
const latest = exactRecord(value, keys);
if (
!latest ||
!(latest.time === null || isoTime(latest.time)) ||
![null, "conversation", "import", "write", "update"].includes(
latest.time_basis as null | string,
) ||
text(latest.excerpt, 200) === null ||
typeof latest.truncated !== "boolean" ||
text(latest.source_id, ID_MAX) === null
) {
return false;
}
if (latest.type === "raw") {
return (
mode === "lite" &&
nonemptyText(latest.raw_id, ID_MAX) &&
nonemptyText(latest.session_id, LABEL_MAX) &&
oneOf(latest.display_state, ["excerpt", "no_user_excerpt"] as const) &&
(latest.display_state !== "no_user_excerpt" ||
(latest.excerpt === "" && latest.truncated === false))
);
}
return (
latest.type === "memory" &&
mode !== "lite" &&
nonemptyText(latest.memory_id, ID_MAX) &&
oneOf(latest.scene, ["chat", "working", "coding"] as const) &&
oneOf(latest.kind, ["chat", "experience"] as const) &&
[null, "working", "coding"].includes(latest.variant as null | string) &&
((latest.kind === "chat" &&
latest.scene === "chat" &&
latest.variant === null) ||
(latest.kind === "experience" &&
latest.scene === latest.variant &&
oneOf(latest.scene, ["working", "coding"] as const)))
);
}
/** The bounded `/api/v2/raw?limit=1` envelope used only for its real total. */
export function validateRawTotal(value: unknown): number | null {
const page = exactRecord(value, [
"user_id",
"items",
"total",
"offset",
"limit",
"time_filtered",
]);
if (
!page ||
typeof page.user_id !== "string" ||
!USER_ID.test(page.user_id) ||
!Array.isArray(page.items) ||
page.items.length > 1 ||
count(page.total) === null ||
page.offset !== 0 ||
page.limit !== 1 ||
page.time_filtered !== false
) {
return null;
}
return page.total as number;
}
function identifiers(value: unknown): string[] | null {
return list(value ?? [], (item) => text(item, ID_MAX));
}
@@ -430,6 +581,30 @@ function record(value: unknown): Record<string, unknown> | null {
: null;
}
function exactRecord(
value: unknown,
keys: readonly string[],
): Record<string, unknown> | null {
const item = record(value);
return item &&
Object.keys(item).length === keys.length &&
keys.every((key) => Object.hasOwn(item, key))
? item
: null;
}
function isoTime(value: unknown): value is string {
return (
typeof value === "string" &&
ISO_TIME.test(value) &&
Number.isFinite(Date.parse(value))
);
}
function nonemptyText(value: unknown, max: number): value is string {
return typeof value === "string" && value.length > 0 && value.length <= max;
}
function text(value: unknown, max: number): string | null {
return typeof value === "string" && value.length <= max ? value : null;
}
@@ -250,6 +250,20 @@ describe("models activate with an embedding change", () => {
code: "embedding_switch_required",
});
});
it("does not treat a domain 404 as an old-Core route miss", async () => {
transport.post.mockResolvedValue({
status: 404,
data: { detail: "embedding_switch_not_found" },
});
const result = await capturedRun(["models", "activate", PROFILE, "--json"]);
expect(result.code).toBe(1);
expect(
transport.post.mock.calls.some(([url]) =>
String(url).endsWith("/activate"),
),
).toBe(false);
});
});
describe("models switch and spaces", () => {
+6 -1
View File
@@ -9,6 +9,7 @@
* makes it live. The CLI shows the plan, asks, starts the job and follows it.
*/
import { isBareNotFound } from "@one-memory/operations";
import type { CoreReader } from "./core-client.js";
import type {
CoreDeleteOperations,
@@ -387,7 +388,11 @@ async function requestPlan(
}
if (outcome.state === "rejected") {
// A Core before the registry has no plan route: activation as before.
if (outcome.http_status === 404 && outcome.code === "http_404") {
if (
outcome.http_status === 404 &&
outcome.code === "http_404" &&
isBareNotFound(outcome.body)
) {
return { state: "unsupported" };
}
const plan = parsePlan(record(outcome.body)?.plan);
+13
View File
@@ -348,12 +348,25 @@ export function parseApiKey(bytes: Buffer | null): string | undefined {
return value;
}
/** What to do about a refusal whose code alone does not say. */
const FLOW_CODE_HINTS: Record<string, string> = {
hermes_root_is_profiles_container:
"--root 指向的是 profiles 目录本身;请指向其中一个 profile,例如 ~/.hermes/profiles/wife。",
hermes_root_overlaps_other_user:
"--root 与另一个用户已接入的 Hermes 目录相同或互相包含,会读到对方的记录;请换一个目录,或先撤销对方的接入。~/.hermes 与 ~/.hermes/profiles/<name> 可以分属两个用户。",
};
export function renderFlowText(document: FlowDocument): string {
const lines = [
`操作状态:${document.state}`,
`操作 ID:${document.operation_id ?? "无"}`,
"code" in document ? `代码:${document.code}` : "已返回有界回执。",
];
const hint =
"code" in document && typeof document.code === "string"
? FLOW_CODE_HINTS[document.code]
: undefined;
if (hint) lines.push(hint);
// A partial success or an all-rejected batch must name each source that was
// not accepted; a bare "partial" leaves the user with nothing to act on.
if (
+52
View File
@@ -233,6 +233,31 @@ describe("memories add-status", () => {
});
});
it("passes the retry code and time of an addition still waiting", async () => {
transport.get.mockResolvedValue({
status: 200,
data: {
...addReceipt("pending"),
error_code: "model_auth_failed",
next_retry_at: "2026-11-02T10:00:00+00:00",
},
});
const json = await capturedRun([
"memories",
"add-status",
REQUEST,
"--json",
]);
expect(JSON.parse(json.stdout).receipt).toMatchObject({
state: "pending",
error_code: "model_auth_failed",
next_retry_at: "2026-11-02T10:00:00+00:00",
});
const text = await capturedRun(["memories", "add-status", REQUEST]);
expect(text.stdout).toContain("error_code: model_auth_failed");
expect(text.stdout).toContain("next_retry_at: 2026-11-02T10:00:00+00:00");
});
it("reports an unknown request id as not_found, exit 1", async () => {
transport.get.mockResolvedValue({
status: 404,
@@ -366,6 +391,33 @@ describe("memories update / delete pass the namespace", () => {
);
});
it("reports a delete answered 404 as already gone and exits 0", async () => {
transport.delete.mockResolvedValue({
status: 404,
data: { detail: "memory_not_found" },
});
const json = await capturedRun([
"memories",
"delete",
"gone-1",
"--yes",
"--json",
]);
expect(json.code).toBe(0);
expect(JSON.parse(json.stdout)).toEqual({
schema_version: 2,
product: expect.any(String),
command: "memories delete",
state: "committed",
operation: "delete",
memory_id: "gone-1",
already_gone: true,
});
const text = await capturedRun(["memories", "delete", "gone-1", "--yes"]);
expect(text.code).toBe(0);
expect(text.stdout).toContain("未找到记忆 gone-1");
});
it("explains an ambiguous id instead of a bare code", async () => {
transport.delete.mockResolvedValue({
status: 409,
+173 -6
View File
@@ -36,15 +36,17 @@ const DEFAULTS: Record<string, boolean> = {
session_episodes: false,
episode_summary: false,
intention: true,
intention_dedup: true,
few_shot: false,
};
const THRESHOLD_DEFAULT = 0.95;
function featuresState(
configured: Record<string, boolean> = {},
configured: Record<string, boolean | number> = {},
mode = "pro",
extra: Record<string, unknown> = {},
) {
const nonDefault: Record<string, boolean> = {};
const nonDefault: Record<string, boolean | number> = {};
const features = Object.entries(DEFAULTS).map(([name, fallback]) => {
const enabled = configured[name] ?? fallback;
if (enabled !== fallback) nonDefault[name] = enabled;
@@ -70,7 +72,34 @@ function featuresState(
description: name,
};
});
return { mode, features, non_default: nonDefault, ...extra };
const threshold =
(configured.intention_dedup_threshold as number | undefined) ??
THRESHOLD_DEFAULT;
if (threshold !== THRESHOLD_DEFAULT)
nonDefault.intention_dedup_threshold = threshold;
const dedupOn =
(configured.intention_dedup as boolean | undefined) ??
DEFAULTS.intention_dedup;
const parameters = [
{
name: "intention_dedup_threshold",
value: threshold,
configured:
"intention_dedup_threshold" in configured
? configured.intention_dedup_threshold
: null,
default: THRESHOLD_DEFAULT,
minimum: 0.8,
maximum: 1,
active: dedupOn,
reason: dedupOn ? null : "requires_intention_dedup",
requires_feature: "intention_dedup",
sdk_field: "recall.intention_dedup_threshold",
env: "MEMORY_INTENTION_DEDUP_THRESHOLD",
description: "threshold",
},
];
return { mode, features, parameters, non_default: nonDefault, ...extra };
}
beforeEach(() => {
@@ -110,11 +139,16 @@ describe("memhub settings features", () => {
expect(result.code).toBe(0);
const json = JSON.parse(result.stdout);
expect(json.state).toBe("ok");
expect(json.features.features).toHaveLength(7);
expect(json.features.features).toHaveLength(8);
expect(json.features.parameters[0].name).toBe("intention_dedup_threshold");
expect(transport.put).not.toHaveBeenCalled();
const text = (await capturedRun(["settings", "features"])).stdout;
expect(text).toContain("graph_rerank:关(默认)");
expect(text).toContain("intention:开(默认)");
expect(text).toContain("intention_dedup:开(默认)");
expect(text).toContain(
"intention_dedup_threshold:0.95(默认,可设 0.8–1)",
);
expect(text).toContain("先开启 session_episodes");
});
@@ -160,6 +194,130 @@ describe("memhub settings features", () => {
);
});
it("turns intention dedup off and sets its threshold, round trip", async () => {
transport.put.mockResolvedValueOnce({
status: 200,
data: featuresState({ intention_dedup: false }, "pro", {
changed: ["intention_dedup"],
}),
});
const off = await capturedRun([
"settings",
"features",
"set",
"intention_dedup=false",
"--yes",
"--json",
]);
expect(off.code).toBe(0);
expect(transport.put).toHaveBeenLastCalledWith(
"/api/v1/settings/memory-features",
{ intention_dedup: false },
undefined,
);
const offJson = JSON.parse(off.stdout);
expect(offJson.state).toBe("committed");
expect(offJson.changed).toEqual(["intention_dedup"]);
expect(offJson.features.parameters[0].reason).toBe(
"requires_intention_dedup",
);
transport.get.mockResolvedValue({
status: 200,
data: featuresState({ intention_dedup: false }),
});
transport.put.mockResolvedValueOnce({
status: 200,
data: featuresState(
{ intention_dedup: true, intention_dedup_threshold: 0.9 },
"pro",
{ changed: ["intention_dedup", "intention_dedup_threshold"] },
),
});
const on = await capturedRun([
"settings",
"features",
"set",
"intention_dedup=true",
"intention_dedup_threshold=0.9",
"--yes",
]);
expect(on.code).toBe(0);
expect(transport.put).toHaveBeenLastCalledWith(
"/api/v1/settings/memory-features",
{ intention_dedup: true, intention_dedup_threshold: 0.9 },
undefined,
);
expect(on.stdout).toContain(
"已更新:intention_dedup、intention_dedup_threshold",
);
expect(on.stdout).toContain("intention_dedup_threshold:0.9(已设置");
transport.get.mockResolvedValue({
status: 200,
data: featuresState({ intention_dedup_threshold: 0.9 }),
});
transport.put.mockResolvedValueOnce({
status: 200,
data: featuresState({}, "pro", {
changed: ["intention_dedup_threshold"],
}),
});
const reset = await capturedRun([
"settings",
"features",
"reset",
"intention_dedup_threshold",
]);
expect(reset.code).toBe(0);
expect(transport.put).toHaveBeenLastCalledWith(
"/api/v1/settings/memory-features",
{ intention_dedup_threshold: null },
undefined,
);
});
it("previews a threshold without --yes and refuses one out of range", async () => {
const preview = await capturedRun([
"settings",
"features",
"set",
"intention_dedup_threshold=0.85",
]);
expect(preview.code).toBe(0);
expect(preview.stdout).toContain("- intention_dedup_threshold = 0.85");
for (const value of ["0.5", "0.79", "1.01", "2", "-0.9", "abc", "1e0"]) {
const result = await capturedRun([
"settings",
"features",
"set",
`intention_dedup_threshold=${value}`,
"--yes",
]);
expect(result.code, value).toBe(2);
}
expect(transport.put).not.toHaveBeenCalled();
});
it("does not send a threshold to a Runtime that does not list it", async () => {
const { parameters: _unused, ...older } = featuresState();
transport.get.mockResolvedValue({ status: 200, data: older });
const result = await capturedRun([
"settings",
"features",
"set",
"intention_dedup_threshold=0.9",
"--yes",
"--json",
]);
expect(result.code).toBe(1);
expect(JSON.parse(result.stdout)).toMatchObject({
state: "rejected",
code: "memory_feature_unsupported",
});
expect(transport.put).not.toHaveBeenCalled();
});
it("does nothing when the values are already set", async () => {
transport.get.mockResolvedValue({
status: 200,
@@ -210,6 +368,8 @@ describe("memhub settings features", () => {
["settings", "features", "set", "graph_rerank=yes"],
["settings", "features", "set", "graph_rerank"],
["settings", "features", "set", "few_shot=true", "few_shot=false"],
["settings", "features", "set", "intention_dedup=0.9"],
["settings", "features", "set", "intention_dedup_threshold=true"],
["settings", "features", "reset", "nope"],
["settings", "features", "toggle"],
]) {
@@ -269,7 +429,11 @@ describe("status pipeline features", () => {
last_evolve_at: null,
graph_build: { state: "idle", users: {} },
evolve: { mode: "pro", schedule: null, running: [], users: {} },
features: { episode_recall: true, intention: false },
features: {
episode_recall: true,
intention: false,
intention_dedup_threshold: 0.9,
},
},
};
const report = await composeStatusReport({
@@ -284,9 +448,12 @@ describe("status pipeline features", () => {
expect(report.pipeline?.features).toEqual({
episode_recall: true,
intention: false,
intention_dedup_threshold: 0.9,
});
const text = renderStatusText(report, "MemHub");
expect(text).toContain("SDK 功能:episode_recall=开、intention=关");
expect(text).toContain(
"SDK 功能:episode_recall=开、intention=关、intention_dedup_threshold=0.9",
);
});
});
+118 -28
View File
@@ -2,7 +2,8 @@
* `memhub settings features [set <name>=<true|false> ... | reset [<name> ...]]`.
*
* The SDK's optional retrieval and extraction features (graph rerank, episode
* recall, the episode write side, intention, few-shot). One global setting in
* recall, the episode write side, intention and its dedup, few-shot), plus the
* intention dedup threshold, the one numeric setting. One global setting in
* the Runtime; an unset feature keeps the SDK default, so an install nobody
* touched behaves as before. `set` prints its plan first and needs `--yes`,
* like `settings mode`, because some features spend model calls; `reset`
@@ -20,10 +21,19 @@ export const MEMORY_FEATURES = [
"session_episodes",
"episode_summary",
"intention",
"intention_dedup",
"few_shot",
] as const;
export type MemoryFeature = (typeof MEMORY_FEATURES)[number];
/** Numeric settings, with the range the Runtime accepts (inclusive). */
export const MEMORY_PARAMETERS = {
intention_dedup_threshold: { minimum: 0.8, maximum: 1 },
} as const;
export type MemoryParameter = keyof typeof MEMORY_PARAMETERS;
export type MemorySetting = MemoryFeature | MemoryParameter;
type SettingValues = Partial<Record<MemorySetting, boolean | number>>;
const FEATURE_LABEL: Record<MemoryFeature, string> = {
graph_rerank: "图谱重排(沿知识图谱的边传播支持度;仅 ultra)",
same_series_timeline:
@@ -33,6 +43,8 @@ const FEATURE_LABEL: Record<MemoryFeature, string> = {
session_episodes: "片段写入(同一会话提取的记忆归入同一片段)",
episode_summary: "片段摘要(片段结束时总结一次;每个片段一次模型调用)",
intention: "意图提取(随事实一起提取用户的计划与意图)",
intention_dedup:
"意图判重(检索时把同一天、相似度达到阈值的意图视为重复,只返回最早一条并删除其余;关闭则不判重、不删除)",
few_shot: "提取示例(在提取提示里附上示例;不加调用,输入更长)",
};
@@ -41,21 +53,42 @@ const REASON_TEXT: Record<string, string> = {
requires_pro: "lite 模式下不生效(需要 pro 或 ultra)",
requires_ultra: "当前模式下不生效(需要 ultra)",
requires_graph_rerank: "需要先开启并生效 graph_rerank",
requires_intention_dedup: "intention_dedup 已关闭,阈值不起作用",
};
const PARAMETER_LABEL: Record<MemoryParameter, string> = {
intention_dedup_threshold: "意图判重阈值(余弦相似度,越高保留越多)",
};
export type FeaturesCommand =
| { kind: "show" }
| {
kind: "set";
values: Partial<Record<MemoryFeature, boolean>>;
values: SettingValues;
confirmed: boolean;
}
| { kind: "reset"; names: MemoryFeature[] };
| { kind: "reset"; names: MemorySetting[] };
function isFeature(value: string): value is MemoryFeature {
return (MEMORY_FEATURES as readonly string[]).includes(value);
}
function isParameter(value: string): value is MemoryParameter {
return Object.hasOwn(MEMORY_PARAMETERS, value);
}
function isSetting(value: string): value is MemorySetting {
return isFeature(value) || isParameter(value);
}
/** A parameter's value: a plain decimal in its range; null = usage. */
function parameterValue(name: MemoryParameter, raw: string): number | null {
if (!/^\d+(\.\d+)?$/.test(raw)) return null;
const value = Number(raw);
const { minimum, maximum } = MEMORY_PARAMETERS[name];
return value >= minimum && value <= maximum ? value : null;
}
/** Arguments after `settings features`; null = usage. */
export function parseFeaturesCommand(
values: readonly string[],
@@ -68,18 +101,28 @@ export function parseFeaturesCommand(
const positional = tail.filter((value) => value !== "--yes");
if (verb === "set") {
if (positional.length === 0) return null;
const parsed: Partial<Record<MemoryFeature, boolean>> = {};
const parsed: SettingValues = {};
for (const item of positional) {
const match = /^([a-z_]+)=(true|false)$/.exec(item);
if (!match || !isFeature(match[1]) || match[1] in parsed) return null;
parsed[match[1]] = match[2] === "true";
const match = /^([a-z_]+)=(.+)$/.exec(item);
if (!match || match[1] in parsed) return null;
const [, name, raw] = match;
if (isFeature(name)) {
if (raw !== "true" && raw !== "false") return null;
parsed[name] = raw === "true";
} else if (isParameter(name)) {
const value = parameterValue(name, raw);
if (value === null) return null;
parsed[name] = value;
} else {
return null;
}
}
return { kind: "set", values: parsed, confirmed: yes === 1 };
}
if (verb === "reset") {
if (new Set(positional).size !== positional.length) return null;
if (!positional.every(isFeature)) return null;
return { kind: "reset", names: positional as MemoryFeature[] };
if (!positional.every(isSetting)) return null;
return { kind: "reset", names: positional as MemorySetting[] };
}
return null;
}
@@ -96,11 +139,26 @@ export interface FeatureState {
description: string;
}
export interface ParameterState {
name: MemoryParameter;
value: number;
configured: number | null;
default: number;
minimum: number;
maximum: number;
active: boolean;
reason: string | null;
requires_feature: string;
description: string;
}
export interface FeaturesState {
mode: string;
features: FeatureState[];
non_default: Partial<Record<MemoryFeature, boolean>>;
changed?: MemoryFeature[];
/** Absent from a Runtime older than the intention dedup threshold. */
parameters?: ParameterState[];
non_default: SettingValues;
changed?: MemorySetting[];
}
function featuresState(value: unknown): FeaturesState | null {
@@ -125,6 +183,20 @@ function featuresState(value: unknown): FeaturesState | null {
return null;
}
}
if (item.parameters !== undefined) {
if (!Array.isArray(item.parameters)) return null;
for (const parameter of item.parameters) {
if (
!parameter ||
typeof parameter !== "object" ||
typeof (parameter as ParameterState).name !== "string" ||
typeof (parameter as ParameterState).value !== "number" ||
typeof (parameter as ParameterState).active !== "boolean"
) {
return null;
}
}
}
return item as unknown as FeaturesState;
}
@@ -137,12 +209,12 @@ export type FeaturesDocument = {
| {
state: "preview";
features: FeaturesState;
target: Partial<Record<MemoryFeature, boolean>>;
target: SettingValues;
applied: false;
next_step: string;
}
| { state: "unchanged"; features: FeaturesState }
| { state: "committed"; features: FeaturesState; changed: MemoryFeature[] }
| { state: "committed"; features: FeaturesState; changed: MemorySetting[] }
| { state: "rejected" | "unknown" | "unavailable"; code: string }
);
@@ -170,13 +242,23 @@ export async function runFeaturesCommand(
if (command.kind === "show") {
return { ...BASE, state: "ok", features: current };
}
// What a person set, per name this Runtime lists; null = the default.
const configured = new Map<string, boolean | number | null>([
...current.features.map(
(feature) => [feature.name, feature.configured] as const,
),
...(current.parameters ?? []).map(
(parameter) => [parameter.name, parameter.configured] as const,
),
]);
let body: Record<string, unknown>;
if (command.kind === "set") {
const configured = new Map(
current.features.map((feature) => [feature.name, feature.configured]),
);
if (Object.keys(command.values).some((name) => !configured.has(name))) {
// An older Runtime does not know the name; it would answer 422.
return { ...BASE, state: "rejected", code: "memory_feature_unsupported" };
}
const differs = Object.entries(command.values).some(
([name, value]) => configured.get(name as MemoryFeature) !== value,
([name, value]) => configured.get(name) !== value,
);
if (!differs) return { ...BASE, state: "unchanged", features: current };
if (!command.confirmed) {
@@ -197,16 +279,10 @@ export async function runFeaturesCommand(
const names =
command.names.length > 0
? command.names
: current.features
.filter((feature) => feature.configured !== null)
.map((feature) => feature.name);
if (
names.every(
(name) =>
current.features.find((feature) => feature.name === name)
?.configured === null,
)
) {
: [...configured]
.filter(([, value]) => value !== null)
.map(([name]) => name as MemorySetting);
if (names.every((name) => (configured.get(name) ?? null) === null)) {
return { ...BASE, state: "unchanged", features: current };
}
body = Object.fromEntries(names.map((name) => [name, null]));
@@ -233,6 +309,16 @@ export function featuresExitCode(document: FeaturesDocument): number {
: 1;
}
function parameterLine(parameter: ParameterState): string {
const label = PARAMETER_LABEL[parameter.name] ?? parameter.name;
const origin = parameter.configured === null ? "默认" : "已设置";
const activity =
!parameter.active && parameter.reason
? `,${REASON_TEXT[parameter.reason] ?? parameter.reason}`
: "";
return `${parameter.name}:${parameter.value}(${origin}${activity},可设 ${parameter.minimum}–${parameter.maximum})— ${label}`;
}
function featureLine(feature: FeatureState): string {
const label = FEATURE_LABEL[feature.name] ?? feature.name;
const value = feature.enabled ? "开" : "关";
@@ -251,6 +337,8 @@ export function renderFeaturesText(document: FeaturesDocument): string {
const state = document.features;
const lines = [`当前模式:${state.mode}`];
for (const feature of state.features) lines.push(featureLine(feature));
for (const parameter of state.parameters ?? [])
lines.push(parameterLine(parameter));
lines.push(
"片段召回(episode_recall)只对带片段的记忆有用:先开启 session_episodes,之后提取的记忆才会带片段。",
);
@@ -258,7 +346,9 @@ export function renderFeaturesText(document: FeaturesDocument): string {
if (document.state === "preview") {
lines.push("将设置:");
for (const [name, value] of Object.entries(document.target)) {
lines.push(`- ${name} = ${value ? "开" : "关"}`);
lines.push(
`- ${name} = ${typeof value === "number" ? value : value ? "开" : "关"}`,
);
}
lines.push(`未执行任何改动。确认后运行:${document.next_step}`);
}
@@ -30,8 +30,7 @@ vi.mock("@one-memory/runtime-control", async (importOriginal) => ({
}));
function languageState(uiLanguage: string | null, extra = {}) {
const sdk =
uiLanguage === null || uiLanguage === "zh-Hans" ? "auto" : uiLanguage;
const sdk = uiLanguage === null ? "auto" : uiLanguage;
return {
ui_language: uiLanguage,
sdk_language: sdk,
@@ -95,7 +94,7 @@ describe("memhub settings language", () => {
});
expect(transport.put).not.toHaveBeenCalled();
const text = (await capturedRun(["settings", "language"])).stdout;
expect(text).toContain("界面语言:跟随系统(未选择)");
expect(text).toContain("界面语言:未设置(SDK 自动识别)");
expect(text).toContain("SDK 提示语言:auto");
});
+7 -6
View File
@@ -2,10 +2,11 @@
* `memhub settings language [<code>|system]`.
*
* The display language the App reports to the Runtime (`ui_language`), and
* the SDK prompt language it implies. zh-Hans and "system" (no explicit
* choice) pass no language, so the SDK keeps detecting Chinese or English as
* before; every other code, en included, makes extraction and the other SDK
* prompts use that language. A change applies at once: a write in flight
* the SDK prompt language it implies. Every code, zh-Hans and en included,
* makes extraction and the other SDK prompts use that language; "system"
* (no language set) passes none, so the SDK keeps detecting Chinese or
* English as before. The App sends the language it shows, so it is only
* unset on an install without the App. A change applies at once: a write in flight
* finishes in the old language, the next ones use the new one. The App sends
* its own choice again whenever it changes there.
*/
@@ -162,14 +163,14 @@ export function renderLanguageText(document: LanguageDocument): string {
const lines = [
`界面语言:${
state.ui_language === null
? "跟随系统(未选择)"
? "未设置(SDK 自动识别)"
: `${state.ui_language}(${NAMES[state.ui_language]})`
}`,
`SDK 提示语言:${sdkText(state.sdk_language)}`,
];
if (document.state === "ok") {
lines.push(
`可选:${UI_LANGUAGES.join("、")}、system。简体中文和 system 不指定语言,其余语言(包括 en)会让 SDK 使用该语言。`,
`可选:${UI_LANGUAGES.join("、")}、system。任一语言(包括 zh-Hans、en)都会让 SDK 使用该语言;system 不指定语言,SDK 自动识别中文或英文。`,
);
}
if (document.state === "unchanged") lines.push("没有需要改动的设置。");
+10
View File
@@ -248,6 +248,10 @@ function addReceipt(receipt: MemoryAddReceipt): Record<string, unknown> {
task_id: receipt.task_id,
memory_ids: receipt.memory_ids,
error: receipt.error,
error_code:
typeof receipt.error_code === "string" ? receipt.error_code : null,
next_retry_at:
typeof receipt.next_retry_at === "string" ? receipt.next_retry_at : null,
};
}
@@ -347,6 +351,12 @@ export function renderMemoryAddText(document: MemoryAddDocument): string {
...((document.receipt.memory_ids as string[]).length > 0
? [`memory_ids: ${(document.receipt.memory_ids as string[]).join(",")}`]
: []),
...(document.receipt.error_code
? [`error_code: ${document.receipt.error_code}`]
: []),
...(document.receipt.next_retry_at
? [`next_retry_at: ${document.receipt.next_retry_at}`]
: []),
document.command === "memories add"
? `查询进度:memhub memories add-status ${document.request_id};重试请带上同一个 --request-id,不会重复添加。`
: "",
+11 -1
View File
@@ -161,6 +161,7 @@ const CORE = new Map<string, { status: number; data: unknown }>([
blocked_stages: [
{ kind: "extract", reason: "model_missing", count: 2 },
],
pipeline: { mode: "pro", configured_mode: "pro" },
},
},
],
@@ -202,7 +203,9 @@ const READ_ONLY_PATHS = [
"/api/v1/live-sync",
"/api/v1/memory-processing",
"/api/v1/model-profiles",
"/api/v2/home-projection",
"/api/v2/home-summary",
"/api/v2/raw",
"/api/v2/memory-library",
"/api/v2/memories/",
"/api/v2/user-profile",
@@ -257,6 +260,9 @@ describe("CLI-20 status summary", () => {
expect(report.sections.model).toMatchObject({ state: "ok", profiles: 1 });
expect(report.sections.memories).toMatchObject({
state: "ok",
content_plane: "memory",
visible_count: 42,
effective_mode: "pro",
memory_count: 42,
});
expect(report.summary.state).toBe("ok");
@@ -903,7 +909,10 @@ describe("CLI-21 memory reads", () => {
expect(isReadOnlyPath("/api/v2/memories/ask")).toBe(false);
expect(isReadOnlyPath("/api/v2/memories/m1")).toBe(true);
expect(isReadOnlyPath("/api/v2/memory-library")).toBe(true);
expect(isReadOnlyPath("/api/v2/home-projection")).toBe(true);
expect(isReadOnlyPath("/api/v2/home-summary")).toBe(true);
expect(isReadOnlyPath("/api/v2/raw")).toBe(true);
expect(isReadOnlyPath("/api/v1/settings/memory-mode/plan")).toBe(true);
// The retired v1 data plane is no longer a CLI read.
expect(isReadOnlyPath("/api/v1/memories/m1")).toBe(false);
expect(isReadOnlyPath("/api/v1/home-summary")).toBe(false);
@@ -1085,7 +1094,8 @@ describe("malformed success responses", () => {
it("refuses a model profile whose string fields are not strings", async () => {
transport.get.mockImplementation(async (url: string) => {
if (url !== "/api/v1/model-profiles") return CORE.get(url);
if (url !== "/api/v1/model-profiles")
return CORE.get(url) ?? { status: 404, data: { detail: "not_found" } };
return {
status: 200,
data: [
+240
View File
@@ -1,5 +1,6 @@
import { describe, expect, it } from "vitest";
import type { CoreReader, ReadOutcome } from "./core-client.js";
import { composeDoctorReport, renderDoctorText } from "./doctor-report.js";
import {
composeStatusReport,
@@ -41,6 +42,108 @@ describe("status default output", () => {
});
});
describe("status on a paused model", () => {
const processing = (backoff: {
llm_backoff_until: string | null;
llm_backoff_error: string | null;
}) => ({
pending: 4,
fresh_pending: 4,
backlog_pending: 0,
outside_window_pending: 0,
running: 0,
waiting_for_model: 0,
completed: 1,
failed: 0,
blocked_stages: [],
suspended: 2,
...backoff,
});
const reader =
(summary: unknown): CoreReader =>
async <T>(path: string) =>
(path === "/api/v1/memory-processing"
? { state: "ok", data: summary }
: { state: "unavailable", code: "x" }) as ReadOutcome<T>;
const compose = (summary: unknown) =>
composeStatusReport({
dataDir: "/tmp/isolated/.one-memory",
runtime: {
state: "running",
code: "runtime_verified",
host: "127.0.0.1",
port: 30001,
},
background: null,
reader: reader(summary),
});
it("names the code and the retry time while the breaker pauses", async () => {
const report = await compose(
processing({
llm_backoff_until: "2026-11-02T10:00:00+00:00",
llm_backoff_error: "model_auth_failed",
}),
);
expect(report.sections.processing).toMatchObject({
state: "ok",
pending: 4,
suspended: 2,
llm_backoff_until: "2026-11-02T10:00:00+00:00",
llm_backoff_error: "model_auth_failed",
});
const text = renderStatusText(report, "MemHub");
expect(text).toContain(
"模型暂停:model_auth_failed,将于 2026-11-02T10:00:00+00:00 自动重试",
);
expect(text).not.toContain("llm_backoff");
});
it("prints no pause line and null fields when nothing is paused", async () => {
const report = await compose(
processing({ llm_backoff_until: null, llm_backoff_error: null }),
);
expect(report.sections.processing).toMatchObject({
llm_backoff_until: null,
llm_backoff_error: null,
});
expect(renderStatusText(report, "MemHub")).not.toContain("模型暂停");
});
it("reports a restarting write drain and its exit code", async () => {
const report = await compose({
...processing({ llm_backoff_until: null, llm_backoff_error: null }),
write_drain: "restarting",
write_drain_error: "operational_error",
});
expect(report.sections.processing).toMatchObject({
write_drain: "restarting",
write_drain_error: "operational_error",
});
const text = renderStatusText(report, "MemHub");
expect(text).toContain(
"写入队列:异常退出(operational_error),正在自动重启",
);
expect(text).not.toContain("write_drain");
});
it("says nothing about a running drain and nulls what an older Core omits", async () => {
const running = await compose({
...processing({ llm_backoff_until: null, llm_backoff_error: null }),
write_drain: "running",
write_drain_error: null,
});
expect(renderStatusText(running, "MemHub")).not.toContain("写入队列");
const older = await compose(
processing({ llm_backoff_until: null, llm_backoff_error: null }),
);
expect(older.sections.processing).toMatchObject({
write_drain: null,
write_drain_error: null,
});
});
});
describe("status and doctor on an unfinished upgrade migration", () => {
it("names the migration log and the issues page", async () => {
const report = await composeStatusReport({
@@ -77,3 +180,140 @@ describe("status and doctor on an unfinished upgrade migration", () => {
expect(report.runtime.log_file).toBeUndefined();
});
});
describe("mode-aware status content projection", () => {
const runtime = {
state: "running" as const,
code: "runtime_verified",
host: "127.0.0.1",
port: 30001,
};
const processing = (mode: "lite" | "pro" | "ultra") => ({
pending: 0,
fresh_pending: 0,
backlog_pending: 0,
outside_window_pending: 0,
running: 0,
waiting_for_model: 0,
completed: 0,
failed: 0,
pipeline: { mode, configured_mode: mode },
});
const projection = {
projection_version: 1,
user_id: "local",
effective_mode: "lite",
capabilities: {
content_plane: "raw",
generation_required: false,
graph_enabled: false,
evolve_available: false,
},
visible_count: 7,
display_state: "excerpt",
latest: {
type: "raw",
raw_id: "raw-1",
session_id: "session-1",
display_state: "excerpt",
time: null,
time_basis: null,
excerpt: "hello",
truncated: false,
source_id: "cursor",
},
observed_at: "2026-11-04T00:00:00Z",
};
it("uses the projection without reading either legacy count endpoint", async () => {
const calls: string[] = [];
const reader: CoreReader = async <T>(path: string) => {
calls.push(path);
const data =
path === "/api/v2/home-projection"
? projection
: path === "/api/v1/memory-processing"
? processing("lite")
: null;
return { state: "ok", data } as ReadOutcome<T>;
};
const report = await composeStatusReport({
dataDir: "/tmp/isolated",
runtime,
background: null,
reader,
});
expect(report.sections.memories).toEqual({
state: "ok",
content_plane: "raw",
visible_count: 7,
effective_mode: "lite",
raw_count: 7,
});
expect(calls).not.toContain("/api/v2/raw");
expect(calls).not.toContain("/api/v2/home-summary");
expect(renderStatusText(report, "MemHub")).toContain(
"原始对话:原始对话 7",
);
});
it("falls back to the real raw total only for an old Lite Core", async () => {
const reader: CoreReader = async <T>(path: string) => {
if (path === "/api/v2/home-projection")
return { state: "not_found", code: "http_404" } as ReadOutcome<T>;
if (path === "/api/v1/memory-processing")
return {
state: "ok",
data: processing("lite"),
} as ReadOutcome<T>;
if (path === "/api/v2/raw")
return {
state: "ok",
data: {
user_id: "local",
items: [],
total: 9,
offset: 0,
limit: 1,
time_filtered: false,
},
} as ReadOutcome<T>;
return { state: "unavailable", code: "x" } as ReadOutcome<T>;
};
const report = await composeStatusReport({
dataDir: "/tmp/isolated",
runtime,
background: null,
reader,
});
expect(report.sections.memories).toMatchObject({
state: "ok",
content_plane: "raw",
visible_count: 9,
raw_count: 9,
});
expect(report.sections.memories).not.toHaveProperty("memory_count");
});
it("rejects extra projection fields instead of falling back or leaking them", async () => {
const reader: CoreReader = async <T>(path: string) =>
({
state: "ok",
data:
path === "/api/v2/home-projection"
? { ...projection, secret: "must-not-leak" }
: processing("lite"),
}) as ReadOutcome<T>;
const report = await composeStatusReport({
dataDir: "/tmp/isolated",
runtime,
background: null,
reader,
});
expect(report.sections.memories).toEqual({
state: "invalid",
code: "response_invalid",
});
expect(JSON.stringify(report)).not.toContain("must-not-leak");
});
});
+178 -16
View File
@@ -1,4 +1,5 @@
import type {
HomeProjectionV1,
LiveSyncHealth,
MemoryProcessingSummary,
ModelProfile,
@@ -12,10 +13,12 @@ import { sdkMigrationLogPath } from "@one-memory/runtime-control";
import type { CoreReader, ReadOutcome } from "./core-client.js";
import {
type SdkHomeSummary,
validateHomeProjection,
validateHomeSummary,
validateLiveSync,
validateModelProfiles,
validateProcessing,
validateRawTotal,
validateSources,
} from "./core-responses.js";
import { describeEmbeddingSwitchSummary } from "./embedding-switch-flow.js";
@@ -85,7 +88,7 @@ export interface PipelineSummary {
>;
graph_build: string | null;
/** SDK features (`memory_features`) whose value differs from the default. */
features: Record<string, boolean>;
features: Record<string, boolean | number>;
/** The App's display language; null = never chosen or "follow the system". */
ui_language: string | null;
/** The SDK prompt language: `auto` (zh/en detection) or the code passed. */
@@ -134,12 +137,13 @@ export function describePipeline(value: unknown): PipelineSummary | null {
};
}
}
const features: Record<string, boolean> = {};
const features: Record<string, boolean | number> = {};
if (item.features && typeof item.features === "object") {
for (const [name, value] of Object.entries(
item.features as Record<string, unknown>,
)) {
if (typeof value === "boolean") features[name] = value;
if (typeof value === "boolean" || typeof value === "number")
features[name] = value;
}
}
const graph =
@@ -210,12 +214,12 @@ export async function composeStatusReport(options: {
for (const name of SECTION_NAMES) sections[name] = { state, code };
} else {
const reader = options.reader;
const [sources, sync, processing, model, memories] = await Promise.all([
const [sources, sync, processing, model, projection] = await Promise.all([
reader<unknown>("/api/v1/sources"),
reader<unknown>("/api/v1/live-sync"),
reader<unknown>("/api/v1/memory-processing"),
reader<unknown>("/api/v1/model-profiles"),
reader<unknown>("/api/v2/home-summary"),
reader<unknown>("/api/v2/home-projection"),
]);
sections.sources = section(sources, validateSources, describeSources);
sections.sync = section(sync, validateLiveSync, describeSync);
@@ -227,11 +231,7 @@ export async function composeStatusReport(options: {
pipeline =
processing.state === "ok" ? describePipeline(processing.data) : null;
sections.model = section(model, validateModelProfiles, describeModel);
sections.memories = section(
memories,
validateHomeSummary,
describeMemories,
);
sections.memories = await memorySection(reader, projection, pipeline);
}
// A bridge we could not inspect is unknown, not broken and not fine.
const bridge: StatusDocument["bridge"] =
@@ -311,7 +311,9 @@ export function renderStatusText(
];
for (const name of SECTION_NAMES) {
lines.push(
`${SECTION_LABELS[name]}:${sectionLine(report.sections[name])}`,
`${sectionLabel(name, report.sections[name])}:${sectionLine(
report.sections[name],
)}`,
);
}
if (report.pipeline) lines.push(`记忆模式:${pipelineLine(report.pipeline)}`);
@@ -319,6 +321,10 @@ export function renderStatusText(
report.sections.processing?.embedding_switch,
);
if (embeddingSwitch !== null) lines.push(`向量模型:${embeddingSwitch}`);
const backoff = modelBackoffLine(report.sections.processing);
if (backoff !== null) lines.push(backoff);
const drain = writeDrainLine(report.sections.processing);
if (drain !== null) lines.push(drain);
if (report.summary.state === "partial") {
const notes = [
report.summary.degraded.length > 0
@@ -334,6 +340,35 @@ export function renderStatusText(
return lines.join("\n");
}
/** "Model paused" while Core's circuit breaker holds model work back; null
* otherwise. Nothing to do here: the queue resumes by itself once the model
* answers again (or after the model is fixed). */
function modelBackoffLine(section: Section | undefined): string | null {
if (section?.state !== "ok") return null;
const error = section.llm_backoff_error;
if (typeof error !== "string" || error.length === 0) return null;
const until = section.llm_backoff_until;
return `模型暂停:${error},${
typeof until === "string" && until.length > 0
? `将于 ${until} 自动重试`
: "稍后自动重试"
}`;
}
/** The durable write drain, when it is not simply running: it exited on an
* error and is coming back (`restarting`), or it is not running at all. */
function writeDrainLine(section: Section | undefined): string | null {
if (section?.state !== "ok") return null;
const state = section.write_drain;
if (state !== "restarting" && state !== "stopped") return null;
const error = section.write_drain_error;
const reason =
typeof error === "string" && error.length > 0 ? `(${error})` : "";
return state === "restarting"
? `写入队列:异常退出${reason},正在自动重启`
: `写入队列:已停止${reason}`;
}
export function pipelineLine(pipeline: PipelineSummary): string {
const parts: string[] = [pipeline.mode];
if (pipeline.forced_by_env)
@@ -357,7 +392,10 @@ export function pipelineLine(pipeline: PipelineSummary): string {
if (changed.length > 0) {
parts.push(
`;SDK 功能:${changed
.map(([name, value]) => `${name}=${value ? "开" : "关"}`)
.map(
([name, value]) =>
`${name}=${typeof value === "number" ? value : value ? "开" : "关"}`,
)
.join("、")}`,
);
}
@@ -380,15 +418,33 @@ function runtimeLine(report: StatusDocument): string {
})`;
}
/** Fields that have their own line (or none) instead of a place in the
* section's one-line summary. */
const SECTION_LINE_OMITTED = new Set([
"state",
"code",
"embedding_switch",
"llm_backoff_until",
"llm_backoff_error",
"write_drain",
"write_drain_error",
"content_plane",
"visible_count",
"effective_mode",
]);
function sectionLabel(name: string, section: Section | undefined): string {
return name === "memories" && section?.content_plane === "raw"
? "原始对话"
: SECTION_LABELS[name];
}
function sectionLine(section: Section | undefined): string {
if (!section) return "未检查";
if (section.state === "not_checked") return "未检查";
if (section.state !== "ok") return "暂时不可用,请稍后重试";
return Object.entries(section)
.filter(
([key]) =>
key !== "state" && key !== "code" && key !== "embedding_switch",
)
.filter(([key]) => !SECTION_LINE_OMITTED.has(key))
.map(([key, value]) => `${fieldLabel(key)} ${formatValue(value)}`)
.join(" ");
}
@@ -424,6 +480,7 @@ function fieldLabel(key: string): string {
backlog_pending: "历史待处理",
outside_window_pending: "稍后处理",
waiting_for_model: "等待模型",
suspended: "来源已暂停",
completed: "已完成",
failed: "失败",
blocked: "卡住",
@@ -433,6 +490,10 @@ function fieldLabel(key: string): string {
profiles: "配置",
active: "当前",
memory_count: "记忆",
raw_count: "原始对话",
visible_count: "可见",
content_plane: "内容层",
effective_mode: "模式",
};
return labels[key] ?? key;
}
@@ -504,10 +565,47 @@ function describeProcessing(
reason: block.reason,
count: block.count,
})),
suspended: typeof summary.suspended === "number" ? summary.suspended : 0,
...llmBackoffFields(summary),
...writeDrainFields(summary),
...embeddingSwitchField(summary),
};
}
/** The model circuit breaker: set only while Core pauses model work because
* the provider keeps failing. Both null when nothing is paused. */
function llmBackoffFields(summary: MemoryProcessingSummary): {
llm_backoff_until: string | null;
llm_backoff_error: string | null;
} {
const until = summary.llm_backoff_until;
const error = summary.llm_backoff_error;
return {
llm_backoff_until:
typeof until === "string" && until.length <= 64 ? until : null,
llm_backoff_error:
typeof error === "string" && /^[a-z_]{1,64}$/.test(error) ? error : null,
};
}
/** Durable write drain liveness. Null when Core does not report it (older
* Core, or recovery has not started the drain yet). */
function writeDrainFields(summary: MemoryProcessingSummary): {
write_drain: "running" | "restarting" | "stopped" | null;
write_drain_error: string | null;
} {
const state = summary.write_drain;
const error = summary.write_drain_error;
return {
write_drain:
state === "running" || state === "restarting" || state === "stopped"
? state
: null,
write_drain_error:
typeof error === "string" && /^[a-z_]{1,64}$/.test(error) ? error : null,
};
}
/** The running switch or merge, projected; absent when none runs. */
function embeddingSwitchField(
summary: MemoryProcessingSummary,
@@ -548,6 +646,70 @@ function describeMemories(summary: SdkHomeSummary): Record<string, unknown> {
};
}
async function memorySection(
reader: CoreReader,
projection: ReadOutcome<unknown>,
pipeline: PipelineSummary | null,
): Promise<Section> {
if (projection.state === "ok") {
const parsed = validateHomeProjection(projection.data);
return parsed === null
? { state: "invalid", code: "response_invalid" }
: { state: "ok", ...describeProjection(parsed) };
}
if (projection.state !== "not_found") {
return {
state: projection.state === "denied" ? "denied" : "unavailable",
code: projection.code,
};
}
if (pipeline === null) {
return { state: "unavailable", code: "effective_mode_unavailable" };
}
if (pipeline.mode === "lite") {
const raw = await reader<unknown>("/api/v2/raw", {
limit: 1,
offset: 0,
});
if (raw.state !== "ok") {
return {
state: raw.state === "denied" ? "denied" : "unavailable",
code: raw.code,
};
}
const total = validateRawTotal(raw.data);
return total === null
? { state: "invalid", code: "response_invalid" }
: {
state: "ok",
content_plane: "raw",
visible_count: total,
effective_mode: "lite",
raw_count: total,
};
}
const legacy = await reader<unknown>("/api/v2/home-summary");
return section(legacy, validateHomeSummary, (summary) => ({
content_plane: "memory",
visible_count: summary.memory_count,
effective_mode: pipeline.mode,
...describeMemories(summary),
}));
}
function describeProjection(
projection: HomeProjectionV1,
): Record<string, unknown> {
return {
content_plane: projection.capabilities.content_plane,
visible_count: projection.visible_count,
effective_mode: projection.effective_mode,
...(projection.capabilities.content_plane === "raw"
? { raw_count: projection.visible_count }
: { memory_count: projection.visible_count }),
};
}
function sum(
sources: SourceResponse[],
key: "imported_conversations" | "imported_messages",
+63
View File
@@ -208,6 +208,27 @@ describe("--user-id on the memory commands", () => {
expect(direct.get.mock.calls[0][1].params).toEqual({ user_id: "wife" });
});
it("prints the retry code and time of a write still waiting", async () => {
direct.get.mockResolvedValue({
data: {
request_id: REQUEST,
state: "accepted",
created_at: "T1",
updated_at: "T2",
error_code: "model_timeout",
next_retry_at: "2026-11-02T10:00:00+00:00",
},
});
const text = await capturedRun(["write", "status", REQUEST]);
expect(text.stdout).toContain("error_code: model_timeout");
expect(text.stdout).toContain("next_retry_at: 2026-11-02T10:00:00+00:00");
const json = await capturedRun(["write", "status", REQUEST, "--json"]);
expect(JSON.parse(json.stdout).write).toMatchObject({
error_code: "model_timeout",
next_retry_at: "2026-11-02T10:00:00+00:00",
});
});
it("passes the user in the body of memory writes and searches", async () => {
await capturedRun(["memories", "add", "Prefers tea", "--user-id", "wife"]);
expect(transport.post.mock.calls[0][0]).toBe("/api/v2/memories");
@@ -373,6 +394,48 @@ describe("sources connect for a Hermes profile", () => {
}
expect(transport.post).not.toHaveBeenCalled();
});
it("says the user id need not equal the Hermes profile name", async () => {
const result = await capturedRun([
"sources",
"connect",
"hermes",
"--user-id",
"wife:work",
"--root",
"~/.hermes/profiles/wife:work",
]);
expect(result.code).toBe(2);
expect(result.stderr).toContain("用户 ID 不必等于 Hermes profile 名");
expect(result.stderr).toContain(
"--user-id wife-work --root ~/.hermes/profiles/wife:work",
);
expect(transport.post).not.toHaveBeenCalled();
});
it("explains a refused overlapping or profiles-container root", async () => {
for (const code of [
"hermes_root_is_profiles_container",
"hermes_root_overlaps_other_user",
]) {
transport.post.mockResolvedValueOnce({
status: code.endsWith("container") ? 422 : 409,
data: { detail: code },
});
const result = await capturedRun([
"sources",
"connect",
"hermes",
"--user-id",
"kid",
"--root",
"/profiles",
]);
expect(result.code).toBe(1);
expect(result.stdout).toContain(code);
expect(result.stdout).toContain("--root");
}
});
});
async function capturedRun(arguments_: string[]) {
+2 -2
View File
@@ -4,7 +4,7 @@ build-backend = "uv_build"
[project]
name = "one-memory"
version = "1.0.5"
version = "1.0.6"
description = "MemHub local-first memory core"
readme = "README.md"
requires-python = ">=3.12"
@@ -12,7 +12,7 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.10",
"hy-memory==2.0.14",
# chromadb arrives through the HY SDK; locked here so an unrelated upgrade
# cannot move the SDK's store. Built-in embedding consumes numpy directly.
"chromadb==1.5.9",
File diff suppressed because it is too large. Load diff
@@ -15,6 +15,20 @@ BULK_WORKERS = 3
SNAPSHOT_WORKERS = 8
INTERACTIVE_READ_WORKERS = 4
SOURCE_READER_WORKERS = 4
# One per source the drain runs side by side (`max_concurrent_sources`), and
# two over for the outcome inquiries that follow them.
SDK_WRITE_WORKERS = 12
# Recall's SDK reads: one search and the `get`s behind it (`_SDK_GET_CONCURRENCY`).
SDK_READ_WORKERS = 8
# The durable write queue's own bookkeeping: claim, settle, the per-job reads
# around an SDK write, and the drain's maintenance passes. Each is one short
# statement on the single writer or a reader; the drain runs at most
# `max_concurrent_sources` jobs, and they wait for the writer, not for CPU.
DURABLE_QUEUE_WORKERS = 4
# Lease renewal alone. Renewal is the drain's ownership fence: a lease that
# lapses while its SDK write is still running hands the job to recovery, so it
# must never wait for a thread behind any other work, its own queue's included.
LEASE_RENEWAL_WORKERS = 2
_pool: ThreadPoolExecutor | None = None
_snapshot_pool: ThreadPoolExecutor | None = None
@@ -22,6 +36,9 @@ _interactive_pool: ThreadPoolExecutor | None = None
_raw_read_pool: ThreadPoolExecutor | None = None
_memory_read_pool: ThreadPoolExecutor | None = None
_source_reader_pool: ThreadPoolExecutor | None = None
_sdk_write_pool: ThreadPoolExecutor | None = None
_durable_queue_pool: ThreadPoolExecutor | None = None
_lease_renewal_pool: ThreadPoolExecutor | None = None
_warmed_readers: weakref.WeakSet[ThreadPoolExecutor] = weakref.WeakSet()
_reader_warm_lock = threading.Lock()
_interactive_reads = 0
@@ -71,15 +88,67 @@ def raw_read_pool() -> ThreadPoolExecutor:
def memory_read_pool() -> ThreadPoolExecutor:
"""Recall's own threads: SDK searches, gets, and what decides between them.
Recall used to hand these to the default pool, which SDK writes share. A
write that times out keeps its thread until the SDK returns (#231), so a
few stuck writes left recall waiting for a thread while `/health`, which
needs none, still answered.
"""
global _memory_read_pool
if _memory_read_pool is None:
_memory_read_pool = ThreadPoolExecutor(
max_workers=INTERACTIVE_READ_WORKERS,
max_workers=SDK_READ_WORKERS,
thread_name_prefix="one-memory-memory-read",
)
return _memory_read_pool
def sdk_write_pool() -> ThreadPoolExecutor:
"""SDK writes, bounded and apart from everything else.
Each thread here only waits for the SDK's own loop thread to answer
(`_LoopThread.run`), so any thread outside that loop may make the call; a
stuck write therefore costs a thread of this pool and nobody else's.
"""
global _sdk_write_pool
if _sdk_write_pool is None:
_sdk_write_pool = ThreadPoolExecutor(
max_workers=SDK_WRITE_WORKERS,
thread_name_prefix="one-memory-sdk-write",
)
return _sdk_write_pool
def durable_queue_pool() -> ThreadPoolExecutor:
"""The durable write queue's database work, apart from bulk work.
The queue used to hand its claims, settles and maintenance to the bulk
pool, whose three threads also carry import heartbeats and job listings.
Under an import peak all three sat in those for tens of seconds, and the
queue's next statement waited for a thread before it could even join the
writer's line, and leases lapsed while their SDK writes were still running.
"""
global _durable_queue_pool
if _durable_queue_pool is None:
_durable_queue_pool = ThreadPoolExecutor(
max_workers=DURABLE_QUEUE_WORKERS,
thread_name_prefix="one-memory-write-queue",
)
return _durable_queue_pool
def lease_renewal_pool() -> ThreadPoolExecutor:
"""Threads that only renew durable write leases (`DurableWriteDrain.renew`)."""
global _lease_renewal_pool
if _lease_renewal_pool is None:
_lease_renewal_pool = ThreadPoolExecutor(
max_workers=LEASE_RENEWAL_WORKERS,
thread_name_prefix="one-memory-lease-renewal",
)
return _lease_renewal_pool
def source_reader_pool() -> ThreadPoolExecutor:
"""Bounded process-wide pool for source iteration and parsing."""
global _source_reader_pool
@@ -99,9 +168,21 @@ def warm_bulk_pool() -> None:
Starting a thread is nothing on an idle machine and 2.5 seconds on a busy
one, which is precisely when bulk work is being handed over. So it is
done here instead, at startup, before there is anything to be late for.
The durable write queue's pools are warmed here too, for the same reason:
their first renewal must not pay for starting a thread.
"""
pool = bulk_pool()
started = tuple(threading.Event() for _ in range(BULK_WORKERS))
_start_every_worker(bulk_pool(), BULK_WORKERS, "bulk_pool_warmup_timeout")
_start_every_worker(
durable_queue_pool(), DURABLE_QUEUE_WORKERS, "durable_queue_pool_warmup_timeout"
)
_start_every_worker(
lease_renewal_pool(), LEASE_RENEWAL_WORKERS, "lease_renewal_pool_warmup_timeout"
)
def _start_every_worker(pool: ThreadPoolExecutor, workers: int, timeout_code: str) -> None:
started = tuple(threading.Event() for _ in range(workers))
release = threading.Event()
def occupy(marker: threading.Event) -> None:
@@ -117,7 +198,7 @@ def warm_bulk_pool() -> None:
try:
for marker in started:
if not marker.wait(10):
raise RuntimeError("bulk_pool_warmup_timeout")
raise RuntimeError(timeout_code)
finally:
release.set()
for future in futures:
@@ -175,6 +256,62 @@ async def in_interactive_read[T](
return await in_read_pool(interactive_read_pool(), work, *args, **kwargs)
async def in_sdk_read[T](
work: Callable[..., T],
/,
*args: object,
**kwargs: object,
) -> T:
"""A recall-path read, in recall's own pool (`memory_read_pool`)."""
return await in_read_pool(memory_read_pool(), work, *args, **kwargs)
async def in_sdk_write[T](
work: Callable[..., T],
/,
*args: object,
**kwargs: object,
) -> T:
"""One SDK write call, in the write pool (`sdk_write_pool`)."""
return await asyncio.get_running_loop().run_in_executor(
sdk_write_pool(),
copy_context().run,
partial(work, *args, **kwargs),
)
async def in_the_write_queue[T](
work: Callable[..., T],
/,
*args: object,
**kwargs: object,
) -> T:
"""One durable write queue statement, in the queue's pool (`durable_queue_pool`).
It still goes through the one writer and its line, like every write; what
it no longer does is wait for a bulk thread first.
"""
return await asyncio.get_running_loop().run_in_executor(
durable_queue_pool(),
copy_context().run,
partial(work, *args, **kwargs),
)
async def in_lease_renewal[T](
work: Callable[..., T],
/,
*args: object,
**kwargs: object,
) -> T:
"""One lease renewal, in the threads kept for nothing else (`lease_renewal_pool`)."""
return await asyncio.get_running_loop().run_in_executor(
lease_renewal_pool(),
copy_context().run,
partial(work, *args, **kwargs),
)
def _retrieve[T](future: "asyncio.Future[T]") -> None:
"""Mark a shielded read's outcome as seen.
@@ -261,7 +398,16 @@ def _warm_reader_pool(reader: ThreadPoolExecutor) -> None:
def shutdown_bulk_pool() -> None:
global _pool, _snapshot_pool, _interactive_pool, _raw_read_pool, _memory_read_pool
global _source_reader_pool
global _source_reader_pool, _sdk_write_pool, _durable_queue_pool, _lease_renewal_pool
if _durable_queue_pool is not None:
_durable_queue_pool.shutdown(wait=False, cancel_futures=True)
_durable_queue_pool = None
if _lease_renewal_pool is not None:
_lease_renewal_pool.shutdown(wait=False, cancel_futures=True)
_lease_renewal_pool = None
if _sdk_write_pool is not None:
_sdk_write_pool.shutdown(wait=False, cancel_futures=True)
_sdk_write_pool = None
if _pool is not None:
_pool.shutdown(wait=False, cancel_futures=True)
_pool = None
@@ -66,6 +66,8 @@ class SdkContextSearchRequest(ContextSearchRequest, SdkSearchOptions):
limit: int = Field(default=10, ge=1, le=20, strict=True)
schema_limit: int = Field(default=3, ge=0, le=20, strict=True)
intention_limit: int = Field(default=1, ge=0, le=20, strict=True)
#: Product callers can request one item per turn instead of SDK raw batches.
raw_unit: Literal["raw", "turn"] = "raw"
#: Whose memories to search. The App never sends it and reads `local`.
user_id: UserId = DEFAULT_USER_ID
@@ -123,6 +125,9 @@ class SdkRawContextItem(BaseModel):
memory_at: datetime | None = None
score: float = Field(ge=0, le=1)
truncated: bool = False
turn_id: str | None = Field(default=None, min_length=1, max_length=96)
turn_index: int | None = Field(default=None, ge=0)
turn_count: int | None = Field(default=None, ge=1)
class SdkMemoryContextItem(BaseModel):
@@ -155,6 +160,8 @@ class SdkContextSearchResponse(BaseModel):
used_characters: int = Field(ge=0, le=20000)
truncated: bool
total_ms: float = Field(ge=0)
#: Whose memories were searched: the request's `user_id`, else `local`.
user_id: str = Field(min_length=1, max_length=64)
class ContextLaneResult(BaseModel):
@@ -30,6 +30,7 @@ from one_memory.hy_sdk_types import (
parse_sdk_search,
)
from one_memory.imports.storage import EvidenceStore
from one_memory.raw_turns import raw_display_turns, turns_for_hit
from one_memory.sdk_search_options import SdkSearchFilters, SearchLane
from one_memory.semantic_deadline import SemanticMemoryHit
from one_memory.user_scope import owns
@@ -70,6 +71,18 @@ def raw_lane_overrides(value: SdkSearchFilters) -> dict[str, object]:
return found
def raw_lane_statuses(*, lite: bool) -> dict[str, list[str]]:
"""Which raw statuses the raw lane searches.
Lite extracts nothing, so its search is plain RAG over every raw,
including the shadow raws pro already processed before the switch to
lite. Pro and ultra keep the SDK default (active only): a shadow raw is
already represented by the memories extracted from it, and returning it
again would repeat that content in the caller's context budget.
"""
return {"statuses": ["active", "shadow"]} if lite else {}
def hit_namespace(hit: SemanticMemoryHit) -> dict[str, str]:
"""The `get` namespace of a hit: nothing for chat, kind and variant otherwise."""
if hit.kind in {"experience", "fact"} and hit.variant is not None:
@@ -120,6 +133,9 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
sources = raw_lane_sources(authority.raw_sources, value.include_sources)
if sources is None:
return ()
# Plain RAG: every vector hit over the floor is an answer, ranked
# by its score. Each hit yields at least one item, so the SDK
# needs to return no more hits than the caller's limit.
raw = await asyncio.to_thread(
client.search,
value.query,
@@ -135,27 +151,60 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
"scene": "chat",
"reader": HY_MEMORY_READER,
"layers": ["l1_raw"],
**raw_lane_statuses(lite=lite),
**raw_lane_overrides(value),
},
)
hits = tuple(
hit for hit in parse_sdk_raw_search(raw) if value.passes_floor("raw", hit.score)
)
return tuple(
SdkRawContextItem(
raw_id=hit.raw_id,
content=hit.content,
source=hit.source,
session_id=hit.session_id,
memory_at=(
datetime.fromtimestamp(hit.memory_at, tz=UTC)
if hit.memory_at is not None
else None
),
score=hit.score,
# One item per hit first, so a hit is never crowded out by another
# hit's sibling turns; the siblings fill whatever room is left.
primary: list[SdkRawContextItem] = []
siblings: list[SdkRawContextItem] = []
for hit in hits:
memory_at = (
datetime.fromtimestamp(hit.memory_at, tz=UTC)
if hit.memory_at is not None
else None
)
for hit in hits
)
turns = raw_display_turns(hit) if value.raw_unit == "turn" else ()
if not turns:
primary.append(
SdkRawContextItem(
raw_id=hit.raw_id,
content=hit.content,
source=hit.source,
session_id=hit.session_id,
memory_at=memory_at,
score=hit.score,
)
)
continue
# The embedding is the batch's: the hit is shown by the turns
# that contain the query, or else by its closest turn, each
# with the hit's score.
shown = [
SdkRawContextItem(
raw_id=hit.raw_id,
turn_id=turn.turn_id,
turn_index=turn.turn_index,
turn_count=turn.turn_count,
content=turn.content,
source=hit.source,
session_id=hit.session_id,
memory_at=memory_at,
score=hit.score,
)
for turn in turns_for_hit(turns, value.query)
]
primary.append(shown[0])
siblings.extend(shown[1:])
chosen = primary[: value.limit]
chosen.extend(siblings[: value.limit - len(chosen)])
# Stable: a hit's turns stay together, in hit-score order.
chosen.sort(key=lambda item: item.score, reverse=True)
return tuple(chosen)
async def memory_lane() -> tuple[SdkMemoryContextItem, ...]:
# lite extracts nothing: its search answers raw turns, which the
@@ -261,6 +310,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
used_characters=0,
truncated=True,
total_ms=(time.monotonic() - started) * 1000,
user_id=value.user_id,
)
await record(final, timed_out=True)
return final
@@ -306,6 +356,7 @@ def sdk_context_router(dependencies: Sequence[Depends]) -> APIRouter:
used_characters=used,
truncated=truncated,
total_ms=(time.monotonic() - started) * 1000,
user_id=value.user_id,
)
await record(final, timed_out=False)
return final
@@ -111,10 +111,26 @@ MAIN_DATABASE_CHECKPOINT_WRITE_INTERVAL = 64
MAIN_DATABASE_CHECKPOINT_SECONDS = 30.0
# Advisory only: PASSIVE cannot force a frame past a long reader. Crossing this
# threshold is a signal that the WAL is not converging and that read snapshots
# or write volume need investigation.
# threshold with frames still waiting to be copied back is a signal that the
# WAL is not converging and that read snapshots or write volume need
# investigation. A large file whose frames are all checkpointed is not: the
# idle TRUNCATE below gives that space back.
MAIN_DATABASE_WAL_WARN_BYTES = 512 * 1024 * 1024
# What a WAL file is cut back to once a checkpoint has copied every frame and
# the writer starts it over. Without a limit SQLite reuses the file in place
# and never shrinks it, so one import peak left a 639 MB WAL behind for good
# (#245). 64 MiB is about sixteen times what the default autocheckpoint (1000
# pages) lets build up between checkpoints, so ordinary write bursts reuse the
# file without growing and shrinking it each time, and it is still a tenth of
# what that user's disk was left holding.
MAIN_DATABASE_JOURNAL_SIZE_LIMIT_BYTES = 64 * 1024 * 1024
# How long an idle TRUNCATE may wait for the writer or for a reader to leave.
# It runs only when nothing holds the database; anyone arriving in the
# meantime wins, and the checkpoint is tried again on the next pass.
MAIN_DATABASE_IDLE_CHECKPOINT_WAIT_MS = 50
def checkpoint_write_interval() -> int:
raw = os.environ.get(
@@ -226,6 +242,75 @@ class _InTurn:
self._line.remove(place)
class _Checkpointer:
"""The one thread that runs cadence checkpoints, for every main database.
A database asks for at most one checkpoint at a time: a request made
while one is already waiting is the same request. The checkpoint goes
through the database's own writer, in its line, like any other write.
A waiting request holds the database weakly: it must not keep alive a
database nobody else holds any more.
"""
def __init__(self) -> None:
self._state = threading.Condition()
self._due: OrderedDict[int, weakref.ref[MainDatabase]] = OrderedDict()
self._running = 0
self._thread: threading.Thread | None = None
def request(self, database: "MainDatabase") -> None:
with self._state:
self._due.setdefault(id(database), weakref.ref(database))
if self._thread is None or not self._thread.is_alive():
self._thread = threading.Thread(
target=self._run,
name="one-memory-checkpoint",
daemon=True,
)
self._thread.start()
self._state.notify_all()
def forget(self, database: "MainDatabase") -> None:
with self._state:
self._due.pop(id(database), None)
self._state.notify_all()
def wait_idle(self, timeout: float) -> bool:
"""Whether every requested checkpoint has run within `timeout` seconds."""
with self._state:
return self._state.wait_for(lambda: not self._due and not self._running, timeout)
def _run(self) -> None:
while True:
with self._state:
self._state.wait_for(lambda: bool(self._due))
_key, reference = self._due.popitem(last=False)
database = reference()
if database is None:
self._state.notify_all()
continue
self._running += 1
try:
database._cadence_checkpoint()
except Exception: # noqa: BLE001 - maintenance must not end the thread
logger.warning(
"main_database_checkpoint_failed file=%s", database.path.name, exc_info=True
)
finally:
del database
with self._state:
self._running -= 1
self._state.notify_all()
_checkpointer = _Checkpointer()
def wait_for_cadence_checkpoints(timeout: float = 5.0) -> bool:
"""Wait for the cadence checkpoints already requested; for tests and shutdown."""
return _checkpointer.wait_idle(timeout)
def _locked() -> sqlite3.OperationalError:
# Callers tell a busy database from a broken one by these, as they would
# for SQLite's own refusal.
@@ -276,6 +361,8 @@ class MainDatabase:
self._writes_since_checkpoint = 0
self._last_checkpoint_at = time.monotonic()
self._wal_warned_at = 0.0
#: Learned from the first checkpoint; turns WAL frames into bytes.
self._page_size = 4096
self._reader_started_at: dict[int, float] = {}
@contextmanager
@@ -542,24 +629,77 @@ class MainDatabase:
def _checkpoint_from_maintenance(self, mode: str) -> tuple[int, int, int]:
return self._checkpoint(mode, maintenance=True)
def _checkpoint(self, mode: str, *, maintenance: bool) -> tuple[int, int, int]:
def _checkpoint(
self,
mode: str,
*,
maintenance: bool,
busy_timeout_ms: int = WRITE_LOCK_WAIT_MS,
) -> tuple[int, int, int]:
normalized = mode.strip().upper()
if normalized not in {"PASSIVE", "FULL", "RESTART", "TRUNCATE"}:
raise ValueError("unsupported WAL checkpoint mode")
with self.write(autocommit=True, maintenance=maintenance) as connection:
with self.write(
autocommit=True, maintenance=maintenance, busy_timeout_ms=busy_timeout_ms
) as connection:
row = connection.execute(f"PRAGMA wal_checkpoint({normalized})").fetchone()
page_size = connection.execute("PRAGMA page_size").fetchone()
if row is None or len(row) != 3:
raise sqlite3.DatabaseError("SQLite returned an invalid checkpoint result")
if page_size is not None:
self._page_size = int(page_size[0])
return int(row[0]), int(row[1]), int(row[2])
def idle(self) -> bool:
"""Nobody holds or waits for a connection to this database right now."""
with self._state:
return (
self._writer_owner is None
and not self._reader_holders
and not len(self._write_line)
and not len(self._read_line)
)
def checkpoint_if_idle(self) -> tuple[int, int, int] | None:
"""Give the WAL file back to the disk while nothing is using the database.
PASSIVE copies frames back but leaves the file its size; only TRUNCATE
empties it, and TRUNCATE has to wait for every reader to finish, so it
is attempted only when this process holds no checkout at all, and with
a wait of a few milliseconds. A busy result, or someone arriving in
between, is not a failure: the next idle pass tries again. Returns the
checkpoint result, or None when it was not attempted or not admitted.
"""
if not self.idle() or self._wal_bytes() == 0:
return None
try:
return self._checkpoint(
"TRUNCATE",
maintenance=True,
busy_timeout_ms=MAIN_DATABASE_IDLE_CHECKPOINT_WAIT_MS,
)
except sqlite3.Error:
logger.debug("main_database_idle_checkpoint_skipped file=%s", self.path.name)
return None
def _maybe_checkpoint_after_write(self) -> None:
"""Checkpoint on a write cadence without opening another connection.
"""Count a write, and hand a due cadence checkpoint to the checkpointer.
SQLite's default autocheckpoint only runs when the writer itself
crosses the page threshold. Production hit a state where the WAL grew
to 13GB, so the main database needs its own bounded, observable
maintenance path. This runs after the caller's checkout has been reset
and only re-enters the same serialized writer queue.
maintenance path: every `checkpoint_write_interval()` writes or
`MAIN_DATABASE_CHECKPOINT_SECONDS`, whichever comes first.
The checkpoint itself used to run right here, on the thread that had
just written, as one more turn in the writer's line. That thread was
an import heartbeat or a durable lease renewal as often as not, and
it then waited behind the whole line a second time before it could
return. The idle TRUNCATE (#278) cannot replace the cadence: under a
sustained import the database is never idle, and that is exactly when
the WAL grows. So the cadence stays, and one checkpointer thread runs
it (`_Checkpointer`); the writer stays the only connection that
writes, and no caller waits for maintenance it did not ask for.
"""
now = time.monotonic()
with self._state:
@@ -571,25 +711,35 @@ class MainDatabase:
if due:
self._writes_since_checkpoint = 0
self._last_checkpoint_at = now
if not due:
return
if due:
_checkpointer.request(self)
def _cadence_checkpoint(self) -> None:
"""One PASSIVE checkpoint through the writer, and the WAL alert after it."""
now = time.monotonic()
try:
busy, log_frames, checkpointed = self._checkpoint_from_maintenance("PASSIVE")
except sqlite3.Error:
logger.warning("main_database_checkpoint_failed file=%s", self.path.name, exc_info=True)
return
wal_bytes = self._wal_bytes()
if wal_bytes >= MAIN_DATABASE_WAL_WARN_BYTES:
# Frames the checkpoint could not copy back yet: what a reader is
# holding up. The file size alone also counts space already
# checkpointed and only waiting to be reused or truncated, which is
# not the WAL failing to converge.
pending_bytes = max(0, log_frames - checkpointed) * self._page_size
if pending_bytes >= MAIN_DATABASE_WAL_WARN_BYTES:
with self._state:
should_warn = now - self._wal_warned_at >= MAIN_DATABASE_REPORT_SECONDS
if should_warn:
self._wal_warned_at = now
if should_warn:
logger.warning(
"main_database_wal_high file=%s wal_bytes=%d busy=%d "
"main_database_wal_high file=%s wal_bytes=%d pending_bytes=%d busy=%d "
"log_frames=%d checkpointed=%d readers_in_use=%d",
self.path.name,
wal_bytes,
pending_bytes,
busy,
log_frames,
checkpointed,
@@ -640,6 +790,8 @@ class MainDatabase:
def close(self) -> None:
"""Close every idle connection now; ones in use close when handed back."""
# A checkpoint still waiting would open the writer again after this.
_checkpointer.forget(self)
with self._state:
self._generation += 1
self._identity = None
@@ -784,6 +936,10 @@ class MainDatabase:
try:
if wal:
enable_write_ahead_log(connection, wait_ms=busy_timeout_ms)
# Only the writer restarts the WAL, so only the writer needs the cap.
connection.execute(
f"PRAGMA journal_size_limit = {int(MAIN_DATABASE_JOURNAL_SIZE_LIMIT_BYTES)}"
)
_seal(connection)
if busy_timeout_ms != WRITE_LOCK_WAIT_MS:
# Opened at the caller's wait; every checkout starts at the default.
@@ -1027,6 +1183,18 @@ def main_database_transaction(
yield connection
def checkpoint_idle_main_databases() -> int:
"""TRUNCATE the WAL of every database nobody is using; how many were emptied."""
with _databases_lock:
databases = list(_alive.values())
emptied = 0
for database in databases:
result = database.checkpoint_if_idle()
if result is not None and result[0] == 0:
emptied += 1
return emptied
def close_main_databases() -> None:
"""Close every connection this process holds; used when the Core stops."""
with _databases_lock:
@@ -1050,6 +1218,9 @@ async def report_main_database(seconds: float) -> None:
while True:
await asyncio.sleep(seconds)
log_main_database_stats("periodic")
# The idle signal is this process's own checkouts: none held, none
# waiting. Off the loop, since a checkpoint is disk work however short.
await asyncio.to_thread(checkpoint_idle_main_databases)
def main_database_stats() -> dict[str, float | int]:
@@ -126,15 +126,23 @@ def embedding_switch_router(dependencies: Sequence[Depends]) -> APIRouter:
@router.post("/api/v1/embedding-switches", status_code=202, response_model=None)
async def start_embedding_switch(value: EmbeddingSwitchStartRequest, request: Request) -> Any:
try:
job = await _service(request).start_switch(
value.profile_id,
expected_revision=value.expected_revision,
plan_token=value.plan_token,
requested_by=value.requested_by,
)
except EmbeddingSwitchError as error:
return refusal(error.code, plan=error.plan)
# One lifecycle lock orders mode/evolve/backfill and embedding
# maintenance. Without it each endpoint could pass its own preflight
# and then begin concurrently.
async with request.app.state.memory_mode_lock:
evolve = request.app.state.evolve_service
backfill = request.app.state.memory_graph_backfill
if evolve.running() or backfill.running:
return refusal("memory_lifecycle_busy")
try:
job = await _service(request).start_switch(
value.profile_id,
expected_revision=value.expected_revision,
plan_token=value.plan_token,
requested_by=value.requested_by,
)
except EmbeddingSwitchError as error:
return refusal(error.code, plan=error.plan)
return job.model_dump(mode="json")
@router.get("/api/v1/embedding-switches/current")
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.10"
HY_MEMORY_VERSION = "2.0.14"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -503,7 +503,7 @@ def _ensure_coding_schema(path: Path) -> None:
CodingMemoryStore,
)
# Relies on the pinned SDK's internals (hy-memory==2.0.10 is pinned exactly).
# Relies on the pinned SDK's internals (hy-memory==2.0.14 is pinned exactly).
store = CodingMemoryStore.__new__(CodingMemoryStore)
store._db_path = str(path)
store._conn = None
@@ -9,7 +9,6 @@ from __future__ import annotations
import asyncio
import hashlib
import logging
import os
import threading
from collections.abc import Callable, Coroutine, Iterable, Iterator, Mapping
@@ -48,8 +47,10 @@ from one_memory.local_embedding import LocalEmbedder
from one_memory.log_file import confine_hy_sdk_logging
from one_memory.memory_features import (
FEATURES,
FEATURES_BY_NAME,
PARAMETERS,
SETTINGS_BY_NAME,
FeatureOverrides,
SettingValue,
feature_overrides,
)
from one_memory.model_profiles import (
@@ -58,8 +59,6 @@ from one_memory.model_profiles import (
disable_thinking_params,
)
logger = logging.getLogger(__name__)
_RUNTIME_LOCK = threading.RLock()
BUILTIN_RETRIEVAL_PROFILE_ID = "builtin-retrieval"
_LOCAL_ONNX_PLACEHOLDER_URL = "http://127.0.0.1/local-onnx"
@@ -324,21 +323,27 @@ def sdk_relevance_floor() -> float:
return min(1.0, max(0.0, value))
def sdk_feature_defaults() -> dict[str, bool]:
"""Each `memory_features` feature's SDK default, as a fresh config has it.
def sdk_feature_defaults() -> dict[str, SettingValue]:
"""Each `memory_features` setting's SDK default, as a fresh config has it.
That is the SDK's code default unless the process environment sets the
SDK's own variable (`FeatureSpec.env`), exactly what an unset feature
leaves on a client.
"""
config = MemoryConfig()
return {
defaults: dict[str, SettingValue] = {
spec.name: bool(getattr(getattr(config, spec.section), spec.field)) for spec in FEATURES
}
for parameter in PARAMETERS:
defaults[parameter.name] = float(
getattr(getattr(config, parameter.section), parameter.field)
)
return defaults
def apply_sdk_features(
config: MemoryConfig, features: Mapping[str, bool] | Iterable[tuple[str, bool]]
config: MemoryConfig,
features: Mapping[str, SettingValue] | Iterable[tuple[str, SettingValue]],
) -> MemoryConfig:
"""Set each feature a person chose on its `MemoryConfig` field.
@@ -349,14 +354,11 @@ def apply_sdk_features(
"""
values = dict(features.items() if isinstance(features, Mapping) else features)
for name, value in feature_overrides(values):
spec = FEATURES_BY_NAME[name]
spec = SETTINGS_BY_NAME[name]
setattr(getattr(config, spec.section), spec.field, value)
return config
_LANGUAGE_UNSUPPORTED_LOGGED = False
class LanguageKwargs(TypedDict, total=False):
language: str
@@ -371,22 +373,14 @@ def language_kwargs(language: str | None) -> LanguageKwargs:
def apply_sdk_language(config: MemoryConfig, language: str | None) -> MemoryConfig:
"""Set the SDK prompt language, when there is one and the SDK has the field.
"""Set the SDK prompt language (`MemoryConfig.language`), when there is one.
None (zh-Hans, or no choice) sets nothing: the SDK detects zh/en as it
always did. An SDK without `MemoryConfig.language` keeps working without
it; that is logged once at debug level. Like a feature, the language never
touches `vector_store`, so the collection identity never depends on it.
None (no language set) sets nothing: the SDK detects zh/en as it always
did. Every display language, zh-Hans included, is set explicitly. Like a feature, the language never touches `vector_store`, so
the collection identity never depends on it.
"""
global _LANGUAGE_UNSUPPORTED_LOGGED
if language is None:
return config
if not hasattr(config, "language"):
if not _LANGUAGE_UNSUPPORTED_LOGGED:
_LANGUAGE_UNSUPPORTED_LOGGED = True
logger.debug("hy_sdk_language_unsupported language=%s", language)
return config
config.language = language
if language is not None:
config.language = language
return config
@@ -397,7 +391,7 @@ def build_sdk_config(
*,
api_key: str | None,
data_dir: Path,
features: Mapping[str, bool] | Iterable[tuple[str, bool]] = (),
features: Mapping[str, SettingValue] | Iterable[tuple[str, SettingValue]] = (),
language: str | None = None,
) -> MemoryConfig:
"""Map a model snapshot and its embedding space to an SDK configuration.
@@ -75,6 +75,9 @@ class HySdkMemoryRecord(BaseModel):
# Epoch seconds. `updated_at` falls back to `created_at`.
created_at: int | None = None
updated_at: int | None = None
# Actual scene=all ordering timestamp before the public fallback above.
# Internal-only: callers keep the existing SdkMemoryRecord wire shape.
sort_updated_at: int | None = Field(default=None, exclude=True, repr=False)
@classmethod
def from_sdk(cls, raw: object) -> HySdkMemoryRecord:
@@ -95,6 +98,10 @@ class HySdkMemoryRecord(BaseModel):
if created_at is None:
created_at = gmt_created
updated_at = _epoch(raw.get("updated_at"))
modified_at = _epoch(raw.get("gmt_modified"))
sort_updated_at = (
modified_at if modified_at is not None else updated_at
)
return cls(
memory_id=str(raw["memory_id"]),
content=str(raw["content"]),
@@ -116,6 +123,7 @@ class HySdkMemoryRecord(BaseModel):
gmt_created=gmt_created,
created_at=created_at,
updated_at=updated_at if updated_at is not None else created_at,
sort_updated_at=sort_updated_at,
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_record_shape_unknown") from error
@@ -180,6 +188,7 @@ class HySdkMemoryRecord(BaseModel):
solution=solution,
created_at=created_at,
updated_at=updated_at if updated_at is not None else created_at,
sort_updated_at=updated_at,
)
except (KeyError, TypeError, ValueError) as error:
raise HySdkReadError("hy_memory_record_shape_unknown") from error
@@ -436,6 +436,11 @@ def root_identity_matches(root: Path, expected: str) -> bool:
return _root_identity_matches_details(resolved, details, expected)
def resolved_source_root(root: Path) -> Path:
"""The real directory a root names (symlinks followed), as a grant records it."""
return _root_details(root)[0]
def _check_root_identity(root: Path, expected: str | None) -> None:
if expected is not None and not root_identity_matches(root, expected):
raise SourceAccessError(SourceAccessErrorCode.SOURCE_IDENTITY_CHANGED)
@@ -1,6 +1,6 @@
import hashlib
import os
from collections.abc import AsyncIterator, Iterator
from collections.abc import AsyncIterator, Callable, Iterator
from dataclasses import dataclass
from pathlib import Path
@@ -27,7 +27,8 @@ def discovered_off_loop(
*,
suffixes: frozenset[str],
max_targets: int,
excluded_top_directories: frozenset[str] = frozenset(),
excluded_directory_names: frozenset[str] = frozenset(),
directory_predicate: Callable[[Path], bool] | None = None,
) -> AsyncIterator[SourceFileDiscoveryEvent]:
"""The same walk, for callers that have an event loop to keep free.
@@ -42,7 +43,8 @@ def discovered_off_loop(
grant,
suffixes=suffixes,
max_targets=max_targets,
excluded_top_directories=excluded_top_directories,
excluded_directory_names=excluded_directory_names,
directory_predicate=directory_predicate,
),
scope_key=grant.source_id,
)
@@ -53,15 +55,18 @@ def discover_source_files(
*,
suffixes: frozenset[str],
max_targets: int,
excluded_top_directories: frozenset[str] = frozenset(),
excluded_directory_names: frozenset[str] = frozenset(),
directory_predicate: Callable[[Path], bool] | None = None,
) -> Iterator[SourceFileDiscoveryEvent]:
"""Metadata only: one lstat per candidate, no open and no per-file grant lookup.
The grant and root identity are checked once before the walk. Each listed
file keeps the grant's check, which runs again when that file is opened;
the open itself refuses symlinks and any inode other than the one listed.
`excluded_top_directories` are directories directly under the root that
are never entered, such as another user's Hermes profiles.
`excluded_directory_names` are directory names never entered, at any
depth under the root, such as the `profiles` container that holds other
users' Hermes profiles. Any depth, because the granted root may sit above
the Hermes home (`~/.hermes` inside it is then `.hermes/profiles`).
"""
grant.check_root()
inspected = 0
@@ -69,12 +74,16 @@ def discover_source_files(
grant.root,
followlinks=False,
):
at_root = Path(directory) == grant.root
current = Path(directory)
child_directories[:] = sorted(
child
for child in child_directories
if not (Path(directory) / child).is_symlink()
and not (at_root and child in excluded_top_directories)
if not (current / child).is_symlink()
and child not in excluded_directory_names
and (
directory_predicate is None
or directory_predicate((current / child).relative_to(grant.root))
)
)
for filename in sorted(filenames):
if Path(filename).suffix.lower() not in suffixes:
+188 -8
View File
@@ -1,6 +1,7 @@
import logging
import re
import sqlite3
from collections.abc import Iterable
from contextlib import AbstractContextManager, nullcontext
from datetime import UTC, datetime
from enum import StrEnum
@@ -18,13 +19,27 @@ from one_memory.imports.adapters import ScanPhase, SourceCursor
#: the drift would only surface on a fresh database.
#:
#: This runs on SQLite today, so the SQLite rendering is used.
from one_memory.memory_server.migrations import import_jobs_ddl
from one_memory.memory_server.migrations import (
import_job_lookup_indexes_ddl,
import_jobs_ddl,
)
logger = logging.getLogger("one_memory.imports.jobs")
JOB_SCHEMA = import_jobs_ddl("sqlite")
#: Migration 63, applied by the store as well so a running home gets it.
JOB_LOOKUP_INDEXES = import_job_lookup_indexes_ddl("sqlite")
MAX_LIVE_BACKLOG_BYTES = 256 * 1024
#: Finished live syncs kept per (user, source). Live sync opens a job for every
#: change it imports, and nothing removed them: hundreds a day on an active
#: machine. The App shows the latest job per source and the active ones; twenty is
#: that plus enough recent history to read a failure back.
SYNC_JOBS_KEPT_PER_SOURCE = 20
#: Rows one pruning transaction may delete, so it holds the single writer for
#: a few milliseconds at most however large the table grew.
PRUNE_BATCH_ROWS = 200
class ImportStatus(StrEnum):
PENDING = "pending"
@@ -36,6 +51,12 @@ class ImportStatus(StrEnum):
CANCELLED = "cancelled"
#: States no job ever leaves (`ImportJobStore._TRANSITIONS`). Failed and
#: interrupted are not among them: both can be resumed, and both still hold the
#: source's one active slot.
FINISHED_STATUSES = (ImportStatus.COMPLETED, ImportStatus.CANCELLED)
class ImportJob(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
@@ -272,6 +293,9 @@ class ImportJobStore:
WHERE status IN ('pending', 'running', 'paused', 'interrupted', 'failed')
"""
)
for statement in JOB_LOOKUP_INDEXES.split(";"):
if statement.strip():
connection.execute(statement)
def create(
self,
@@ -334,8 +358,8 @@ class ImportJobStore:
read again.
"""
opened: list[ImportJob] = []
for job in self.list_jobs(status=ImportStatus.FAILED):
if job.operation != "history" or job.stored != 0:
for job in self.find_jobs(operation="history", statuses=(ImportStatus.FAILED,)):
if job.stored != 0:
continue
reasons = {issue.reason for issue in self.list_issues(job.job_id)}
if not reasons or not reasons <= self._UNPUBLISHED_SNAPSHOT_FAILURES:
@@ -799,16 +823,172 @@ class ImportJobStore:
return self.require(job_id)
def list_jobs(self, *, status: ImportStatus | None = None) -> list[ImportJob]:
"""Every job, oldest first. Hot paths ask `find_jobs` a narrower question."""
return self.find_jobs(statuses=(status,) if status is not None else None)
def find_jobs(
self,
*,
user_id: str | None = None,
source_id: str | None = None,
operation: Literal["history", "sync"] | None = None,
statuses: Iterable[ImportStatus] | None = None,
newest_first: bool = False,
limit: int | None = None,
) -> list[ImportJob]:
"""The jobs matching every given filter, oldest first unless asked otherwise.
The filters are SQL, served by `import_jobs_lane` / `import_jobs_status`
(migration 63). Reading the whole table and filtering here was what
three bulk threads and three request threads were all inside, for tens
of seconds, while the loop waited on them.
"""
clauses: list[str] = []
parameters: list[str | int] = []
if user_id is not None:
clauses.append("user_id = ?")
parameters.append(user_id)
if source_id is not None:
clauses.append("source_id = ?")
parameters.append(source_id)
if operation is not None:
clauses.append("operation = ?")
parameters.append(operation)
if statuses is not None:
wanted = tuple(dict.fromkeys(ImportStatus(status) for status in statuses))
if not wanted:
return []
clauses.append(f"status IN ({', '.join('?' for _ in wanted)})")
parameters.extend(wanted)
query = "SELECT * FROM import_jobs"
parameters: tuple[str, ...] = ()
if status is not None:
query += " WHERE status = ?"
parameters = (status,)
query += " ORDER BY created_at, job_id"
if clauses:
query += " WHERE " + " AND ".join(clauses)
query += (
" ORDER BY created_at DESC, job_id DESC"
if newest_first
else " ORDER BY created_at, job_id"
)
if limit is not None:
if limit < 1:
raise ValueError("limit must be positive")
query += " LIMIT ?"
parameters.append(limit)
with self._connect() as connection:
rows = connection.execute(query, parameters).fetchall()
return [self._to_job(row) for row in rows]
def latest_job(
self,
*,
user_id: str,
source_id: str,
operation: Literal["history", "sync"] | None = None,
statuses: Iterable[ImportStatus] | None = None,
) -> ImportJob | None:
jobs = self.find_jobs(
user_id=user_id,
source_id=source_id,
operation=operation,
statuses=statuses,
newest_first=True,
limit=1,
)
return jobs[0] if jobs else None
def prune_finished_sync_jobs(
self,
*,
keep: int = SYNC_JOBS_KEPT_PER_SOURCE,
max_rows: int = PRUNE_BATCH_ROWS,
) -> int:
"""Delete finished live-sync jobs beyond the newest `keep` of each source.
One bounded transaction: at most `max_rows` jobs, with their issues and
skip counts. Returns how many jobs went; fewer than `max_rows` means
nothing is left to prune, so calling again until then finishes the job.
Only `sync` jobs in a state nothing leaves (completed, cancelled) are
ever deleted. Failed and interrupted jobs hold the source's active slot
and can be resumed; history jobs are named by connection receipts, by
the onboarding gate and by the first-import review, and there are a
handful per source, so they are all kept. Cursors live in their own
tables keyed by file, not by job, so a pruned job takes no position
with it. The newest job of every source is always among those kept,
which is the one the App and live sync health name.
"""
if keep < 1:
raise ValueError("keep must be positive")
if max_rows < 1:
raise ValueError("max_rows must be positive")
finished = tuple(FINISHED_STATUSES)
placeholders = ", ".join("?" for _ in finished)
deleted = 0
with self._connect(writing=True) as connection:
lanes = connection.execute(
f"""
SELECT user_id, source_id, COUNT(*) AS finished
FROM import_jobs
WHERE operation = 'sync' AND status IN ({placeholders})
GROUP BY user_id, source_id
HAVING COUNT(*) > ?
""",
(*finished, keep),
).fetchall()
for lane in lanes:
if deleted >= max_rows:
break
# The oldest job still kept: everything strictly older goes.
boundary = connection.execute(
f"""
SELECT created_at, job_id FROM import_jobs
WHERE user_id = ? AND source_id = ? AND operation = 'sync'
AND status IN ({placeholders})
ORDER BY created_at DESC, job_id DESC
LIMIT 1 OFFSET ?
""",
(lane["user_id"], lane["source_id"], *finished, keep - 1),
).fetchone()
if boundary is None:
continue
victims = [
str(row["job_id"])
for row in connection.execute(
f"""
SELECT job_id FROM import_jobs
WHERE user_id = ? AND source_id = ? AND operation = 'sync'
AND status IN ({placeholders})
AND (created_at < ? OR (created_at = ? AND job_id < ?))
ORDER BY created_at, job_id
LIMIT ?
""",
(
lane["user_id"],
lane["source_id"],
*finished,
boundary["created_at"],
boundary["created_at"],
boundary["job_id"],
max_rows - deleted,
),
)
]
if not victims:
continue
marks = ", ".join("?" for _ in victims)
# Foreign keys cascade these too; deleting them by name keeps
# the outcome the same on a connection that has them off.
connection.execute(f"DELETE FROM import_issues WHERE job_id IN ({marks})", victims)
connection.execute(
f"DELETE FROM import_skip_reason_counts WHERE job_id IN ({marks})", victims
)
removed = connection.execute(
f"DELETE FROM import_jobs WHERE job_id IN ({marks}) "
f"AND operation = 'sync' AND status IN ({placeholders})",
(*victims, *finished),
)
deleted += removed.rowcount
return deleted
def get_source_cursors(
self,
*,
@@ -28,7 +28,7 @@ from one_memory.imports.adapters import (
)
from one_memory.imports.jobs import ImportJob, ImportJobStore, ImportStatus
from one_memory.imports.supervisor import ImportSupervisor
from one_memory.user_scope import DEFAULT_USER_ID, lane_key, split_lane
from one_memory.user_scope import DEFAULT_USER_ID, split_lane
#: How long an automatic retry will wait for a wedged worker to notice cancel.
ABANDON_TIMEOUT_SECONDS = 5.0
@@ -249,13 +249,15 @@ class PollingLiveSyncSupervisor:
),
)
return None
source_jobs = await in_the_background(self.job_store.list_jobs)
# This lane's unfinished jobs only: the newest unfinished sync, and
# whatever history shares its cursor. Asked in SQL; listing every
# job ever run and filtering here was the poll's biggest read.
source_jobs = await in_the_background(self._lane_jobs, source_id)
active_sync = next(
(
job
for job in reversed(source_jobs)
if lane_key(job.user_id, job.source_id) == source_id
and job.operation == "sync"
if job.operation == "sync"
and job.status
in {
ImportStatus.PENDING,
@@ -275,7 +277,9 @@ class PollingLiveSyncSupervisor:
active_sync.job_id
):
attempted_at = datetime.now(UTC)
self.import_supervisor.enqueue(active_sync.job_id)
await self.import_supervisor.enqueue_off_loop(
active_sync.job_id, job=active_sync
)
if not wait_for_completion:
self._observe_sync(source_id, attempted_at, active_sync.job_id)
return active_sync
@@ -295,22 +299,20 @@ class PollingLiveSyncSupervisor:
# by a process restart between yield and claim. Enqueue is
# the operation that actually creates the task.
attempted_at = datetime.now(UTC)
self.import_supervisor.resume(active_sync.job_id)
self.import_supervisor.enqueue(active_sync.job_id)
resumed = await self.import_supervisor.resume_off_loop(active_sync.job_id)
await self.import_supervisor.enqueue_off_loop(active_sync.job_id, job=resumed)
completed = await self.import_supervisor.wait(active_sync.job_id)
self._record_outcome(source_id, attempted_at, completed)
return completed
competing = [
job
for job in source_jobs
if lane_key(job.user_id, job.source_id) == source_id
and job.operation != "sync"
if job.operation != "sync"
and job.status
in {ImportStatus.PENDING, ImportStatus.RUNNING, ImportStatus.INTERRUPTED}
]
bounded_history_incomplete = any(
lane_key(job.user_id, job.source_id) == source_id
and job.operation == "history"
job.operation == "history"
and job.history_after is not None
and job.status is not ImportStatus.COMPLETED
for job in source_jobs
@@ -354,6 +356,38 @@ class PollingLiveSyncSupervisor:
# automatically after the realtime slice completes.
return await self._run_sync(source_id, wait_for_completion=wait_for_completion)
def _lane_jobs(self, source_id: str) -> list[ImportJob]:
"""The lane's jobs `_sync` decides on, oldest first.
Everything not completed, of either operation: an unfinished sync to
pick up, a history that competes for the cursor, and a bounded history
that has not completed yet (which includes a cancelled one). Finished
syncs, the rows that grow without end, are never read.
"""
user_id, bare_source_id = split_lane(source_id)
unfinished = tuple(
status for status in ImportStatus if status is not ImportStatus.COMPLETED
)
return [
*self.job_store.find_jobs(
user_id=user_id,
source_id=bare_source_id,
operation="history",
statuses=unfinished,
),
*self.job_store.find_jobs(
user_id=user_id,
source_id=bare_source_id,
operation="sync",
statuses=(
ImportStatus.PENDING,
ImportStatus.RUNNING,
ImportStatus.PAUSED,
ImportStatus.INTERRUPTED,
),
),
]
async def _has_work(self, source_id: str) -> bool:
"""Ask the source whether a scan would write anything, before taking a job out.
@@ -452,7 +486,7 @@ class PollingLiveSyncSupervisor:
),
)
self._active_jobs[source_id] = job.job_id
self.import_supervisor.enqueue(job.job_id)
await self.import_supervisor.enqueue_off_loop(job.job_id, job=job)
if not wait_for_completion:
self._observe_sync(source_id, attempted_at, job.job_id)
return job
@@ -557,7 +591,11 @@ class PollingLiveSyncSupervisor:
async def _poll_source(self, source_id: str) -> None:
while True:
await asyncio.sleep(self.interval_seconds)
# A watched source is normally triggered by its filesystem event.
# Its timer is only a missed-event backstop; recursively walking
# every multi-gigabyte host root once a minute starved unrelated
# Core reads even when no source had changed.
await asyncio.sleep(self._poll_interval(source_id))
try:
await self.trigger(source_id)
except asyncio.CancelledError:
@@ -574,6 +612,11 @@ class PollingLiveSyncSupervisor:
except Exception as again:
self._note_internal_error(source_id, again)
def _poll_interval(self, source_id: str) -> float:
if source_id in (self._watched_sources or ()):
return max(self.interval_seconds, 15 * 60)
return self.interval_seconds
def _note_worker_still_running(self, source_id: str) -> None:
current = self.health(source_id)
self._set_health(
@@ -23,6 +23,7 @@ from pathlib import Path
from watchfiles import awatch
from one_memory.background_work import in_the_background
from one_memory.diagnostic_log import diagnostic_event
from one_memory.imports.jobs import ImportStatus
from one_memory.imports.live_sync import PollingLiveSyncSupervisor
@@ -90,7 +91,8 @@ class FileWatchLiveSync:
# turn that caused setup. The watcher cannot observe a change
# from before it existed, so immediately run the same bounded
# trigger once instead of waiting for the 60-second backstop.
self._queue_changes(set(self.roots.values()))
# The roots are this watcher's own sources: nothing to resolve.
self._queue_sources(tuple(self.roots), count=len(self.roots))
async def shutdown(self) -> None:
task = self._task
@@ -139,28 +141,54 @@ class FileWatchLiveSync:
observed_at: datetime | None = None,
) -> tuple[str, ...]:
"""Queue affected sources and wait for their coalesced work to settle."""
workers = self._queue_changes(changed, observed_at=observed_at)
workers = await self._queue_changes(changed, observed_at=observed_at)
await asyncio.gather(*(asyncio.shield(worker) for worker in workers.values()))
return tuple(workers)
def _queue_changes(
def sources_for(self, changed: set[Path]) -> tuple[str, ...]:
"""The sources owning `changed`, in path order. Resolves every path."""
return tuple(
dict.fromkeys(
source
for source in (self.source_for(path) for path in sorted(changed))
if source is not None
)
)
async def _queue_changes(
self, changed: set[Path], *, observed_at: datetime | None = None
) -> dict[str, asyncio.Task[None]]:
"""Attribute a batch of changed paths to sources and queue them.
Attributing resolves each path, a `stat` per component, and a busy
host reports hundreds of paths at once. On the loop that was the main
thread inside `source_for` while nothing else was answered, so the
resolving happens off it and only the queueing here.
"""
if self._stop.is_set():
return {}
observed = observed_at or datetime.now(UTC)
sources = await in_the_background(self.sources_for, changed)
return self._queue_sources(sources, count=len(changed), observed_at=observed)
def _queue_sources(
self,
sources: tuple[str, ...],
*,
count: int,
observed_at: datetime | None = None,
) -> dict[str, asyncio.Task[None]]:
if self._stop.is_set():
return {}
observed = observed_at or datetime.now(UTC)
workers: dict[str, asyncio.Task[None]] = {}
for source_id in dict.fromkeys(
source
for source in (self.source_for(path) for path in sorted(changed))
if source is not None
):
for source_id in sources:
diagnostic_event(
level="debug",
component="import.live_watch",
event="live_sync.hook_received",
outcome="queued",
details={"source_id": source_id, "count": len(changed)},
details={"source_id": source_id, "count": count},
)
self._pending[source_id] = max(self._pending.get(source_id, observed), observed)
worker = self._workers.get(source_id)
@@ -239,7 +267,7 @@ class FileWatchLiveSync:
stop_event=self._stop,
ignore_permission_denied=True,
):
self._queue_changes({Path(path) for _change, path in changes})
await self._queue_changes({Path(path) for _change, path in changes})
except asyncio.CancelledError:
raise
except Exception:
@@ -209,6 +209,7 @@ class JsonlSourceAdapter:
file_predicate: Callable[[AuthorizedSourceFile], bool] | None = None,
suffixes: frozenset[str] = frozenset({".jsonl"}),
header_selector: Callable[[dict[str, JsonValue]], bool] | None = None,
header_gate: Callable[[dict[str, JsonValue] | None], bool] | None = None,
) -> None:
if descriptor.source_id != grant.source_id:
raise ValueError("Descriptor and authorized root source IDs must match")
@@ -219,8 +220,15 @@ class JsonlSourceAdapter:
self.file_predicate = file_predicate or (lambda _: True)
self.suffixes = suffixes
self.header_selector = header_selector
# False refuses the whole file before any record is read. It reports
# and leaves no cursor, so a release that learns the format reads the
# file from the top instead of finding it already behind a checkpoint.
self.header_gate = header_gate
self._header_cache: dict[str, tuple[str, dict[str, JsonValue] | None]] = {}
self._history_listing_cache: SourceListing | None = None
self._probe_listing_cache: tuple[
dict[str, SourceCursor], int | None, SourceListing
] | None = None
@property
def descriptor(self) -> SourceDescriptor:
@@ -276,6 +284,11 @@ class JsonlSourceAdapter:
predicate=self.file_predicate,
cursors=options.cursors,
)
self._probe_listing_cache = (
dict(options.cursors),
options.max_scan_targets,
listing,
)
return bool(listing.changed or listing.failures)
async def live_baseline(self) -> dict[str, SourceCursor]:
@@ -310,9 +323,11 @@ class JsonlSourceAdapter:
async for event in self._scan_frozen(options):
yield event
return
listing = await asyncio.to_thread(
self._history_listing if options.reuse_listing else self._fresh_listing,
options,
listing = (
await asyncio.to_thread(self._history_listing, options)
if options.reuse_listing
else self._take_probe_listing(options)
or await asyncio.to_thread(self._fresh_listing, options)
)
for failure in listing.failures:
yield ScanFailure(reason_code=discovery_failure_code(failure))
@@ -344,6 +359,16 @@ class JsonlSourceAdapter:
self._history_listing_cache = listing
return listing
def _take_probe_listing(self, options: ScanOptions) -> "SourceListing | None":
cached = self._probe_listing_cache
self._probe_listing_cache = None
if cached is None:
return None
cursors, max_scan_targets, listing = cached
if cursors != options.cursors or max_scan_targets != options.max_scan_targets:
return None
return listing
def _fresh_listing(self, options: ScanOptions) -> "SourceListing":
return list_changed_files(
self.grant,
@@ -371,6 +396,9 @@ class JsonlSourceAdapter:
source_file, checkpoint, header = await asyncio.to_thread(
self._prepare, listed, options
)
if self.header_gate is not None and not self.header_gate(header):
yield ScanFailure(reason_code=ScanIssueCode.UNSUPPORTED_SCHEMA)
continue
_log.debug(
"import_file_read source=%s file=%d/%d file_bytes=%d planned_bytes=%d",
self.descriptor.source_id,
@@ -574,6 +602,9 @@ class JsonlSourceAdapter:
# resumed scan starts past the header, and the records after it need
# it just as much as the first batch did.
header = await _read_header(reader, source_file, self.header_selector)
if self.header_gate is not None and not self.header_gate(header):
yield ScanFailure(reason_code=ScanIssueCode.UNSUPPORTED_SCHEMA)
continue
last: SourceCursor = _start_cursor(source_file, options, checkpoint)
async for raw in iterate_in_worker(
partial(reader, source_file, checkpoint=checkpoint),
@@ -9,6 +9,7 @@ from collections.abc import AsyncIterator, Iterator
from dataclasses import dataclass, replace
from datetime import UTC, datetime, timedelta, timezone
from functools import partial
from pathlib import Path
from pydantic import JsonValue
@@ -175,6 +176,9 @@ class CursorUnifiedSourceAdapter:
self.grant = grant
self.source_version = source_version
self._history_listing_cache: _CursorListing | None = None
self._probe_listing_cache: tuple[
dict[str, SourceCursor], int | None, _CursorListing
] | None = None
@property
def descriptor(self) -> SourceDescriptor:
@@ -207,6 +211,20 @@ class CursorUnifiedSourceAdapter:
state = await self.install_state()
return min(state.session_count, limit) if limit is not None else state.session_count
async def has_changes(self, options: ScanOptions) -> bool:
listing = await asyncio.to_thread(self._fresh_listing, options)
self._probe_listing_cache = (
dict(options.cursors),
options.max_scan_targets,
listing,
)
return bool(
listing.transcripts
or listing.stores
or listing.legacy
or listing.failures
)
async def live_baseline(self) -> dict[str, SourceCursor]:
"""Capture Cursor's current tail so the realtime lane never backfills history."""
listing = await asyncio.to_thread(
@@ -222,9 +240,11 @@ class CursorUnifiedSourceAdapter:
async for event in self._scan_frozen(options):
yield event
return
listing = await asyncio.to_thread(
self._history_listing if options.reuse_listing else self._fresh_listing,
options,
listing = (
await asyncio.to_thread(self._history_listing, options)
if options.reuse_listing
else self._take_probe_listing(options)
or await asyncio.to_thread(self._fresh_listing, options)
)
for failure in listing.failures:
yield ScanFailure(reason_code=discovery_failure_code(failure))
@@ -329,6 +349,16 @@ class CursorUnifiedSourceAdapter:
self._history_listing_cache = listing
return listing
def _take_probe_listing(self, options: ScanOptions) -> "_CursorListing | None":
cached = self._probe_listing_cache
self._probe_listing_cache = None
if cached is None:
return None
cursors, max_scan_targets, listing = cached
if cursors != options.cursors or max_scan_targets != options.max_scan_targets:
return None
return listing
def _fresh_listing(self, options: ScanOptions) -> "_CursorListing":
return _list_cursor_files(
self.grant,
@@ -376,6 +406,7 @@ class CursorUnifiedSourceAdapter:
self.grant,
suffixes=frozenset({".db"}),
max_targets=options.max_scan_targets or 100_000,
directory_predicate=_cursor_data_directory,
):
if isinstance(discovered, SourceFileDiscoveryFailure):
yield ScanFailure(reason_code=discovery_failure_code(discovered))
@@ -477,7 +508,10 @@ def _list_cursor_files(
legacy: list[AuthorizedSourceFile] = []
failures: list[SourceFileDiscoveryFailure] = []
for item in discover_source_files(
grant, suffixes=frozenset({".jsonl", ".db", ".vscdb"}), max_targets=max_targets
grant,
suffixes=frozenset({".jsonl", ".db", ".vscdb"}),
max_targets=max_targets,
directory_predicate=_cursor_data_directory,
):
if isinstance(item, SourceFileDiscoveryFailure):
failures.append(item)
@@ -586,6 +620,25 @@ def _is_cursor_data_file(source_file: AuthorizedSourceFile) -> bool:
)
def _cursor_data_directory(relative_path: Path) -> bool:
"""Enter only Cursor directories that can contain conversation authority.
The granted root also contains extensions, skills, caches, logs and agent
artifacts. Recursively walking all of them every live-sync poll made the
reader monopolize Core even though none can pass the file predicates.
"""
parts = relative_path.parts
if not parts:
return False
if parts[0] in {"chats", "acp-sessions"}:
return True
if parts[0] == "projects":
return len(parts) <= 2 or parts[2] == "agent-transcripts"
if parts[0] == "User":
return len(parts) == 1 or parts[1] in {"globalStorage", "workspaceStorage"}
return False
def _is_transcript_file(source_file: AuthorizedSourceFile) -> bool:
parts = source_file.relative_path.parts
return source_file.relative_path.suffix == ".jsonl" and "agent-transcripts" in parts
@@ -634,6 +687,7 @@ def _select_transcript_files(
grant,
suffixes=frozenset({".jsonl"}),
max_targets=100_000,
directory_predicate=_cursor_data_directory,
):
if not isinstance(item, AuthorizedSourceFile) or not _is_transcript_file(item):
continue
@@ -654,6 +708,7 @@ def _subagent_conversation_ids(grant: AuthorizedSourceRoot) -> set[str]:
grant,
suffixes=frozenset({".jsonl"}),
max_targets=100_000,
directory_predicate=_cursor_data_directory,
)
if isinstance(item, AuthorizedSourceFile)
and _is_transcript_file(item)
@@ -1,3 +1,4 @@
import re
from pathlib import PurePosixPath
from pydantic import JsonValue
@@ -37,6 +38,19 @@ DEEPSEEK_HARNESS_DESCRIPTOR = SourceDescriptor(
),
)
#: One session is one file, and the name carries the format generation it was
#: written in: `session.jsonl[.zstd]` for the first format, then
#: `session.v3.jsonl.zstd` and `session.v4.jsonl.zstd`. The version segment is
#: part of the published on-disk contract, so discovery must accept it — the
#: earlier exact-name match imported none of the versioned files.
SESSION_FILE = re.compile(r"^session(?:\.v[0-9]+)?\.jsonl(?:\.zstd)?$")
#: Session generations this parser has been verified against: the first format
#: (header `version: 0`), v3 and v4. Discovery is intentionally wider than
#: parsing: a future generation is found by pattern and then skipped from its
#: header, so it is reported as unsupported instead of silently vanishing.
SUPPORTED_SESSION_VERSIONS = frozenset({0, 3, 4})
def create_deepseek_harness_adapter(
grant: AuthorizedSourceRoot,
@@ -49,25 +63,43 @@ def create_deepseek_harness_adapter(
grant=grant,
parser=parse_deepseek_harness_record,
source_version=source_version,
file_predicate=lambda source_file: (
source_file.relative_path.name in {"session.jsonl", "session.jsonl.zstd"}
),
file_predicate=lambda source_file: bool(SESSION_FILE.match(source_file.relative_path.name)),
suffixes=frozenset({".jsonl", ".zstd"}),
header_selector=_is_session_header,
header_gate=_supported_session,
)
def _is_session_header(record: dict[str, JsonValue]) -> bool:
"""Harness writes the session's `cwd` once, in the first line only.
Verified against a session this machine produced: the header is
`{"type":"session","version":0,"id":...,"cwd":"/private/tmp/dsh-verify"}`
and no later record repeats it. Read per-record, `cwd` is therefore always
absent and every turn imports with no workspace.
Verified against sessions this machine produced: the header is
`{"type":"session","version":0|4,"id":...,"cwd":...}` and no later record
repeats it. Read per-record, `cwd` is therefore always absent and every turn
imports with no workspace.
"""
return record.get("type") == "session"
def _supported_session(header: dict[str, JsonValue] | None) -> bool:
"""Whether the session's declared format generation is one we can read.
A missing header, or a header without `version`, is the first format — the
earliest files this adapter already read. Any declared version outside
`SUPPORTED_SESSION_VERSIONS` is refused rather than interpreted with today's
field rules: a future format may change field semantics, and importing it
wrongly is harder to undo than not importing it yet.
"""
if header is None:
return True
version = header.get("version")
if version is None:
return True
if not isinstance(version, int) or isinstance(version, bool):
return False
return version in SUPPORTED_SESSION_VERSIONS
def parse_deepseek_harness_record(
record: dict[str, JsonValue],
context: SourceRecordContext,
@@ -80,6 +112,11 @@ def parse_deepseek_harness_record(
)
event_type = record.get("type")
data = as_record(record.get("data"))
if not _supported_session(context.header):
return ParsedSkip(
reason_code=ScanIssueCode.UNSUPPORTED_SCHEMA,
conversation_id=conversation_id,
)
if event_type == "session":
return ParsedSkip(reason_code=ScanIssueCode.UNKNOWN_RECORD)
if data is None:
@@ -148,11 +185,18 @@ def _tool_message(
"""`tool/call` carries name and arguments; `tool/result` a user-role message.
The same call also appears as a `tool-call` block in `assistant/message`;
only the event is read, so it is stored once.
only the event is read, so it is stored once. The first format and v3 wrap
the result in `tool-result` blocks; v4 writes plain `text` blocks, so both
shapes are read through the same clipping and credential-scrubbing path.
"""
message = as_record(data.get("message"))
if record.get("type") == "tool/result" and message is not None:
text = tool_blocks_evidence(message.get("content"), block_types=frozenset({"tool-result"}))
if text is None:
text = tool_blocks_evidence(
[{"type": "tool-result", "content": message.get("content")}],
block_types=frozenset({"tool-result"}),
)
message_id = first_string(message.get("id"), data.get("callId"))
else:
text = tool_evidence_text(data)
@@ -1,9 +1,9 @@
import json
import re
import sqlite3
from collections.abc import AsyncIterator, Iterator
from collections.abc import AsyncIterator, Iterable, Iterator
from functools import partial
from pathlib import PurePosixPath
from pathlib import Path, PurePosixPath
from pydantic import JsonValue
@@ -88,7 +88,7 @@ class HermesSourceAdapter:
self.grant,
suffixes=frozenset({".jsonl", ".db"}),
max_targets=10_000,
excluded_top_directories=frozenset({HERMES_PROFILES_DIRECTORY}),
excluded_directory_names=frozenset({HERMES_PROFILES_DIRECTORY}),
):
if isinstance(item, AuthorizedSourceFile) and _is_hermes_target(
item.relative_path.as_posix()
@@ -102,7 +102,7 @@ class HermesSourceAdapter:
self.grant,
suffixes=frozenset({".jsonl", ".db"}),
max_targets=options.max_scan_targets or 100_000,
excluded_top_directories=frozenset({HERMES_PROFILES_DIRECTORY}),
excluded_directory_names=frozenset({HERMES_PROFILES_DIRECTORY}),
):
yield ScanProgressEvent(
phase=ScanPhase.DISCOVER,
@@ -253,16 +253,62 @@ def parse_hermes_rollout_record(
#: own `state.db` and sessions. A profile is another person's history: it is
#: read only through its own grant (its own MemHub user), never as part of the
#: root above it. Hermes' own export skips this directory for the same reason.
#:
#: Skipped at any depth, not only directly under the root: a root granted
#: above the Hermes home holds `.hermes/profiles/`. Hermes keeps nothing of
#: its own under a directory of that name (its history is `state.db` and
#: `sessions/`), so nothing real is lost.
HERMES_PROFILES_DIRECTORY = "profiles"
def _is_hermes_target(relative_path: str) -> bool:
path = PurePosixPath(relative_path)
if path.parts and path.parts[0] == HERMES_PROFILES_DIRECTORY:
if HERMES_PROFILES_DIRECTORY in path.parts[:-1]:
return False
return path.name == "state.db" or (path.suffix == ".jsonl" and "sessions" in path.parts)
def hermes_root_is_profiles_container(root: Path) -> bool:
"""`profiles` itself holds everyone's profiles; it is never one person's history."""
return root.name == HERMES_PROFILES_DIRECTORY
def hermes_root_conflict(mine: Path, other_users_roots: Iterable[Path]) -> str | None:
"""Why the resolved root `mine` may not be granted beside other users' Hermes roots.
Paths are compared resolved (symlinks followed; the other users' roots are
resolved here). Two roots that are equal or nest would read the same files
for two users, except when one is exactly the other's `profiles/<name>`:
the scan above never enters `profiles/`, so the two stay apart (`local` on
`~/.hermes`, a user on `~/.hermes/profiles/<name>`). None when it may.
"""
if hermes_root_is_profiles_container(mine):
return HERMES_ROOT_IS_PROFILES_CONTAINER
for other in other_users_roots:
theirs = _real(other)
if mine == theirs:
return HERMES_ROOT_OVERLAPS_OTHER_USER
for parent, child in ((mine, theirs), (theirs, mine)):
if parent in child.parents:
relative = child.relative_to(parent).parts
if len(relative) != 2 or relative[0] != HERMES_PROFILES_DIRECTORY:
return HERMES_ROOT_OVERLAPS_OTHER_USER
return None
#: 422: the root is the `profiles` container, not a profile.
HERMES_ROOT_IS_PROFILES_CONTAINER = "hermes_root_is_profiles_container"
#: 409: the root is, holds or sits inside another user's Hermes root.
HERMES_ROOT_OVERLAPS_OTHER_USER = "hermes_root_overlaps_other_user"
def _real(path: Path) -> Path:
try:
return path.expanduser().resolve(strict=False)
except (OSError, RuntimeError):
return path.expanduser().absolute()
def _read_rollout(
source_file: AuthorizedSourceFile,
options: ScanOptions,
@@ -10,7 +10,10 @@ from pydantic import JsonValue
from one_memory.input_sanitization import sanitize_model_input
_REMINDER_TAGS = r"additional_data|system[-_]reminder|working_memory_reminder"
_CONTEXT_TAGS = r"user_info|rules|agent_transcripts|system[-_]notification|pipeline_log"
_CONTEXT_TAGS = (
r"user_info|rules|agent_transcripts|attached_files|image_local_path|"
r"system[-_]notification|pipeline_log"
)
_TAG_ATTRIBUTES = r"""(?:\s+[\w:-]+\s*=\s*(?:"[^"<>]*"|'[^'<>]*'))*\s*"""
_HOST_PREFIX = re.compile(rf"<(?P<tag>{_REMINDER_TAGS}|{_CONTEXT_TAGS}){_TAG_ATTRIBUTES}>")
_REMINDER_START = re.compile(rf"<(?P<tag>{_REMINDER_TAGS}){_TAG_ATTRIBUTES}>")
@@ -1,6 +1,7 @@
import asyncio
from collections.abc import Awaitable, Callable, Mapping
from contextlib import suppress
from functools import partial
from time import monotonic
from typing import Literal
@@ -82,6 +83,10 @@ class ImportSupervisor:
)
self._tasks: dict[str, asyncio.Task[None]] = {}
self._drains: dict[str, asyncio.Task[None]] = {}
#: Pause and cancel settle a stopped job with a read and a transition.
#: Made on the loop, two of them could not interleave; made off it,
#: they are serialized here so they still land in the order they woke.
self._settling = asyncio.Lock()
def register_adapter(self, adapter: SourceAdapter, *, user_id: str = "local") -> None:
"""Adapters are keyed by lane: the source id for `local`, `source@user`
@@ -116,11 +121,44 @@ class ImportSupervisor:
raise UnknownSourceError(lane)
if self.source_enabled is not None and not self.source_enabled(lane):
return job
existing = self._tasks.get(job_id)
self._start(job)
return job
async def enqueue_off_loop(self, job_id: str, *, job: ImportJob | None = None) -> ImportJob:
"""`enqueue`, with its job read and its grant check made off the loop.
Both are database reads, and the grant check resolves the source root
on disk. Live sync enqueues from its poll, so on the loop they were
the main thread waiting for a reader behind an import peak. `job` is
the row the caller has just read, when it has one.
"""
if job_id in self._drains:
raise InvalidJobTransitionError("import_cleanup_in_progress")
def admit() -> tuple[ImportJob, bool]:
current = job if job is not None and job.job_id == job_id else None
if current is None:
current = self.job_store.require(job_id)
lane = self.lane_of(current)
return current, self.source_enabled is None or self.source_enabled(lane)
current, enabled = await in_the_background(admit)
lane = self.lane_of(current)
if lane not in self.adapters:
raise UnknownSourceError(lane)
if job_id in self._drains:
raise InvalidJobTransitionError("import_cleanup_in_progress")
if enabled:
self._start(current)
return current
def _start(self, job: ImportJob) -> None:
lane = self.lane_of(job)
existing = self._tasks.get(job.job_id)
if existing is None or existing.done():
self._tasks[job_id] = asyncio.create_task(
self._tasks[job.job_id] = asyncio.create_task(
self._execute(
job_id,
job.job_id,
source_id=lane,
operation=job.operation,
)
@@ -130,10 +168,24 @@ class ImportSupervisor:
component="import.supervisor",
event="import.queued",
outcome="queued",
job_id=job_id,
job_id=job.job_id,
details={"source_id": job.source_id, "mode": job.operation},
)
return job
async def resume_interrupted_off_loop(self) -> None:
"""`resume_interrupted` for the running Core: every read and write off the loop."""
for status in (ImportStatus.INTERRUPTED, ImportStatus.PENDING):
jobs = await in_the_background(partial(self.job_store.find_jobs, statuses=(status,)))
for job in jobs:
if self.lane_of(job) not in self.adapters:
continue
if job.status is not ImportStatus.PENDING:
if self.source_enabled is not None and not await in_the_background(
self.source_enabled, self.lane_of(job)
):
continue
job = await in_the_background(self.job_store.queue_for_resume, job.job_id)
await self.enqueue_off_loop(job.job_id, job=job)
def resume_interrupted(self) -> None:
# A process restart must not place recoverable work behind every newly
@@ -166,40 +218,64 @@ class ImportSupervisor:
self.enqueue(job_id)
return queued
async def resume_off_loop(self, job_id: str) -> ImportJob:
"""`resume`, with its reads and its write made off the event loop."""
if job_id in self._drains:
raise InvalidJobTransitionError("import_cleanup_in_progress")
queued = await in_the_background(self._queue_resumable, job_id)
if queued.status is not ImportStatus.PENDING:
return queued
return await self.enqueue_off_loop(job_id, job=queued)
def _queue_resumable(self, job_id: str) -> ImportJob:
job = self.job_store.require(job_id)
if job.status not in {
ImportStatus.PENDING,
ImportStatus.PAUSED,
ImportStatus.INTERRUPTED,
ImportStatus.FAILED,
}:
return job
return self.job_store.queue_for_resume(job_id)
async def wait(self, job_id: str) -> ImportJob:
task = self._tasks.get(job_id)
if task is not None:
await asyncio.shield(task)
return self.job_store.require(job_id)
return await in_the_background(self.job_store.require, job_id)
async def pause(self, job_id: str) -> ImportJob:
job = self.job_store.require(job_id)
job = await in_the_background(self.job_store.require, job_id)
if job.status in {ImportStatus.PAUSED, ImportStatus.INTERRUPTED}:
return job
await self._cancel_task(job_id)
current = self.job_store.require(job_id)
if current.status is ImportStatus.COMPLETED:
return current
return self.job_store.transition(job_id, ImportStatus.PAUSED)
async with self._settling:
return await in_the_background(self._settle_stopped, job_id, ImportStatus.PAUSED)
async def cancel(self, job_id: str) -> ImportJob:
job = self.job_store.require(job_id)
job = await in_the_background(self.job_store.require, job_id)
if job.status is ImportStatus.CANCELLED:
return job
await self._cancel_task(job_id)
async with self._settling:
return await in_the_background(self._settle_stopped, job_id, ImportStatus.CANCELLED)
def _settle_stopped(self, job_id: str, target: ImportStatus) -> ImportJob:
"""A stopped runner's job moves to `target`, unless it completed first."""
current = self.job_store.require(job_id)
if current.status is ImportStatus.COMPLETED:
return current
return self.job_store.transition(job_id, ImportStatus.CANCELLED)
return self.job_store.transition(job_id, target)
async def shutdown(self) -> None:
running = [
job
for job in self.job_store.list_jobs(status=ImportStatus.RUNNING)
if job.job_id in self._tasks
]
for job in running:
self.job_store.interrupt_running(job.job_id)
mine = frozenset(self._tasks)
def interrupt_mine() -> None:
for job in self.job_store.find_jobs(statuses=(ImportStatus.RUNNING,)):
if job.job_id in mine:
self.job_store.interrupt_running(job.job_id)
await in_the_background(interrupt_mine)
await asyncio.gather(
*(self._cancel_task(job_id) for job_id in tuple(self._tasks)),
)
@@ -355,9 +431,7 @@ class ImportSupervisor:
try:
await asyncio.wait_for(asyncio.shield(task), timeout)
except TimeoutError:
current = self.job_store.require(job_id)
if current.status is ImportStatus.RUNNING:
self.job_store.interrupt_running(job_id)
await in_the_background(self._interrupt_if_running, job_id)
return False
except asyncio.CancelledError:
# The worker noticed cancel and exited. A cancel of this
@@ -365,10 +439,16 @@ class ImportSupervisor:
caller = asyncio.current_task()
if caller is not None and caller.cancelling():
raise
# A read and a write on the single writer: off the loop. Live sync
# calls this from its poll, and on the loop it was the main thread
# waiting its turn for the writer behind an import peak.
await in_the_background(self._interrupt_if_running, job_id)
return True
def _interrupt_if_running(self, job_id: str) -> None:
current = self.job_store.require(job_id)
if current.status is ImportStatus.RUNNING:
self.job_store.interrupt_running(job_id)
return True
async def _cancel_task(self, job_id: str) -> None:
drain = self._drains.get(job_id)
@@ -34,6 +34,7 @@ from one_memory.memory_governance import MemoryGovernanceStore, MemoryStatus
from one_memory.memory_server.durable_queue import DurableWriteQueue
from one_memory.memory_server.unified_entry import UnifiedWriteIntake
from one_memory.memory_tasks import MemoryTaskStatus, MemoryTaskStore
from one_memory.model_backoff import next_retry_at
from one_memory.secrets import SecretStoreUnavailableError
from one_memory.user_scope import DEFAULT_USER_ID, USER_ID_PATTERN, UserId
@@ -77,6 +78,12 @@ class MemoryAddReceipt(BaseModel):
task_id: str | None = None
memory_ids: list[str] = Field(default_factory=list)
error: str | None = None
#: Why a note still being worked on last failed (`model_auth_failed`, ...),
#: or why a failed one failed. A provider failure keeps it `pending`.
error_code: str | None = None
#: When a pending note is next tried: its own retry wait, or the model
#: provider's pause while the provider is failing.
next_retry_at: str | None = None
class LegacyMemoryAdditionReader:
@@ -258,6 +265,8 @@ class MemoryAdditionStore:
durable_state = durable.state if durable is not None else "failed"
mapped: MemoryAddState = {
"pending": "pending",
# Re-run automatically within seconds (`recover_unknown`).
"unknown": "pending",
"running": "running",
"waiting_for_model": "waiting_for_model",
"waiting_for_permission": "waiting_for_permission",
@@ -272,6 +281,12 @@ class MemoryAdditionStore:
state=mapped,
memory_ids=(list(durable.sdk_memory_ids) if durable is not None else []),
error=durable.detail if durable is not None else "write_missing",
error_code=(
durable.error_code
if durable is not None and durable.state != "completed"
else None
),
next_retry_at=durable.retry_at if durable is not None else None,
)
if self.processor is None:
raise RuntimeError("legacy_memory_add_processor_unavailable")
@@ -402,10 +417,15 @@ def memory_add_router(dependencies: Sequence[Depends]) -> APIRouter:
user_id: str = Query(default=DEFAULT_USER_ID, pattern=USER_ID_PATTERN),
) -> MemoryAddReceipt:
try:
return store(request).receipt_for_request(request_id, user_id=user_id)
receipt = store(request).receipt_for_request(request_id, user_id=user_id)
except KeyError as error:
raise HTTPException(404, "memory_add_not_found") from error
except ValueError as error:
raise HTTPException(409, str(error)) from error
return receipt.model_copy(
update={
"next_retry_at": next_retry_at(request.app, receipt.state, receipt.next_retry_at)
}
)
return router
@@ -465,12 +465,16 @@ class EvolveScheduler:
service: EvolveService,
schedule: Callable[[], EvolveSchedule],
users_with_memories: Callable[[], Awaitable[Iterable[str]]],
lifecycle_lock: asyncio.Lock | None = None,
lifecycle_open: Callable[[], bool] = lambda: True,
now: Callable[[], datetime] = lambda: datetime.now().astimezone(),
check_seconds: float = SCHEDULE_CHECK_SECONDS,
) -> None:
self.service = service
self.schedule = schedule
self.users_with_memories = users_with_memories
self.lifecycle_lock = lifecycle_lock
self.lifecycle_open = lifecycle_open
self._now = now
self.check_seconds = check_seconds
self._task: asyncio.Task[None] | None = None
@@ -492,7 +496,15 @@ class EvolveScheduler:
if not evolve_due(now, schedule, last, anchor):
continue
try:
receipt = self.service.start(user_id, trigger="scheduled")
if self.lifecycle_lock is None:
if not self.lifecycle_open():
break
receipt = self.service.start(user_id, trigger="scheduled")
else:
async with self.lifecycle_lock:
if not self.lifecycle_open():
break
receipt = self.service.start(user_id, trigger="scheduled")
except EvolveModeError:
break
await self.service.wait(receipt.run_id)
@@ -533,10 +545,12 @@ class MemoryGraphBackfill:
*,
coordinator: EvolveCoordinator,
state: EvolveStateStore,
lifecycle_open: Callable[[], bool] = lambda: True,
now: Callable[[], datetime] = lambda: datetime.now(UTC),
) -> None:
self.coordinator = coordinator
self.state = state
self.lifecycle_open = lifecycle_open
self._now = now
self._task: asyncio.Task[None] | None = None
@@ -557,6 +571,35 @@ class MemoryGraphBackfill:
self._task = asyncio.create_task(self._run(users), name="one-memory-graph-backfill")
return state
def recover(self, user_ids: Iterable[str]) -> GraphBuildState:
"""Resume an interrupted backfill without repeating completed users."""
if self.running:
return self.state.graph_build()
saved = self.state.graph_build()
if saved.state != "running":
return saved
users = tuple(dict.fromkeys((*saved.users.keys(), *user_ids)))
pending = tuple(
user_id
for user_id in users
if saved.users.get(user_id, GraphBuildUser()).state != "succeeded"
)
resumed = saved.model_copy(deep=True)
for user_id in pending:
resumed.users[user_id] = GraphBuildUser()
if not pending:
resumed.state = "succeeded"
resumed.finished_at = self._now()
self.state.set_graph_build(resumed)
return resumed
resumed.state = "running"
resumed.finished_at = None
self.state.set_graph_build(resumed)
self._task = asyncio.create_task(
self._run(pending), name="one-memory-graph-backfill-recovery"
)
return resumed
async def wait(self) -> None:
if self._task is not None:
await asyncio.shield(self._task)
@@ -564,6 +607,8 @@ class MemoryGraphBackfill:
async def _run(self, users: tuple[str, ...]) -> None:
state = self.state.graph_build()
for user_id in users:
while not self.lifecycle_open():
await asyncio.sleep(0.2)
state.users[user_id] = GraphBuildUser(state="running")
await asyncio.to_thread(self.state.set_graph_build, state.model_copy(deep=True))
try:
@@ -21,6 +21,11 @@ exercises (query rewrite needs `queries=[...]`, session narratives need
cross-encoder download, access write-back); and everything MemHub already
controls elsewhere (pipeline log, trace, memory operations, telemetry, mode).
Beside the on/off features there is one numeric parameter,
`intention_dedup_threshold` (`PARAMETERS`): the cosine similarity at which the
read-time intention dedup treats two intentions as copies. It is set, stored,
reset and keyed exactly like a feature.
No SDK import here: only `hy_sdk_runtime` and `hy_sdk_migration` import it.
`config_state` imports this module, so it imports nothing from there.
"""
@@ -31,7 +36,7 @@ from collections.abc import Mapping
from dataclasses import dataclass
from typing import Literal
from pydantic import BaseModel, ConfigDict, StrictBool
from pydantic import BaseModel, ConfigDict, Field, StrictBool
MemoryFeatureName = Literal[
"graph_rerank",
@@ -40,8 +45,13 @@ MemoryFeatureName = Literal[
"session_episodes",
"episode_summary",
"intention",
"intention_dedup",
"few_shot",
]
#: Numeric settings beside the on/off features.
MemoryParameterName = Literal["intention_dedup_threshold"]
#: Every key of the setting: a feature or a parameter.
MemorySettingName = MemoryFeatureName | MemoryParameterName
#: Read features change search; write features change what extraction stores.
FeatureSide = Literal["read", "write"]
#: none: no extra call. extra_query: one more store query per search.
@@ -49,9 +59,25 @@ FeatureSide = Literal["read", "write"]
#: a longer extraction prompt.
FeatureCost = Literal["none", "extra_query", "llm_call", "prompt_tokens"]
#: Why an enabled feature does nothing right now (`active: false`).
FeatureReason = Literal["disabled", "requires_pro", "requires_ultra", "requires_graph_rerank"]
FeatureReason = Literal[
"disabled",
"requires_pro",
"requires_ultra",
"requires_graph_rerank",
"requires_intention_dedup",
]
#: A feature is on or off; a parameter is a number.
type SettingValue = bool | float
#: A client's features as a hashable key: sorted (name, value) pairs, set ones only.
type FeatureOverrides = tuple[tuple[str, bool], ...]
type FeatureOverrides = tuple[tuple[str, SettingValue], ...]
#: The range a person may set `intention_dedup_threshold` to. With dedup on,
#: the SDK deletes every intention at or above it from the store at read time,
#: which cannot be undone; below 0.8 the embedding models MemHub uses score
#: different plans on one topic as that close, so a lower value would delete
#: intentions that are not copies. 1.0 keeps only exact copies' dedup.
INTENTION_DEDUP_THRESHOLD_MIN = 0.8
INTENTION_DEDUP_THRESHOLD_MAX = 1.0
_MODE_RANK: dict[str, int] = {"lite": 0, "pro": 1, "ultra": 2}
@@ -169,6 +195,23 @@ FEATURES: tuple[FeatureSpec, ...] = (
"No extra call."
),
),
FeatureSpec(
name="intention_dedup",
section="recall",
field="intention_dedup_enabled",
env="MEMORY_INTENTION_DEDUP_ENABLED",
side="read",
requires_mode="pro",
requires_feature=None,
cost="none",
description=(
"Treat same-day intentions (L7) with the same expiry and a cosine "
"similarity at or above intention_dedup_threshold as copies: a "
"search returns the earliest and deletes the others from the store. "
"On by default; off hides, drops and deletes no intention, at read "
"time or in a write batch. No extra call."
),
),
FeatureSpec(
name="few_shot",
section="llm",
@@ -189,12 +232,57 @@ FEATURE_NAMES: tuple[MemoryFeatureName, ...] = tuple(spec.name for spec in FEATU
FEATURES_BY_NAME: dict[str, FeatureSpec] = {spec.name: spec for spec in FEATURES}
@dataclass(frozen=True)
class ParameterSpec:
"""A numeric setting; it acts only while `requires_feature` acts."""
name: MemoryParameterName
section: Literal["recall"]
field: str
env: str
minimum: float
maximum: float
requires_feature: MemoryFeatureName
description: str
@property
def sdk_field(self) -> str:
return f"{self.section}.{self.field}"
PARAMETERS: tuple[ParameterSpec, ...] = (
ParameterSpec(
name="intention_dedup_threshold",
section="recall",
field="intention_dedup_threshold",
env="MEMORY_INTENTION_DEDUP_THRESHOLD",
minimum=INTENTION_DEDUP_THRESHOLD_MIN,
maximum=INTENTION_DEDUP_THRESHOLD_MAX,
requires_feature="intention_dedup",
description=(
"Cosine similarity at or above which intention_dedup treats two "
"intentions as copies. Higher keeps more. The SDK default is 0.95 "
"(MEMORY_DEDUP_THRESHOLD also sets it when its own env var is unset)."
),
),
)
PARAMETER_NAMES: tuple[MemoryParameterName, ...] = tuple(spec.name for spec in PARAMETERS)
PARAMETERS_BY_NAME: dict[str, ParameterSpec] = {spec.name: spec for spec in PARAMETERS}
#: Every key of the setting, features first, each in its catalog order.
SETTING_NAMES: tuple[MemorySettingName, ...] = (*FEATURE_NAMES, *PARAMETER_NAMES)
SETTINGS_BY_NAME: dict[str, FeatureSpec | ParameterSpec] = {
**FEATURES_BY_NAME,
**PARAMETERS_BY_NAME,
}
class MemoryFeatures(BaseModel):
"""`config.json` `memory_features`, and the `PUT` body.
Every key is optional; absent (or null) means the SDK default. In a `PUT`
an absent key is left as it is and null resets it. Unknown keys and
non-boolean values are refused (422).
an absent key is left as it is and null resets it. Unknown keys,
non-boolean feature values and a threshold that is not a number in its
range are refused (422).
"""
model_config = ConfigDict(frozen=True, extra="forbid", strict=True)
@@ -205,7 +293,14 @@ class MemoryFeatures(BaseModel):
session_episodes: StrictBool | None = None
episode_summary: StrictBool | None = None
intention: StrictBool | None = None
intention_dedup: StrictBool | None = None
few_shot: StrictBool | None = None
intention_dedup_threshold: float | None = Field(
default=None,
ge=INTENTION_DEDUP_THRESHOLD_MIN,
le=INTENTION_DEDUP_THRESHOLD_MAX,
allow_inf_nan=False,
)
def overrides(self) -> FeatureOverrides:
"""The features a person set, as the client key."""
@@ -221,24 +316,47 @@ class MemoryFeatures(BaseModel):
return MemoryFeatures(**{key: value for key, value in values.items() if value is not None})
def feature_overrides(values: Mapping[str, bool]) -> FeatureOverrides:
unknown = set(values) - set(FEATURES_BY_NAME)
def feature_overrides(values: Mapping[str, SettingValue]) -> FeatureOverrides:
unknown = set(values) - set(SETTINGS_BY_NAME)
if unknown:
raise ValueError("unknown_memory_feature")
return tuple(sorted((name, bool(value)) for name, value in values.items()))
return tuple(sorted((name, _setting_value(name, value)) for name, value in values.items()))
def _setting_value(name: str, value: SettingValue) -> SettingValue:
parameter = PARAMETERS_BY_NAME.get(name)
if parameter is None:
return bool(value)
if isinstance(value, bool) or not parameter.minimum <= float(value) <= parameter.maximum:
raise ValueError("invalid_memory_feature_value")
return float(value)
def effective_features(
overrides: FeatureOverrides, defaults: Mapping[str, bool]
) -> dict[str, bool]:
"""Every feature's value: what a person set, else the SDK default."""
values: dict[str, bool] = {spec.name: bool(defaults.get(spec.name, False)) for spec in FEATURES}
overrides: FeatureOverrides, defaults: Mapping[str, SettingValue]
) -> dict[str, SettingValue]:
"""Every setting's value: what a person set, else the SDK default."""
values: dict[str, SettingValue] = {
spec.name: bool(defaults.get(spec.name, False)) for spec in FEATURES
}
for parameter in PARAMETERS:
if parameter.name in defaults:
values[parameter.name] = float(defaults[parameter.name])
values.update(dict(overrides))
return values
def feature_activity(name: str, effective: Mapping[str, bool], mode: str) -> FeatureReason | None:
"""None when the feature acts in this mode; else why it does not."""
def feature_activity(
name: str, effective: Mapping[str, SettingValue], mode: str
) -> FeatureReason | None:
"""None when the feature acts in this mode; else why it does not.
A parameter acts exactly when the feature it tunes acts.
"""
parameter = PARAMETERS_BY_NAME.get(name)
if parameter is not None:
reason = feature_activity(parameter.requires_feature, effective, mode)
return "requires_intention_dedup" if reason == "disabled" else reason
spec = FEATURES_BY_NAME[name]
if not effective.get(name, False):
return "disabled"
@@ -252,11 +370,12 @@ def feature_activity(name: str, effective: Mapping[str, bool], mode: str) -> Fea
def non_default_features(
overrides: FeatureOverrides, defaults: Mapping[str, bool]
) -> dict[str, bool]:
"""The features whose value differs from the SDK default, for status."""
overrides: FeatureOverrides, defaults: Mapping[str, SettingValue]
) -> dict[str, SettingValue]:
"""The settings whose value differs from the SDK default, for status."""
baseline = effective_features((), defaults)
return {
name: value
for name, value in effective_features(overrides, defaults).items()
if value != bool(defaults.get(name, False))
if value != baseline.get(name)
}
@@ -74,9 +74,9 @@ class ConfigDocument(BaseModel):
#: SDK retrieval and extraction features a person set; an unset one keeps
#: the SDK default. See `one_memory.memory_features`.
memory_features: MemoryFeatures = Field(default_factory=MemoryFeatures)
#: The display language a person chose in the App; None = never chosen or
#: "follow the system". See `one_memory.ui_language` for what it does to
#: the SDK.
#: The display language the App shows (its choice, or the system language
#: it follows); None = none set. See `one_memory.ui_language` for what it
#: does to the SDK.
ui_language: UiLanguage | None = None
@field_validator("sources")
@@ -153,6 +153,11 @@ class ConfigStore:
with self._lock:
return self._document.memory_mode
def memory_mode_explicitly_configured(self) -> bool:
"""Whether a person chose the mode, including choosing the default."""
with self._lock:
return "memory_mode" in self._document.model_fields_set
def set_memory_mode(self, mode: MemoryMode) -> ConfigDocument:
if mode not in MEMORY_MODES:
raise ValueError("invalid_memory_mode")
File diff suppressed because it is too large. Load diff
@@ -3163,6 +3163,137 @@ async def _060_memory_note_requests(cursor: Any) -> None:
await cursor.execute(statement)
def write_job_retries_ddl(dialect: str = "mysql") -> str:
"""Per-job retry state: which SDK request id is next, and retries spent.
The SDK replays a request id's first outcome forever, so every re-run of
a job needs an id of its own (`sdk_attempt`). `retries` counts automatic
retries after failures of the job itself; provider failures spend none.
"""
if dialect == "sqlite":
return """
ALTER TABLE memory_write_jobs
ADD COLUMN sdk_attempt INTEGER NOT NULL DEFAULT 0;
ALTER TABLE memory_write_jobs
ADD COLUMN retries INTEGER NOT NULL DEFAULT 0;
"""
return """
ALTER TABLE memory_write_jobs
ADD COLUMN sdk_attempt INT NOT NULL DEFAULT 0;
ALTER TABLE memory_write_jobs
ADD COLUMN retries INT NOT NULL DEFAULT 0;
"""
async def _061_write_job_retries(cursor: Any) -> None:
for statement in _split_statements(write_job_retries_ddl(_dialect(cursor))):
await cursor.execute(statement)
ONBOARDING_PROGRESS_MIGRATION_VERSION = 62
def onboarding_progress_ddl(dialect: str = "mysql") -> str:
"""Append resumable onboarding choices without rewriting migration 12."""
if dialect == "sqlite":
return """
ALTER TABLE onboarding_state
ADD COLUMN mode_intent TEXT NULL;
ALTER TABLE onboarding_state
ADD COLUMN setup_step TEXT NULL;
ALTER TABLE onboarding_state
ADD COLUMN model_profile_id TEXT NULL;
"""
return """
ALTER TABLE onboarding_state
ADD COLUMN mode_intent VARCHAR(16) NULL AFTER updated_at;
ALTER TABLE onboarding_state
ADD COLUMN setup_step VARCHAR(16) NULL AFTER mode_intent;
ALTER TABLE onboarding_state
ADD COLUMN model_profile_id VARCHAR(36) NULL AFTER setup_step;
"""
async def _062_onboarding_progress(cursor: Any) -> None:
if _dialect(cursor) == "sqlite":
existing = await cursor.columns("onboarding_state")
statements = _split_statements(onboarding_progress_ddl("sqlite"))
for column, statement in zip(
("mode_intent", "setup_step", "model_profile_id"),
statements,
strict=True,
):
if column not in existing:
await cursor.execute(statement)
return
for statement in _split_statements(onboarding_progress_ddl("mysql")):
try:
await cursor.execute(statement)
except Exception as error: # noqa: BLE001 - MySQL driver error type varies
if _is_duplicate_column(error):
continue
raise
IMPORT_JOB_LOOKUP_INDEXES_MIGRATION_VERSION = 63
def import_job_lookup_indexes_ddl(dialect: str = "mysql") -> str:
"""Index the per-source and per-status import job lookups.
Migration 7 gave SQLite `import_jobs` no index beyond its primary key (the
one-active-job index is partial), so every "this source's jobs" or "the
jobs in this state" question read and sorted the whole table. Live sync
adds a row per change it imports and nothing removed them, so on a machine
a few weeks old that table was thousands of rows read several times a poll.
`import_jobs_lane` serves the lane lookups (one user, one source, one
operation, a status, newest first) and the pruning of finished syncs;
`import_jobs_status` serves restart recovery and the status filters.
MySQL's migration 7 already has a status index and no `operation` column,
so it only gains the ordered source index.
"""
if dialect == "sqlite":
return """
CREATE INDEX IF NOT EXISTS import_jobs_lane
ON import_jobs(user_id, source_id, operation, status, created_at);
CREATE INDEX IF NOT EXISTS import_jobs_status
ON import_jobs(status, created_at);
"""
return """
CREATE INDEX idx_import_jobs_source_created
ON import_jobs(user_id, source_id, created_at);
"""
def _is_duplicate_index(error: Exception) -> bool:
"""MySQL's "index name already there" (1061): the index is in place."""
args: tuple[object, ...] = getattr(error, "args", ())
return bool(args) and args[0] == 1061
async def _063_import_job_lookup_indexes(cursor: Any) -> None:
dialect = _dialect(cursor)
if dialect == "sqlite":
# `operation` was added by the job store, not by migration 7, so a
# database built from the catalog alone may not have it yet. Same
# declaration as the store's own addition.
if "operation" not in await cursor.columns("import_jobs"):
await cursor.execute(
"ALTER TABLE import_jobs ADD COLUMN operation TEXT NOT NULL DEFAULT 'history'"
)
for statement in _split_statements(import_job_lookup_indexes_ddl(dialect)):
await cursor.execute(statement)
return
for statement in _split_statements(import_job_lookup_indexes_ddl(dialect)):
try:
await cursor.execute(statement)
except Exception as error: # noqa: BLE001 - MySQL driver error type varies
if _is_duplicate_index(error):
continue
raise
def _split_statements(sql: str) -> list[str]:
"""Split a multi-statement script, dropping blanks, PRAGMA and comments.
@@ -3462,4 +3593,16 @@ MIGRATIONS: tuple[Migration, ...] = (
_060_memory_note_requests,
("memory_note_requests",),
),
Migration(61, "write_job_retries", _061_write_job_retries),
Migration(
ONBOARDING_PROGRESS_MIGRATION_VERSION,
"onboarding_progress",
_062_onboarding_progress,
("onboarding_state",),
),
Migration(
IMPORT_JOB_LOOKUP_INDEXES_MIGRATION_VERSION,
"import_job_lookup_indexes",
_063_import_job_lookup_indexes,
),
)
@@ -57,6 +57,11 @@ class WriteRecord:
detail: str = ""
raw_ids: tuple[str, ...] = ()
memory_ids: tuple[str, ...] = ()
#: The stable code of the last failure of a job still being worked on
#: (or of a failed one).
error_code: str | None = None
#: When a waiting job is next tried, if a retry wait holds it.
next_retry_at: str | None = None
def _now() -> str:
@@ -44,6 +44,7 @@ from one_memory.memory_server.durable_queue import (
)
from one_memory.memory_server.registry import WriteRegistry, WriteState
from one_memory.memory_server.submit import WriteConflictError
from one_memory.model_backoff import next_retry_at
from one_memory.user_scope import DEFAULT_USER_ID, USER_ID_PATTERN, UserId
#: Durable keys of another user's writes start with this. A `local` request
@@ -72,6 +73,12 @@ class MemoryWriteReceipt(BaseModel):
detail: str = ""
raw_ids: tuple[str, ...] = ()
memory_ids: tuple[str, ...] = ()
#: Why a job still being worked on last failed (`model_auth_failed`, ...),
#: or why a failed one failed. A provider failure keeps `state` queued.
error_code: str | None = None
#: When it is next tried: the job's own retry wait, or the model
#: provider's pause (`model_backoff`) while the provider is failing.
next_retry_at: str | None = None
class OpenAIMessage(BaseModel):
@@ -235,6 +242,8 @@ def sdk_router(dependencies: list[Any] | None = None) -> APIRouter:
detail=record.detail,
raw_ids=record.raw_ids,
memory_ids=record.memory_ids,
error_code=record.error_code,
next_retry_at=next_retry_at(request.app, record.state, record.next_retry_at),
)
return instance
@@ -0,0 +1,330 @@
"""One pause for every model request while the provider keeps failing (#245).
A broken model configuration (a revoked key, a wrong base URL, a model that
does not exist, a provider that is down) used to fail each queued job on its
own: the drain claimed the next job the moment the last one failed, so a
backlog of hundreds of jobs turned into requests as fast as the provider could
refuse them. The pause is global, not per job: a failure says something about
the provider, not about the conversation that happened to hit it.
After the first failure no further model work is claimed for 5 seconds; each
failed probe after that doubles the pause, up to 3 minutes. A pause admits one
probe job at a time. The first success, or any change the user makes to the
model configuration, ends it at once. Jobs not yet sent stay `pending`.
"""
from __future__ import annotations
import logging
import threading
import time
from collections.abc import Callable, Iterator
from contextlib import contextmanager
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from typing import Any
import httpx
logger = logging.getLogger(__name__)
BACKOFF_FIRST_SECONDS = 5.0
BACKOFF_CAP_SECONDS = 180.0
#: How long one probe may hold the pause open before another is admitted. A
#: probe job that never reaches the provider (nothing to remember, a refusal
#: before the SDK) reports nothing, and the queue must not wait on it forever.
PROBE_HOLD_SECONDS = 30.0
#: Requests already in flight when the pause began are refused without going
#: out once this many failures in a row confirm it is not a one-off.
REFUSE_IN_FLIGHT_AFTER = 2
#: Statuses that say the configuration or the provider is wrong, not the one
#: request: authentication, a missing model or route, throttling, a timeout
#: and the provider's own failures. A 400/413/422 is about the request's
#: content and is left to that job.
_PROVIDER_STATUSES = {401: "model_auth_failed", 403: "model_auth_failed", 404: "model_not_found"}
#: Every code a provider failure is recorded under: a job that failed with one
#: of these waits for the provider instead of spending a retry.
MODEL_FAILURE_CODES = frozenset(
{
"model_auth_failed",
"model_not_found",
"model_timeout",
"model_rate_limited",
"model_server_error",
"model_unreachable",
"model_transport_failed",
}
)
def backoff_seconds(level: int) -> float:
"""The pause after the `level`-th failed attempt in a row (1-based)."""
if level < 1:
return 0.0
return min(BACKOFF_FIRST_SECONDS * float(2 ** min(level - 1, 16)), BACKOFF_CAP_SECONDS)
def failure_code(status: int) -> str | None:
"""The stable code for a provider HTTP status, or None if it is no provider failure."""
if status in _PROVIDER_STATUSES:
return _PROVIDER_STATUSES[status]
if status in {408, 504}:
return "model_timeout"
if status == 429:
return "model_rate_limited"
if status >= 500:
return "model_server_error"
return None
def transport_failure_code(error: BaseException) -> str:
if isinstance(error, httpx.TimeoutException):
return "model_timeout"
if isinstance(error, httpx.ConnectError | httpx.NetworkError):
return "model_unreachable"
return "model_transport_failed"
def request_kind(request: httpx.Request) -> str:
"""`embedding` for the embeddings route, `chat` for everything else."""
return "embedding" if request.url.path.rstrip("/").endswith("/embeddings") else "chat"
@dataclass(frozen=True)
class ModelBackoffState:
#: When the pause ends; None when no pause is in force.
until: datetime | None
#: Failed attempts in a row.
failures: int
#: The last failure's stable code (`model_auth_failed`, ...).
last_error: str | None
profile_id: str | None
class ModelBackoffRefusal(httpx.TransportError):
"""A request refused locally because the provider is being given a rest."""
@dataclass(eq=False)
class ModelAttempt:
"""One job's write, as the provider saw it.
`failure` is the code of the first provider failure (or local refusal)
that happened while this write was in flight, None if there was none.
"""
failure: str | None = None
class ModelBackoff:
"""Process-wide circuit breaker for the active model profile.
Thread-safe: the transport records from the SDK's own loop thread, the
drain asks from its claim thread, and status reads from the event loop.
"""
def __init__(
self,
*,
clock: Callable[[], float] = time.monotonic,
wall_clock: Callable[[], datetime] = lambda: datetime.now(UTC),
) -> None:
self._clock = clock
self._wall_clock = wall_clock
self._lock = threading.Lock()
#: Writes in flight (`attempt`); never cleared by a reset.
self._attempts: set[ModelAttempt] = set()
self._reset_locked()
def _reset_locked(self) -> None:
#: Which kinds of request are failing (`chat`, `embedding`): a working
#: embedding endpoint says nothing about a chat model that is gone.
self._failing: set[str] = set()
self._level = 0
self._failures = 0
self._until = 0.0
self._probe_until = 0.0
self._last_error: str | None = None
self._profile_id: str | None = None
def reset(self, reason: str = "model_configuration_changed") -> None:
"""Forget every failure: the user changed what is being called."""
with self._lock:
was_paused = self._failures > 0
self._reset_locked()
if was_paused:
logger.info("model_backoff_reset reason=%s", reason)
def record_success(self, profile_id: str | None = None, kind: str = "model") -> None:
with self._lock:
if self._failures == 0:
return
if profile_id is not None and self._profile_id not in {None, profile_id}:
return
self._failing.discard(kind)
if self._failing:
return
self._reset_locked()
logger.info("model_backoff_cleared profile_id=%s", profile_id)
def record_failure(self, code: str, profile_id: str | None = None, kind: str = "model") -> None:
now = self._clock()
with self._lock:
if profile_id is not None and self._profile_id not in {None, profile_id}:
# Another profile's history says nothing about this one.
self._reset_locked()
self._profile_id = profile_id
self._failing.add(kind)
self._mark_attempts_locked(code)
self._failures += 1
self._last_error = code
self._probe_until = 0.0
# Requests that were already out when the pause began fail too;
# they are the same outage, not a reason to wait longer.
if now >= self._until:
self._level += 1
self._until = now + backoff_seconds(self._level)
level, pause = self._level, self._until - now
logger.warning(
"model_backoff_failure error_code=%s level=%d pause_s=%.0f profile_id=%s",
code,
level,
pause,
profile_id,
)
@contextmanager
def attempt(self) -> Iterator[ModelAttempt]:
"""Hold one job's write open, and learn whether the provider failed it.
Every provider failure and every local refusal while the write is in
flight is charged to it. A request cannot be traced to the job that
made it: the SDK sends embeddings from one shared batching worker, so
neither the calling thread nor a context variable identifies the job.
Charging every write then in flight can name a write the provider did
not fail (another job's request failed beside it); that write waits
for the provider and is retried with the rest. It can never miss the
write the provider did fail, which would spend that job's retries on
an outage.
"""
record = ModelAttempt()
with self._lock:
self._attempts.add(record)
try:
yield record
finally:
with self._lock:
self._attempts.discard(record)
def record_refusal(self) -> str:
"""A request refused locally; charged to the writes in flight like a failure."""
with self._lock:
code = self._last_error or "model_transport_failed"
self._mark_attempts_locked(code)
return code
def _mark_attempts_locked(self, code: str) -> None:
for record in self._attempts:
if record.failure is None:
record.failure = code
def paused(self) -> bool:
with self._lock:
return self._failures > 0 and self._clock() < self._until
def refuses_requests(self, kind: str = "model") -> bool:
"""Whether a request of this kind should not go out at all right now.
Only the failing kind is refused: a chat model that is gone must not
take recall's embedding calls down with it.
"""
with self._lock:
return (
kind in self._failing
and self._failures >= REFUSE_IN_FLIGHT_AFTER
and self._clock() < self._until
)
def allowance(self, *, kind: str | None = None) -> int | None:
"""How many matching provider jobs may be claimed now.
0 while paused; once the pause is over, one probe at a time until it
reports or `PROBE_HOLD_SECONDS` pass. ``kind`` keeps a failed chat
model from pausing raw-only work that needs embeddings, while an
embedding outage still paces that work.
"""
now = self._clock()
with self._lock:
if self._failures == 0 or (kind is not None and kind not in self._failing):
return None
if now < self._until or now < self._probe_until:
return 0
self._probe_until = now + PROBE_HOLD_SECONDS
return 1
def state(self) -> ModelBackoffState:
now = self._clock()
with self._lock:
remaining = self._until - now if self._failures else 0.0
return ModelBackoffState(
until=(
self._wall_clock() + timedelta(seconds=remaining) if remaining > 0 else None
),
failures=self._failures,
last_error=self._last_error,
profile_id=self._profile_id,
)
def next_retry_at(app: Any, state: str, own: str | None) -> str | None:
"""When a queued job is tried next: its own retry wait, or the provider's pause.
While the breaker pauses, nothing is claimed at all, so the pause holds
every queued job, whatever it last failed with.
"""
if state not in {"accepted", "pending"}:
return None
backoff = getattr(getattr(app, "state", None), "model_backoff", None)
until = backoff.state().until if isinstance(backoff, ModelBackoff) else None
paused = until.isoformat() if until is not None else None
candidates = [value for value in (own, paused) if value]
return max(candidates) if candidates else None
class BackedOffModelTransport(httpx.AsyncBaseTransport):
"""Report each provider exchange to the breaker; refuse while it confirms a pause."""
def __init__(
self,
backoff: ModelBackoff,
*,
profile_id: str,
inner: httpx.AsyncBaseTransport | None = None,
) -> None:
self.backoff = backoff
self.profile_id = profile_id
self.inner = inner or httpx.AsyncHTTPTransport(retries=0, trust_env=False)
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
kind = request_kind(request)
if self.backoff.refuses_requests(kind):
self.backoff.record_refusal()
raise ModelBackoffRefusal("model_backoff_active", request=request)
try:
response = await self.inner.handle_async_request(request)
except ModelBackoffRefusal:
raise
except httpx.TransportError as error:
self.backoff.record_failure(transport_failure_code(error), self.profile_id, kind)
raise
code = failure_code(response.status_code)
if code is not None:
self.backoff.record_failure(code, self.profile_id, kind)
elif response.status_code < 400:
self.backoff.record_success(self.profile_id, kind)
return response
async def aclose(self) -> None:
await self.inner.aclose()
+16 -3
View File
@@ -26,6 +26,7 @@ if TYPE_CHECKING:
_operation: ContextVar[tuple["ModelProbeOperations", str] | None] = ContextVar(
"model_probe_operation", default=None
)
_dispatch_started: ContextVar[bool] = ContextVar("model_probe_dispatch_started", default=False)
class ProbeCode(StrEnum):
@@ -113,16 +114,26 @@ class ModelProber:
if existing is not None:
return existing
token = _operation.set((operations, request.operation_id))
dispatch_token = _dispatch_started.set(False)
try:
await self._probe(profile_id)
except ProbeOperationConflict:
await asyncio.to_thread(
operations.fail, request.operation_id, "model_probe_profile_changed"
)
except Exception:
# Unknown dispatch/storage outcomes stay reserved, including across restart.
raise ProbeOperationUnavailable("model_probe_operation_unavailable") from None
except Exception as exc:
if _dispatch_started.get():
# Once a request may have reached the provider, preserve the
# reservation so a retry cannot duplicate a possibly billed call.
raise ProbeOperationUnavailable("model_probe_operation_unavailable") from None
logging.getLogger(__name__).warning(
"model probe failed before dispatch: %s", type(exc).__name__
)
await asyncio.to_thread(
operations.fail, request.operation_id, "model_probe_execution_failed"
)
finally:
_dispatch_started.reset(dispatch_token)
_operation.reset(token)
return await asyncio.to_thread(operations.lookup, request.operation_id)
@@ -317,6 +328,8 @@ class ModelProber:
listing_optional: bool = False,
) -> ProbeStage:
await self._check_operation()
if _operation.get() is not None:
_dispatch_started.set(True)
started = monotonic()
try:
response = await client.request(
+171 -32
View File
@@ -4,9 +4,10 @@ from collections.abc import Iterator
from contextlib import contextmanager
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from typing import Any, Literal
from uuid import UUID
from pydantic import BaseModel, ConfigDict
from pydantic import BaseModel, ConfigDict, Field, field_validator
from one_memory.database import (
take_the_write_lock_first,
@@ -14,11 +15,25 @@ from one_memory.database import (
from one_memory.database_access import main_database
#: Schema owned by `memory_server.migrations` (migration 12). No local copy.
from one_memory.memory_server.migrations import onboarding_state_ddl
from one_memory.memory_server.sql_compat import execute, is_mysql
from one_memory.memory_server.migrations import (
onboarding_progress_ddl,
onboarding_state_ddl,
)
from one_memory.memory_server.sql_compat import execute, is_mysql, row_to_dict
from one_memory.memory_server.store_binding import store_for
SCHEMA = onboarding_state_ddl("sqlite")
ONBOARDING_COLUMNS = (
"singleton",
"status",
"selected_source_ids_json",
"started_at",
"completed_at",
"updated_at",
"mode_intent",
"setup_step",
"model_profile_id",
)
#: The start/complete upserts, per engine.
#:
@@ -51,11 +66,13 @@ START_MYSQL = """
updated_at = VALUES(updated_at)
"""
COMPLETE_UPDATE_SQLITE = (
"UPDATE onboarding_state SET status = 'completed', completed_at = ?, updated_at = ? "
"UPDATE onboarding_state SET status = 'completed', completed_at = ?, updated_at = ?, "
"mode_intent = NULL, setup_step = NULL, model_profile_id = NULL "
"WHERE singleton = 1"
)
COMPLETE_UPDATE_MYSQL = (
"UPDATE onboarding_state SET status = 'completed', completed_at = %s, updated_at = %s "
"UPDATE onboarding_state SET status = 'completed', completed_at = %s, updated_at = %s, "
"mode_intent = NULL, setup_step = NULL, model_profile_id = NULL "
"WHERE singleton = 1"
)
COMPLETE_INSERT_SQLITE = (
@@ -71,13 +88,27 @@ COMPLETE_INSERT_MYSQL = (
RESET_UPDATE_SQLITE = (
"UPDATE onboarding_state SET status = 'not_started', "
"selected_source_ids_json = '[]', started_at = NULL, completed_at = NULL, "
"mode_intent = NULL, setup_step = NULL, model_profile_id = NULL, "
"updated_at = ? WHERE singleton = 1"
)
RESET_UPDATE_MYSQL = (
"UPDATE onboarding_state SET status = 'not_started', "
"selected_source_ids_json = '[]', started_at = NULL, completed_at = NULL, "
"mode_intent = NULL, setup_step = NULL, model_profile_id = NULL, "
"updated_at = %s WHERE singleton = 1"
)
ENSURE_PROGRESS_SQLITE = (
"INSERT INTO onboarding_state ("
"singleton,status,selected_source_ids_json,started_at,completed_at,updated_at"
") VALUES (1,'not_started','[]',NULL,NULL,?) "
"ON CONFLICT(singleton) DO NOTHING"
)
ENSURE_PROGRESS_MYSQL = (
"INSERT INTO onboarding_state ("
"singleton,status,selected_source_ids_json,started_at,completed_at,updated_at"
") VALUES (1,'not_started','[]',NULL,NULL,%s) "
"ON DUPLICATE KEY UPDATE singleton=VALUES(singleton)"
)
class OnboardingState(BaseModel):
@@ -87,6 +118,35 @@ class OnboardingState(BaseModel):
selected_source_ids: tuple[str, ...]
started_at: datetime | None
completed_at: datetime | None
mode_intent: Literal["lite", "pro", "ultra"] | None
setup_step: Literal["review", "mode", "model"] | None
model_profile_id: str | None
class OnboardingProgressUpdate(BaseModel):
"""Only non-sensitive, resumable choices accepted from the onboarding UI."""
model_config = ConfigDict(frozen=True, extra="forbid")
mode_intent: Literal["lite", "pro", "ultra"] | None = None
setup_step: Literal["review", "mode", "model"] | None = None
model_profile_id: str | None = Field(default=None, min_length=36, max_length=36)
@field_validator("model_profile_id")
@classmethod
def validate_model_profile_id(cls, value: str | None) -> str | None:
if value is None:
return None
try:
if str(UUID(value)) != value:
raise ValueError
except (ValueError, TypeError, AttributeError):
raise ValueError("invalid_model_profile_id") from None
return value
class OnboardingCompletedError(RuntimeError):
pass
class OnboardingStore:
@@ -103,12 +163,31 @@ class OnboardingStore:
"""
with self._connect(initialize=True) as connection:
execute(connection, is_mysql(self._store), SCHEMA)
if not is_mysql(self._store):
existing = {
str(row[1]) for row in connection.execute("PRAGMA table_info(onboarding_state)")
}
statements = tuple(
statement.strip()
for statement in onboarding_progress_ddl("sqlite").split(";")
if statement.strip()
)
for column, statement in zip(
("mode_intent", "setup_step", "model_profile_id"),
statements,
strict=True,
):
if column not in existing:
connection.execute(statement)
def get(self) -> OnboardingState:
mysql = is_mysql(self._store)
with self._connect() as connection:
row = execute(
connection, mysql, "SELECT * FROM onboarding_state WHERE singleton = 1"
connection,
mysql,
"SELECT " + ", ".join(ONBOARDING_COLUMNS) + " "
"FROM onboarding_state WHERE singleton = 1",
).fetchone()
if row is None:
return OnboardingState(
@@ -116,8 +195,11 @@ class OnboardingStore:
selected_source_ids=(),
started_at=None,
completed_at=None,
mode_intent=None,
setup_step=None,
model_profile_id=None,
)
return self._to_state(row)
return self._to_state(row, mysql=mysql)
def start(self, source_ids: tuple[str, ...]) -> OnboardingState:
now = datetime.now(UTC).isoformat()
@@ -150,6 +232,43 @@ class OnboardingStore:
)
return self.get()
def update_progress(self, update: OnboardingProgressUpdate) -> OnboardingState:
fields = tuple(
field
for field in ("mode_intent", "setup_step", "model_profile_id")
if field in update.model_fields_set
)
if not fields:
state = self.get()
if state.status == "completed":
raise OnboardingCompletedError("onboarding_completed")
return state
mysql = is_mysql(self._store)
placeholder = "%s" if mysql else "?"
assignments = [f"{field} = {placeholder}" for field in fields]
assignments.append(f"updated_at = {placeholder}")
values = [getattr(update, field) for field in fields]
now = datetime.now(UTC).isoformat()
values.append(now)
with self._connect(writing=True) as connection:
execute(
connection,
mysql,
ENSURE_PROGRESS_MYSQL if mysql else ENSURE_PROGRESS_SQLITE,
(now,),
)
updated = execute(
connection,
mysql,
"UPDATE onboarding_state SET "
+ ", ".join(assignments)
+ " WHERE singleton = 1 AND status != 'completed'",
tuple(values),
)
if updated.rowcount != 1:
raise OnboardingCompletedError("onboarding_completed")
return self.get()
def reset(self) -> OnboardingState:
"""Return onboarding to its first-run state, touching nothing else.
@@ -207,33 +326,53 @@ class OnboardingStore:
yield connection
@staticmethod
def _to_state(row: Any) -> OnboardingState:
"""Read the row by position, so it works for `sqlite3.Row` and tuples.
`SELECT *` makes the column order part of the contract here, which is
why the query is `SELECT *` and the indices are named: relying on
`row["..."]` would only work on SQLite, and relying on position alone
without saying which column is which would be unreadable.
Timestamps are a further engine difference: SQLite hands back the stored
string, MySQL hands back a `datetime`. Accepting both is not padding —
parsing a `datetime` raises, and treating a string as one would be
silently wrong.
"""
# singleton, status, selected_source_ids_json, started_at, completed_at, updated_at
raw_ids = json.loads(row[2])
def _to_state(row: Any, *, mysql: bool) -> OnboardingState:
"""Map SQLite rows and MySQL tuples through the shared database adapter."""
if mysql:
if not isinstance(row, tuple):
raise ValueError("onboarding_row_shape_invalid")
elif not isinstance(row, sqlite3.Row):
raise ValueError("onboarding_row_shape_invalid")
if len(row) != len(ONBOARDING_COLUMNS):
raise ValueError("onboarding_row_shape_invalid")
if mysql:
values = dict(zip(ONBOARDING_COLUMNS, tuple(row), strict=True))
else:
values = row_to_dict(row, ONBOARDING_COLUMNS, mysql=False)
if set(values) != set(ONBOARDING_COLUMNS):
raise ValueError("onboarding_row_shape_invalid")
_as_datetime(values["updated_at"], mysql=mysql, nullable=False)
raw_ids = json.loads(values["selected_source_ids_json"])
return OnboardingState(
status=row[1],
status=values["status"],
selected_source_ids=tuple(value for value in raw_ids if isinstance(value, str)),
started_at=_as_datetime(row[3]),
completed_at=_as_datetime(row[4]),
started_at=_as_datetime(values["started_at"], mysql=mysql),
completed_at=_as_datetime(values["completed_at"], mysql=mysql),
mode_intent=values["mode_intent"],
setup_step=values["setup_step"],
model_profile_id=values["model_profile_id"],
)
def _as_datetime(value: object) -> datetime | None:
"""SQLite stores these as text; MySQL returns `datetime` objects."""
if value is None or value == "":
def _as_datetime(
value: object,
*,
mysql: bool,
nullable: bool = True,
) -> datetime | None:
"""Validate the engine's timestamp shape and normalize it to UTC."""
if value is None:
if not nullable:
raise ValueError("onboarding_datetime_invalid")
return None
if isinstance(value, datetime):
return value
return datetime.fromisoformat(str(value))
if mysql:
if not isinstance(value, datetime):
raise ValueError("onboarding_datetime_invalid")
parsed = value
else:
if not isinstance(value, str) or value == "":
raise ValueError("onboarding_datetime_invalid")
parsed = datetime.fromisoformat(value)
if parsed.tzinfo is None:
return parsed.replace(tzinfo=UTC)
return parsed.astimezone(UTC)
@@ -0,0 +1,123 @@
"""Turn-level read projection for SDK raw batches.
HY keeps several complete conversation turns in one L1 raw node so model
writes stay batched. Product readers must not expose that storage batch as
one conversation item: this module gives every contained turn a stable
identity while retaining the parent raw id for traceability.
"""
from __future__ import annotations
import hashlib
import re
from dataclasses import dataclass
from one_memory.hy_sdk_types import HySdkRawRecord, raw_turn_messages
@dataclass(frozen=True)
class RawDisplayTurn:
turn_id: str
turn_index: int
turn_count: int
content: str
messages: tuple[tuple[str, str], ...]
def raw_display_turns(record: HySdkRawRecord) -> tuple[RawDisplayTurn, ...]:
"""Split one stored raw batch into user-opened turns.
A user message opens a new turn; assistant/tool/system messages stay with
that user message until the next user message. Legacy content with no user
marker remains one turn so it never disappears from the product.
"""
messages = raw_turn_messages(record.content)
grouped: list[list[tuple[str, str]]] = []
for message in messages:
role, _ = message
has_user = bool(grouped) and any(item[0] == "user" for item in grouped[-1])
if not grouped or (role == "user" and has_user):
grouped.append([message])
else:
grouped[-1].append(message)
if not grouped:
grouped = [[("", record.content)]]
digest = hashlib.sha256(record.raw_id.encode("utf-8")).hexdigest()
count = len(grouped)
return tuple(
RawDisplayTurn(
turn_id=f"raw-turn:{digest}:{index}",
turn_index=index,
turn_count=count,
content=_turn_content(turn),
messages=tuple(turn),
)
for index, turn in enumerate(grouped)
)
def _turn_content(messages: list[tuple[str, str]]) -> str:
return "\n".join(f"[{role}]: {content}" if role else content for role, content in messages)
def raw_prefix_added(
cached: tuple[tuple[str, str | None], ...],
current: tuple[tuple[str, str | None], ...],
cached_total: int,
current_total: int,
) -> int | None:
"""How many raws were only put on top of a cached newest-first listing.
`cached` and `current` are the (raw id, status) of the first page of each
listing. The answer is a count when the current page is the cached one
with new raws before it and nothing else changed; None when anything else
happened (a delete, a status change, a raw dated into the middle), which
needs a full re-listing.
"""
added = current_total - cached_total
if added <= 0 or added > len(current):
return None
kept = current[added:]
if not kept and cached_total:
return None
if kept != cached[: len(kept)]:
return None
return added
_WORD = re.compile(r"\w+")
def _query_terms(query: str) -> frozenset[str]:
"""Words of the query, plus character bigrams of unspaced (CJK) words."""
terms: set[str] = set()
for word in _WORD.findall(query.casefold()):
terms.add(word)
if not word.isascii() and len(word) > 1:
terms.update(word[index : index + 2] for index in range(len(word) - 1))
return frozenset(terms)
def turns_for_hit(turns: tuple[RawDisplayTurn, ...], query: str) -> tuple[RawDisplayTurn, ...]:
"""The turns one semantic raw hit shows, never none while it has turns.
The SDK embeds a whole raw batch, so a vector hit cannot say which of its
turns matched. Every turn containing the query text is shown; when none
does, the turn sharing the most query terms (the earliest on a tie) stands
for the hit. A semantic hit is never dropped for lacking the literal query.
"""
if not turns:
return ()
needle = query.casefold().strip()
if needle:
lexical = tuple(turn for turn in turns if needle in turn.content.casefold())
if lexical:
return lexical
terms = _query_terms(query)
def overlap(turn: RawDisplayTurn) -> tuple[int, int]:
content = turn.content.casefold()
return (sum(1 for term in terms if term in content), -turn.turn_index)
return (max(turns, key=overlap),)
Loaded 100 of 367 files, more files were not shown because too many files have changed in this diff. Show more