Compare commits

...
Author SHA1 Message Date
alvinfeiandCursor a60b2d366c release: publish Hy Memory 2.0.5.dev4
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 21:46:33 +08:00
0244ebe38c feat(sdk): process existing raw records (#136)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 21:45:12 +08:00
e3b5a1bc9c release: publish Hy Memory 2.0.5.dev3 (#135)
* release: prepare Hy Memory 2.0.5.dev3

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(sdk): record 2.0.5.dev3 publication

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 20:50:45 +08:00
8aeb042bb7 feat(sdk): search raw through existing search API (#134)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 17:58:29 +08:00
635d2572c9 release: prepare Hy Memory 2.0.5.dev2 (#133)
* release: prepare Hy Memory 2.0.5.dev2

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(sdk): record 2.0.5.dev2 publication

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 17:19:09 +08:00
6113e2351e feat(sdk): import semantic memories exactly (#131)
* feat(sdk): import semantic memories exactly

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(sdk): record exact memory migration API

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 17:05:24 +08:00
b5a8a78c69 release: prepare Hy Memory 2.0.5.dev1 (#130)
* release: prepare Hy Memory 2.0.5.dev1

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(sdk): record 2.0.5.dev1 publication

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 17:01:12 +08:00
fd12c8f1c0 feat(sdk): add MemHub authority migration contracts (#129)
* feat(sdk): expose memory lifecycle status filters

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sdk): persist idempotent write outcomes

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sdk): add explicit exact raw APIs

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sdk): snapshot quiesced local data plane

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sdk): restore quiesced local data plane

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sdk): report incomplete physical deletes

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(sdk): record authority migration contracts

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 16:43:25 +08:00
e089d4b800 fix(coding): default-off search chain_block (diff) injection (#128)
Medium/heavy coding users pay large prompt tokens for the version-evolution
diff block; make MEMORY_CODING_CHAIN_BLOCK_ENABLED opt-in (default false).
Write-side observe is unchanged; HCM_BLOCK_OFF remains an advanced kill switch.

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 14:46:23 +08:00
b370663870 fix(ci): locate nested Linux release artifacts (#36)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 14:14:24 +08:00
8ea6996859 chore(core): retire memory evidence freshness on read paths (#127)
* chore(core): retire memory evidence freshness on read paths

Stop re-validating support_refs against live evidence during library,
context, and call-reference reads. The check hid memories when evidence
changed but caused 503s and segment-key breakage; proofs may still be
written but are ignored until a later teardown removes them.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(docs): align install-docs pin with hy-memory 2.0.4

CI memhub failed because README already documents 2.0.4 / v1.0.4 while
the install-docs contract test still expected 2.0.3 / v1.0.3.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 14:11:57 +08:00
dc740604d2 fix(mysql): merge minute metrics under one row lock (#97)
store_metrics_minute() read the row, merged counters in Python, then wrote
with a separate SELECT + UPDATE/INSERT. Two instances flushing the same
minute could both read the old value and overwrite each other, and a
concurrent insert could lose the whole batch to a duplicate-key error that
the outer except swallowed.

Run the read-merge-write in one transaction with SELECT ... FOR UPDATE and
retry once on a duplicate key or deadlock. Add unit regressions for the
insert and merge paths.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 12:19:55 +08:00
55bf946bcc fix(episode): read series timestamps back in local time (#94)
Series episode memory_at used datetime.utcfromtimestamp() on values produced
by datetime.timestamp(). That yields a naive UTC datetime, which the vector
payload later re-interprets as local time, shifting the stored timestamp by
the machine's UTC offset (8 hours on Asia/Shanghai).

Round-trip through datetime.fromtimestamp() instead, and cover the wall-clock
round trip with a non-UTC TZ regression.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 12:19:51 +08:00
14fcc6fe38 fix(history): query every agent when agent_ids is omitted (#93)
get_recent_history() documented "agent_ids 为空时查询该用户所有 agent" but
constructed the isolation key with the literal "default". History records
are written as "{user_id}:{agent_id}", and add() defaults to
"default_agent", so the documented path usually returned nothing.

Add HistoryStore.get_recent_for_user(), which matches the literal user
prefix with an escaped LIKE, and use it from the client. Add regressions
for multiple agents, a sibling user id and LIKE wildcards in the user id.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 12:19:47 +08:00
1ef0766891 fix(agent): restore the SessionSummaryResult dataclass decorator (#92)
The @dataclass decorator was left at the end of the section-divider comment
line, so SessionSummaryResult was a plain class without __init__. Every
construction raised "takes no arguments" - including the error branch inside
generate_session_summary(), whose TypeError escaped the surrounding except.

Move the decorator onto its own line and add a constructor regression.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 12:19:42 +08:00
3370a715dd fix(client): stop both owned event loops on close (#91)
A solo client creates a write loop and a separate search loop, but close()
stopped only the write loop and __aexit__ stopped neither. Every client
instance against a long-lived process therefore leaked the search thread,
its selector and its 50ms self-wakeup timer. _LoopThread.stop() also left
the loop object unclosed.

Stop every owned loop from one idempotent helper used by both close() and
__aexit__, leave shared-runtime loops to the runtime, and close the loop
after its thread exits. Add lifecycle regressions for both modes.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 12:19:38 +08:00
638d1b7bde fix(server): avoid deadlock when shutting down on signals (#89)
BaseServer.shutdown() waits for serve_forever() to return. The SIGINT /
SIGTERM handlers called it directly, and signal handlers run on the same
thread that is executing serve_forever(), so the server could never exit:
the process hung until SIGKILL and the client was never closed.

Ask a helper thread to call shutdown() and move server_close()/client.close()
into a finally block after serve_forever() returns. Add an in-process test
and a subprocess test that fires SIGALRM while serving on the main thread.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 12:19:33 +08:00
d6890793ec fix(sdk): repair FAISS embedding lookup, dedup pairing, and episode summaries (#84)
* fix(faiss): reconstruct embeddings for dedup and episode summaries

Normal upsert paths store only the payload and the FAISS index position.
get_embeddings() looked for an "embedding" key that only update_payload
ever wrote, so it returned an empty mapping for every upserted node. Search
dedup and episode summaries therefore saw no vectors on the FAISS backend.

Reconstruct the vector from the live index position, guarded by the same
_idx_to_id check search() uses, and keep the stored fallback for nodes
updated through update_payload. Add a regression test.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(search): pair dedup markers with their source search result

The public items are filtered by the requested layers, but the private
dedup view was built with zip(result.memories, public_items). Once one
result is filtered out every later _memory_kind shifts by one, so a
canonical_state can enter fuzzy dedup and be shadowed, or an ordinary fact
can silently skip dedup.

Build both views in the same loop, from the same source element, and add a
regression test for the filtered case.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(episode): use the writer's vector store for background summaries

_schedule_episode_summary accessed self.vector_store, which MemoryWriter
never defines. The AttributeError was swallowed by the task's except block,
so every closed episode silently skipped its L3 summary.

Resolve the store through the existing async _get_vector_store() accessor
and cover the scheduled task with a regression test.

Co-Authored-By: Claude Code <noreply@anthropic.com>

---------

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 12:19:28 +08:00
u7-u7andqiye.sjh 32bf05f821 docs: point install instructions at the current releases (#113)
The install blocks still asked for hy-memory 2.0.3 (and 2.0.1 extras),
MemHub 1.0.3 (build 7), and said memhub-cli was unpublished. pyproject.toml
ships 2.0.4, the MemHub workspace ships 1.0.4 (build 13), and README already
announces memhub-cli 1.0.5.

Update the install commands and the release metadata in the root READMEs,
the usage/search guides, the monorepo policy, and the MemHub docs. Historical
CHANGELOG entries are deliberately unchanged.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
2026-10-21 12:19:23 +08:00
u7-u7andqiye.sjh b6e6a3fb43 test(core): skip fixtures that are absent from the public snapshot (#112)
Four test modules import artifacts that are not in this repository:
one_memory.processing_requests and three scripts/bench_*.py files. A public
checkout therefore fails during collection, so the whole core-fast suite
cannot start even though every collected test is fine.

Skip only when the module or script is missing. Builds that carry the private
artifacts keep running the same tests unchanged.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
2026-10-21 12:19:18 +08:00
u7-u7andqiye.sjh 4e44f59026 fix(import): report oversized source files as file_too_large (#108)
The access gate refuses a file over the per-root max_file_bytes with
FILE_TOO_LARGE, but both discovery_failure_code() and _scan_failure_code()
collapsed everything except budget and symlink failures into
UNSUPPORTED_SCHEMA. A user with a 3 GiB Codex rollout therefore saw
"unsupported schema" while the parser was fine; the only problem was size.

Map the access gate's FILE_TOO_LARGE onto the existing ScanIssueCode.
FILE_TOO_LARGE on both paths, and update the enum comment to say it covers
both the import's per-file ceiling and the access gate.

The per-file limit itself is unchanged.

Fixes #102.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
2026-10-21 12:19:14 +08:00
35bd56ddd0 fix(runtime): start the reader when the supervisor owns the Core (#99)
readerEnabled was only honored on the attach path. The normal first launch
spawns its own Core and returned from start() without calling startReader(),
so the read-only Reader never ran: no reader_port in the receipt and every
read fell back to Core.

Start the Reader after every successful owned start (first launch, race
recovery and relaunch). Clear readerChild when it exits so a later restart
can spawn a fresh one, and stop the old Reader before a relaunch. Add a
public-start regression using fake child processes.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 12:19:10 +08:00
2cea39da4f fix(core): drop native store write gate on HY search (#126)
MemHub expects concurrent native reads during precipitation. The
process-local read slot mostly existed for a Chroma/GIL fail-fast and
was never held by writers on main; activating it (as in #98) would only
push recall onto weaker fallbacks. Remove the lock module and the
hy_memory_store_busy path instead.

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 12:15:26 +08:00
059e3a4903 fix(runtime): accept profile-scoped host ids in the host permission layer (#70)
* fix(runtime): accept profile-scoped host ids in the host permission layer

DeepSeek Harness configures recall per profile, so recall-hosts.ts reports
`deepseek-harness:<profile>` as the host id. The CLI accepts that shape and
prints it from `host list`, but the runtime rejected it twice:

  - HostPermissions.supported() tested `[a-z][a-z0-9-]*`, so preview/install
    answered host_permission_transport_unsupported for every Harness profile.
  - HostGrantStore.validHostId() applied the same restriction, so even a
    supported host could not create its installation or pass the load-time
    revalidation.

A host id is an opaque label: the installation directory is a digest of it,
so a colon never becomes a path component. Both checks now allow one optional
`:<profile>` segment, matching the CLI's existing HOST_ID.

Adds an integration test that previews a real `cordis.patch.yml` install for
`deepseek-harness:desktop` and a grant-store test that the colon survives a
reload, plus a regression for malformed ids.

* fix(runtime): share one host-id grammar across CLI, permissions and grants

Review follow-up: the three layers had three different host-id rules, so the
CLI could list an id the permission layer (or the grant store) then refused.
They now import `isHostId` from `@one-memory/contracts`, which is the only
definition:

  - host: 1..64 chars, alphanumeric first, lower-case letters/digits/hyphens
  - optional `:<profile>`: 1..32 chars, alphanumeric first
  - total <= 97, and the pattern asserts true end-of-input so a trailing
    newline cannot pass

Digit-leading ids stay accepted (the CLI always took them), and the profile
segment is the only colon allowed. Boundary cases live in
`packages/contracts/src/host-id.test.ts`; the grant-store and permission tests
cover the same grammar from their side.

Packaging: contracts is a source-only workspace package, so every bundle that
now imports it for a value inlines it (`noExternal`): host-bridge entries,
the built Claude Code hook assets, and the CLI. Without that the emitted CJS
asks Node for a `.js` file that only exists as `.ts` and the MCP entry fails
at startup.

---------

Co-authored-by: Vesper <226942762+VE5PER-17@users.noreply.github.com>
Co-authored-by: Vesper <hb.ran@foxmail.com>
2026-10-21 11:40:28 +08:00
0a9b10dbfc fix(auth): require the local token on reader internal endpoints (#82)
The /internal/semantic-ids, /internal/call-event and
/internal/graph-snapshot routes were registered directly on the app without
the local_api_guard dependency used by every other route. Any local process
could read memory graph content, probe semantic IDs, or forge call events
without authorization.

Mount the same guard on all three routes. The Reader already sends
Authorization: Bearer <core_api_token>, so its calls keep working. Add
regression tests for both the 401 and authorized paths.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 11:40:23 +08:00
3a74a33456 feat(memhub): pair unlisted harnesses with one-time codes (#124)
* feat: pair custom harnesses with one-time codes

* fix(memhub): satisfy biome format checks for custom harness pairing

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-21 11:36:55 +08:00
514bf5c47b fix(intention): preserve isolation scope in L7 recall and dedup (#81)
Session-scoped searches build only isolation_keys and leave user_ids /
agent_ids unset. recall_intentions had no isolation_keys parameter, so the
VDB query ran without any tenancy filter and could return every user's
intentions.

The duplicate bucket also ignored user_id / agent_id. Two users with a
similar intention on the same day were treated as duplicates, and the
loser was hard-deleted through vector_store.delete().

Pass isolation_key / isolation_keys through to the vector search and add
user_id / agent_id to the dedup bucket key. Add regression tests for the
parameter forwarding, cross-user recall, and cross-user dedup.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 11:02:58 +08:00
4bc918af7a fix(coding): apply agent and session scope to coding retrieval (#87)
The public search API accepts agent_ids / session_ids, and the chat reader
applies them, but the coding path dropped both before calling the store.
A caller that relies on agent or session boundaries could therefore recall
facts and experiences written by another agent or session of the same user.

Accept the id lists in the coding store, filter them with IN clauses and
in the post-fetch checks, and forward them from _search_coding. Singular
agent_id / session_id stay supported for existing callers. Add store-level
and client-level regressions for the filtered scopes.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 11:02:52 +08:00
75163a0d1e fix(import): truncate oversized verbatim instead of wedging the job (#95)
A single record above MAX_VERBATIM_CHARACTERS failed EvidenceInput
validation. The import retried it three times, failed the whole job at the
read phase, and the byte checkpoint resumed onto the same record, so every
later record in the source stayed unimported.

Clamp verbatim and raw_verbatim in to_evidence(), keeping the head and an
explicit "...[truncated: original N chars]" marker. The identity hash still
uses the clamped raw_verbatim, and records that were previously rejected
never had a stored identity to collide with.

Add regressions for the oversized and normal paths, and update the Core test
inventory.

Refs #68

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 11:02:44 +08:00
09e4bc8a30 fix(logging): keep user_id namespaces inside the pipeline log root (#83)
PipelineLogWriter derives its subdirectory from the caller-controlled
user_id prefix before "__". That prefix was joined directly to the log
root, so an absolute prefix or ".." escaped it: user_id="/tmp/evil__u1"
created /tmp/evil and appended the daily log there.

Whitelist a single safe path component and, as defence in depth, verify
the resolved directory stays under the configured root. Sanitize
default_subdir the same way. Add regressions for absolute, parent and
nested prefixes plus a malicious default subdir.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 11:02:36 +08:00
4dbf7b6ed2 fix(auth): validate loopback origins as IP addresses (#90)
is_loopback_origin() accepted any hostname starting with "127.". That
includes ordinary DNS names such as 127.0.0.1.attacker.example, which an
attacker can point at 127.0.0.1 and use for DNS rebinding against the
local Core.

Keep the exact loopback names and parse everything else with
ipaddress.ip_address(...).is_loopback. Add regressions for rebinding-style
names and IPv6 loopback.

Co-authored-by: qiye.sjh <shijiahao_0802@youzan.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-21 11:02:29 +08:00
Jiazheng Kang 8f1c4045e9 Merge pull request #104 from Tencent-Hunyuan/docs/quantized-models-news
docs: announce Hy-Memory-30B-A3B FP8 and GGUF quantized models
2026-10-20 12:06:49 +08:00
kkjzkangandCursor 8918098fa8 docs: announce Hy-Memory-30B-A3B FP8 and GGUF quantized models
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-20 11:22:16 +08:00
0afe787c95 fix: stop authority-invalid HY retries (#78)
* fix: prevent authority-invalid HY retries

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: align coding authority with conversations

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-19 15:30:32 +08:00
67ae43f920 fix(memhub): repair visibility dirty trigger conflicts (#77)
* fix(memhub): repair visibility link trigger

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(memhub): make trigger upgrade atomic

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-19 15:30:11 +08:00
3ce89ea560 fix(sdk): serialize sqlite connection lifecycle (#73)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-19 13:02:22 +08:00
7b7fd217a8 fix(sdk): clarify metadata and allow opt-in raw get (#65)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-18 22:30:23 +08:00
7efe054e83 fix(core): park writes until a model is active (#63)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-18 21:43:24 +08:00
fdbcd9934d fix(web): distinguish scanned and saved imports (#60)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-18 21:10:10 +08:00
3171007b3c fix(core): keep initial sync behind bounded history (#59)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-18 21:00:54 +08:00
a04fcb50d5 docs: announce MemHub and CLI releases (#56)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-18 19:56:32 +08:00
a28ade2faf fix(release): hash Runtime symlinks consistently (#55)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-18 19:56:07 +08:00
1c424928a2 fix(cli): trust GitHub release asset redirects (#54)
Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-18 19:49:40 +08:00
kristinhe1111 688878171e chore(release): bump MemHub build number to 13 (#52) 2026-10-18 18:03:08 +08:00
a3f09f4cac release: merge MemHub 1.0.4 development fixes (#50)
* feat(memhub): add model usage audit and cost controls

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(memhub): typed source authorization state and managed-key registry ownership

Closes #29

Issue #29 surfaced as "WorkBuddy plugin cannot be uninstalled / source sync
cannot be turned on", but it is two independent failure domains that were both
being reported as a single opaque `desktop_action_failed`:

1. Source sync domain: the UI could not tell whether a source failed to sync
   because the grant was missing, revoked, or because the source identity or
   root path changed. Every failure collapsed into one error, so the UI could
   only say "failed".

2. Host plugin install domain: WorkBuddy shares three registry files
   (`known_marketplaces.json`, `installed_plugins.json`, `settings.json`) with
   the host. MemHub recorded a whole-file hash at install time and then refused
   to uninstall whenever the host (or a reformat) changed any byte anywhere in
   the file, even in entries MemHub does not own.

What changes

Domain errors (new `packages/host-bridge/src/domain-error.ts`)
- Closed allowlist `DOMAIN_CODES`: grant_required, source_identity_changed,
  host_uninstall_modified_managed_value, host_uninstall_unknown_owned_file,
  host_uninstall_needs_review, host_install_journal_invalid,
  host_install_recovery_required, host_permission_storage_unsafe.
- `DomainError` + `domainCodeOf()` so callers stop string-matching on
  `Error.message`. `diagnosticDomainError()` emits only the code and a
  repo-relative safe stack, never message text, so diagnostics cannot leak
  paths or payloads.
- Re-exported from `runtime-control` (host-bridge owns the type to avoid a
  dependency cycle); desktop `observeDesktopAction` now uses
  `domainCodeOf(error) ?? fallbackCode`.

Source authorization state
- `SourceAuthorizationState` enum: ok / missing_grant / revoked_grant /
  source_identity_changed / source_root_unavailable / candidate_changed, plus
  `SourceGrantStore.authorization_state()` which is read-only and fails closed.
- `/api/v1/sources` now returns `authorization_state` per source; contracts and
  the web app render a concrete next action next to the sync switch instead of
  a generic failure.
- `_change_source_sync` distinguishes sync_prepare_failed,
  sync_config_write_failed, sync_resume_failed, sync_watcher_failed (409 with
  the code as detail) while keeping the existing rollback behaviour.
- Capture refuses silently but now records a fail-open rejection marker
  (`sync_disabled` / grant_missing / grant_revoked) that counts occurrences in a
  time window and carries no payload, path, or session content.

WorkBuddy registry ownership (journal v1 -> v2)
- Ownership is now the value at MemHub's own key, not the whole file. Added
  `canonicalJson`, `registryManagedDigest`, `registryWithoutOurs`,
  `registryHoldsOnlyOurs`.
- Install: the plan records `keyPath` + managed digest; apply compares the
  managed value (CAS) instead of the whole file; the journal writes both the v1
  `installed_hash` and the v2 `installed_managed`.
- Uninstall: `resolveRegistryOwnership()` compares the v2 digest, falls back to
  the v1 hash and `isManagedRegistryEntry`, and otherwise produces a review
  finding instead of throwing. `managedRegistryRemoval()` deletes a file MemHub
  created when it still holds only our entry; removes just our key when the host
  has added its own content; and restores recorded bytes only when the rest of
  the document is unchanged, otherwise patches the current document.
- v1 journals migrate: a whole-file hash mismatch no longer blocks uninstall.

Tests
- `packages/host-bridge/src/domain-error.test.ts` (5): code is carried, message
  never reaches diagnostics, no guessing on unclassified errors, refuses to
  parse codes out of free text, allowlist is closed.
- `workbuddy-plugin-installer.test.ts` (34 total): uninstall survives a host
  reformat, removes only our key when the host changed something else, asks for
  review when a managed entry was changed, migrates a v1 journal, keeps a
  registry the host now uses, installs over a post-plan change.
- `apps/core/tests/test_source_authorization_state.py` (12): every state,
  fail-closed ordering, capture rejection names, marker contains no secret or
  path, recording never raises.
- `apps/web`: authorization note rendered for revoked grant and hidden when ok.

Verification
- host-bridge WorkBuddy suite 34/34; domain-error 5/5.
- apps/web 1906 tests across 105 files pass.
- pnpm -r typecheck clean; biome check clean on touched files.
- core-fast: 232 failed / 4131 passed, identical failure count to the clean
  baseline (232 failed / 4119 passed) with +12 from the new tests; the failures
  are pre-existing and unrelated (confirmed by stashing and re-running).

* feat(memhub): add a start/stop switch for model sedimentation

Shipped builds gave the user no way to stop model calls: the model content
permission switch is gone (removed with its endpoint in #20) and there is no
pause or remove entry either, so a configured model keeps turning
conversations into memories and keeps costing money.

Adds a user switch in settings: stopping keeps the model configuration and
the already imported conversations, leaves queued work pending instead of
failing it, and stops new work, enqueueing and backfill so nothing further
reaches the model. Starting again resumes the same work without re-entering
a key.

The flag lives in Core's own user config (config.json), so no schema
migration is needed, and the supervisor takes it as an optional callback so
every existing caller and test keeps its current behaviour.

* test(desktop): keep host-bridge mocks whole and assert the review contract

The memhub CI job failed four desktop tests after the issue #29 change. Both
were test-side contracts that the new ownership model intentionally moves, not
regressions in the fix itself.

1. `host-control-ownership.test.ts` mocked `@one-memory/host-bridge` with a
   whitelist that carried only `RecallBroker`. Main now imports `domainCodeOf`,
   which runtime-control re-exports from host-bridge, so the mock handed back
   undefined and every error path that called it threw a TypeError instead of
   the error the test was asserting on. The mock now spreads the real module
   and overrides only `RecallBroker`, which is the part it was written for.

2. `host-permissions.test.ts` asserted the pre-#29 contract, where a change
   anywhere in a shared registry made uninstall throw
   `host_uninstall_modified_files`. Ownership is now the value at our own key,
   so a change to that value is reported instead of guessed at: the plan
   schedules no removal, names the file in a review finding, and applying it
   refuses with `host_uninstall_needs_review`. The test keeps the guarantee it
   existed for, that those bytes are never overwritten, and now also asserts
   that apply refuses, so the safety is checked at both steps.

Verification: full desktop suite 38 files / 436 tests pass, `pnpm -r typecheck`
clean, biome reports only two pre-existing warnings on lines this change does
not touch.

* fix(logging): keep info diagnostics actionable

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(memhub): register the sedimentation switch test in the core test inventory

New core test files must appear in TEST_INDEX.md and in the classification
inventory's expected count and digest, and the backfill case needs a settled
turn because an unanswered user tail is not backfill material.

* feat(diagnostics): expose build info and fullscreen logs

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(settings): align help controls and tighten spacing

Co-authored-by: Cursor <cursoragent@cursor.com>

* release: prepare MemHub 1.0.4 build 8

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(memhub): clear pre-existing dev lint errors and flaky SDK dedup tests

Three things blocking CI that came in with dev and are unrelated to the
sedimentation switch itself:

- biome errors in DebugLogPanel.tsx (import order) and ModelUsageView.tsx
  (refreshToken is an intentional refresh trigger, annotated like the
  existing modelVersion trigger in R6Settings).
- test_extractor_prompt_override.py's fake LLM now accepts keyword args the
  real pipeline passes since the usage audit landed (audit_stage).
- the fake embeddings in the intention dedup tests used the process-random
  hash(), so two different texts collided roughly one run in 32 and were
  misread as duplicates; switched to a deterministic md5 index.

* fix(cli): count recall characters as code points

A recall whose hits contain an emoji outside the BMP was reported as
response_invalid even though the data and the search were fine: the CLI
cross-checked the response by summing String.prototype.length, which counts
UTF-16 code units, against Core's used_characters, which Python's len()
counts in code points. One emoji is two units here and one character there,
so any hit set with an emoji failed the check.

The sibling validator in host-bridge (context-protocol.ts) already counted
code points, so the CLI was the odd one out rather than a deliberate choice.

- validateContextResponse now sums through characterCount(), which counts
  code points, matching Core.
- Two regression tests: an emoji hit whose used_characters follows code
  points must stay grounded, and the same response counted as UTF-16 units
  must still be refused, so the cross-check is not simply loosened.
- The contract now states used_characters counts Unicode code points and is
  not an encoding length, so a client in another language does not repeat
  the comparison.

Verified by reverting the source change: the first test fails on the old
implementation and passes with it.

Closes #30

* fix(web): keep the model recovery entry reachable while a probe is pending

Closes #40

A pending model probe fenced the configuration editor, and the only
recovery entry ("核对测试结果") lived inside the area the fence kept
shut. Reaching it required opening the form; opening the form required
having recovered. People with a stuck probe could not change or repair
their model at all.

The fence itself was doing its job and still is. What was wrong is that
refusal also hid the way out.

R6Settings
- openConfiguration() now opens the model area whether or not the editor
  accepts, so the recovery entry inside it is reachable. editProfile()
  and add() still return false while the fence is up; nothing about the
  protection changed.
- The refusal notice is reworded to match: the configuration cannot be
  changed yet, and the next step is down in the area that just opened.
  Saying "cannot be opened" was both untrue now and unhelpful before.

ModelSettings
- A refused open is remembered instead of dropped. Once the fence clears,
  the intent is replayed, so the person lands in the form they asked for.
  Without this they would get a blank new-configuration form the moment
  the probe reconciled, which invites a duplicate entry.
- Replaying only fills fields in. It never sends a request, so no test is
  re-sent implicitly; "核对测试结果" remains the only thing that does.

Tests
- R6Settings: with a current model and without one, the recovery entry is
  reachable while the editor is fenced, and the refusal is still reported.
- ModelSettings: after reconciling, the form reopens on the profile that
  was asked for, and the probe is sent exactly once.
- Both new R6Settings cases and the ModelSettings case fail on the
  previous code.

* perf(runtime-control): decide runtime reuse before staging the payload

Every App launch copied the whole 373MB bundle into a temp directory and
hashed it twice before asking whether the selected runtime could be reused.
On a machine that already runs this build the answer is yes and all of that
work is thrown away: measured on a release Mac bundle, 6.9s to copy and
about 3.8s of hashing (fileDigests, then verifyInstalledDistribution over
the same tree).

installPackagedRuntime now describes the payload in place first. The staged
tree is a byte for byte copy of resources/{core,host-bridge,runtime,web}
plus two generated launchers, so the same relative paths and digests can be
read without copying anything. reuseSelectedSameBuild takes that description
instead of a staged directory, and only a payload that is going to be
installed is ever staged.

The decision itself is unchanged: it still compares every file digest except
bin/node, still repairs the packaged launcher when the App moved, and still
raises installed_version_conflict or distribution_hash_mismatch exactly as
before. Nothing was replaced by a cheaper fingerprint, because a wrong reuse
is what those errors exist to prevent.

Hashing also runs through a small worker pool now; the digests are identical
and the manifest keeps its sorted key order.

New test: describeBundlePayload must equal the manifest the staged copy
produces, so moving the decision ahead of the copy cannot decide something
different. Verified read-only against the installed release bundle:
1534 files, identical file sets, zero digest mismatches.

* perf(desktop): raise a startup window before the runtime and Core come up

Opening the App currently shows nothing at all until initialize finishes.
On this machine that is 55.6s from desktop.initializing to desktop.ready
(about 19s installing and starting the managed runtime, then about 36s for
Core to become ready), and window.requested is logged after desktop.ready.
A launch that slow with no window reads as a launch that failed, and people
click the icon again.

startDesktopLifecycle now takes two optional hooks. showStartupWindow runs
before initialize and loads a placeholder; revealStartupWindow runs after
initialize and swaps that same window onto the real app. Because the window
is reused rather than replaced, there is no second window and no flash.

The placeholder is served from the app protocol as /__startup, in memory,
not as a file in the web bundle. It performs no IPC: it renders before
initialize has registered a single handler, so anything it asked for would
fail and leave the page in an error state for the whole wait. That route is
also reserved, so a file of the same name inside the bundle can never be
reached through it.

Raising the placeholder is best effort. It only fills a wait that would
otherwise be blank; failing to draw it must not stop the services or the
real window.

Tests: five lifecycle cases cover the ordering (placeholder before
initialize, reveal after, reuse of the same window, nothing for a background
launch, and a real window when the placeholder was closed or could not be
raised). The four suites that mock ./app-protocol now spread the real module
instead of whitelisting it, which is what broke when the protocol gained
exports. host-control-ownership gains the BrowserWindow methods the
placeholder path uses, and its "no window" assertion now names the
placeholder, since a placeholder window is expected there by design.

* fix(core): stop the sedimentation switch failing its own request

Turning the switch off answered "停止未能确认,模型可能仍在整理记忆,请重试"
every time. The value was saved and the error was a lie the endpoint told
afterwards: `put_sedimentation` logged `memory_sedimentation.toggled` with
`details={"enabled": ...}`, `enabled` is not in the diagnostic allow-list, and
`_details` raises. That ValueError escaped the endpoint, so the request is
recorded in this machine's log three times as `api.request.failed` /
`core.request.unhandled` with `diagnostic_detail_key_invalid`, and the UI took
the 500 as an unconfirmed stop.

Because the raise sits between the save and the resume, starting again was
broken the same way: `resume_allowed()` never ran, so work parked by a stop
stayed parked.

Three changes, in the order they matter:

1. `enabled` is now a legal detail key. It is a bounded boolean, the same
   shape as `spawned` and `index_complete`.
2. `diagnostic_event` no longer lets a rejected payload reach the caller. It
   already swallowed `OSError` with the comment "Diagnostics must not become a
   new product outage"; an unlisted key is the same failure seen from the
   caller, and it is exactly what turned every external recall into a 500 and
   then killed the process last time. The payload is still wrong and still
   caught in CI — now by every module rather than five.
3. `test_diagnostic_payload_keys.py` walks every module instead of a
   hand-picked list. The two payloads that caused real outages were in files
   nobody thought to add to that list; the only shape of the check that holds
   the line is "every module, always". This file was already failing on dev
   with `api.py:3673: unlisted details key 'enabled'` — the bug was known to
   the suite and shipped anyway.

`DiagnosticLog.write` still raises, so the rejection itself is unchanged and
still tested; only the callers are protected.

* fix(core): let the drain claim a manual note's job

A memory added by hand stayed on "原话已保存,等待整理" indefinitely, with
`task_id: null`: the note's durable job was never claimed, so no task was ever
created for it. On this machine the job is still `pending` with `attempts=0`
more than eight hours after it was saved, while the write gate admits its
source without complaint.

The drain had two different answers to one question. `source_writes_permitted`
is `_source_writes_on`, which admits `one-memory-manual` by its grant. The
claim gate was `_config_store(app).sync_enabled`, the per-app toggle, and
`one-memory-manual` is not a connected app: it never appears in the
configuration document, so that predicate answers False for it. The job
survived the `enabled_sources` prefilter and was then dropped row by row, on
every tick, forever. Nothing ever failed and nothing ever ran.

The claim gate now asks the same question the write gate asks. Admission is
one decision; giving it two predicates is what parked the work.

Two tests pin it. One builds the drain as the composition root does and shows
the job is claimed, and shows the shipped wiring stranding the same job, so the
regression is visible if the toggle comes back. The other records the
asymmetry itself: `_source_writes_on` is True for the manual source while
`sync_enabled` is False, which is why the pair could disagree at all.

No migration and no reinstall: existing pending jobs are picked up on the next
drain tick.

* fix(memhub): make sedimentation pause stop model dispatch

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(memhub): schedule memory writes by event time

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(core): answer the sedimentation switch before the backlog walk

Turning "用模型沉淀记忆" back on wrote the choice to the store and then
awaited `resume_allowed()` inside the PUT handler. That call is not a nudge:
it runs maintenance over every unfinished task and then a whole-queue scan,
and the bounded scan waited -- holding the scan lock -- for the queue to drop
under its cohort ceiling. With a real backlog the queue never drops, so the
request never returned: one toggle in the field is still open 20+ minutes
later with no `api.request.completed`, while the renderer gave up at its 30s
timeout and showed "启用未能确认,仍显示上次确认的状态;请重试。" for a switch
that had in fact been saved.

The choice is the durable fact; the walk is follow-up work:

- The handler now stores the choice and returns it, and starts the resume as a
  watched background task (same idiom as the other Core background walks).
  Repeated toggles reuse the running walk instead of stacking.
- A bounded scan now stops at its ceiling instead of sleeping under the scan
  lock waiting for capacity, which also un-starves the periodic scan that was
  queued behind it. It reports `queue_saturated` rather than `no_tasks`, so
  the status stops claiming there is nothing to do while hundreds wait.

Verified on the field log first: the failing toggle is the only request in
the window with `api.request.started` and no `api.request.completed`, and its
trace shows `memory_sedimentation.toggled` succeeded 6ms in at
`pending_count` ~500. New tests fail on the old code (timeout) and pass now.

* fix(core): stop a blank author from erasing a whole conversation

A user turn whose host did not say who wrote it carries
`written_by=unknown`. The write binding's allow-list had every other
role/author pair but not that one, so verifying the context raised
`hy_memory_write_verification_context_scope` and the task ended as
`model_content_outcome_unknown` after two attempts: a memory the model may
well have written, that can never be confirmed, and a conversation that is
never sedimented again.

This is not a rare shape. On this machine 1078 user records across 1002
WorkBuddy conversations are `unknown`, and the failures were 42 per 15
minutes in the field log — every one of them a task reporting
`result_count=0` while 402 sat pending.

The author is missing, not the turn: the record is still inside the same
conversation, workspace and source, which is what the binding actually
checks, and the turn assembly already sent it. Accepting it here puts the
verification back in agreement with what was sent, the same reasoning that
added ("user", AGENT) for sub agent questions.

New test fails on the old code at hy_source_context.py:76 — the same frame
as the field stack — and passes now.

* fix(core): stop an abandoned read reporting a failure nobody owns

`in_read_pool` shields the wrapped worker, so a lane that runs out of budget
gives up on the answer without yanking a query out from under SQLite. The
caller cancels; the read runs on. When such an abandoned read then failed,
nothing awaited its future any more: `shield` drops its inner callback as soon
as the outer future is cancelled, so the inner future finished with an
exception that nobody retrieved. asyncio reports that at GC, and the loop
handler files it as `core.loop.task_exception` -- a red error sitting next to a
lane that had already degraded on its own budget and already said so.

On this machine that is the `raw_context_timeout` pair in the field log: a
Future, not a Task, carrying TimeoutError('raw_context_timeout'), reported
three times in half an hour. The lane result itself was right; only the
diagnostics were arguing with themselves.

Retrieving the outcome on completion keeps the diagnostics honest without
hiding anything: awaiting the read still raises for a caller who waited, which
the second case pins down.

The new case fails on the previous code (it reports the TimeoutError) and
passes now.

* fix(core): one task with no inputs no longer strands every unknown task

`MemoryTaskRunner._request` falls back to `chunk[0]` when no record is a
person-written user turn. A task whose inputs are gone yields an empty chunk,
so that fallback raised IndexError -- an error no caller can read. A reconcile
pass catches domain errors per task and moves on, so IndexError did not skip
the task: it ended the pass, and every unknown task behind it stayed unknown
while the retry backed off 1, 2, 4 ... 128 and gave up.

On this machine 68 of 303 unknown tasks had no inputs, which is exactly the
`reconcile_unknown / error_type=IndexError` backing off in the field log. Those
68 cannot be reconciled either way -- there is nothing to ask HY about -- but
the 235 that can must not be held hostage to them, and this is what stands
between a task whose model write may already have landed and its confirmation.

An empty chunk is now a domain error, so the pass skips the task and continues.
Also folds in one formatting hunk left over in this file from the previous
change, so the formatter is clean again.

The new case fails on the previous code at memory_processing.py:1036 with the
same IndexError the log shows, and passes now.

* chore(memhub): bump release build number to 10

The installer refuses a package whose bytes differ while its build number
stays the same, so the four Core fixes on dev need their own number to be
installable over the 1.0.4 / build 9 package already on the machine.

* fix(core): retire a window whose remaining rows FIFO refuses

The event-time window asked only whether its ten-minute range still held
*some* pending row before deciding the anchor was still active. A pending
row that FIFO holds back is not claimable: it may be blocked by an earlier
sibling of the same Harness whose own event time sits outside the range, and
that sibling can never be claimed from inside this window. The anchor then
stayed pinned forever and the whole queue starved.

Live shape on one machine: the anchor sat on 2026-10-07T03:24Z while the four
pending rows inside it were blocked by 929/930/1243/2347 earlier workbuddy
jobs, the earliest of them dated 2026-03-12. Completions stopped at 09:59 and
3.8k queued turns sat at zero output; restarting does not help because the
anchor is persisted.

The contract (docs/contracts/memory-event-time-scheduling-v1.md) retires a
window once every *eligible* pending/running row in the range has settled or
parked, and re-anchors on the next oldest eligible job. Reuse the claim's own
predicate — free lease, no busy Harness, no earlier unsent sibling — so "the
window is busy" and "the window can still be worked" finally agree. Anchor
selection and in-window selection already used it; only the liveness check
did not.

* fix(core): a refused source context is a failure, not an unknown write

Issue #44: an OpenCode import carried `role=user, written_by=unknown`. The
role/author set now admits it, but the row still never moved: `HyWriteBindings`
is built inside `ingest`, and the invocation marker is recorded *before* that
call, so the refusal arrived with the marker already set. The classifier
checked `hy_invoked` first and parked the task as `unknown` — a state that
means "HY may hold a write we cannot confirm" about a request that provably
never reached HY. The reason stayed off the row, the task could not be
re-driven, and the queue behind it stopped.

`hy_memory_write_verification_context_scope` is raised from that construction
alone, and `_check_source_context` (which does run after the call) raises
`hy_memory_evidence_changed` instead, so the code is unambiguous: nothing was
sent. It joins `PRE_SEND_REFUSAL_CODES`, and both classifiers now test that set
before the marker: the durable path settles `failed` with the marker cleared so
reclaim does not treat a row HY never saw as possibly-sent, and the task path
fails with the code instead of marking unknown.

Regression covers both directions: a pre-call refusal must end `failed` with
`hy_invoked=0`, and a post-call refusal (`authority_invalid`) must still be
`unknown` — the fix narrows the unknown set, it does not widen the failure set.

* chore(memhub): bump release build number to 11

Carries the window starvation fix (dc0ba49) and the pre-send refusal
fix (0df84c0). A new build number keeps the package distinguishable from
build 10 and lets it install over the current one.

* refactor(memhub): simplify durable memory scheduling

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(memhub): bump release build number to 12

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(security): parse diagnostic stack paths linearly

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(web): sync sedimentation switch status to home (#51)

---------

Co-authored-by: alvinfei <alvinfei@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: kristinhe <kristinhe@tencent.com>
2026-10-18 17:52:59 +08:00
kristinhe1111 0c59413ad8 Merge pull request #43 from Tencent-Hunyuan/kristinhe1111-patch-1
Revise README for clarity and installation instructions
2026-10-18 00:37:45 +08:00
kristinhe1111 625f877b43 Revise README for clarity and installation instructions
Updated README to clarify MemHub's functionality and provide installation guidance.
2026-10-18 00:27:38 +08:00
kristinhe1111 1608a4cfc2 feat(memhub): restore the one-click log export entry in settings (#38)
* feat(memhub): show the help and diagnostics group in settings by default

The settings group that carries the one-click log export was gated behind
VITE_ENABLE_DIAGNOSTICS=true at open-sourcing time, so shipped builds never
showed it and users could not export a diagnostics bundle. Flip the default
to visible and keep VITE_ENABLE_DIAGNOSTICS=false as the opt-out for any
build that must not offer it.

* style(memhub): format rerender call per biome
2026-10-17 22:05:24 +08:00
281 changed files with 23505 additions and 2615 deletions

No files matched your search

+1
View File
@@ -86,6 +86,7 @@ MEMORY_VECTOR_STORE=chroma
# MEMORY_CODING_DB_PATH= # coding_memory.db 路径(默认 {DATA_DIR}/data/coding_memory.db)
# MEMORY_CODING_TOOL_RESULT_MAX_BYTES=2048 # 单条 tool_result 截断字节上限
# MEMORY_CODING_VERBATIM_FALLBACK_ENABLED=false # 模型无结果时是否保存原文 fact;默认关,避免 tool trace 外泄
# MEMORY_CODING_CHAIN_BLOCK_ENABLED=false # search 是否注入版本演进链块(diff);默认关,省 token
# MEMORY_EXTRACT_HISTORY_TURNS=5 # 除本次 write 外,额外注入 extractor 的历史 user-turn 数
# ── 写入流程开关 ──
+13 -7
View File
@@ -122,19 +122,25 @@ jobs:
shell: bash
run: |
set -euo pipefail
shopt -s globstar nullglob
version=$(node -p 'require("./package.json").version')
node scripts/verify-release-version.mjs \
--channel stable \
--version "$version" \
--build-number "$(node -p 'require("./package.json").release.build_number')" \
--tag "$GITHUB_REF_NAME"
mkdir -p publish-assets
for arch in x64 arm64; do
archive="release-assets/memhub-runtime-v${version}-linux-${arch}.tar.gz"
test -f "$archive"
test -f "$archive.sha256"
name="memhub-runtime-v${version}-linux-${arch}.tar.gz"
archives=(release-assets/**/"$name")
checksums=(release-assets/**/"$name.sha256")
test "${#archives[@]}" -eq 1
test "${#checksums[@]}" -eq 1
cp -- "${archives[0]}" "publish-assets/$name"
cp -- "${checksums[0]}" "publish-assets/$name.sha256"
(
cd release-assets
sha256sum -c "$(basename "$archive.sha256")"
cd publish-assets
sha256sum -c "$name.sha256"
)
done
- name: Upload Linux assets to a draft GitHub Release
@@ -160,7 +166,7 @@ jobs:
--title "$GITHUB_REF_NAME"
fi
gh release upload "$GITHUB_REF_NAME" \
release-assets/memhub-runtime-v*-linux-*.tar.gz \
release-assets/memhub-runtime-v*-linux-*.tar.gz.sha256 \
publish-assets/memhub-runtime-v*-linux-*.tar.gz \
publish-assets/memhub-runtime-v*-linux-*.tar.gz.sha256 \
--repo "$GITHUB_REPOSITORY" \
--clobber
+69
View File
@@ -2,6 +2,40 @@
Hy Memory and MemHub are versioned and released independently.
## Hy Memory 2.0.5.dev4 — 2026-10-21
- Development preview; not the final `2.0.5` release.
- Adds idempotent `process_raw` / `async_process_raw` APIs that extract and
reconcile ordered existing L1 raw IDs without embedding or inserting raw
again.
- Returns durable raw/memory ID receipts for MemHub queue recovery.
## Hy Memory 2.0.5.dev3 — 2026-10-21
- Development preview; not the final `2.0.5` release.
- Published as
[`hy-memory==2.0.5.dev3`](https://pypi.org/project/hy-memory/2.0.5.dev3/).
- Lets the existing `search` API explicitly query SDK raw with
`layers=["l1_raw"]`, including existing source filters.
## Hy Memory 2.0.5.dev2 — 2026-10-21
- Development preview; not the final `2.0.5` release.
- Published as
[`hy-memory==2.0.5.dev2`](https://pypi.org/project/hy-memory/2.0.5.dev2/).
- Adds exact, idempotent semantic-memory import for legacy MemHub cutover
without extraction, reconciliation or identity changes.
## Hy Memory 2.0.5.dev1 — 2026-10-21
- Development preview for the MemHub SDK-authority migration; this is not the
final `2.0.5` release.
- Published as
[`hy-memory==2.0.5.dev1`](https://pypi.org/project/hy-memory/2.0.5.dev1/).
- Adds lifecycle status filtering, durable idempotent write outcomes, exact raw
APIs, offline local data-plane snapshot/restore, and fail-closed physical
delete receipts.
## Hy Memory 2.0.3 — 2026-10-17
- Added an optional runtime HTTP transport factory so embedded products can
@@ -14,6 +48,41 @@ Hy Memory and MemHub are versioned and released independently.
## Unreleased
- `search(..., layers=["l1_raw"])` now explicitly searches SDK raw records in
pro/ultra mode and keeps existing user/agent/session/source filters; default
search continues to hide raw.
- Added `import_memory_exact` for idempotent migration of an existing semantic
memory with its canonical ID, lifecycle and supersession links, without
extraction or reconciliation.
- Added SDK-authoritative lifecycle listing with explicit status filters,
idempotent durable write outcomes, exact raw import/get/list APIs, and
quiesced local data-plane snapshot/restore contracts for the MemHub
authority migration.
- Physical delete now reports incomplete VDB/Graph cleanup as a partial failure
instead of claiming success after only one store was changed.
- Coding search no longer injects the version-evolution diff block
(`chain_block`) by default. Enable with
`MEMORY_CODING_CHAIN_BLOCK_ENABLED=true` or
`CodingConfig.chain_block_enabled=True` (write-side observe unchanged).
- Fixed governed-memory revisions and repeated evidence links failing when the
visibility dirty-marker trigger encounters the same memory twice. Startup
now replaces the affected legacy trigger without changing schema versions
or governed and visibility data.
- Serialized each SDK SQLite store's shared connection lifecycle so concurrent
history/cache operations cannot race with `close()` and crash the process.
- Scoped Coding reconciliation targets to the current HY authority,
conversation, and workspace, preventing near-duplicate sessions from
mutating an experience or working fact owned by another evidence context.
- Kept post-write authority verification failures available for read-only
reconciliation while preventing automatic or generic confirmed retries from
invoking the model again. Automatic reconciliation skips those sticky rows
before applying its batch limit, and linked durable jobs exclusively own
retry scheduling for their memory tasks.
- Clarified that `add(metadata=...)` carries legacy SDK processing options,
while persistent caller business context belongs in `meta`.
- Added an opt-in `get(..., include_raw=True)` path for reading `L1_RAW`
memories, including memories written in lite mode. The default remains
hidden for backward compatibility.
- Removed MemHub's task-scoped HyPermission store wrappers and global
model-content consent switch. The trusted embedded Hy Memory SDK now uses
its native local stores directly.
+11 -9
View File
@@ -32,7 +32,9 @@ English | [中文](README.zh.md)
## Latest News
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.3`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.3/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** quantized **FP8** (**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**) and **GGUF** (**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**) versions of **Hy-Memory-30B-A3B** on Hugging Face!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **Released [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** and **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**, with a signed macOS desktop app and verified macOS/Linux Runtimes!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Released** **Hy Memory 2.0 (`hy-memory==2.0.4`)** on [PyPI](https://pypi.org/project/hy-memory/2.0.4/) and open-sourced the **Hy-Memory** monorepo together with **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **Open-sourced** **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** on Hugging Face along with its [technical report](https://arxiv.org/abs/2610.22196) on attributed reinforcement learning for long-term memory writing!
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **Released** **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)** (submitted to **ICLR 2027**).
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **Accepted** at **EMNLP 2026**: **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)**!
@@ -63,13 +65,13 @@ Welcome to the **Hy Memory Research Family**, covering cognitive chat memory, lo
### 1. Installation
```bash
pip install hy-memory==2.0.3 # core (includes ChromaDB and Kuzu)
pip install "hy-memory[qdrant]==2.0.3" # + Qdrant
pip install "hy-memory[faiss]==2.0.3" # + FAISS
pip install "hy-memory[graph]==2.0.3" # + Neo4j
pip install "hy-memory[mysql]==2.0.3" # + MySQL cache
pip install "hy-memory[rerank]==2.0.3" # + local multilingual cross-encoder rerank
pip install "hy-memory[all]==2.0.3" # everything
pip install hy-memory==2.0.4 # core (includes ChromaDB and Kuzu)
pip install "hy-memory[qdrant]==2.0.4" # + Qdrant
pip install "hy-memory[faiss]==2.0.4" # + FAISS
pip install "hy-memory[graph]==2.0.4" # + Neo4j
pip install "hy-memory[mysql]==2.0.4" # + MySQL cache
pip install "hy-memory[rerank]==2.0.4" # + local multilingual cross-encoder rerank
pip install "hy-memory[all]==2.0.4" # everything
```
### 2. Configuration
@@ -151,7 +153,7 @@ result = client.search(
result["facts"] # fact layer
result["experiences"] # experience layer
result["chain_block"] # coding only: version-evolution context
result["chain_block"] # coding only: version-evolution context (empty unless MEMORY_CODING_CHAIN_BLOCK_ENABLED)
```
### 5. Routing, Source Isolation & Advanced APIs
+11 -9
View File
@@ -32,7 +32,9 @@
## 最新动态
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.3`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.3/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **Hy-Memory-30B-A3B** 的 **FP8**(**[Hy-Memory-30B-A3B-FP8](https://huggingface.co/tencent/Hy-Memory-30B-A3B-FP8)**)与 **GGUF**(**[Hy-Memory-30B-A3B-GGUF](https://huggingface.co/tencent/Hy-Memory-30B-A3B-GGUF)**)量化版本模型!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 📦 **正式发布 [MemHub v1.0.4](https://github.com/Tencent-Hunyuan/Hy-Memory/releases/tag/v1.0.4)** 与 **[memhub-cli 1.0.5](https://www.npmjs.com/package/memhub-cli/v/1.0.5)**,提供已签名的 macOS 桌面应用及经过验证的 macOS/Linux Runtime!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **正式发布** **Hy Memory 2.0 (`hy-memory==2.0.4`)**(已上线 [PyPI](https://pypi.org/project/hy-memory/2.0.4/)),并开源 **Hy-Memory** monorepo 与 **[MemHub](memhub/README.md)**!
* *<mark>[new]</mark>* 🔥🔥🔥 **[2026-10]**: 🚀 **模型开源** 在 Hugging Face 发布 **[Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B)** 长期记忆写入模型及[技术报告](https://arxiv.org/abs/2610.22196)!
* *<mark>[new]</mark>* 🔥🔥 **[2026-09]**: 📊 **基准与方法发布** 推出长周期与交错式编程记忆评测基准 **[CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents](docs/papers/CMB_A_Coding_Memory_Benchmark_for_Long_Horizon_and_Interleaved_Coding_Agents.pdf)**(已投稿 **ICLR 2027**)。
* *<mark>[new]</mark>* 🔥🔥 **[2026-08]**: 🎉 **论文录用** 双系统认知记忆架构论文 **[Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agents](https://arxiv.org/abs/2606.09483)** 被 **EMNLP 2026** 接收!
@@ -63,13 +65,13 @@
### 1. 安装
```bash
pip install hy-memory==2.0.3 # 核心(含 ChromaDB 与 Kuzu)
pip install "hy-memory[qdrant]==2.0.3" # + Qdrant
pip install "hy-memory[faiss]==2.0.3" # + FAISS
pip install "hy-memory[graph]==2.0.3" # + Neo4j
pip install "hy-memory[mysql]==2.0.3" # + MySQL 缓存
pip install "hy-memory[rerank]==2.0.3" # + 本地多语言 cross-encoder rerank
pip install "hy-memory[all]==2.0.3" # 全部
pip install hy-memory==2.0.4 # 核心(含 ChromaDB 与 Kuzu)
pip install "hy-memory[qdrant]==2.0.4" # + Qdrant
pip install "hy-memory[faiss]==2.0.4" # + FAISS
pip install "hy-memory[graph]==2.0.4" # + Neo4j
pip install "hy-memory[mysql]==2.0.4" # + MySQL 缓存
pip install "hy-memory[rerank]==2.0.4" # + 本地多语言 cross-encoder rerank
pip install "hy-memory[all]==2.0.4" # 全部
```
### 2. 配置
@@ -151,7 +153,7 @@ result = client.search(
result["facts"] # 事实层
result["experiences"] # 经验层
result["chain_block"] # 仅 coding:版本演进上下文
result["chain_block"] # 仅 coding:版本演进上下文(默认空;需 MEMORY_CODING_CHAIN_BLOCK_ENABLED)
```
### 5. 自动路由、来源过滤与进阶 API
+1
View File
@@ -197,6 +197,7 @@ SDK 不设置 Rerank Search 并发上限。开启后模型在 `HyMemoryClient`
| `MEMORY_HISTORY_DB_PATH` | `{DATA_DIR}/data/history.db` | History SQLite 路径 |
| `MEMORY_HISTORY_RECORD_SEARCHES` | `false` | 是否记录 Search 操作 |
| `MEMORY_CODING_CHAIN_REPO_ROOT` | — | coding 版本链读取本地 Git diff 的仓库根目录 |
| `MEMORY_CODING_CHAIN_BLOCK_ENABLED` | `false` | search(scene=coding) 是否注入版本演进链块(diff / `chain_block`);默认关 |
## HTTP Server
+3 -2
View File
@@ -78,7 +78,8 @@ default 多-query 超过 `MEMORY_QUERY_REWRITE_MAX_CHARS` 时不改写:Coding
Coding 关闭则直接使用最后一条 query。
路由到 `working` / `coding` 后,Search 读取独立生产力记忆库,返回 `facts` /
`experiences`,不使用 chat 三通道结构。`coding` 还返回 `chain_block` / `chain_used`。
`experiences`,不使用 chat 三通道结构。`coding` 还返回 `chain_block` / `chain_used`
(默认空;需 `MEMORY_CODING_CHAIN_BLOCK_ENABLED` 才注入 diff 块)。
## Chat 成功返回结构
@@ -152,7 +153,7 @@ client = HyMemoryClient.from_config({
安装可选依赖:
```bash
pip install 'hy-memory[rerank]==2.0.1'
pip install 'hy-memory[rerank]==2.0.4'
```
启用后,`HyMemoryClient(...)` 构造阶段会完成模型下载/读取、device 放置和一次 dummy
+2 -3
View File
@@ -37,8 +37,7 @@ do not belong in the public source tree.
- Root GitHub workflows run SDK and MemHub checks from their respective working
directories. Native lifecycle workflows produce reviewable Actions artifacts;
tag releases remain fail-closed on public identity, signing and notarization.
- Hy Memory is published as `hy-memory==2.0.3`. MemHub pins that public
distribution; its selected npm package is `memhub-cli`, which is not yet
published.
- Hy Memory is published as `hy-memory==2.0.4`. MemHub pins that public
distribution; its selected npm package is `memhub-cli` 1.0.5.
- Hy-Memory source uses Apache-2.0 at the repository root. Third-party
components retain their own licenses and attribution notices.
+6 -2
View File
@@ -168,9 +168,11 @@ working/coding 搜索返回:
`limit` 分别限制 facts 与 experiences,因此两层最多各返回 `limit` 条。
`scene="coding"` 额外返回 `chain_block` 和 `chain_used`:
`scene="coding"` 额外返回 `chain_block` 和 `chain_used` 字段(键始终存在;默认内容为空):
- `chain_block` 是与已召回 fact 对齐的版本演进上下文;无法挂链时为 `""`。
- 注入 diff / 版本演进块需显式开启 `MEMORY_CODING_CHAIN_BLOCK_ENABLED=true`
(或 `CodingConfig.chain_block_enabled=True`)。默认关闭,以节约 prompt token。
- 开启后,`chain_block` 是与已召回 fact 对齐的版本演进上下文;无法挂链时为 `""`。
- `chain_used` 是已经进入 chain block 的 `facts` 下标。
- SDK 不会从 `facts` 删除这些条目。调用方若要拼成“chain block + 剩余 facts”,需自行按 `chain_used` 过滤。
@@ -222,10 +224,12 @@ working/coding 搜索返回:
| `MEMORY_CODING_DB_PATH` | `{DATA_DIR}/data/coding_memory.db` | working 基础 SQLite 路径;coding/trace 路径由此派生 |
| `MEMORY_CODING_TOOL_RESULT_MAX_BYTES` | `2048` | 单条 tool result 的字节上限 |
| `MEMORY_CODING_VERBATIM_FALLBACK_ENABLED` | `false` | 模型无结果时是否保存原文 fact;默认关闭,避免 assistant/tool trace 直接展示 |
| `MEMORY_CODING_CHAIN_BLOCK_ENABLED` | `false` | search 是否注入版本演进链块(`chain_block` / diff);默认关闭(opt-in) |
| `MEMORY_EXTRACT_HISTORY_TURNS` | `5` | 回读的历史 user-turn 数;当前 prompt 最终最多使用 5 |
| `MEMORY_CODING_CHAIN_REPO_ROOT` | 空 | coding 版本链读取本地 Git diff 的仓库路径 |
`HCM_*` 变量属于版本链的高级调优项,不是稳定的公共配置契约;普通接入不应依赖其默认值。
其中 `HCM_BLOCK_OFF=1` 可在已开启 `MEMORY_CODING_CHAIN_BLOCK_ENABLED` 时做消融(完全不出块)。
## 8. 定向验证
+23 -7
View File
@@ -5,14 +5,14 @@
## 安装与快速开始
```bash
pip install hy-memory==2.0.3
pip install hy-memory==2.0.4
# 可选后端
pip install 'hy-memory[qdrant]==2.0.1'
pip install 'hy-memory[faiss]==2.0.1'
pip install 'hy-memory[graph]==2.0.1' # Neo4j;Kuzu 已在核心依赖中
pip install 'hy-memory[mysql]==2.0.1'
pip install 'hy-memory[rerank]==2.0.1' # 本地多语言 cross-encoder rerank
pip install 'hy-memory[qdrant]==2.0.4'
pip install 'hy-memory[faiss]==2.0.4'
pip install 'hy-memory[graph]==2.0.4' # Neo4j;Kuzu 已在核心依赖中
pip install 'hy-memory[mysql]==2.0.4'
pip install 'hy-memory[rerank]==2.0.4' # 本地多语言 cross-encoder rerank
```
默认 LLM 和 Embedding 均使用 OpenAI-compatible 接口。使用 OpenAI 时可只设置:
@@ -106,6 +106,15 @@ result = client.add(
)
```
`metadata` 与 `meta` 不是同一字段:
- `metadata` 是兼容旧调用方保留的 SDK 处理选项,会进入内部
`WriteRequest.extra`,不保证持久化或随查询结果返回。
- `meta` 是调用方的业务上下文,会持久化并在 `search()`、`get()` 等结果中返回。
同一记忆被多次合并时,各次 `meta` 会按写入顺序保存在列表中。
需要以后读回的数据必须放在 `meta`,不要放在 `metadata`。
`scene` 的公开取值:
- `None`:普通文本直接走 chat;messages 含 tool 时执行快速场景判断。
@@ -201,7 +210,8 @@ result = client.search(
```
`working` / `coding` 路径返回 `scene="productivity"`、`facts`、`experiences`,不使用
chat 的三桶结构;`coding` 还会返回 `chain_block` 和 `chain_used`。
chat 的三桶结构;`coding` 还会返回 `chain_block` 和 `chain_used` 字段(默认空;
需 `MEMORY_CODING_CHAIN_BLOCK_ENABLED=true` 才注入 diff 块)。
#### `search_mode`
@@ -263,6 +273,8 @@ catalog 也可以在没有基础 L0 时形成只读的合成画像。
```python
memory = client.get("memory_id")
# lite 模式写入的是 L1_RAW;按 ID 读取时必须显式选择返回原文。
raw_memory = client.get("memory_id", include_raw=True)
result = client.update("memory_id", "更新后的内容")
result = client.delete("memory_id")
result = client.delete_all(user_id="user_001")
@@ -282,6 +294,10 @@ result = client.clone_user("user_001", "user_002", agent_id=None)
对应异步方法为 `async_get()`、`async_update()`、`async_delete()`、
`async_delete_all()`、`async_list_memories()`、`async_clone_user()`。
`get()` / `async_get()` 默认继续隐藏 `L1_RAW`,因此不会改变现有调用方的原文访问
边界。只有显式传入 `include_raw=True` 才返回该层;这也是 lite 模式按 ID 读取
写入内容及其 `meta` 的方式。MemHub 的 HTTP 读取接口不开放此参数。
普通 `list_memories()` 返回 `vdb` 桶,并在 Graph 有结果时附带 `graph`;传入
`workspace_id` 时返回生产力记忆的 `scene="productivity"` 和 `experiences`。
+12
View File
@@ -55,6 +55,13 @@ except Exception:
from .client import HyMemoryClient
from .inspector import MemoryInspector
from .runtime import SharedRuntime
from .snapshot import (
DataPlaneRestore,
DataPlaneSnapshot,
DataPlaneSnapshotRejected,
restore_local_data_plane,
snapshot_local_data_plane,
)
# ====== 配置 ======
from .config import MemoryConfig
@@ -111,6 +118,11 @@ __all__ = [
"HyMemoryClient",
"MemoryInspector",
"SharedRuntime",
"DataPlaneRestore",
"DataPlaneSnapshot",
"DataPlaneSnapshotRejected",
"restore_local_data_plane",
"snapshot_local_data_plane",
# 配置
"MemoryConfig",
# Embedding 端点覆盖
+2 -1
View File
@@ -22,7 +22,8 @@ logger = logging.getLogger(__name__)
# ================================================================
# 数据结构
# ================================================================@dataclass
# ================================================================
@dataclass
class SessionSummaryResult:
"""Session 摘要结果 (V2)"""
success: bool
+13 -3
View File
@@ -39,6 +39,7 @@ import logging
import hashlib
import json
import re
from datetime import datetime
from typing import Any, Dict, List, Optional
import uuid
@@ -252,6 +253,17 @@ def _series_episode_id(
return str(uuid.uuid5(uuid.NAMESPACE_URL, "hy-memory:series-episode:" + raw))
def _series_memory_at(times: List[int]) -> Optional[datetime]:
"""Earliest member timestamp as the same local wall clock it came from.
``_node_timestamp`` produces ``datetime.timestamp()`` values. Reading them
back with ``utcfromtimestamp`` turns them into a naive *UTC* datetime, which
the vector payload later re-interprets as local time and shifts by the
machine's UTC offset.
"""
return datetime.fromtimestamp(min(times)) if times else None
def _node_timestamp(node: Any) -> Optional[int]:
value = getattr(node, "memory_at", None) or getattr(node, "gmt_created", None)
if value is None:
@@ -974,8 +986,6 @@ async def finalize_series_episode_containers(
Stored text is only a title plus a structural intro. Exact L2 bodies remain in
their canonical nodes and are referenced by ID from dated timeline segments.
"""
from datetime import datetime
from ..models.memory import MemoryLayer, MemoryNode, MemoryStatus, SourceType
from ..pipelines._retrieval.session_episode import (
add_parent_episode_membership,
@@ -1061,7 +1071,7 @@ async def finalize_series_episode_containers(
segment["memory_at"] for segment in plan["timeline"]
if segment.get("memory_at") is not None
]
memory_at = datetime.utcfromtimestamp(min(times)) if times else None
memory_at = _series_memory_at(times)
content = f"{plan['title']}. {plan['intro']}".strip()
series = MemoryNode(
node_id=series_id, user_id=user_id,
+2
View File
@@ -1123,6 +1123,7 @@ class Extractor:
max_tokens=self._llm_config.agent_max_tokens,
tools=None,
tool_choice=None,
audit_stage="extract",
)
self._call_count += 1
total_tokens += response.tokens_used
@@ -1233,6 +1234,7 @@ class Extractor:
max_tokens=self._llm_config.agent_max_tokens,
tools=None,
tool_choice=None,
audit_stage="extract",
)
self._call_count += 1
total_tokens += response.tokens_used
+21 -16
View File
@@ -34,6 +34,7 @@ import os
import weakref
from ..config import MemoryConfig
from ..utils.model_call_context import model_call_stage
from ..core.workload import llm_slot
logger = logging.getLogger(__name__)
@@ -348,6 +349,7 @@ class LLMProvider:
stop: List[str] = None,
tools: Optional[List[Dict[str, Any]]] = None,
tool_choice: Optional[Any] = None,
audit_stage: Optional[str] = None,
**kwargs
) -> LLMResponse:
"""
@@ -386,15 +388,16 @@ class LLMProvider:
len(prompt), _content_fingerprint(prompt),
)
response = await self._call_llm(
prompt=prompt,
max_tokens=max_tokens,
temperature=temperature,
stop=stop,
tools=tools,
tool_choice=tool_choice,
**kwargs
)
with model_call_stage(audit_stage):
response = await self._call_llm(
prompt=prompt,
max_tokens=max_tokens,
temperature=temperature,
stop=stop,
tools=tools,
tool_choice=tool_choice,
**kwargs
)
self._total_calls += 1
self._total_tokens += response.tokens_used
@@ -595,6 +598,7 @@ class LLMProvider:
temperature: Optional[float] = None,
tools: Optional[List[Dict[str, Any]]] = None,
tool_choice: Optional[Any] = None,
audit_stage: Optional[str] = None,
) -> LLMResponse:
"""
多轮 messages 调用(支持 tool results 回传)。
@@ -622,13 +626,14 @@ class LLMProvider:
f"msgs={len(messages)} tools={len(tools) if tools else 0}"
)
response = await self._chat_completions(
messages,
max_tokens=max_tokens,
temperature=temperature,
tools=tools,
tool_choice=tool_choice,
)
with model_call_stage(audit_stage):
response = await self._chat_completions(
messages,
max_tokens=max_tokens,
temperature=temperature,
tools=tools,
tool_choice=tool_choice,
)
self._total_calls += 1
self._total_tokens += response.tokens_used
+2
View File
@@ -669,6 +669,7 @@ class MemoryReconciler:
response = await llm.complete(
prompt=prompt,
max_tokens=500,
audit_stage="reconcile",
)
text = response.content.strip()
if "```json" in text:
@@ -1383,6 +1384,7 @@ class MemoryReconciler:
response = await llm.complete(
prompt=prompt,
max_tokens=dyn_max_tokens,
audit_stage="reconcile",
)
except Exception as e:
# LLM 彻底不可用(provider 已按 max_retries 重试过,网关 500 /
+1116 -121
View File
File diff suppressed because it is too large. Load diff
+3 -1
View File
@@ -647,7 +647,9 @@ class HyCodeMem:
top-k,但它们的 diff 对当前工作区不可能有用,因此挂链前必须做同仓过滤。
"""
facts = facts or []
# 开关: HCM_BLOCK_OFF=1 → 完全不出块, 也不从 bullet 删任何 fact。
# 高级消融开关: HCM_BLOCK_OFF=1 → 完全不出块, 也不从 bullet 删任何 fact。
# 公共 opt-in 见 CodingConfig.chain_block_enabled /
# MEMORY_CODING_CHAIN_BLOCK_ENABLED(默认关;客户端在调用 decorate 前拦截)。
if os.environ.get("HCM_BLOCK_OFF") == "1":
return "", []
picked, used = [], []
+1
View File
@@ -919,6 +919,7 @@ class CodingMemoryExtractor:
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=self.max_tokens,
audit_stage="coding_extract",
)
except Exception as e:
logger.warning(f"[coding-extract] LLM call failed: {e}")
+35 -2
View File
@@ -466,7 +466,13 @@ class WorkingFactReconciler:
return [], []
candidates = await self._retrieve_candidates(
drafts, user_id=user_id, source=source,
drafts,
user_id=user_id,
agent_id=agent_id,
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
source=source,
)
try:
@@ -499,7 +505,15 @@ class WorkingFactReconciler:
# ------------------------------------------------------------
async def _retrieve_candidates(
self, drafts: List[WorkingFactDraft], *, user_id: str, source: str
self,
drafts: List[WorkingFactDraft],
*,
user_id: str,
agent_id: str,
session_id: Optional[str],
workspace_id: Optional[str],
branch: Optional[str],
source: str,
) -> List[List[WorkingFact]]:
"""每条 draft 用 statement 召回同 user 的 top-N active 事实。
@@ -512,6 +526,10 @@ class WorkingFactReconciler:
关掉时整份召回原样交给 LLM。适用于 subject 不是稳定实体名的数据集(自由文本、
位置标签),那里精确字符串相等会把同实体判成不同主体,反而拦掉真实更新。
"""
# Empty/legacy session metadata remains readable, but cannot prove
# conversation authority for a destructive SUPERSEDE.
if not session_id:
return [[] for _ in drafts]
try:
embeddings = await self.embed_service.embed_batch(
[d.statement for d in drafts]
@@ -525,6 +543,10 @@ class WorkingFactReconciler:
try:
hits = await self.store.search_facts(
emb, user_id=user_id, top=self.candidates_per_draft,
agent_id=agent_id,
session_id=session_id,
workspace_id=workspace_id,
branch=branch,
# 写入侧同样接词面通道:候选选错了 LLM 就没有机会判对。这里不传
# min_score——写入侧要的是"有哪些可能的前身",让 LLM 看更宽的名单,
# 阈值过滤是读取侧的事。
@@ -571,6 +593,7 @@ class WorkingFactReconciler:
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=self.max_tokens,
audit_stage="coding_fact_reconcile",
)
return self._parse_ops(resp.content)
@@ -637,6 +660,7 @@ class WorkingFactReconciler:
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=_CHAIN_SUMMARY_MAX_TOKENS,
audit_stage="coding_fact_summary",
)
except Exception as e:
logger.warning(f"[fact-reconcile] chain summary failed: {e}")
@@ -792,6 +816,15 @@ class WorkingFactReconciler:
old = await self.store.get_fact(old_id, user_id=user_id)
if old is None:
continue
if (
old.agent_id != agent_id
or old.session_id != session_id
or old.workspace_id != workspace_id
):
logger.warning(
f"[fact-reconcile] refused cross-authority SUPERSEDE: {old.fact_id}"
)
continue
# 闸门在候选阶段已经拦过一道,这里再拦一次:候选是按 statement 召回的,
# 而 LLM 只能覆盖候选名单里的 id,理论上到不了这儿。真到了就说明前面
# 哪一环漏了,宁可让这次覆盖不生效——多留一条旧事实是可恢复的,删掉
+11 -3
View File
@@ -29,6 +29,9 @@ logger = logging.getLogger(__name__)
_JUDGE_MAX_TURNS = 20
_JUDGE_HEAD_TURNS = 5
_JUDGE_USER_CONTENT_LIMIT = 1_000
_JUDGE_USER_CONTENT_HEAD = 200
_JUDGE_USER_CONTENT_TAIL = 200
_JUDGE_MAX_QUERY_CHARS = 300
_JUDGE_MAX_QUERIES = 20
@@ -186,6 +189,7 @@ async def classify_messages_scene(
prompt=prompt,
max_tokens=max_tokens,
temperature=temperature,
audit_stage="coding_judge",
)
_elapsed = round((time.perf_counter() - _t0) * 1000, 2)
result = _parse_scene_json(resp.content)
@@ -631,9 +635,13 @@ def _format_turns(summary: List[Dict[str, Any]]) -> str:
rows = []
for t in summary:
user = t.get("user", "") or ""
# 单 turn 用户文本截到 200 字符
if len(user) > 200:
user = user[:200] + "..."
if len(user) > _JUDGE_USER_CONTENT_LIMIT:
omitted = len(user) - _JUDGE_USER_CONTENT_HEAD - _JUDGE_USER_CONTENT_TAIL
user = (
user[:_JUDGE_USER_CONTENT_HEAD]
+ f"\n[...{omitted} chars omitted...]\n"
+ user[-_JUDGE_USER_CONTENT_TAIL:]
)
tools = t.get("tools", []) or []
rows.append(
f"- turn {t.get('turn', '?')}: user={user!r}, tools={tools}"
+34 -1
View File
@@ -240,7 +240,7 @@ class CodingMemoryReconciler:
)
# 3. 校验 + 执行
executed = await self._execute(ops, drafts)
executed = await self._execute(ops, drafts, candidates_per_draft)
logger.info(
f"[coding-reconcile] drafts={len(drafts)} ops={len(executed)} "
+ ", ".join(f"{a}={sum(1 for o in executed if o.action == a)}"
@@ -263,6 +263,12 @@ class CodingMemoryReconciler:
"""
result: List[List[CodingMemory]] = []
for d in drafts:
# A coding mutation must be attributable to one conversation.
# Empty/legacy session metadata stays readable through ordinary
# search, but it cannot authorize UPDATE/DELETE.
if not d.session_id:
result.append([])
continue
try:
# task 为主入口,search_keys 为附加入口(去重、去空)
queries = [d.task] + [k for k in (d.search_keys or []) if k and k != d.task]
@@ -274,6 +280,8 @@ class CodingMemoryReconciler:
hits = await self.store.search_by_query_embedding(
emb,
user_id=d.user_id,
agent_id=d.agent_id,
session_id=d.session_id,
workspace_id=d.workspace_id,
branch=d.branch,
top=self.candidates_per_draft * 4,
@@ -310,6 +318,7 @@ class CodingMemoryReconciler:
resp = await self.llm_provider.complete(
prompt=prompt,
max_tokens=self.max_tokens,
audit_stage="coding_reconcile",
)
return self._parse_ops(resp.content), prompt, resp.content, resp
@@ -414,6 +423,7 @@ class CodingMemoryReconciler:
self,
ops: List[ReconcileOp],
drafts: List[CodingMemoryDraft],
candidates_per_draft: List[List[CodingMemory]],
) -> List[ReconcileOp]:
"""
校验 + 执行 ops。返回成功执行的 ops(filter 掉无效 op)。
@@ -453,6 +463,11 @@ class CodingMemoryReconciler:
if not op.target_experience_id:
logger.warning(f"[coding-reconcile] drop UPDATE missing target: {op.to_dict()}")
continue
if not self._target_was_offered(op, candidates_per_draft):
logger.warning(
f"[coding-reconcile] drop UPDATE outside authority scope: {op.to_dict()}"
)
continue
if op.draft_idx in consumed:
logger.warning(f"[coding-reconcile] drop UPDATE on consumed draft: {op.to_dict()}")
continue
@@ -475,6 +490,11 @@ class CodingMemoryReconciler:
if not op.target_experience_id:
logger.warning(f"[coding-reconcile] drop DELETE missing target: {op.to_dict()}")
continue
if not self._target_was_offered(op, candidates_per_draft):
logger.warning(
f"[coding-reconcile] drop DELETE outside authority scope: {op.to_dict()}"
)
continue
# DELETE 需要知道 user_id;通过 draft_idx(如有)或第一个 draft 推断
user_id = self._infer_user_id(op, drafts)
if not user_id:
@@ -511,6 +531,19 @@ class CodingMemoryReconciler:
return executed
@staticmethod
def _target_was_offered(
op: ReconcileOp,
candidates_per_draft: List[List[CodingMemory]],
) -> bool:
"""Only mutate a target frozen into this draft's scoped candidate set."""
if op.draft_idx is None or not (0 <= op.draft_idx < len(candidates_per_draft)):
return False
return any(
candidate.experience_id == op.target_experience_id
for candidate in candidates_per_draft[op.draft_idx]
)
@staticmethod
def _infer_user_id(
op: ReconcileOp, drafts: List[CodingMemoryDraft]
+98 -15
View File
@@ -209,6 +209,14 @@ def _scope_where(
return sql, params
def _in_clause(column: str, values: Optional[List[str]]) -> Tuple[str, List[Any]]:
"""SQL ``column IN (...)`` + parameters; empty values mean no condition."""
if not values:
return "", []
placeholders = ",".join("?" for _ in values)
return f"{column} IN ({placeholders})", list(values)
def _scope_matches(
row_workspace_id: Optional[str],
row_branch: Optional[str],
@@ -529,10 +537,20 @@ class CodingMemoryStore:
"""覆盖式更新一条 coding memory。"""
# 拿旧 keys 长度,先删旧 VDB rows
prev = await self.get_by_id(memory.experience_id, user_id=memory.user_id)
prev_key_count = (1 + len(prev.search_keys)) if prev else 0
if prev is None:
raise ValueError("coding_memory_update_target_missing")
if (
not prev.session_id
or not memory.session_id
or prev.agent_id != memory.agent_id
or prev.session_id != memory.session_id
or prev.workspace_id != memory.workspace_id
):
raise ValueError("coding_memory_update_authority_mismatch")
prev_key_count = 1 + len(prev.search_keys)
memory.updated_at = datetime.utcnow()
if memory.created_at is None and prev is not None:
if memory.created_at is None:
memory.created_at = prev.created_at
await _run_in_pool(self._upsert_sql_sync, memory)
# 删旧 VDB
@@ -671,6 +689,10 @@ class CodingMemoryStore:
query_embedding: List[float],
user_id: str,
*,
agent_id: Optional[str] = None,
session_id: Optional[str] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
workspace_id: Optional[str] = None,
branch: Optional[str] = None,
top: int = 40,
@@ -688,21 +710,34 @@ class CodingMemoryStore:
},
...
]
workspace_id / branch 是作用域过滤,规则见 _scope_where:不传则纯语义召回
(行为与加过滤前完全一致);传了则只保留 SQLite 里落在作用域内的经验,并按
作用域的选择率放大向量取数,避免"先取 top-N 再过滤"把召回吃掉。
agent_id / session_id(或 agent_ids / session_ids)是写授权作用域,
传入时精确匹配;workspace_id / branch 的规则见 _scope_where。
不传任何作用域则维持纯语义召回;传了则只保留 SQLite 里落在作用域内的经验,
并按选择率放大向量取数,避免"先取 top-N 再过滤"把召回吃掉。
"""
if self._vec is None:
return []
effective_agent_ids = list(agent_ids) if agent_ids else (
[agent_id] if agent_id else []
)
effective_session_ids = list(session_ids) if session_ids else (
[session_id] if session_id else []
)
allowed: Optional[Set[str]] = None
if workspace_id:
if effective_agent_ids or effective_session_ids or workspace_id:
allowed, total = await _run_in_pool(
self._scoped_experience_ids_sync, user_id, workspace_id, branch
self._authority_scoped_experience_ids_sync,
user_id,
effective_agent_ids,
effective_session_ids,
workspace_id,
branch,
)
if not allowed:
logger.info(
f"[coding-store] no experience in scope: "
f"agent_id={agent_id!r} session_id={session_id!r} "
f"workspace_id={workspace_id!r} branch={branch!r} "
f"(user has {total}); skip vdb search"
)
@@ -778,6 +813,10 @@ class CodingMemoryStore:
query_embedding: List[float],
user_id: str,
*,
agent_id: Optional[str] = None,
session_id: Optional[str] = None,
agent_ids: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
workspace_id: Optional[str] = None,
branch: Optional[str] = None,
top: int = 5,
@@ -791,8 +830,9 @@ class CodingMemoryStore:
SUPERSEDE 时向量行已被删掉,理论上召不到 superseded;这里再按 SQLite 的
status 过一道,避免两边不一致时把过期结论放出去。
workspace_id / branch 是作用域过滤,规则见 _scope_where,语义与经验层一致:
不传不过滤,传了按 SQLite 里的值筛,并按选择率放大向量取数。
agent_id / session_id(或 agent_ids / session_ids)是写授权作用域,
传入时精确匹配;workspace_id / branch 的规则见 _scope_where。
不传任何作用域则不过滤,传了按 SQLite 里的值筛,并按选择率放大向量取数。
向量库按 top 的 4 倍过采样:向量行和 SQLite 行不是一个事务里写的,中间挂掉
就会留下"向量还在、SQLite 没了"的孤儿,软删时删向量失败也会留下 superseded
@@ -808,14 +848,27 @@ class CodingMemoryStore:
if self._fact_vec is None:
return []
effective_agent_ids = list(agent_ids) if agent_ids else (
[agent_id] if agent_id else []
)
effective_session_ids = list(session_ids) if session_ids else (
[session_id] if session_id else []
)
fetch = top * 4
if workspace_id:
if effective_agent_ids or effective_session_ids or workspace_id:
scoped, total = await _run_in_pool(
self._count_active_facts_sync, user_id, workspace_id, branch
self._count_active_facts_sync,
user_id,
effective_agent_ids,
effective_session_ids,
workspace_id,
branch,
)
if not scoped:
logger.info(
f"[coding-store] no active fact in scope: "
f"agent_ids={effective_agent_ids!r} "
f"session_ids={effective_session_ids!r} "
f"workspace_id={workspace_id!r} branch={branch!r} "
f"(user has {total}); skip fact vdb search"
)
@@ -852,6 +905,8 @@ class CodingMemoryStore:
alive = [
(f, scores[f.fact_id]) for f in facts
if f.status == "active"
and (not effective_agent_ids or f.agent_id in effective_agent_ids)
and (not effective_session_ids or f.session_id in effective_session_ids)
and _scope_matches(f.workspace_id, f.branch, workspace_id, branch)
]
kept = [(f, s) for f, s in alive if s >= min_score]
@@ -1050,14 +1105,28 @@ class CodingMemoryStore:
by_id = {r["experience_id"]: _row_to_memory(r) for r in rows}
return [by_id[eid] for eid in experience_ids if eid in by_id]
def _scoped_experience_ids_sync(
self, user_id: str, workspace_id: str, branch: Optional[str]
def _authority_scoped_experience_ids_sync(
self,
user_id: str,
agent_ids: Optional[List[str]],
session_ids: Optional[List[str]],
workspace_id: Optional[str],
branch: Optional[str],
) -> Tuple[Set[str], int]:
"""返回 (落在作用域内的 experience_id 集合, 该 user 的经验总数)。
"""返回 (落在授权作用域内的 experience_id 集合, 该 user 的经验总数)。
一次查询同时拿到过滤结果和选择率分母:前者供向量命中做交集,后者定放大倍数。
agent_ids / session_ids 为空表示该维度不筛;非空按 IN 精确匹配。
"""
scope_sql, scope_params = _scope_where(workspace_id, branch)
agent_sql, agent_params = _in_clause("agent_id", agent_ids)
if agent_sql:
scope_sql = f"({agent_sql}) AND ({scope_sql})"
scope_params = [*agent_params, *scope_params]
session_sql, session_params = _in_clause("session_id", session_ids)
if session_sql:
scope_sql = f"({session_sql}) AND ({scope_sql})"
scope_params = [*session_params, *scope_params]
with self._lock:
assert self._conn is not None
cur = self._conn.execute(
@@ -1069,13 +1138,27 @@ class CodingMemoryStore:
return {r["experience_id"] for r in rows if r["in_scope"]}, len(rows)
def _count_active_facts_sync(
self, user_id: str, workspace_id: str, branch: Optional[str]
self,
user_id: str,
agent_ids: Optional[List[str]],
session_ids: Optional[List[str]],
workspace_id: Optional[str],
branch: Optional[str],
) -> Tuple[int, int]:
"""返回 (作用域内 active 事实数, 该 user 的 active 事实总数)。
事实层的过滤在 get_facts 拿到对象之后用 _scope_matches 做,这里只要选择率。
agent_ids / session_ids 为空表示该维度不筛;非空按 IN 精确匹配。
"""
scope_sql, scope_params = _scope_where(workspace_id, branch)
agent_sql, agent_params = _in_clause("agent_id", agent_ids)
if agent_sql:
scope_sql = f"({agent_sql}) AND ({scope_sql})"
scope_params = [*agent_params, *scope_params]
session_sql, session_params = _in_clause("session_id", session_ids)
if session_sql:
scope_sql = f"({session_sql}) AND ({scope_sql})"
scope_params = [*session_params, *scope_params]
with self._lock:
assert self._conn is not None
row = self._conn.execute(
+7
View File
@@ -1136,6 +1136,9 @@ class CodingConfig:
# LLM 失败或没有抽出 fact 时,是否把对话原文存成 working fact。
# 默认关闭,避免 assistant/tool 轨迹以 statement 形式直接暴露给消费方。
verbatim_fallback_enabled: bool = None
# search(scene="coding") 是否注入版本演进链块(diff / chain_block)。
# 默认关闭(opt-in):中重度 coding 用户可省下大量 prompt token;写入侧仍可 observe。
chain_block_enabled: bool = None
def __post_init__(self):
if self.enable is None:
@@ -1153,6 +1156,10 @@ class CodingConfig:
self.verbatim_fallback_enabled = _get_env_bool(
"MEMORY_CODING_VERBATIM_FALLBACK_ENABLED", False
)
if self.chain_block_enabled is None:
self.chain_block_enabled = _get_env_bool(
"MEMORY_CODING_CHAIN_BLOCK_ENABLED", False
)
@dataclass
+31
View File
@@ -114,6 +114,37 @@ class CacheBase(ABC):
"""查询 pipeline 中间结果日志"""
...
# ================================================================
# Durable write outcomes
# ================================================================
@abstractmethod
async def claim_write_outcome(
self,
request_id: str,
user_id: str,
agent_id: str,
session_id: str,
operation: str,
) -> Optional[Dict[str, Any]]:
"""Claim one idempotent write; return the existing row if already claimed."""
...
@abstractmethod
async def finish_write_outcome(
self,
request_id: str,
status: str,
result: Dict[str, Any],
) -> bool:
"""Persist the terminal/unknown result for a claimed write."""
...
@abstractmethod
async def get_write_outcome(self, request_id: str) -> Optional[Dict[str, Any]]:
"""Read one durable write outcome by exact request id."""
...
# ================================================================
# Summary Buffer(rolling summary 的未消费 turn 暂存)
#
+130 -16
View File
@@ -81,6 +81,21 @@ _CREATE_TABLES_SQL = [
)
""",
"""
CREATE TABLE IF NOT EXISTS write_outcomes (
request_id VARCHAR(256) NOT NULL,
user_id VARCHAR(255) NOT NULL,
agent_id VARCHAR(255) NOT NULL DEFAULT '',
session_id VARCHAR(255) NOT NULL DEFAULT '',
operation VARCHAR(32) NOT NULL,
status VARCHAR(32) NOT NULL,
result_json LONGTEXT NOT NULL,
created_at VARCHAR(64) NOT NULL,
updated_at VARCHAR(64) NOT NULL,
PRIMARY KEY (request_id),
KEY idx_write_outcomes_owner (user_id, agent_id, session_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
""",
"""
CREATE TABLE IF NOT EXISTS system_metrics (
minute_ts VARCHAR(32) NOT NULL,
data LONGTEXT NOT NULL,
@@ -459,29 +474,128 @@ class MysqlCache(CacheBase):
logger.warning(f"get_pipeline_logs failed: {e}")
return []
async def claim_write_outcome(
self,
request_id: str,
user_id: str,
agent_id: str,
session_id: str,
operation: str,
) -> Optional[Dict[str, Any]]:
now = datetime.now().isoformat()
inserted = await self._execute(
"""
INSERT IGNORE INTO write_outcomes (
request_id, user_id, agent_id, session_id, operation,
status, result_json, created_at, updated_at
) VALUES (%s, %s, %s, %s, %s, 'running', '{}', %s, %s)
""",
(request_id, user_id, agent_id, session_id, operation, now, now),
)
if inserted == 1:
return None
return await self.get_write_outcome(request_id)
async def finish_write_outcome(
self,
request_id: str,
status: str,
result: Dict[str, Any],
) -> bool:
updated = await self._execute(
"""
UPDATE write_outcomes
SET status = %s, result_json = %s, updated_at = %s
WHERE request_id = %s
""",
(
status,
json.dumps(result, ensure_ascii=False, default=str),
datetime.now().isoformat(),
request_id,
),
)
return updated == 1
async def get_write_outcome(self, request_id: str) -> Optional[Dict[str, Any]]:
row = await self._fetchone(
"SELECT * FROM write_outcomes WHERE request_id = %s",
(request_id,),
)
if row is None:
return None
try:
row["result"] = json.loads(row.pop("result_json") or "{}")
except (TypeError, json.JSONDecodeError):
row["result"] = {}
return row
# ================================================================
# System Metrics(分钟粒度落盘)
# ================================================================
async def store_metrics_minute(self, minute_ts: str, data: dict) -> None:
import aiomysql
try:
now = datetime.now().isoformat()
# 尝试 UPSERT
existing = await self._fetchone(
"SELECT data FROM system_metrics WHERE minute_ts = %s", (minute_ts,)
)
if existing:
old_data = json.loads(existing["data"])
merged = self._merge_metric_buckets(old_data, data)
await self._execute(
"UPDATE system_metrics SET data = %s, created_at = %s WHERE minute_ts = %s",
(json.dumps(merged, default=str), now, minute_ts),
)
else:
await self._execute(
"INSERT INTO system_metrics (minute_ts, data, created_at) VALUES (%s, %s, %s)",
(minute_ts, json.dumps(data, default=str), now),
)
# Read-merge-write must be one transaction: two instances flushing
# the same minute used to overwrite each other's counters, and a
# concurrent insert could lose the whole batch to a duplicate key.
for attempt in range(2):
try:
async with self._pool.acquire() as conn:
await conn.begin()
try:
async with conn.cursor(aiomysql.DictCursor) as cur:
await cur.execute(
"SELECT data FROM system_metrics "
"WHERE minute_ts = %s FOR UPDATE",
(minute_ts,),
)
row = await cur.fetchone()
if row is not None:
old_raw = (
row["data"]
if isinstance(row, dict)
else row[0]
)
merged = self._merge_metric_buckets(
json.loads(old_raw), data,
)
await cur.execute(
"UPDATE system_metrics SET data = %s, "
"created_at = %s WHERE minute_ts = %s",
(
json.dumps(merged, default=str),
now,
minute_ts,
),
)
else:
await cur.execute(
"INSERT INTO system_metrics "
"(minute_ts, data, created_at) "
"VALUES (%s, %s, %s)",
(
minute_ts,
json.dumps(data, default=str),
now,
),
)
await conn.commit()
return
except Exception:
await conn.rollback()
raise
except (aiomysql.IntegrityError, aiomysql.OperationalError) as error:
code = error.args[0] if error.args else None
deadlock = code in (1205, 1213)
if attempt or (isinstance(error, aiomysql.OperationalError) and not deadlock):
raise
# A competing writer inserted the row, or InnoDB chose us
# as the deadlock victim. Retry so this batch merges.
continue
except Exception as e:
logger.warning(f"store_metrics_minute failed: {e}")
+232 -125
View File
@@ -7,7 +7,7 @@ Agent Memory V2 - SqliteCache
设计模式跟随 history_store.py:
- 标准库 sqlite3 + _run_in_sqlite_pool()
- WAL 模式 + check_same_thread=False
- threading.Lock 保护写操作
- 每实例 threading.RLock 保护共享连接的全部操作与关闭
三张表:
- memory_operations / pipeline_logs / system_metrics: 审计与观测
@@ -87,6 +87,20 @@ CREATE TABLE IF NOT EXISTS pipeline_logs_v2 (
CREATE INDEX IF NOT EXISTS idx_plog_request ON pipeline_logs_v2(request_id);
CREATE INDEX IF NOT EXISTS idx_plog_user ON pipeline_logs_v2(user_id);
CREATE TABLE IF NOT EXISTS write_outcomes (
request_id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
agent_id TEXT NOT NULL DEFAULT '',
session_id TEXT NOT NULL DEFAULT '',
operation TEXT NOT NULL,
status TEXT NOT NULL,
result_json TEXT NOT NULL DEFAULT '{}',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_write_outcomes_owner
ON write_outcomes(user_id, agent_id, session_id);
CREATE TABLE IF NOT EXISTS system_metrics (
minute_ts TEXT PRIMARY KEY,
data TEXT NOT NULL,
@@ -122,7 +136,10 @@ class SqliteCache(CacheBase):
def __init__(self, config: MemoryConfig):
self.config = config
self._conn: Optional[sqlite3.Connection] = None
self._lock = threading.Lock()
# One connection is shared by the executor workers. Use the same lock
# for reads, writes, initialization and close so close() cannot free
# the connection while another worker is inside sqlite3.
self._lock = threading.RLock()
# 读取 db_path
cache_cfg = getattr(config, "cache", None)
@@ -144,35 +161,37 @@ class SqliteCache(CacheBase):
logger.debug(f"SqliteCache initialized: {self._db_path}")
def _init_sync(self) -> None:
db_dir = os.path.dirname(self._db_path)
if db_dir:
os.makedirs(db_dir, exist_ok=True)
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
self._conn.row_factory = sqlite3.Row
self._conn.execute("PRAGMA journal_mode=WAL;")
self._conn.executescript(_CREATE_TABLES_SQL)
self._migrate_pipeline_logs_columns()
self._conn.commit()
with self._lock:
db_dir = os.path.dirname(self._db_path)
if db_dir:
os.makedirs(db_dir, exist_ok=True)
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
self._conn.row_factory = sqlite3.Row
self._conn.execute("PRAGMA journal_mode=WAL;")
self._conn.executescript(_CREATE_TABLES_SQL)
self._migrate_pipeline_logs_columns()
self._conn.commit()
def _migrate_pipeline_logs_columns(self) -> None:
"""给旧库的 pipeline_logs_v2 补 namespace/session/event time 列。"""
try:
cols = {
row["name"]
for row in self._conn.execute("PRAGMA table_info(pipeline_logs_v2)")
}
_text_cols = ("namespace", "session_id")
for col in _text_cols:
if col not in cols:
with self._lock:
try:
cols = {
row["name"]
for row in self._conn.execute("PRAGMA table_info(pipeline_logs_v2)")
}
_text_cols = ("namespace", "session_id")
for col in _text_cols:
if col not in cols:
self._conn.execute(
f"ALTER TABLE pipeline_logs_v2 ADD COLUMN {col} TEXT NOT NULL DEFAULT ''"
)
if "event_at_ms" not in cols:
self._conn.execute(
f"ALTER TABLE pipeline_logs_v2 ADD COLUMN {col} TEXT NOT NULL DEFAULT ''"
"ALTER TABLE pipeline_logs_v2 ADD COLUMN event_at_ms INTEGER NOT NULL DEFAULT 0"
)
if "event_at_ms" not in cols:
self._conn.execute(
"ALTER TABLE pipeline_logs_v2 ADD COLUMN event_at_ms INTEGER NOT NULL DEFAULT 0"
)
except Exception as e:
logger.debug(f"[sqlite] pipeline_logs_v2 column migration skipped: {e}")
except Exception as e:
logger.debug(f"[sqlite] pipeline_logs_v2 column migration skipped: {e}")
async def close(self) -> None:
if self._conn is not None:
@@ -180,9 +199,10 @@ class SqliteCache(CacheBase):
logger.info("SqliteCache closed")
def _close_sync(self) -> None:
if self._conn:
self._conn.close()
self._conn = None
with self._lock:
if self._conn:
self._conn.close()
self._conn = None
# ================================================================
# 统计
@@ -255,39 +275,40 @@ class SqliteCache(CacheBase):
"""
try:
def _query():
conditions = []
params = []
if request_id:
conditions.append("request_id = ?")
params.append(request_id)
if memory_id:
conditions.append("memory_id = ?")
params.append(memory_id)
if user_id:
conditions.append("user_id = ?")
params.append(user_id)
with self._lock:
conditions = []
params = []
if request_id:
conditions.append("request_id = ?")
params.append(request_id)
if memory_id:
conditions.append("memory_id = ?")
params.append(memory_id)
if user_id:
conditions.append("user_id = ?")
params.append(user_id)
if not conditions:
return []
if not conditions:
return []
where = " AND ".join(conditions)
rows = self._conn.execute(
f"SELECT * FROM memory_operations_v2 WHERE {where} "
f"ORDER BY id DESC LIMIT ?",
params + [limit],
).fetchall()
result = []
for r in rows:
row = dict(r)
# 反序列化 supersedes JSON 字符串 → list
import json as _json
raw_sup = row.get("supersedes", "")
try:
row["supersedes"] = _json.loads(raw_sup) if raw_sup else []
except Exception:
row["supersedes"] = []
result.append(row)
return result
where = " AND ".join(conditions)
rows = self._conn.execute(
f"SELECT * FROM memory_operations_v2 WHERE {where} "
f"ORDER BY id DESC LIMIT ?",
params + [limit],
).fetchall()
result = []
for r in rows:
row = dict(r)
# 反序列化 supersedes JSON 字符串 → list
import json as _json
raw_sup = row.get("supersedes", "")
try:
row["supersedes"] = _json.loads(raw_sup) if raw_sup else []
except Exception:
row["supersedes"] = []
result.append(row)
return result
return await _run_in_sqlite_pool(_query)
except Exception as e:
@@ -369,43 +390,126 @@ class SqliteCache(CacheBase):
"""
try:
def _query():
conditions = []
params = []
if request_id:
conditions.append("request_id = ?")
params.append(request_id)
if user_id:
conditions.append("user_id = ?")
params.append(user_id)
if step:
conditions.append("step = ?")
params.append(step)
with self._lock:
conditions = []
params = []
if request_id:
conditions.append("request_id = ?")
params.append(request_id)
if user_id:
conditions.append("user_id = ?")
params.append(user_id)
if step:
conditions.append("step = ?")
params.append(step)
if not conditions:
return []
if not conditions:
return []
where = " AND ".join(conditions)
rows = self._conn.execute(
f"SELECT * FROM pipeline_logs_v2 WHERE {where} "
f"ORDER BY id ASC LIMIT ?",
params + [limit],
).fetchall()
out = [dict(r) for r in rows]
# 派生可读的 event_at(ISO ms)供查询接口直接用;event_at_ms 为 0 时留空
from datetime import datetime as _dt
for _r in out:
_ms = _r.get("event_at_ms") or 0
_r["event_at"] = (
_dt.fromtimestamp(_ms / 1000.0).isoformat(timespec="milliseconds")
if _ms else ""
)
return out
where = " AND ".join(conditions)
rows = self._conn.execute(
f"SELECT * FROM pipeline_logs_v2 WHERE {where} "
f"ORDER BY id ASC LIMIT ?",
params + [limit],
).fetchall()
out = [dict(r) for r in rows]
# 派生可读的 event_at(ISO ms)供查询接口直接用;event_at_ms 为 0 时留空
from datetime import datetime as _dt
for _r in out:
_ms = _r.get("event_at_ms") or 0
_r["event_at"] = (
_dt.fromtimestamp(_ms / 1000.0).isoformat(timespec="milliseconds")
if _ms else ""
)
return out
return await _run_in_sqlite_pool(_query)
except Exception as e:
logger.warning(f"get_pipeline_logs failed: {e}")
return []
# ================================================================
# Durable write outcomes
# ================================================================
async def claim_write_outcome(
self,
request_id: str,
user_id: str,
agent_id: str,
session_id: str,
operation: str,
) -> Optional[Dict[str, Any]]:
def _claim():
with self._lock:
now = datetime.now().isoformat()
cursor = self._conn.execute(
"""
INSERT OR IGNORE INTO write_outcomes (
request_id, user_id, agent_id, session_id, operation,
status, result_json, created_at, updated_at
) VALUES (?, ?, ?, ?, ?, 'running', '{}', ?, ?)
""",
(request_id, user_id, agent_id, session_id, operation, now, now),
)
self._conn.commit()
if cursor.rowcount == 1:
return None
row = self._conn.execute(
"SELECT * FROM write_outcomes WHERE request_id = ?",
(request_id,),
).fetchone()
return dict(row) if row else None
existing = await _run_in_sqlite_pool(_claim)
if existing is None:
return None
return await self.get_write_outcome(request_id)
async def finish_write_outcome(
self,
request_id: str,
status: str,
result: Dict[str, Any],
) -> bool:
def _finish():
with self._lock:
cursor = self._conn.execute(
"""
UPDATE write_outcomes
SET status = ?, result_json = ?, updated_at = ?
WHERE request_id = ?
""",
(
status,
json.dumps(result, ensure_ascii=False, default=str),
datetime.now().isoformat(),
request_id,
),
)
self._conn.commit()
return cursor.rowcount == 1
return await _run_in_sqlite_pool(_finish)
async def get_write_outcome(self, request_id: str) -> Optional[Dict[str, Any]]:
def _get():
with self._lock:
row = self._conn.execute(
"SELECT * FROM write_outcomes WHERE request_id = ?",
(request_id,),
).fetchone()
return dict(row) if row else None
row = await _run_in_sqlite_pool(_get)
if row is None:
return None
try:
row["result"] = json.loads(row.pop("result_json") or "{}")
except (TypeError, json.JSONDecodeError):
row["result"] = {}
return row
# ================================================================
# System Metrics(分钟粒度落盘)
# ================================================================
@@ -414,26 +518,27 @@ class SqliteCache(CacheBase):
"""存储一个分钟桶的增量指标数据(UPSERT: 同一分钟多次写入会合并)"""
try:
def _store():
now = datetime.now().isoformat()
# UPSERT: 如果同一分钟已存在,合并数据
existing = self._conn.execute(
"SELECT data FROM system_metrics WHERE minute_ts = ?",
(minute_ts,),
).fetchone()
if existing:
import json as _json
old_data = _json.loads(existing["data"])
merged = self._merge_metric_buckets(old_data, data)
self._conn.execute(
"UPDATE system_metrics SET data = ?, created_at = ? WHERE minute_ts = ?",
(_json.dumps(merged, default=str), now, minute_ts),
)
else:
self._conn.execute(
"INSERT INTO system_metrics (minute_ts, data, created_at) VALUES (?, ?, ?)",
(minute_ts, json.dumps(data, default=str), now),
)
self._conn.commit()
with self._lock:
now = datetime.now().isoformat()
# UPSERT: 如果同一分钟已存在,合并数据
existing = self._conn.execute(
"SELECT data FROM system_metrics WHERE minute_ts = ?",
(minute_ts,),
).fetchone()
if existing:
import json as _json
old_data = _json.loads(existing["data"])
merged = self._merge_metric_buckets(old_data, data)
self._conn.execute(
"UPDATE system_metrics SET data = ?, created_at = ? WHERE minute_ts = ?",
(_json.dumps(merged, default=str), now, minute_ts),
)
else:
self._conn.execute(
"INSERT INTO system_metrics (minute_ts, data, created_at) VALUES (?, ?, ?)",
(minute_ts, json.dumps(data, default=str), now),
)
self._conn.commit()
await _run_in_sqlite_pool(_store)
except Exception as e:
logger.warning(f"store_metrics_minute failed: {e}")
@@ -442,17 +547,18 @@ class SqliteCache(CacheBase):
"""读取指定时间范围内的分钟指标数据"""
try:
def _load():
rows = self._conn.execute(
"SELECT data FROM system_metrics WHERE minute_ts >= ? AND minute_ts <= ? ORDER BY minute_ts",
(start_ts, end_ts),
).fetchall()
results = []
for row in rows:
try:
results.append(json.loads(row["data"]))
except (json.JSONDecodeError, TypeError):
pass
return results
with self._lock:
rows = self._conn.execute(
"SELECT data FROM system_metrics WHERE minute_ts >= ? AND minute_ts <= ? ORDER BY minute_ts",
(start_ts, end_ts),
).fetchall()
results = []
for row in rows:
try:
results.append(json.loads(row["data"]))
except (json.JSONDecodeError, TypeError):
pass
return results
return await _run_in_sqlite_pool(_load)
except Exception as e:
logger.warning(f"load_metrics_range failed: {e}")
@@ -462,11 +568,12 @@ class SqliteCache(CacheBase):
"""删除指定时间之前的 metrics 数据"""
try:
def _cleanup():
self._conn.execute(
"DELETE FROM system_metrics WHERE minute_ts < ?",
(before_ts,),
)
self._conn.commit()
with self._lock:
self._conn.execute(
"DELETE FROM system_metrics WHERE minute_ts < ?",
(before_ts,),
)
self._conn.commit()
await _run_in_sqlite_pool(_cleanup)
except Exception as e:
logger.warning(f"cleanup_old_metrics failed: {e}")
+113 -69
View File
@@ -16,6 +16,7 @@ import json
import logging
import os
import sqlite3
import threading
from datetime import datetime, timezone
from typing import Dict, List, Any, Optional
@@ -65,6 +66,11 @@ def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def _escape_like(value: str) -> str:
"""Escape LIKE wildcards so a user id is matched literally."""
return value.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
class HistoryStore:
"""
SQLite 历史存储 - 记录所有内存操作的审计日志。
@@ -89,6 +95,10 @@ class HistoryStore:
"""
self._db_path = config.history.db_path
self._conn: Optional[sqlite3.Connection] = None
# sqlite3.Connection.close() must never race with an execute() on the
# same connection. The lock also serializes operations on this shared
# connection; separate HistoryStore instances remain independent.
self._conn_lock = threading.RLock()
async def initialize(self) -> None:
"""创建表和索引(如果不存在)"""
@@ -97,20 +107,21 @@ class HistoryStore:
def _init_sync(self) -> None:
"""同步初始化:创建目录、连接数据库、建表建索引"""
# 确保目录存在
db_dir = os.path.dirname(self._db_path)
if db_dir:
os.makedirs(db_dir, exist_ok=True)
with self._conn_lock:
# 确保目录存在
db_dir = os.path.dirname(self._db_path)
if db_dir:
os.makedirs(db_dir, exist_ok=True)
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
self._conn.row_factory = sqlite3.Row
# 开启 WAL 模式,提高并发读写性能
self._conn.execute("PRAGMA journal_mode=WAL;")
self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
self._conn.row_factory = sqlite3.Row
# 开启 WAL 模式,提高并发读写性能
self._conn.execute("PRAGMA journal_mode=WAL;")
self._conn.execute(_CREATE_TABLE_SQL)
for idx_sql in _CREATE_INDEXES_SQL:
self._conn.execute(idx_sql)
self._conn.commit()
self._conn.execute(_CREATE_TABLE_SQL)
for idx_sql in _CREATE_INDEXES_SQL:
self._conn.execute(idx_sql)
self._conn.commit()
# ================================================================
# 记录操作
@@ -218,13 +229,14 @@ class HistoryStore:
return await _run_in_sqlite_pool(self._get_history_sync, memory_id)
def _get_history_sync(self, memory_id: str) -> List[Dict[str, Any]]:
if self._conn is None:
return []
cursor = self._conn.execute(
"SELECT * FROM memory_history WHERE memory_id = ? ORDER BY created_at ASC",
(memory_id,),
)
return [dict(row) for row in cursor.fetchall()]
with self._conn_lock:
if self._conn is None:
return []
cursor = self._conn.execute(
"SELECT * FROM memory_history WHERE memory_id = ? ORDER BY created_at ASC",
(memory_id,),
)
return [dict(row) for row in cursor.fetchall()]
async def get_recent(
self,
@@ -244,14 +256,43 @@ class HistoryStore:
return await _run_in_sqlite_pool(self._get_recent_sync, isolation_key, limit)
def _get_recent_sync(self, isolation_key: str, limit: int) -> List[Dict[str, Any]]:
if self._conn is None:
return []
cursor = self._conn.execute(
"SELECT * FROM memory_history WHERE isolation_key = ? "
"ORDER BY created_at DESC LIMIT ?",
(isolation_key, limit),
with self._conn_lock:
if self._conn is None:
return []
cursor = self._conn.execute(
"SELECT * FROM memory_history WHERE isolation_key = ? "
"ORDER BY created_at DESC LIMIT ?",
(isolation_key, limit),
)
return [dict(row) for row in cursor.fetchall()]
async def get_recent_for_user(
self,
user_id: str,
limit: int = 50,
) -> List[Dict[str, Any]]:
"""获取某 user 下所有 agent 的最近操作。
isolation_key 形如 ``"{user_id}:{agent_id}"``。user_id 里的 LIKE
通配符会被转义,保证只匹配这个 user,不会漏到 ``u1`` 之外。
"""
prefix = _escape_like(f"{user_id}:") + "%"
return await _run_in_sqlite_pool(
self._get_recent_prefix_sync, prefix, limit,
)
return [dict(row) for row in cursor.fetchall()]
def _get_recent_prefix_sync(
self, pattern: str, limit: int,
) -> List[Dict[str, Any]]:
with self._conn_lock:
if self._conn is None:
return []
cursor = self._conn.execute(
"SELECT * FROM memory_history WHERE isolation_key LIKE ? ESCAPE '\\' "
"ORDER BY created_at DESC LIMIT ?",
(pattern, limit),
)
return [dict(row) for row in cursor.fetchall()]
async def count(self, isolation_key: str = "") -> int:
"""
@@ -266,16 +307,17 @@ class HistoryStore:
return await _run_in_sqlite_pool(self._count_sync, isolation_key)
def _count_sync(self, isolation_key: str) -> int:
if self._conn is None:
return 0
if isolation_key:
cursor = self._conn.execute(
"SELECT COUNT(*) FROM memory_history WHERE isolation_key = ?",
(isolation_key,),
)
else:
cursor = self._conn.execute("SELECT COUNT(*) FROM memory_history")
return cursor.fetchone()[0]
with self._conn_lock:
if self._conn is None:
return 0
if isolation_key:
cursor = self._conn.execute(
"SELECT COUNT(*) FROM memory_history WHERE isolation_key = ?",
(isolation_key,),
)
else:
cursor = self._conn.execute("SELECT COUNT(*) FROM memory_history")
return cursor.fetchone()[0]
async def close(self) -> None:
"""关闭数据库连接"""
@@ -284,9 +326,10 @@ class HistoryStore:
logger.info("[HistoryStore] closed")
def _close_sync(self) -> None:
if self._conn:
self._conn.close()
self._conn = None
with self._conn_lock:
if self._conn:
self._conn.close()
self._conn = None
# ================================================================
# 内部
@@ -339,33 +382,34 @@ class HistoryStore:
role: str = "",
extra: Optional[Dict[str, Any]] = None,
) -> int:
if self._conn is None:
return -1
now = _now_iso()
extra_json = json.dumps(extra or {}, ensure_ascii=False)
cursor = self._conn.execute(
"""INSERT INTO memory_history
(memory_id, isolation_key, event,
old_memory, new_memory, old_status, new_status,
change_reason, layer, actor_id, role, extra,
created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
memory_id,
isolation_key,
event,
old_memory,
new_memory,
old_status,
new_status,
change_reason,
layer,
actor_id,
role,
extra_json,
now,
now,
),
)
self._conn.commit()
return cursor.lastrowid
with self._conn_lock:
if self._conn is None:
return -1
now = _now_iso()
extra_json = json.dumps(extra or {}, ensure_ascii=False)
cursor = self._conn.execute(
"""INSERT INTO memory_history
(memory_id, isolation_key, event,
old_memory, new_memory, old_status, new_status,
change_reason, layer, actor_id, role, extra,
created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
memory_id,
isolation_key,
event,
old_memory,
new_memory,
old_status,
new_status,
change_reason,
layer,
actor_id,
role,
extra_json,
now,
now,
),
)
self._conn.commit()
return cursor.lastrowid
+33 -8
View File
@@ -462,14 +462,39 @@ class FaissVectorStore(VectorStoreBase):
return None
async def get_embeddings(self, node_ids: List[str]) -> Dict[str, List[float]]:
"""批量取向量(in-memory _data 直接读 embedding),供去重算 cosine。"""
out: Dict[str, List[float]] = {}
for nid in node_ids:
point_id = self._node_id_to_point_id(nid)
entry = self._data.get(point_id)
if entry and entry.get("embedding"):
out[nid] = list(entry["embedding"])
return out
"""批量取向量,供去重 / episode 摘要计算 cosine。
``_data`` 只保存 payload 与 FAISS 内部下标;普通 upsert 不额外存一份
embedding,向量本体在 FAISS index 里。这里按当前有效下标 reconstruct,
并用 ``_idx_to_id`` 校验,避免读到已被替换的旧下标。
"""
def _get() -> Dict[str, List[float]]:
out: Dict[str, List[float]] = {}
with self._lock:
for nid in node_ids:
point_id = self._node_id_to_point_id(nid)
entry = self._data.get(point_id)
if entry is None:
continue
embedding = entry.get("embedding")
if embedding:
out[nid] = list(embedding)
continue
idx = entry.get("idx")
if (
idx is None
or self._index is None
or self._idx_to_id.get(int(idx)) != point_id
):
continue
try:
out[nid] = self._index.reconstruct(int(idx)).tolist()
except Exception:
continue
return out
return await _run_in_vdb_pool(_get)
# ================================================================
# 删除
+20 -5
View File
@@ -10,8 +10,8 @@ HY Memory - Intention (L7) 召回 + 惰性过期转换 + 复本判定
节点的 layer 改写为 l2_fact(持久化 update_payload),并从意图结果里剔除——
过期意图变成普通历史事实,下次走 normal 召回;
- **判重(重点)**:对幸存意图按下述规则分组,同组只留一条本次返回:
同 user_id + 同一天 memory_at + valid_until 相同(都为 None 或都相等)
+ cosine 相似度 ≥ MEMORY_DEDUP_THRESHOLD (默认 0.95)
同 user_id + 同 agent_id + 同一天 memory_at + valid_until 相同
(都为 None 或都相等)+ cosine 相似度 ≥ MEMORY_DEDUP_THRESHOLD (默认 0.95)
→ 视为同一个意图的复本(extractor 复读机 / 多 session 重复表达)。
保留组里 `gmt_created` 最早的(老的先来先得,稳定 id);
- **异步删库**:其余复本 fire-and-forget 从 VDB 删除,不影响本次调用 RT。
@@ -112,11 +112,19 @@ def _partition_duplicates(
if len(hits) < 2:
return list(hits), []
# 1. 按 (memory_at day, valid_until day) 分桶
buckets: Dict[Tuple[Any, Any], List[int]] = {}
# 1. 按 (user_id, agent_id, memory_at day, valid_until day) 分桶。
# user_id / agent_id 必须进键:多用户检索时不同用户的同一天意图可能
# 文本、日期都相同,但它们是两个独立的租户数据;只按日期分桶会把其中
# 一个判成复本并硬删除。
buckets: Dict[Tuple[Any, Any, Any, Any], List[int]] = {}
for idx, h in enumerate(hits):
node = h.get("node")
key = (_memory_at_day(node), _valid_until_key(node))
key = (
getattr(node, "user_id", None) if node else None,
getattr(node, "agent_id", None) if node else None,
_memory_at_day(node),
_valid_until_key(node),
)
buckets.setdefault(key, []).append(idx)
def _gmt(h: Dict[str, Any]) -> float:
@@ -229,6 +237,8 @@ async def recall_intentions(
*,
user_ids: Optional[List[str]] = None,
agent_ids: Optional[List[str]] = None,
isolation_key: str = "",
isolation_keys: Optional[List[str]] = None,
limit: int = 10,
now: Optional[datetime] = None,
stats_out: Optional[Dict[str, Any]] = None,
@@ -243,6 +253,9 @@ async def recall_intentions(
query_embedding: 查询向量
user_ids: 用户隔离(单/多用户)
agent_ids: 可选 agent 过滤
isolation_key / isolation_keys: 精确的 (user, agent, session) 隔离键。
与其它召回通道一致:调用方在按 session 过滤时只生成这两项,
user_ids / agent_ids 为 None;不把它们透传下去就会退化成全库检索。
limit: 召回上限(返回给上游的条数)
now: 当前时间(默认 datetime.now(),测试可注入)
stats_out: 可选 dict,本函数把分段耗时与数量**原地回填**进去,供调用方落
@@ -283,6 +296,8 @@ async def recall_intentions(
query_embedding=query_embedding,
user_ids=user_ids or None,
agent_ids=agent_ids,
isolation_key=isolation_key,
isolation_keys=isolation_keys,
layers=[MemoryLayer.L7_INTENTION],
limit=fetch_limit,
score_threshold=0.0, # intention 不设语义门槛
+4
View File
@@ -231,6 +231,10 @@ class WriteRequest:
# 已有记忆 (V1 合并检测用)
existing_memories: Optional[List[Dict[str, Any]]] = None
# Public client process_raw() only: reuse these exact L1 nodes as input.
# Ordinary add() never populates this field.
existing_raw_ids: List[str] = field(default_factory=list)
# 记忆时间戳(不传则用当前时间 datetime.now())
memory_at: Optional[datetime] = None
+2
View File
@@ -1840,6 +1840,8 @@ class BaseReadPipeline(ReadPipeline):
query_embedding,
user_ids=isolation_params.get("user_ids"),
agent_ids=isolation_params.get("agent_ids"),
isolation_key=isolation_params.get("isolation_key") or "",
isolation_keys=isolation_params.get("isolation_keys"),
limit=request.intention_limit,
stats_out=_intent_stats,
include_sources=request.include_sources,
+86 -32
View File
@@ -280,6 +280,27 @@ class MemoryWriter(WritePipeline):
custom["source_raw_memory_ids"] = source_ids
return custom
async def _attach_existing_raw_ids(
self,
vector_store,
memory_ids: List[str],
raw_ids: List[str],
) -> None:
"""Attach every reused raw record to canonical outputs."""
for memory_id in memory_ids:
node = await vector_store.get_by_id(memory_id)
if node is None:
continue
custom = dict(node.custom or {})
source_ids = list(custom.get("source_raw_memory_ids") or [])
legacy_source = str(getattr(node, "source_raw_memory_id", "") or "")
for raw_id in (legacy_source, *raw_ids):
if raw_id and raw_id not in source_ids:
source_ids.append(raw_id)
if source_ids:
custom["source_raw_memory_ids"] = source_ids
await vector_store.update_payload(memory_id, {"custom": custom})
@staticmethod
def _attach_client_meta(custom: Dict[str, Any], request: Optional[WriteRequest]) -> None:
"""给新建节点挂上调用方 meta(单份 list)。
@@ -968,7 +989,7 @@ class MemoryWriter(WritePipeline):
async def _run():
try:
nid = await summarize_closed_episode(
vector_store=self.vector_store,
vector_store=await self._get_vector_store(),
embed_service=self.embed_service,
llm_provider=self.llm_provider,
episode_id=episode_id,
@@ -2596,13 +2617,23 @@ class MemoryWriter(WritePipeline):
from ..metrics import MetricsCollector
MetricsCollector.get().sys1_start()
# 2. 向量化 + 持久化(L1_RAW)
existing_raw_ids = request.existing_raw_ids
if (
not isinstance(existing_raw_ids, list)
or any(not isinstance(raw_id, str) or not raw_id for raw_id in existing_raw_ids)
):
raise ValueError("_existing_raw_ids must be a list of non-empty ids")
# 2. 向量化(处理已有 raw 时复用已存节点,不再次 embed)
_t_embed = datetime.now()
with tracer.span("embed") as s:
embedding, embed_meta = await self.embed_service.embed_queued_with_meta(
request.content
)
s.set_output({"dims": len(embedding), "content_len": len(request.content)})
if existing_raw_ids:
embedding, embed_meta = [], {"reused_raw": True}
else:
with tracer.span("embed") as s:
embedding, embed_meta = await self.embed_service.embed_queued_with_meta(
request.content
)
s.set_output({"dims": len(embedding), "content_len": len(request.content)})
_embed_ms = (datetime.now() - _t_embed).total_seconds() * 1000
response.extra["embedding"] = embedding
await self._emit_pipeline_step(
@@ -2625,27 +2656,43 @@ class MemoryWriter(WritePipeline):
with tracer.span("qdrant_upsert") as s:
try:
vector_store = await self._get_vector_store()
layer_enum = MemoryLayer.from_string(suggested_layer)
_raw_custom: Dict[str, Any] = {}
self._attach_client_meta(_raw_custom, request)
mem_node = MemoryNode(
user_id=request.user_id,
agent_id=request.agent_id or "default_agent",
session_id=request.session_id or "default_session",
source=request.source,
layer=layer_enum,
content=request.content,
source_type=SourceType.EXPLICIT,
status=MemoryStatus.ACTIVE,
is_latest=True,
embedding=embedding,
memory_at=request.memory_at,
custom=_raw_custom,
)
memory_id = await vector_store.upsert(mem_node)
if existing_raw_ids:
memory_id = existing_raw_ids[-1]
mem_node = await vector_store.get_by_id(memory_id)
if (
mem_node is None
or mem_node.layer != MemoryLayer.L1_RAW
or mem_node.user_id != request.user_id
or mem_node.agent_id != (request.agent_id or "default_agent")
or mem_node.session_id != (request.session_id or "default_session")
or mem_node.source != request.source
):
raise ValueError("existing raw record is outside the write scope")
else:
layer_enum = MemoryLayer.from_string(suggested_layer)
_raw_custom: Dict[str, Any] = {}
self._attach_client_meta(_raw_custom, request)
mem_node = MemoryNode(
user_id=request.user_id,
agent_id=request.agent_id or "default_agent",
session_id=request.session_id or "default_session",
source=request.source,
layer=layer_enum,
content=request.content,
source_type=SourceType.EXPLICIT,
status=MemoryStatus.ACTIVE,
is_latest=True,
embedding=embedding,
memory_at=request.memory_at,
custom=_raw_custom,
)
memory_id = await vector_store.upsert(mem_node)
response.memory_id = memory_id
s.set_output({"memory_id": memory_id, "layer": suggested_layer})
logger.debug(f"[write] persisted: id={memory_id} layer={suggested_layer}")
logger.debug(
f"[write] {'reused' if existing_raw_ids else 'persisted'}: "
f"id={memory_id} layer={suggested_layer}"
)
except Exception as persist_err:
_l1_error = str(persist_err)
s.set_error(_l1_error)
@@ -2709,6 +2756,12 @@ class MemoryWriter(WritePipeline):
recent_history_count=_recent_history_count,
)
total_tokens += response.extra.get("_agent_tokens", 0)
if existing_raw_ids and response.extra.get("agent_stored_ids"):
await self._attach_existing_raw_ids(
vector_store,
list(response.extra["agent_stored_ids"]),
existing_raw_ids,
)
if response.extra.get("agent_status") == "failed":
await self._emit_pipeline_step(
request,
@@ -2718,12 +2771,13 @@ class MemoryWriter(WritePipeline):
"error": response.extra.get("agent_error", ""),
},
)
await self._rollback_failed_l1(
request=request,
response=response,
vector_store=vector_store,
memory_id=memory_id,
)
if not existing_raw_ids:
await self._rollback_failed_l1(
request=request,
response=response,
vector_store=vector_store,
memory_id=memory_id,
)
response.tokens_used = total_tokens
_failed_workflow_ms = (
datetime.now() - _t_workflow
+19 -6
View File
@@ -29,7 +29,7 @@ import logging
import os
import re
import signal
import sys
import threading
from datetime import datetime
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Any, Dict, Optional
@@ -495,6 +495,16 @@ class MemoryHTTPHandler(BaseHTTPRequestHandler):
_json_response(self, 200, result)
def _request_shutdown(server: ThreadingHTTPServer) -> None:
"""Ask ``serve_forever`` to stop without blocking the calling thread.
``BaseServer.shutdown()`` waits for the ``serve_forever`` loop to exit. A
signal handler runs on the very thread executing that loop, so calling it
there deadlocks forever. Hand the call to a helper thread instead.
"""
threading.Thread(target=server.shutdown, daemon=True).start()
def run_server(port: int = 19527, host: str = "127.0.0.1"):
"""Start the HTTP server (blocking)."""
# Configure logging — basicConfig for root; hy_memory level set after client init
@@ -523,13 +533,12 @@ def run_server(port: int = 19527, host: str = "127.0.0.1"):
logger.info(f"[server] Listening on http://{host}:{port}")
logger.info(f"[server] Health check: http://{host}:{port}/healthz")
# Graceful shutdown
# Graceful shutdown. The signal handler only asks a helper thread to stop
# the loop; cleanup runs after serve_forever() returns so it cannot race
# the request threads or deadlock the serving thread.
def _shutdown(signum, frame):
logger.info("[server] Shutting down...")
server.shutdown()
if _client is not None:
_client.close()
sys.exit(0)
_request_shutdown(server)
signal.signal(signal.SIGINT, _shutdown)
signal.signal(signal.SIGTERM, _shutdown)
@@ -538,6 +547,10 @@ def run_server(port: int = 19527, host: str = "127.0.0.1"):
server.serve_forever()
except KeyboardInterrupt:
_shutdown(None, None)
finally:
server.server_close()
if _client is not None:
_client.close()
def main():
+150
View File
@@ -0,0 +1,150 @@
"""Public, offline snapshot support for the SDK-owned local data plane."""
from __future__ import annotations
import hashlib
import os
import shutil
from dataclasses import dataclass
from pathlib import Path
class DataPlaneSnapshotRejected(RuntimeError):
def __init__(self, code: str) -> None:
super().__init__(code)
self.code = code
@dataclass(frozen=True)
class DataPlaneSnapshot:
source_root: Path
snapshot_root: Path
files: tuple[dict[str, object], ...]
total_bytes: int
@dataclass(frozen=True)
class DataPlaneRestore:
restored_root: Path
previous_root: Path | None
files_restored: int
total_bytes: int
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def snapshot_local_data_plane(
*,
source_root: Path,
target_root: Path,
writers_stopped: bool,
) -> DataPlaneSnapshot:
"""Copy one quiesced local SDK root and return a verified inventory.
Remote providers require their own export implementation and must not call
this local-path contract.
"""
if not writers_stopped:
raise DataPlaneSnapshotRejected("data_plane_snapshot_writers_running")
source = source_root.resolve()
target = target_root.resolve()
if source.is_symlink() or not source.is_dir():
raise DataPlaneSnapshotRejected("data_plane_snapshot_source_invalid")
if target.exists():
raise DataPlaneSnapshotRejected("data_plane_snapshot_target_exists")
try:
target.mkdir(parents=True)
for root, directories, files in os.walk(source, followlinks=False):
root_path = Path(root)
relative = root_path.relative_to(source)
target_directory = target / relative
target_directory.mkdir(parents=True, exist_ok=True)
for name in directories:
if (root_path / name).is_symlink():
raise DataPlaneSnapshotRejected("data_plane_snapshot_symlink")
for name in files:
item = root_path / name
if item.is_symlink():
raise DataPlaneSnapshotRejected("data_plane_snapshot_symlink")
shutil.copy2(item, target_directory / name)
inventory: list[dict[str, object]] = []
for path in sorted(item for item in target.rglob("*") if item.is_file()):
inventory.append(
{
"path": path.relative_to(target).as_posix(),
"bytes": path.stat().st_size,
"sha256": _sha256(path),
}
)
for item in inventory:
copied = target / str(item["path"])
if copied.stat().st_size != item["bytes"] or _sha256(copied) != item["sha256"]:
raise DataPlaneSnapshotRejected("data_plane_snapshot_verify_failed")
except DataPlaneSnapshotRejected:
shutil.rmtree(target, ignore_errors=True)
raise
except OSError as error:
shutil.rmtree(target, ignore_errors=True)
raise DataPlaneSnapshotRejected("data_plane_snapshot_failed") from error
return DataPlaneSnapshot(
source_root=source,
snapshot_root=target,
files=tuple(inventory),
total_bytes=sum(int(item["bytes"]) for item in inventory),
)
def restore_local_data_plane(
*,
snapshot_root: Path,
target_root: Path,
writers_stopped: bool,
protective_root: Path | None = None,
) -> DataPlaneRestore:
"""Replace one local SDK root from a verified snapshot with rollback."""
if not writers_stopped:
raise DataPlaneSnapshotRejected("data_plane_restore_writers_running")
source = snapshot_root.resolve()
target = target_root.resolve()
protective = (
protective_root.resolve()
if protective_root is not None
else target.parent / f".{target.name}.pre-restore"
)
staging = target.parent / f".{target.name}.restore.tmp"
if target.is_symlink() or protective.exists() or staging.exists():
raise DataPlaneSnapshotRejected("data_plane_restore_target_unsafe")
copied = snapshot_local_data_plane(
source_root=source,
target_root=staging,
writers_stopped=True,
)
previous: Path | None = None
try:
if target.exists():
target.replace(protective)
previous = protective
staging.replace(target)
except OSError as error:
if previous is not None and previous.exists() and not target.exists():
previous.replace(target)
previous = None
shutil.rmtree(staging, ignore_errors=True)
raise DataPlaneSnapshotRejected("data_plane_restore_replace_failed") from error
return DataPlaneRestore(
restored_root=target,
previous_root=previous,
files_restored=len(copied.files),
total_bytes=copied.total_bytes,
)
+24
View File
@@ -0,0 +1,24 @@
"""Context propagated to host-provided HTTP transports for cost accounting."""
from collections.abc import Iterator
from contextlib import contextmanager
from contextvars import ContextVar, Token
_model_call_stage: ContextVar[str] = ContextVar(
"hy_memory_model_call_stage",
default="other",
)
def current_model_call_stage() -> str:
return _model_call_stage.get()
@contextmanager
def model_call_stage(stage: str | None) -> Iterator[None]:
normalized = (stage or "other").strip().lower() or "other"
token: Token[str] = _model_call_stage.set(normalized)
try:
yield
finally:
_model_call_stage.reset(token)
+35 -4
View File
@@ -18,12 +18,32 @@ from __future__ import annotations
import json
import logging
import re
from datetime import datetime
from pathlib import Path
from typing import Any, List, Optional
logger = logging.getLogger(__name__)
#: A namespace is a single path component, not a path. `\w` keeps Unicode
#: names working while excluding separators, drive letters and NUL.
_SAFE_SUBDIR = re.compile(r"[\w.-]{1,64}", re.UNICODE)
def _safe_subdir(value: str) -> Optional[str]:
"""Return ``value`` if it is safe as one directory name, else ``None``.
``user_id`` is caller-controlled. Without this check a value such as
``"/tmp/evil__u1"`` makes ``Path(base) / "/tmp/evil"`` discard ``base``
entirely; ``"../.."`` escapes it the same way.
"""
candidate = (value or "").strip()
if not candidate or candidate in {".", ".."}:
return None
if not _SAFE_SUBDIR.fullmatch(candidate):
return None
return candidate
# 兼容旧 import
TraceFileWriter = None # set after class definition
@@ -40,7 +60,7 @@ class PipelineLogWriter:
def __init__(self, log_dir: str, default_subdir: str = "default"):
self._log_dir = log_dir
self._default_subdir = default_subdir or "default"
self._default_subdir = _safe_subdir(default_subdir) or "default"
try:
(Path(self._log_dir) / self._default_subdir).mkdir(parents=True, exist_ok=True)
except Exception:
@@ -49,7 +69,7 @@ class PipelineLogWriter:
def _resolve_subdir(self, user_id: str) -> str:
"""从 user_id 的 "{prefix}__{user_id}" 前缀取 prefix 作为子目录;无前缀则回落。"""
if user_id and "__" in user_id:
prefix = user_id.split("__", 1)[0].strip()
prefix = _safe_subdir(user_id.split("__", 1)[0])
if prefix:
return prefix
return self._default_subdir
@@ -80,7 +100,11 @@ class PipelineLogWriter:
split_namespace_user,
)
tokens = (prompt_tokens or 0) + (completion_tokens or 0) + (total_tokens or 0)
# Providers define total_tokens as prompt+completion. Adding all
# three fields doubled every cost estimate in this JSONL.
tokens = (total_tokens or 0) or (
(prompt_tokens or 0) + (completion_tokens or 0)
)
namespace, uid = split_namespace_user(user_id or "")
if not is_llm_content_logging_enabled():
prompt = redact_content(prompt)
@@ -111,7 +135,14 @@ class PipelineLogWriter:
line = json.dumps(record, ensure_ascii=False, default=str)
date_str = datetime.now().strftime("%Y-%m-%d")
subdir = self._resolve_subdir(user_id)
log_dir = Path(self._log_dir) / subdir
base_dir = Path(self._log_dir).resolve()
log_dir = (base_dir / subdir).resolve()
# Defence in depth: even if a future change lets a suspicious
# component through, never write outside the configured root.
try:
log_dir.relative_to(base_dir)
except ValueError:
log_dir = base_dir / self._default_subdir
log_dir.mkdir(parents=True, exist_ok=True)
log_file = log_dir / f"{date_str}.log"
with open(log_file, "a", encoding="utf-8") as f:
+20 -8
View File
@@ -1,18 +1,30 @@
# MemHub / One Memory
# MemHub
MemHub 是一个本地优先的共享记忆产品:把用户授权的多个 AI 产品中的历史对话和新增内容
汇总到同一份记忆,并让这些 AI 在独立授权后复用它。Hy Memory 是 MemHub 使用的 Python
SDK 和记忆引擎。One Memory / `one-memory` 是历史模块、数据目录、服务和兼容命令名称。
MemHub是一个本地优先的共享记忆产品。它把用户明确授权的AI对话和编码记录汇总到一份本地记忆中,再由用户分别授权给支持的AI工具调用。你可以继续使用熟悉的客户端,同时保留跨工具延续的上下文。Hy Memory是MemHub使用的Python SDK和记忆引擎。
# 💡❤️ MemHub 刚刚开源,欢迎一起优化共建! ❤️💡
**# 💭 MemHub处于preview阶段,当前公开版本可以下载和体验,部分平台制品、安装路径与宿主体验仍在持续优化验证。安装前请先查看Release里的实际制品和版本说明。**
> 🙋 如果你遇到问题/有建议:欢迎提交Issue,并附上MemHub版本、操作系统与芯片架构、复现步骤、预期结果、实际结果和经过脱敏的日志。请不要上传APIKey、真实对话、个人路径或其他敏感信息。
> 我们会公开跟进可复现的问题,持续发布修复,并根据真实反馈完善安装、同步和召回体验。
**# 感谢每一条反馈,期待你和我们一起打磨MemHub的体验 :)**
> 💗 MemHub has just been open-sourced and is under active development. The current public release is available for download andevaluation. Some platform packages, installation paths, and host integrations are still being completed and verified. Check the assets
and release notes before installing.
If you run into a problem, please open an Issue with the version, platform,reproduction steps, and sanitized logs, We wilreviewreports, publish fixes, and keep the support matrix up to date.
## 发行与平台合同
公共 Python SDK 为 `hy-memory==2.0.3`。MemHub 源码版本为 `1.0.3`
(build 7),GitHub 仓库为
公共 Python SDK 为 `hy-memory==2.0.4`。MemHub 源码版本为 `1.0.4`
(build 13),GitHub 仓库为
[`Tencent-Hunyuan/Hy-Memory`](https://github.com/Tencent-Hunyuan/Hy-Memory),npm 包名为
`memhub-cli`。当前代码架构见
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)。
`v1.0.3` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
`v1.0.4` 的发行目标是 Linux x64/arm64 Runtime 与 macOS arm64/x64 App 和 Runtime。
Linux 制品由 GitHub Actions 在原生架构 runner 上构建并完成生命周期验收;macOS
制品由发行流水线完成 Developer ID 签名、公证和验收后手动上传。Windows 不支持。
只有同时出现在 GitHub Release 与 `release-manifest.json` 中、且通过公开校验的制品才属于
@@ -101,7 +113,7 @@ MemHub 的 SQLite 数据默认位于 `~/.one-memory`,不在 npm 包或 Runtime
## Hy Memory
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.3`。SDK 配置、权限边界及 search
当前开发依赖固定为公开 PyPI 包 `hy-memory==2.0.4`。SDK 配置、权限边界及 search
返回格式见 [`MEMORY.md`](MEMORY.md)。HY search 返回的 `content` 已包含 SDK 统一生成的
逻辑时间和演化上下文;调用方不应再次转换时间戳或重复拼接 `When`。
+3 -3
View File
@@ -3,9 +3,9 @@
"dataLicense": "CC0-1.0",
"SPDXID": "SPDXRef-DOCUMENT",
"name": "one-memory",
"documentNamespace": "https://one-memory.local/spdx/2026-10-17T11:07:57Z",
"documentNamespace": "https://one-memory.local/spdx/2026-10-17T13:52:48Z",
"creationInfo": {
"created": "2026-10-17T11:07:57Z",
"created": "2026-10-17T13:52:48Z",
"creators": [
"Tool: one-memory-generate-sbom"
]
@@ -2335,7 +2335,7 @@
{
"SPDXID": "SPDXRef-Package-hy-memory",
"name": "hy-memory",
"versionInfo": "2.0.3",
"versionInfo": "2.0.4",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"licenseConcluded": "Apache-2.0",
+2 -2
View File
@@ -2,14 +2,14 @@
MemHub itself is licensed under Apache-2.0 (see `LICENSE`).
It bundles the runtime dependencies listed below.
Generated from the installed dependency tree on 2026-10-17T11:07:57Z
Generated from the installed dependency tree on 2026-10-17T13:52:48Z
by `python3 scripts/generate_sbom.py`. Do not edit by hand.
Per-package license texts are not reproduced here yet; the license
column is filled in as part of release review, and the SBOM records
`NOASSERTION` until then rather than guessing.
## hy-memory 2.0.3
## hy-memory 2.0.4
License: Apache-2.0
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@one-memory/cli",
"version": "1.0.3",
"version": "1.0.4",
"description": "MemHub bootstrap CLI for installing and managing the local-first shared memory runtime",
"license": "Apache-2.0",
"repository": {
@@ -12,7 +12,6 @@
* It is generated rather than hand-written so the two cannot drift: this reads
* the real package.json and only strips what is provably not needed at runtime.
*/
import { execFileSync } from "node:child_process";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { createRequire } from "node:module";
import { dirname, resolve } from "node:path";
@@ -20,12 +19,14 @@ import { fileURLToPath } from "node:url";
const here = dirname(fileURLToPath(import.meta.url));
const root = resolve(here, "..");
const repositoryRoot = resolve(root, "../..");
const require = createRequire(import.meta.url);
const { resolveBuildVersion } = require("../../../scripts/release-version.cjs");
const releaseVersion = resolveBuildVersion(process.env, repositoryRoot);
const { assertSemver } = require("../../../scripts/release-version.cjs");
const pkgPath = resolve(root, "package.json");
const pkg = JSON.parse(readFileSync(pkgPath, "utf8"));
const publishVersion = assertSemver(
process.env.MEMHUB_NPM_VERSION?.trim() || pkg.version,
"MEMHUB_NPM_VERSION",
);
const publishName = process.env.MEMHUB_NPM_PACKAGE?.trim();
if (!publishName || !/^(?:@[a-z0-9._-]+\/)?[a-z0-9._-]+$/.test(publishName)) {
throw new Error(
@@ -53,7 +54,7 @@ mkdirSync(outDir, { recursive: true });
const published = { ...pkg };
published.name = publishName;
published.version = releaseVersion;
published.version = publishVersion;
for (const section of ["dependencies", "devDependencies", "peerDependencies"]) {
if (!published[section]) continue;
published[section] = Object.fromEntries(
+18 -13
View File
@@ -4,7 +4,11 @@ import { createReadStream } from "node:fs";
import { access } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import type { HealthResponse, ImportJob } from "@one-memory/contracts";
import {
type HealthResponse,
type ImportJob,
isHostId,
} from "@one-memory/contracts";
import {
type AcquiredRuntimeDistribution,
acquireRuntimeDistribution,
@@ -331,7 +335,7 @@ import {
renderVersion,
renderVersionDocument,
} from "./usage.js";
import { CLI_VERSION, PRODUCT } from "./version.js";
import { CLI_VERSION, PRODUCT, RUNTIME_RELEASE_VERSION } from "./version.js";
import { runWatchLoop, type WatchEvent, type WatchRead } from "./watch-loop.js";
export interface RunControls {
@@ -417,7 +421,8 @@ async function selectInstallSource(
throw new Error("release_manifest_url_unavailable");
}
const manifestUrl =
explicitManifestUrl ?? template.replaceAll("{version}", CLI_VERSION);
explicitManifestUrl ??
template.replaceAll("{version}", RUNTIME_RELEASE_VERSION);
const allowedHosts = parseAllowedReleaseHosts(
environment.MEMHUB_RELEASE_ALLOWED_HOSTS,
);
@@ -462,7 +467,7 @@ async function resolveRemoteInstallPreview(
): Promise<ResolvedRuntimeRelease> {
return (controls.resolveRuntimeRelease ?? resolveRuntimeRelease)({
manifestUrl: selection.manifestUrl,
expectedVersion: CLI_VERSION,
expectedVersion: RUNTIME_RELEASE_VERSION,
channel: selection.channel,
platform: process.platform,
arch: process.arch,
@@ -476,7 +481,7 @@ async function acquireRemoteInstallSource(
): Promise<AcquiredRuntimeDistribution> {
return (controls.acquireRuntimeDistribution ?? acquireRuntimeDistribution)({
manifestUrl: selection.manifestUrl,
expectedVersion: CLI_VERSION,
expectedVersion: RUNTIME_RELEASE_VERSION,
channel: selection.channel,
platform: process.platform,
arch: process.arch,
@@ -5155,10 +5160,10 @@ type HostCommand =
args: unknown[];
};
// `deepseek-harness:web` is a real id — that host is configured per profile —
// and `host list` prints it. Rejecting it here would mean listing a host the
// CLI then refuses to act on.
const HOST_ID = /^[a-z0-9][a-z0-9-]{0,31}(:[a-z0-9][a-z0-9-]{0,31})?$/;
// Host ids are read with the same grammar as the permission layer and the
// grant store (`@one-memory/contracts`): `deepseek-harness:web` is a real id —
// that host is configured per profile — and `host list` prints it, so rejecting
// it here would mean listing a host the CLI then refuses to act on.
const FINGERPRINT = /^[0-9a-f]{64}$/;
/** Host subcommands.
@@ -5181,7 +5186,7 @@ function parseHostCommand(rest: readonly string[]): HostCommand | null {
const token = tokens[index];
if (token === "--host") {
const value = tokens[index + 1];
if (!value || !HOST_ID.test(value)) return null;
if (!value || !isHostId(value)) return null;
if (hostId !== undefined) return null;
hostId = value;
index += 1;
@@ -5214,7 +5219,7 @@ function parseHostCommand(rest: readonly string[]): HostCommand | null {
qualifier !== undefined &&
(verb === "status" || verb === "install" || verb === "uninstall")
) {
if (!HOST_ID.test(qualifier)) return null;
if (!isHostId(qualifier)) return null;
hostId = qualifier;
// Drop the consumed host positional so the verb/qualifier checks below see
// the same shape as the flag form.
@@ -5230,7 +5235,7 @@ function parseHostCommand(rest: readonly string[]): HostCommand | null {
positional[2] !== undefined
) {
const trailing = positional[2];
if (!HOST_ID.test(trailing)) return null;
if (!isHostId(trailing)) return null;
hostId = trailing;
positional.splice(2, 1);
verb = positional[0];
@@ -6520,7 +6525,7 @@ function parseInstallOptions(
...(releaseManifestUrl ? { releaseManifestUrl } : {}),
releaseChannel:
releaseChannel ??
(CLI_VERSION.includes("-") ? "prerelease" : "stable"),
(RUNTIME_RELEASE_VERSION.includes("-") ? "prerelease" : "stable"),
installRoot: resolvedInstallRoot,
home: resolvedHome,
uid: uid ?? process.getuid?.(),
@@ -384,6 +384,101 @@ describe("unified context search", () => {
expect(streams.stdout()).not.toContain("must-not-print");
});
it("counts characters as code points so an emoji hit stays grounded", async () => {
// Core reports used_characters with Python's len(), which counts code
// points: this text is 10 characters there and 12 UTF-16 units here.
const text = "消费保 📍 扩展 📡";
expect(text.length).toBe(12);
expect(Array.from(text).length).toBe(10);
transport.post.mockResolvedValue({
status: 200,
data: {
...RESPONSE,
paths: {
...RESPONSE.paths,
memory: {
...RESPONSE.paths.memory,
status: "empty",
returned_count: 0,
},
},
items: [
{
...(RESPONSE.items[0] as Record<string, unknown>),
messages: [
{
...(
RESPONSE.items[0] as {
messages: Array<Record<string, unknown>>;
}
).messages[0],
text,
},
],
},
],
used_characters: 10,
},
});
const streams = captureStreams();
try {
expect(await run(["recall", "test", "--json"])).toBe(0);
} finally {
streams.restore();
}
expect(JSON.parse(streams.stdout())).toMatchObject({
state: "grounded",
used_characters: 10,
items: [{ kind: "raw", messages: [{ text }] }],
});
});
it("still rejects a response whose character total does not match", async () => {
// The same text counted as UTF-16 units is exactly what Core does not mean,
// so the cross-check must keep failing instead of accepting both totals.
const text = "消费保 📍 扩展 📡";
transport.post.mockResolvedValue({
status: 200,
data: {
...RESPONSE,
paths: {
...RESPONSE.paths,
memory: {
...RESPONSE.paths.memory,
status: "empty",
returned_count: 0,
},
},
items: [
{
...(RESPONSE.items[0] as Record<string, unknown>),
messages: [
{
...(
RESPONSE.items[0] as {
messages: Array<Record<string, unknown>>;
}
).messages[0],
text,
},
],
},
],
used_characters: 12,
},
});
const streams = captureStreams();
try {
expect(await run(["recall", "test", "--json"])).toBe(1);
} finally {
streams.restore();
}
expect(JSON.parse(streams.stdout())).toMatchObject({
state: "invalid",
code: "response_invalid",
});
});
it("reports endpoint denial without exposing the response body", async () => {
transport.post.mockResolvedValue({
status: 403,
+13 -2
View File
@@ -181,9 +181,9 @@ function validateContextResponse(
items.push(item);
characters +=
item.kind === "memory"
? String(item.content).length
? characterCount(item.content)
: (item.messages as Array<Record<string, unknown>>).reduce(
(sum, message) => sum + String(message.text).length,
(sum, message) => sum + characterCount(message.text),
0,
);
}
@@ -330,6 +330,17 @@ function record(value: unknown): Record<string, unknown> | null {
: null;
}
/**
* Count characters the way Core does. Python's len() counts Unicode code
* points; String.prototype.length counts UTF-16 code units, so an emoji outside
* the BMP is one character to Core and two here. Comparing the two totals made
* every response containing an emoji fail the used_characters cross-check and
* surface as response_invalid.
*/
function characterCount(value: unknown): number {
return Array.from(String(value)).length;
}
function boundedText(
value: unknown,
maximum: number,
+6 -1
View File
@@ -48,7 +48,11 @@ describe("publish manifest is production-safe", () => {
execFileSync("node", ["scripts/prepare-publish-manifest.mjs"], {
cwd: root,
stdio: "pipe",
env: { ...process.env, MEMHUB_NPM_PACKAGE: "@example/memhub-cli" },
env: {
...process.env,
MEMHUB_NPM_PACKAGE: "@example/memhub-cli",
MEMHUB_NPM_VERSION: "1.0.5",
},
});
const published = JSON.parse(
readFileSync(path.join(root, "dist-publish", "package.json"), "utf8"),
@@ -56,6 +60,7 @@ describe("publish manifest is production-safe", () => {
expect(published.dependencies).toBeUndefined();
expect(published.private).toBe(false);
expect(published.name).toBe("@example/memhub-cli");
expect(published.version).toBe("1.0.5");
expect(published.bin).toMatchObject({
memhub: "dist/one-memory.cjs",
"memhub-cli": "dist/one-memory.cjs",
+15 -6
View File
@@ -1,5 +1,9 @@
declare const __ONE_MEMORY_VERSION__: string;
declare const __ONE_MEMORY_BUILD_NUMBER__: string;
declare const __MEMHUB_DEFAULT_RUNTIME_VERSION__: string;
const SEMVER =
/^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$/;
const embeddedVersion =
typeof __ONE_MEMORY_VERSION__ === "string"
@@ -9,13 +13,18 @@ const embeddedBuildNumber =
typeof __ONE_MEMORY_BUILD_NUMBER__ === "string"
? __ONE_MEMORY_BUILD_NUMBER__.trim()
: "";
const embeddedRuntimeVersion =
typeof __MEMHUB_DEFAULT_RUNTIME_VERSION__ === "string"
? __MEMHUB_DEFAULT_RUNTIME_VERSION__.trim()
: "";
export const CLI_VERSION =
/^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$/.test(
embeddedVersion,
)
? embeddedVersion
: "development";
export const CLI_VERSION = SEMVER.test(embeddedVersion)
? embeddedVersion
: "development";
export const RUNTIME_RELEASE_VERSION = SEMVER.test(embeddedRuntimeVersion)
? embeddedRuntimeVersion
: CLI_VERSION;
export const CLI_BUILD_NUMBER = /^\d+$/.test(embeddedBuildNumber)
? Number(embeddedBuildNumber)
+17 -2
View File
@@ -2,11 +2,22 @@ import { createRequire } from "node:module";
import { defineConfig } from "tsup";
const require = createRequire(import.meta.url);
const { resolveBuildVersion } =
const { assertSemver, resolveBuildVersion } =
require("../../scripts/release-version.cjs") as {
assertSemver(value: string, label?: string): string;
resolveBuildVersion(environment?: NodeJS.ProcessEnv, root?: string): string;
};
const releaseVersion = resolveBuildVersion(process.env);
const packageVersion = (require("./package.json") as { version: string })
.version;
const cliVersion = assertSemver(
process.env.MEMHUB_NPM_VERSION?.trim() || packageVersion,
"MEMHUB_NPM_VERSION",
);
const runtimeReleaseVersion = assertSemver(
process.env.MEMHUB_DEFAULT_RUNTIME_VERSION?.trim() || releaseVersion,
"MEMHUB_DEFAULT_RUNTIME_VERSION",
);
export default defineConfig({
entry: { "one-memory": "src/entry.ts" },
@@ -15,7 +26,7 @@ export default defineConfig({
outDir: "dist",
clean: true,
define: {
__ONE_MEMORY_VERSION__: JSON.stringify(releaseVersion),
__ONE_MEMORY_VERSION__: JSON.stringify(cliVersion),
__ONE_MEMORY_BUILD_NUMBER__: JSON.stringify(
process.env.ONE_MEMORY_BUILD_NUMBER ?? "unknown",
),
@@ -25,11 +36,15 @@ export default defineConfig({
__MEMHUB_DEFAULT_RELEASE_MANIFEST_URL_TEMPLATE__: JSON.stringify(
process.env.MEMHUB_DEFAULT_RELEASE_MANIFEST_URL_TEMPLATE ?? "",
),
__MEMHUB_DEFAULT_RUNTIME_VERSION__: JSON.stringify(runtimeReleaseVersion),
},
// The shared operations package ships TypeScript sources, so it must be
// inlined into the CJS bundle the same way runtime-control is; otherwise the
// emitted bundle tries to `require` a `.js` file that only exists as `.ts`.
// `@one-memory/contracts` is the same kind of package and is imported for a
// value (the host-id grammar), so it has to be inlined too.
noExternal: [
"@one-memory/contracts",
"@one-memory/operations",
"@one-memory/runtime-control",
"axios",
+1
View File
@@ -11,6 +11,7 @@ export default defineConfig({
__ONE_MEMORY_BUILD_NUMBER__: JSON.stringify("unknown"),
__ONE_MEMORY_COMMIT__: JSON.stringify("unknown"),
__MEMHUB_DEFAULT_RELEASE_MANIFEST_URL_TEMPLATE__: JSON.stringify(""),
__MEMHUB_DEFAULT_RUNTIME_VERSION__: JSON.stringify(version),
},
test: {
testTimeout: 120_000,
+2 -2
View File
@@ -4,7 +4,7 @@ build-backend = "uv_build"
[project]
name = "one-memory"
version = "1.0.3"
version = "1.0.4"
description = "MemHub local-first memory core"
readme = "README.md"
requires-python = ">=3.12"
@@ -12,7 +12,7 @@ dependencies = [
"fastapi>=0.141.1",
"filelock>=3.32.4",
# Public SDK version validated against MemHub's permission and recall contracts.
"hy-memory==2.0.3",
"hy-memory==2.0.4",
# Retrieval's derived vector index directly consumes these locked versions.
"chromadb==1.5.9",
"numpy==2.5.2",
+312 -44
View File
@@ -100,7 +100,13 @@ from one_memory.imports.access import (
SourceAccessError,
root_identity_matches,
)
from one_memory.imports.adapters import SourceAdapter, SyncCompleteness
from one_memory.imports.adapters import (
ScanFailure,
ScannedEvidence,
ScanOptions,
SourceAdapter,
SyncCompleteness,
)
from one_memory.imports.candidates import (
SourceCandidateError,
SourceCandidateSelection,
@@ -111,7 +117,11 @@ from one_memory.imports.connections import (
ConnectionIntentReceipt,
SourceConnectionStore,
)
from one_memory.imports.grants import SourceGrantReceipt, SourceGrantStore
from one_memory.imports.grants import (
SourceAuthorizationState,
SourceGrantReceipt,
SourceGrantStore,
)
from one_memory.imports.host_packages import CapabilityState, HostPackageCapabilities
from one_memory.imports.jobs import (
ImportIssueRecord,
@@ -175,7 +185,12 @@ from one_memory.memory_queue import (
resolve_extraction_state,
)
from one_memory.memory_server.config_state import ConfigStore
from one_memory.memory_tasks import MemoryTask, MemoryTaskStatus, MemoryTaskStore
from one_memory.memory_tasks import (
NON_RETRYABLE_POST_WRITE_CODES,
MemoryTask,
MemoryTaskStatus,
MemoryTaskStore,
)
from one_memory.model_presets import seed_preset_profile
from one_memory.model_probe import ModelProber, ModelProbeResult
from one_memory.model_profiles import (
@@ -190,6 +205,7 @@ from one_memory.model_profiles import (
ModelProfileStore,
ModelProfileUnavailableError,
)
from one_memory.model_request_audit import AuditedModelTransport, ModelRequestAuditStore
from one_memory.model_request_budget import BudgetedModelTransport, ModelRequestBudget
from one_memory.onboarding import OnboardingState, OnboardingStore
from one_memory.recall import GovernedRecallTarget, RecallReceipt, RecallRequest
@@ -281,6 +297,9 @@ class SourceResponse(BaseModel):
candidate_revision: int = 0
candidate_change_allowed: bool = False
candidate_change_reason: str | None = None
#: Why `authorized` is false, so the UI can ask for the one thing that
#: would fix it instead of offering every recovery at once.
authorization_state: SourceAuthorizationState = SourceAuthorizationState.OK
class PreviewSourceCandidateRequest(BaseModel):
@@ -313,6 +332,22 @@ class SourceSyncState(BaseModel):
sync_enabled: bool
class SedimentationRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
enabled: StrictBool
class SedimentationState(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
enabled: bool
#: What the switch means for the model bill and for what still runs. Read
#: by the app so the setting can say so without hardcoding the wording.
pending_retained: bool = True
imports_unaffected: bool = True
class DetectionResponse(BaseModel):
source_id: str
detected: bool
@@ -339,7 +374,7 @@ class ConnectSourcesRequest(BaseModel):
import_upload_id: UUID | None = None
ONBOARDING_LARGE_HISTORY_SESSIONS = 200
ONBOARDING_LARGE_HISTORY_USER_TURNS = 200
ONBOARDING_HISTORY_WINDOW = timedelta(days=3)
@@ -886,6 +921,7 @@ def create_app(
)
app.state.import_job_store = ImportJobStore(database_path)
app.state.onboarding_store = OnboardingStore(database_path)
app.state.history_import_gate_released = False
app.state.extraction_settings_store = ExtractionSettingsStore(database_path)
app.state.source_grant_store = SourceGrantStore(
database_path,
@@ -908,6 +944,7 @@ def create_app(
app.state.source_permission_locks = {
source_id: asyncio.Lock() for source_id in app.state.source_catalog
}
app.state.sedimentation_lock = asyncio.Lock()
_follow_workbuddy_home(app, runtime_settings.source_home)
app.state.source_detection = {}
app.state.secret_store = secret_store or KeyringSecretStore()
@@ -974,6 +1011,7 @@ def create_app(
else None
)
app.state.model_request_budget = model_request_budget
app.state.model_request_audit_store = ModelRequestAuditStore(database_path)
app.state.memory_answerer = MemoryAnswerer(
governance=app.state.memory_governance_store,
filter_memories=app.state.source_read_access.memories,
@@ -1016,10 +1054,17 @@ def create_app(
profile=profile,
api_key=api_key,
data_dir=data_dir,
http_transport_factory=(
(lambda: BudgetedModelTransport(model_request_budget))
if model_request_budget is not None
else None
http_transport_factory=lambda: AuditedModelTransport(
app.state.model_request_audit_store,
profile_id=profile.profile_id,
profile_revision=profile.revision,
provider_kind=str(profile.provider_kind),
model=profile.model,
inner=(
BudgetedModelTransport(model_request_budget)
if model_request_budget is not None
else None
),
),
),
# `None` means "whatever the active profile allows". Pinned to
@@ -1032,6 +1077,7 @@ def create_app(
max_concurrent_tasks=runtime_settings.precipitation_max_concurrent_tasks,
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
service_levels=app.state.service_levels,
sedimentation_enabled=lambda: _config_store(app).sedimentation_enabled(),
)
# C2 owns the single production budget (handle 5s + reserve 250ms,
# total 5.25s, lease 10s, renew 1s, claim 250ms). The composition
@@ -1041,8 +1087,14 @@ def create_app(
drain_budget = production_drain_budget()
async def reconcile_unknown_writes() -> None:
await app.state.memory_processing_supervisor.reconcile_unknown_outputs()
unknown = await asyncio.to_thread(memory_write_queue.unknown_jobs, limit=128)
await app.state.memory_processing_supervisor.reconcile_unknown_outputs(
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
)
unknown = await asyncio.to_thread(
memory_write_queue.unknown_jobs,
limit=128,
exclude_error_codes=NON_RETRYABLE_POST_WRITE_CODES,
)
for job in unknown:
await asyncio.to_thread(
memory_write_queue.resolve_unknown_from_linked_output,
@@ -1064,8 +1116,12 @@ def create_app(
mark_invoked: Callable[[], Awaitable[bool]],
) -> bool:
"""Serialize pause with the last boundary before an HY call."""
lock = app.state.source_permission_locks.setdefault(source_id, asyncio.Lock())
async with lock:
source_lock = app.state.source_permission_locks.setdefault(
source_id, asyncio.Lock()
)
async with app.state.sedimentation_lock, source_lock:
if not _config_store(app).sedimentation_enabled():
raise RuntimeError("model_sedimentation_disabled")
if not _source_writes_on(app, source_id):
raise RuntimeError("model_content_source_disabled")
return await mark_invoked()
@@ -1103,8 +1159,18 @@ def create_app(
+ 10
),
reconcile_unknown=reconcile_unknown_writes,
source_writes_permitted=lambda source_id: _source_writes_on(app, source_id),
source_claim_enabled=lambda source_id: _config_store(app).sync_enabled(source_id),
source_writes_permitted=lambda source_id: _sedimentation_claim_enabled(
app, source_id
),
# The claim gate has to be the same question the write gate
# asks, not a second answer to it. It used to be
# `sync_enabled`, the per-app toggle, and `one-memory-manual` is
# not a connected app: it has no entry in the configuration
# document, so that predicate answers False for it while the
# write gate admits it by its grant. Every manual note's job was
# therefore skipped on every drain tick — the note stayed
# "saved, waiting" with no task behind it, forever.
source_claim_enabled=lambda source_id: _sedimentation_claim_enabled(app, source_id),
)
app.state.memory_write_drain = memory_write_drain
app.state.memory_write_drain_task = watched_task(
@@ -1133,10 +1199,7 @@ def create_app(
for chunk in _conversation_chunks(conversation, max_turns=12, completed_only=True):
if (
record not in chunk
or not any(
item.role == "user" and item.written_by.value == "person"
for item in chunk
)
or not any(item.role == "user" for item in chunk)
or not any(item.role == "assistant" for item in chunk)
):
continue
@@ -1622,7 +1685,7 @@ def create_app(
# --- Reader-only internal endpoints. Registered here, before the public
# routes below; no early return, so every route after this still registers.
@app.get("/internal/semantic-ids")
@app.get("/internal/semantic-ids", dependencies=protected)
async def internal_semantic_ids(
query: str = Query(min_length=1, max_length=2_000),
limit: int = Query(default=12, ge=1, le=20),
@@ -1634,7 +1697,7 @@ def create_app(
return {"ids": []}
return {"ids": [memory_id for memory_id, _score in hits]}
@app.post("/internal/call-event")
@app.post("/internal/call-event", dependencies=protected)
async def internal_call_event(request: Request) -> dict[str, bool]:
"""Side effects the Reader cannot write: it opens with query_only."""
try:
@@ -1653,7 +1716,7 @@ def create_app(
return {"recorded": False}
return {"recorded": bool(recorded)}
@app.get("/internal/graph-snapshot")
@app.get("/internal/graph-snapshot", dependencies=protected)
async def internal_graph_snapshot(
limit: int = Query(default=160, ge=1, le=300),
) -> dict[str, object]:
@@ -1749,6 +1812,30 @@ def create_app(
"events": events,
}
@app.get("/api/v1/diagnostics/model-usage", dependencies=protected)
def diagnostic_model_usage(
since: datetime | None = Query(default=None), # noqa: B008
request_id: str | None = Query( # noqa: B008
default=None, min_length=1, max_length=512
),
limit: int = Query(default=200, ge=1, le=2_000), # noqa: B008
) -> dict[str, object]:
"""Return content-free precipitation usage for provider reconciliation."""
normalized_since = (
(since if since.tzinfo is not None else since.replace(tzinfo=UTC))
if since is not None
else None
)
report = app.state.model_request_audit_store.report(
since=normalized_since.astimezone(UTC).isoformat()
if normalized_since is not None
else None,
request_id=request_id,
limit=limit,
)
return {"schema_version": 1, **report}
@app.get(
"/api/v1/runtime/identity",
response_model=HealthResponse,
@@ -1937,6 +2024,11 @@ def create_app(
candidate_change_reason=app.state.source_candidate_store.change_reason(
registration
),
authorization_state=grant_store.authorization_state(
user_id="local",
source_id=source_id,
candidate=registration.candidate.path,
),
)
for source_id, registration in _catalog(app).items()
]
@@ -2248,24 +2340,39 @@ def create_app(
raise HTTPException(status_code=409, detail=error.code) from None
if grant is None:
raise HTTPException(status_code=409, detail="grant_required")
await _prepare_source_resume(source_id)
try:
await _prepare_source_resume(source_id)
except BaseException as error:
raise _sync_refused("sync_prepare_failed") from error
else:
await _live_sync(app).stop_source(source_id)
try:
await _live_sync(app).stop_source(source_id)
except BaseException as error:
raise _sync_refused("sync_prepare_failed") from error
try:
await asyncio.to_thread(_config_store(app).set_sync, source_id, request.enabled)
except BaseException:
except BaseException as error:
if request.enabled:
await _live_sync(app).stop_source(source_id)
else:
await _prepare_source_resume(source_id)
await _activate_source_resume(source_id)
await _refresh_live_watch(app)
raise
raise _sync_refused("sync_config_write_failed") from error
try:
if request.enabled:
await _activate_source_resume(source_id)
await _refresh_live_watch(app)
try:
await _activate_source_resume(source_id)
except BaseException as error:
raise _sync_refused("sync_resume_failed") from error
try:
await _refresh_live_watch(app)
except BaseException as error:
raise _sync_refused("sync_watcher_failed") from error
except BaseException:
# Rolls back the desired state and settles the watcher exactly as
# before; the only change is that the raise above now carries a
# code saying which stage failed.
await asyncio.to_thread(_config_store(app).set_sync, source_id, previous)
if request.enabled:
await _live_sync(app).stop_source(source_id)
@@ -2396,6 +2503,8 @@ def create_app(
async def connect_sources(
request: ConnectSourcesRequest,
) -> ConnectSourcesResponse:
if request.onboarding:
app.state.history_import_gate_released = False
source_ids = tuple(dict.fromkeys(request.source_ids))
registrations = [_require_registration(app, source_id) for source_id in source_ids]
if request.request_id is None:
@@ -2695,6 +2804,12 @@ def create_app(
# D75: work queued for the previous model continues with this one.
# Only tasks that never reached the model are re-bound.
await asyncio.to_thread(app.state.memory_task_store.rebind_unstarted_profile, profile)
# A missing profile is an event-driven wait, not a provider outage.
# Release those durable rows only now that activation has changed the
# condition; the ordinary drain poll will claim them without a restart.
await asyncio.to_thread(
app.state.memory_write_queue.readmit_model_profile_waiters
)
# Not awaited: the catch-up reads every conversation ever imported, and
# the answer to "which profile is active" does not depend on it.
_start_backfill(app)
@@ -3561,6 +3676,48 @@ def create_app(
def put_extraction_settings(request: ExtractionWindowUpdate) -> ExtractionSettings:
return _memory_processing_supervisor(app).set_extraction_window(request.window_preset)
@app.get(
"/api/v1/sedimentation",
response_model=SedimentationState,
dependencies=protected,
)
def get_sedimentation() -> SedimentationState:
"""Whether the model may still turn conversations into memories."""
return SedimentationState(enabled=_config_store(app).sedimentation_enabled())
@app.put(
"/api/v1/sedimentation",
response_model=SedimentationState,
dependencies=protected,
)
async def put_sedimentation(request: SedimentationRequest) -> SedimentationState:
"""Start or stop model-driven sedimentation, keeping the configuration.
Stopping is the user's cost control: nothing further goes to the model
and already queued work waits rather than failing, so starting again
resumes it. Importing conversations is not affected either way.
"""
store = _config_store(app)
changed = False
async with app.state.sedimentation_lock:
previous = store.sedimentation_enabled()
if previous is not request.enabled:
await asyncio.to_thread(store.set_sedimentation_enabled, request.enabled)
changed = True
diagnostic_event(
level="info",
component="memory.sedimentation",
event="memory_sedimentation.toggled",
outcome="succeeded",
details={"enabled": request.enabled},
)
if changed and request.enabled:
# Work parked by the stop is picked up again rather than waiting
# for the next natural dispatch, but not on this request's time:
# see _resume_sedimentation.
_resume_sedimentation(app)
return SedimentationState(enabled=store.sedimentation_enabled())
@app.post(
"/api/v1/memory-processing/manual-extract",
response_model=ManualExtractReceipt,
@@ -3921,6 +4078,7 @@ async def _authorize_answer_generation(
# The explicit question is this operation's input, not a standing permission for history.
if not question.strip() or not grounding:
raise MemoryAnswerError("memory_answer_generation_not_authorized")
def check() -> None:
if _model_profile_store(app).active_identity() != (
profile.profile_id,
@@ -3944,6 +4102,7 @@ async def _authorize_answer_generation(
await recheck()
return recheck
class _LazyASGIApp:
"""Compatibility ASGI app that is built only when a server imports it.
@@ -3975,6 +4134,16 @@ class _LazyASGIApp:
app = _LazyASGIApp()
def _sync_refused(code: str) -> HTTPException:
"""A sync refusal that says which stage refused.
Turning the switch on takes four steps: authorize, prepare, write, resume.
The UI used to be told only that the switch had sprung back, and every
stage needs a different fix, so each one gets its own code.
"""
return HTTPException(status_code=409, detail=code)
def _granted_roots(
source_ids: Iterable[str],
grant_store: SourceGrantStore,
@@ -4026,6 +4195,7 @@ def _source_response(
totals: tuple[int, int],
*,
candidate_change_reason: str | None = None,
authorization_state: SourceAuthorizationState = SourceAuthorizationState.OK,
) -> SourceResponse:
location = inspect_source_location(
registration.descriptor.source_id, registration.candidate.path
@@ -4238,7 +4408,7 @@ async def _onboarding_history_after(
home: Path,
now: datetime | None = None,
) -> datetime | None:
"""Return one shared cutoff when any selected harness has large history.
"""Return one shared cutoff when any selected harness has large, old history.
The cutoff is decided before any history job is created, then persisted on
every accepted job. This is deliberately global: independently limiting
@@ -4247,6 +4417,8 @@ async def _onboarding_history_after(
"""
if not request.onboarding:
return None
anchor = now or datetime.now(UTC)
cutoff = anchor - ONBOARDING_HISTORY_WINDOW
revisions = request.source_candidate_revisions or {}
for registration in registrations:
source_id = registration.descriptor.source_id
@@ -4274,29 +4446,52 @@ async def _onboarding_history_after(
home=home,
)
adapter = registration.create_adapter(grant)
count_sessions = cast(
Callable[..., Awaitable[int]] | None,
getattr(adapter, "count_sessions", None),
)
if count_sessions is not None:
session_count = await count_sessions(limit=ONBOARDING_LARGE_HISTORY_SESSIONS + 1)
else:
pending = _install_state(adapter)
if pending is None:
continue
session_count = (await pending).session_count
if await _has_large_old_history(adapter, cutoff=cutoff):
return cutoff
except (OSError, SourceAccessError):
# A supported source whose metadata count cannot be completed is
# treated conservatively. The later connection still owns the
# precise authorization error, while onboarding never falls back
# to an unbounded history import.
session_count = ONBOARDING_LARGE_HISTORY_SESSIONS + 1
if session_count > ONBOARDING_LARGE_HISTORY_SESSIONS:
anchor = now or datetime.now(UTC)
return anchor - ONBOARDING_HISTORY_WINDOW
return cutoff
return None
async def _has_large_old_history(adapter: SourceAdapter, *, cutoff: datetime) -> bool:
"""Probe one authorized Harness with the same parsers the import will use.
Stop as soon as both policy facts are proven: more than 200 genuine person
turns and at least one record older than the three-day cutoff. Counting
parsed Evidence instead of transcript files prevents one giant coding
session from bypassing the limit, while reusing the adapters keeps host
envelopes and tool-shaped user records out of the count.
"""
user_turns = 0
has_old_data = False
async for event in adapter.scan(ScanOptions()):
if isinstance(event, ScanFailure):
# A partial probe must never turn into an unbounded first import.
return True
if not isinstance(event, ScannedEvidence):
continue
observed = event.evidence.created_at
if observed is None and event.cursor.observed_modified_ns > 0:
observed = datetime.fromtimestamp(
event.cursor.observed_modified_ns / 1_000_000_000,
tz=UTC,
)
if observed is not None:
if observed.tzinfo is None:
observed = observed.replace(tzinfo=UTC)
has_old_data = has_old_data or observed.astimezone(UTC) < cutoff
if event.evidence.role == "user":
user_turns += 1
if user_turns > ONBOARDING_LARGE_HISTORY_USER_TURNS and has_old_data:
return True
return False
def _install_state(adapter: object) -> Awaitable[SourceInstallState] | None:
"""Ask an adapter for the two-level answer when it has one.
@@ -4588,7 +4783,7 @@ async def _keep_loop_heartbeat(watchdog: LoopWatchdog) -> None:
async def _emit_resource_snapshots(interval_seconds: float = 60.0) -> None:
while True:
diagnostic_event(
level="info",
level="debug",
component="core.resources",
event="core.resource_snapshot",
outcome="observed",
@@ -4613,6 +4808,33 @@ def _start_backfill(app: FastAPI) -> None:
)
def _resume_sedimentation(app: FastAPI) -> None:
"""Pick the work parked by a stop back up, off the caller's clock.
Turning the switch back on is a decision, not a job the caller waits for:
the choice is already stored, and what follows it is maintenance over every
unfinished task followed by a scan of the whole queue. On a real backlog
that runs for many minutes, so awaiting it here left the PUT unanswered
until the client gave up — the renderer showed "启用未能确认" for a switch
that had in fact been saved, and the switch only appeared to turn on once
the follow-up read came back. Measured: one toggle, twenty-plus minutes,
no response, client timeout at thirty seconds.
Repeated toggles reuse the running walk; it reads the switch as it goes, so
a stop part-way through is honoured on the next batch instead of needing a
cancel.
"""
running = cast("asyncio.Task[None] | None", getattr(app.state, "sedimentation_resume", None))
if running is not None and not running.done():
return
app.state.sedimentation_resume = watched_task(
asyncio.get_running_loop(),
lambda: _memory_processing_supervisor(app).resume_allowed(),
component="memory.sedimentation",
name="one-memory-sedimentation-resume",
)
def _start_visibility_recompute(app: FastAPI) -> None:
"""Keep the stored visibility conclusions current, in the background.
@@ -4856,6 +5078,52 @@ def _source_writes_on(app: FastAPI, source_id: str) -> bool:
return _source_processing_enabled(app, source_id)
def _sedimentation_claim_enabled(app: FastAPI, source_id: str) -> bool:
"""Claim queued model work only when policy, source and model all admit it."""
return (
_config_store(app).sedimentation_enabled()
and _model_profile_store(app).active_identity() is not None
and _history_import_gate_released(app)
and _source_writes_on(app, source_id)
)
def _history_import_gate_released(app: FastAPI) -> bool:
"""Hold first sedimentation until every accepted history import is terminal."""
if getattr(app.state, "history_import_gate_released", False):
return True
onboarding = getattr(app.state, "onboarding_store", None)
jobs = getattr(app.state, "import_job_store", None)
if onboarding is None or jobs is None:
return True
state = onboarding.get()
if state.status != "importing" or not state.selected_source_ids:
app.state.history_import_gate_released = True
return True
selected = set(state.selected_source_ids)
matching = [
job
for job in jobs.list_jobs()
if job.operation == "history"
and job.source_id in selected
and (state.started_at is None or job.created_at >= state.started_at)
]
if {job.source_id for job in matching} != selected:
return False
active = {
ImportStatus.PENDING,
ImportStatus.RUNNING,
ImportStatus.PAUSED,
ImportStatus.INTERRUPTED,
}
released = all(job.status not in active for job in matching)
if released:
app.state.history_import_gate_released = True
return released
def _model_prober(app: FastAPI) -> ModelProber:
return cast(ModelProber, app.state.model_prober)
+10 -1
View File
@@ -1,4 +1,5 @@
import hmac
import ipaddress
from collections.abc import Callable
from urllib.parse import urlsplit
@@ -35,7 +36,15 @@ def is_loopback_origin(origin: str) -> bool:
host = parsed.hostname
if not host:
return False
return host in LOOPBACK_HOSTNAMES or host.startswith("127.")
if host in LOOPBACK_HOSTNAMES:
return True
# A prefix check is not enough: "127.attacker.example" is a perfectly valid
# DNS name that resolves wherever its owner wants (including 127.0.0.1 via
# DNS rebinding). Parse the address and ask the standard library.
try:
return ipaddress.ip_address(host).is_loopback
except ValueError:
return False
def local_api_guard(
@@ -175,6 +175,22 @@ async def in_interactive_read[T](
return await in_read_pool(interactive_read_pool(), work, *args, **kwargs)
def _retrieve[T](future: "asyncio.Future[T]") -> None:
"""Mark a shielded read's outcome as seen.
`in_read_pool` shields the wrapped worker, so a caller that gives up and
cancels leaves the read running to completion. When that read then fails,
nothing awaits the future any more: asyncio reports the orphan at GC as
"Future exception was never retrieved", the loop handler turns it into a
`core.loop.task_exception` error diagnostic, and a lane that degraded on
its own budget -- correctly, and already reported -- looks like a crashed
task on top of it. Retrieving here keeps the diagnostics honest. Awaiting
still raises for callers that are still there, so no failure is hidden.
"""
if not future.cancelled():
future.exception()
async def in_read_pool[T](
pool: ThreadPoolExecutor,
work: Callable[..., T],
@@ -193,7 +209,9 @@ async def in_read_pool[T](
_interactive_reads -= 1
worker.add_done_callback(released)
return await asyncio.shield(asyncio.wrap_future(worker))
pending = asyncio.wrap_future(worker)
pending.add_done_callback(_retrieve)
return await asyncio.shield(pending)
async def wait_for_interactive_reads() -> None:
@@ -76,8 +76,7 @@ def _memory(db: sqlite3.Connection, reference: CallReference) -> SourceRecordRef
if row["status"] == "deleted" or row["deleted_at"] is not None:
return _hidden(reference)
# Same local revocation/ownership predicate as SourceReadAccess.memories,
# evaluated BEFORE freshness (which can read support text), in this snapshot.
# Same local revocation/ownership predicate as SourceReadAccess.memories.
# Its public methods open their own connections, so cannot be used here.
access = db.execute(
"SELECT EXISTS (SELECT 1 FROM governed_memory_evidence "
@@ -121,8 +120,8 @@ def resolve_call_references(database_path: Path, event: CallEvent) -> list[Sourc
Local-owner only: event host attribution is NOT authorization. Explicit local
source revocations deny access; absent grants retain legacy local semantics.
A memory needs at least one accessible active citation. No proof preserves
shared freshness compatibility, not a claim of verified historical freshness.
A memory needs at least one accessible active citation. Evidence freshness
proofs are retired and do not affect availability.
Any read/parse/budget failure raises a fixed whole-request error, never a
partial result or per-item deletion. The route owner must map this separately
+21 -3
View File
@@ -266,7 +266,13 @@ def serve(*, maintenance: bool = False, reader: bool = False) -> None:
)
application = create_reader_app(reader_settings)
print(f"reader_port:{port}", file=sys.stderr, flush=True)
uvicorn.run(application, host=settings.host, port=port, reload=False)
uvicorn.run(
application,
host=settings.host,
port=port,
reload=False,
access_log=False,
)
return
from one_memory.api import create_app
@@ -297,9 +303,21 @@ def serve(*, maintenance: bool = False, reader: bool = False) -> None:
)
if maintenance:
with CoreRuntime(settings, publish_receipt=False):
uvicorn.run(application, host=settings.host, port=settings.port, reload=False)
uvicorn.run(
application,
host=settings.host,
port=settings.port,
reload=False,
access_log=False,
)
else:
uvicorn.run(application, host=settings.host, port=settings.port, reload=False)
uvicorn.run(
application,
host=settings.host,
port=settings.port,
reload=False,
access_log=False,
)
if __name__ == "__main__":
@@ -196,7 +196,7 @@ def read_raw_context(
if _prefers_declared_facts(request.query)
else "1"
)
person_authored = "CASE WHEN e.written_by='person' THEN 1 ELSE 0 END"
person_authored = "CASE WHEN e.role='user' THEN 1 ELSE 0 END"
query_echo = (
"CASE WHEN trim(e.verbatim)=? OR trim(e.verbatim)=? THEN 0 ELSE 1 END"
if _prefers_declared_facts(request.query)
@@ -470,9 +470,8 @@ def revalidate_raw_context(
)
except (ValueError, TypeError):
break
if (row["role"], row["written_by"], created) != (
if (row["role"], created) != (
message.role,
message.written_by,
message.created_at,
):
break
@@ -66,12 +66,12 @@ def _field(text: str) -> ResumeField | None:
return None
def _raw_class(written_by: str, evidence_id: str, accepted: frozenset[str]) -> EvidenceClass:
if written_by == "person":
def _raw_class(role: str, evidence_id: str, accepted: frozenset[str]) -> EvidenceClass:
if role == "user":
return "person_statement"
if written_by == "agent":
if role == "assistant":
return "agent_report"
if written_by == "tool" and evidence_id in accepted:
if role == "tool" and evidence_id in accepted:
return "system_acceptance"
return "tool_output"
@@ -118,7 +118,7 @@ def assemble_resume_packet(
field=_field(message.text) or "unclassified",
text=message.text,
evidence_class=_raw_class(
message.written_by,
message.role,
message.evidence_id,
verified_system_evidence_ids,
),
@@ -345,7 +345,7 @@ def _lexical_raw_rows(
if context_raw._prefers_declared_facts(query.query)
else "1"
)
person_authored = "CASE WHEN e.written_by='person' THEN 1 ELSE 0 END"
person_authored = "CASE WHEN e.role='user' THEN 1 ELSE 0 END"
query_echo = (
"CASE WHEN trim(e.verbatim)=? OR trim(e.verbatim)=? THEN 0 ELSE 1 END"
if context_raw._prefers_declared_facts(query.query)
@@ -507,7 +507,7 @@ def _raw_item(row: sqlite3.Row, start1: int, matched_by: tuple[str, ...]) -> Raw
"source_id": row["source_id"],
"conversation_id": row["conversation_id"],
"role": row["role"],
"written_by": "person" if row["written_by"] == "person" else "agent",
"written_by": "person" if row["role"] == "user" else "agent",
"event_time": event_time.isoformat() if event_time else None,
"spans": [{"start_cp": start, "end_cp": start + len(text), "text": text}],
"matched_by": list(matched_by),
@@ -1039,7 +1039,7 @@ def close_main_databases() -> None:
def log_main_database_stats(moment: str) -> None:
"""One line in core.log: how many connections are open and who waited."""
stats = main_database_stats()
logger.info(
logger.debug(
"main_database_connections moment=%s %s",
moment,
" ".join(f"{key}={value}" for key, value in stats.items()),
@@ -19,6 +19,7 @@ from __future__ import annotations
import atexit
import json
import logging
import os
import re
import sys
@@ -139,6 +140,10 @@ _DETAIL_KEYS = frozenset(
"accounted_ms",
"unaccounted_ms",
"conversation_hash",
# The sedimentation switch: a bounded boolean, the same shape as
# `spawned` and `index_complete`. Left out of this list it did not
# drop a log line, it turned the switch's own request into a 500.
"enabled",
}
)
_trace_id: ContextVar[str | None] = ContextVar("one_memory_trace_id", default=None)
@@ -649,12 +654,24 @@ def diagnostic_event(**values: object) -> None:
active = _active
if active is None:
return
if values.get("level") == "debug" and not logging.getLogger().isEnabledFor(
logging.DEBUG
):
return
try:
active.write(**values) # type: ignore[arg-type]
except OSError:
# Diagnostics must not become a new product outage when a disk is full
# or the log directory is temporarily unavailable.
return
except ValueError:
# Same promise, same reason, for a rejected payload instead of a failed
# write. An unlisted `details` key once turned every external recall
# into a 500 and then killed the process, and the sedimentation switch
# into a permanent failure. The payload itself is still wrong and is
# still caught in CI by tests/test_diagnostic_payload_keys.py; what
# must not happen is the caller paying for it.
return
def current_session_id() -> str | None:
@@ -31,8 +31,8 @@ from one_memory.hy_runtime_config import (
diagnostic_log_level,
)
from one_memory.hy_source_context import HySourceContext, HyWriteBindings
from one_memory.hy_store_concurrency import native_store_read_slot
from one_memory.model_profiles import ModelProfileSnapshot, ProviderKind, attach_disable_thinking
from one_memory.model_request_audit import audit_precipitation_request
from one_memory.semantic_deadline import SemanticMemoryHit
#: D102: how relevant HY has to call a hit before it counts. Raised from 0.25
@@ -143,6 +143,22 @@ class HyMemoryBridgeError(RuntimeError):
pass
MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES = 512
def coding_tool_result_max_bytes() -> int:
"""Return the per-tool payload ceiling used by MemHub's HY client."""
raw = os.environ.get("MEMORY_CODING_TOOL_RESULT_MAX_BYTES")
if raw is None:
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
try:
value = int(raw)
except ValueError:
return MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
return value if value > 0 else MEMHUB_DEFAULT_CODING_TOOL_RESULT_MAX_BYTES
def _exclusive[**P, T](
method: Callable[P, Coroutine[Any, Any, T]],
) -> Callable[P, Coroutine[Any, Any, T]]:
@@ -414,8 +430,9 @@ class HyMemoryBridge:
@_exclusive
async def ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
async with self._write_lock:
return await self._ingest(request)
with audit_precipitation_request(request.request_id):
async with self._write_lock:
return await self._ingest(request)
async def _ingest(self, request: HyMemoryIngestRequest) -> HyMemoryIngestResult:
started = monotonic()
@@ -431,9 +448,7 @@ class HyMemoryBridge:
guards = getattr(client, "_one_memory_write_guards", ())
from one_memory.hy_response_guard import HyReconcileGuard
reconcile_guards = tuple(
guard for guard in guards if isinstance(guard, HyReconcileGuard)
)
reconcile_guards = tuple(guard for guard in guards if isinstance(guard, HyReconcileGuard))
for guard in reconcile_guards:
guard.begin_write()
before = [len(guard.failures) for guard in guards]
@@ -544,8 +559,9 @@ class HyMemoryBridge:
request: HyMemoryIngestRequest,
contents: Sequence[str],
) -> HyMemoryIngestResult:
async with self._write_lock:
return await self._ingest_extracted(request, contents)
with audit_precipitation_request(request.request_id):
async with self._write_lock:
return await self._ingest_extracted(request, contents)
async def _ingest_extracted(
self,
@@ -967,27 +983,29 @@ class HyMemoryBridge:
caller could only rank by position — which forced semantic hits to be
placed as a block ahead of everything the keyword lane found. The
score is what lets the two lanes be ranked against each other.
Native search runs without a process-local write gate. MemHub expects
concurrent read during precipitation; store work is a short burst after
network-bound embed/LLM steps, and refusing reads with
``hy_memory_store_busy`` only pushed callers onto weaker fallbacks.
"""
async with native_store_read_slot() as admitted:
if not admitted:
raise HyMemoryBridgeError("hy_memory_store_busy")
client = await self._get_client()
try:
result = await _joined_thread(
client.search,
query,
user_ids=[user_id],
agent_ids=[] if agent_id == "*" else [agent_id],
limit=limit,
min_score=hy_relevance_floor(),
schema_min_score=hy_relevance_floor(),
schema_limit=limit,
intention_limit=limit,
scene="normal",
reader=HY_MEMORY_READER,
)
except Exception as error:
raise HyMemoryBridgeError("hy_memory_search_failed") from error
client = await self._get_client()
try:
result = await _joined_thread(
client.search,
query,
user_ids=[user_id],
agent_ids=[] if agent_id == "*" else [agent_id],
limit=limit,
min_score=hy_relevance_floor(),
schema_min_score=hy_relevance_floor(),
schema_limit=limit,
intention_limit=limit,
scene="normal",
reader=HY_MEMORY_READER,
)
except Exception as error:
raise HyMemoryBridgeError("hy_memory_search_failed") from error
hits = _replied_memories(result, "hy_memory_search_shape_unknown")
found: list[SemanticMemoryHit] = []
seen: set[str] = set()
@@ -1110,7 +1128,7 @@ class HyMemoryBridge:
self.profile,
api_key=self.api_key,
data_dir=self.data_dir,
http_transport_factory=self._http_transport_factory,
http_transport_factory=self._http_transport_factory,
)
await _joined_thread(initialize)
@@ -1137,7 +1155,7 @@ class HyMemoryBridge:
)
raise HyMemoryBridgeError("hy_memory_init_failed") from error
diagnostic_event(
level="info",
level="debug",
component="hy.client",
event="hy.client_initialization.completed",
outcome="succeeded",
@@ -1328,9 +1346,7 @@ async def _with_native_mutation_delta(
"""
retired = set(result.retired_memory_ids)
superseded = {
key: tuple(values) for key, values in result.superseded_memory_ids.items()
}
superseded = {key: tuple(values) for key, values in result.superseded_memory_ids.items()}
chat_targets = {
key
for guard in reconcile_guards
@@ -1338,7 +1354,8 @@ async def _with_native_mutation_delta(
if isinstance(key, str) and key
}
for item in raw.get("ops", ()) if isinstance(raw.get("ops"), list) else ():
operations = raw.get("ops")
for item in operations if isinstance(operations, list) else ():
if not isinstance(item, Mapping):
continue
if item.get("action") == "DELETE":
@@ -1346,7 +1363,8 @@ async def _with_native_mutation_delta(
if isinstance(key, str) and key:
retired.add(key)
for item in raw.get("fact_ops", ()) if isinstance(raw.get("fact_ops"), list) else ():
fact_operations = raw.get("fact_ops")
for item in fact_operations if isinstance(fact_operations, list) else ():
if not isinstance(item, Mapping) or item.get("action") != "SUPERSEDE":
continue
old, new = item.get("fact_id"), item.get("superseded_by")
@@ -1367,9 +1385,7 @@ async def _with_native_mutation_delta(
if isinstance(replacement, str) and replacement:
superseded[key] = (replacement,)
elif isinstance(replacement, list):
values = tuple(
value for value in replacement if isinstance(value, str) and value
)
values = tuple(value for value in replacement if isinstance(value, str) and value)
if values:
superseded[key] = values
@@ -1697,6 +1713,7 @@ def build_hy_memory_config(
config.history.db_path = str(memory_root / "history.sqlite3")
config.coding.enable = True
config.coding.db_path = str(memory_root / "coding.sqlite3")
config.coding.tool_result_max_bytes = coding_tool_result_max_bytes()
config.recall.episode_recall_enabled = True
config.recall.episode_summary_enabled = False
config.recall.dynamic_episode_enabled = True
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Literal, cast
HY_MEMORY_DISTRIBUTION = "hy-memory"
HY_MEMORY_VERSION = "2.0.3"
HY_MEMORY_VERSION = "2.0.4"
HY_MEMORY_NAMESPACE = "governed-v2"
DiagnosticLogLevel = Literal["error", "warning", "info", "debug"]
@@ -31,11 +31,12 @@ _COLLECTION_LOGGERS = (
"httpx",
"httpcore",
)
_NOISY_LIBRARY_LOGGERS = ("httpx", "httpcore", "watchfiles", "jieba")
# One Memory deliberately stays on ultra because its existing, user-visible
# profile/graph flow calls digest(), which the pinned public SDK exposes only
# for ultra clients. Reader and non-reranker behavior remain pinned for
# compatibility with public SDK 2.0.3.
# compatibility with public SDK 2.0.4.
HY_MEMORY_MODE = "ultra"
# The SDK only recognises `legacy` and `hybrid` (see
# hy_memory/pipelines/_retrieval/config.py: `ALL_READERS = (READER_LEGACY,
@@ -139,6 +140,18 @@ def apply_diagnostic_log_level(level: DiagnosticLogLevel) -> None:
for name in tuple(logging.Logger.manager.loggerDict):
if any(name == root or name.startswith(f"{root}.") for root in _COLLECTION_LOGGERS):
logging.getLogger(name).setLevel(selected)
# Successful HTTP exchanges, file-watch notifications and tokenizer
# initialization are useful only while actively debugging. At INFO they
# dominated core.log and the in-app panel without describing product work.
noisy_level = selected if selected == logging.DEBUG else max(selected, logging.WARNING)
for root in _NOISY_LIBRARY_LOGGERS:
logging.getLogger(root).setLevel(noisy_level)
for name in tuple(logging.Logger.manager.loggerDict):
if any(
name == root or name.startswith(f"{root}.")
for root in _NOISY_LIBRARY_LOGGERS
):
logging.getLogger(name).setLevel(noisy_level)
os.environ["MEMORY_LOG_LEVEL"] = logging.getLevelName(selected)
os.environ["MEMORY_PIPELINE_TRACE_ENABLED"] = "true" if level == "debug" else "false"
@@ -7,7 +7,7 @@ from typing import Any, Literal
from pydantic import BaseModel, ConfigDict
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord, WrittenBy
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
from one_memory.input_sanitization import sanitize_model_input
@@ -61,17 +61,7 @@ class HyWriteBindings:
seed.workspace,
)
or record.content_kind is not EvidenceContentKind.RAW_CONVERSATION
or (record.role, record.written_by)
not in {
# D98: the same change as the turn assembly. A sub agent's
# question is a `user` message marked written by the
# agent; excluding it here would make the source context
# disagree with what was actually sent.
("user", WrittenBy.PERSON),
("user", WrittenBy.AGENT),
("assistant", WrittenBy.AGENT),
("tool", WrittenBy.TOOL),
}
or record.role not in {"user", "assistant", "tool"}
):
raise ValueError("hy_memory_write_verification_context_scope")
self.all_sources: list[dict[str, Any]] = [
@@ -152,7 +152,7 @@ class HyStepTimer:
if value is None:
continue
diagnostic_event(
level="info",
level="debug",
component=STEP_COMPONENTS[name],
event="hy.step.completed",
outcome="succeeded",
@@ -168,7 +168,7 @@ class HyStepTimer:
)
model = measured.get("model")
diagnostic_event(
level="info",
level="debug",
component="hy.bridge",
event="hy.steps.summary",
outcome="succeeded",
@@ -1,41 +0,0 @@
"""Process-local coordination for HY's native stores."""
from __future__ import annotations
import asyncio
import weakref
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
_WRITE_LOCKS: weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
weakref.WeakKeyDictionary()
)
def native_store_write_lock() -> asyncio.Lock:
"""Return the write lock for the current event loop."""
loop = asyncio.get_running_loop()
lock = _WRITE_LOCKS.get(loop)
if lock is None:
lock = asyncio.Lock()
_WRITE_LOCKS[loop] = lock
return lock
@asynccontextmanager
async def native_store_read_slot() -> AsyncIterator[bool]:
"""Admit a native HY read only while no write is active.
Chroma's native reader may retain the GIL after an asyncio timeout. Reads
therefore share the physical-store lock, but never wait behind a writer:
host recall can immediately fall back to the governance ledger.
"""
lock = native_store_write_lock()
if lock.locked():
yield False
return
await lock.acquire()
try:
yield True
finally:
lock.release()
@@ -34,10 +34,11 @@ class ScanIssueCode(StrEnum):
SOURCE_BUSY = "source_busy"
UNSUPPORTED_SCHEMA = "unsupported_schema"
BUDGET_EXCEEDED = "budget_exceeded"
#: The file is bigger than the import's own per-file ceiling. Skipped with
#: a reason rather than read, because one oversized history file can hold
#: the import open long enough to take the process with it. Distinct from
#: the access layer's `FILE_TOO_LARGE`, which is a per-root limit.
#: The file is bigger than a per-file ceiling: either the import's own
#: ceiling or the access gate's per-root `max_file_bytes`. Both are skipped
#: with this reason rather than read, because one oversized history file
#: can hold the import open long enough to take the process with it, and
#: the user-facing cause is the same size limit.
FILE_TOO_LARGE = "file_too_large"
HISTORY_WINDOW_EXCLUDED = "history_window_excluded"
@@ -105,7 +105,7 @@ async def iterate_in_worker[T](
# was unwinding with, and a diagnostic is not worth an import.
try:
diagnostic_event(
level="info",
level="debug",
component="import.async_bridge",
event="import.source_read_finished",
outcome="succeeded",
@@ -1,6 +1,7 @@
import sqlite3
from contextlib import AbstractContextManager
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
from pydantic import BaseModel, ConfigDict
@@ -20,6 +21,23 @@ CURSOR_MAX_SOURCE_FILE_BYTES = 64 * 1024 * 1024 * 1024
GRANT_SCHEMA = source_grants_ddl("sqlite")
class SourceAuthorizationState(StrEnum):
"""Why a source cannot be read right now, or `ok` when it can.
The API used to answer `authorized=false` for all of these at once, which
is how a revoked grant, a rebuilt history directory and a moved candidate
all looked like the same bug and all got the same advice. The values carry
no paths and no user content, because they are written to diagnostics.
"""
OK = "ok"
MISSING_GRANT = "missing_grant"
REVOKED_GRANT = "revoked_grant"
SOURCE_IDENTITY_CHANGED = "source_identity_changed"
SOURCE_ROOT_UNAVAILABLE = "source_root_unavailable"
CANDIDATE_CHANGED = "candidate_changed"
class SourceGrantReceipt(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
@@ -146,6 +164,46 @@ class SourceGrantStore:
raise SourceAccessError(SourceAccessErrorCode.SOURCE_IDENTITY_CHANGED)
return authorized
def authorization_state(
self,
*,
user_id: str,
source_id: str,
candidate: Path,
) -> SourceAuthorizationState:
"""Why reading this source would be refused.
Read-only and fails the same way `authorized_root` does: this names a
state for the UI and for diagnostics, it does not widen access. The
candidate is compared by path, so a grant that still points somewhere
else reads as a moved candidate rather than as a rebuilt directory —
the two need different fixes.
"""
with self._connect() as connection:
row = connection.execute(
"SELECT * FROM source_grants WHERE user_id=? AND source_id=?",
(user_id, source_id),
).fetchone()
if row is None:
return SourceAuthorizationState.MISSING_GRANT
if row["revoked_at"] is not None or row["revision"] <= 0:
return SourceAuthorizationState.REVOKED_GRANT
granted = Path(row["requested_path"])
if not granted.exists():
return SourceAuthorizationState.SOURCE_ROOT_UNAVAILABLE
try:
if not root_identity_matches(granted, row["root_identity"]):
return SourceAuthorizationState.SOURCE_IDENTITY_CHANGED
except SourceAccessError:
# Unreadable or gone between the two calls: there is no grant to
# repair, only a directory to make available again.
return SourceAuthorizationState.SOURCE_ROOT_UNAVAILABLE
# Same directory, but the app now looks somewhere else. Re-authorizing
# the old grant would not help; the candidate has to be granted.
if granted != candidate:
return SourceAuthorizationState.CANDIDATE_CHANGED
return SourceAuthorizationState.OK
def revoke(self, *, user_id: str, source_id: str) -> SourceGrantReceipt:
now = datetime.now(UTC).isoformat()
with self._connect(writing=True) as connection:
@@ -106,7 +106,7 @@ class PollingLiveSyncSupervisor:
level=(
"warn"
if value.state in {LiveSyncState.DEGRADED, LiveSyncState.UNAVAILABLE}
else "info"
else "debug"
),
component="import.live_sync",
event="live_sync.state_changed",
@@ -282,11 +282,24 @@ class PollingLiveSyncSupervisor:
and job.status
in {ImportStatus.PENDING, ImportStatus.RUNNING, ImportStatus.INTERRUPTED}
]
bounded_history_incomplete = any(
job.source_id == source_id
and job.operation == "history"
and job.history_after is not None
and job.status is not ImportStatus.COMPLETED
for job in source_jobs
)
# Establish the realtime EOF independently even while history owns
# this source. Otherwise the P1 job starts from the P2 checkpoint
# (or from zero) and becomes a second full-history scan.
adapter = self.import_supervisor.adapters[source_id]
await self._live_cursors(source_id, adapter)
live_cursors = await self._live_cursors(source_id, adapter)
# A bounded onboarding history is the authority for everything up
# to its EOF. An adapter without a native live baseline has no safe
# realtime cursor until that history finishes; starting a sync from
# zero here would bypass `history_after` and import the old corpus.
if bounded_history_incomplete and not live_cursors:
return None
# Only one scan of a source may be in flight: they share the
# cursor, and two readers means every file read twice. With a
# catch-up still holding the source there is no free moment to ask
@@ -156,7 +156,7 @@ class FileWatchLiveSync:
if source is not None
):
diagnostic_event(
level="info",
level="debug",
component="import.live_watch",
event="live_sync.hook_received",
outcome="queued",
@@ -190,7 +190,7 @@ class FileWatchLiveSync:
try:
await self._trigger_past(source_id, observed)
diagnostic_event(
level="info",
level="debug",
component="import.live_watch",
event="live_sync.hook_completed",
outcome="written",
@@ -79,6 +79,8 @@ class EvidenceInput(BaseModel):
source_ordinal: str | None = Field(default=None, min_length=1, max_length=256)
content_kind: EvidenceContentKind = EvidenceContentKind.RAW_CONVERSATION
role: str = Field(min_length=1, max_length=128)
# Legacy import/storage metadata only. Protocol `role` is authoritative;
# this value must not affect identity, admission, scheduling or ownership.
written_by: WrittenBy
created_at: datetime | None = None
verbatim: str = Field(min_length=1, max_length=MAX_VERBATIM_CHARACTERS)
@@ -149,17 +151,22 @@ class EvidenceInput(BaseModel):
)
)
else:
# `role` is the protocol-level speaker identity. `written_by` was a
# parser guess and differed between two imports of the same host
# message (notably sub-agent/user and unknown/user), minting two
# Evidence rows. Keep it out of every new identity.
#
# No workspace and no source_ordinal in the derived key: the hook
# lane has neither (it reports no workspace and a constant ordinal),
# so including either gave the same message a different key per lane
# and stored it twice. Both are record metadata here, not identity.
identity = "\0".join(
(
"derived-v3",
"derived-v4",
self.user_id,
self.source_id,
self.conversation_id,
self.written_by,
self.role,
self.content_hash,
)
)
@@ -168,6 +175,54 @@ class EvidenceInput(BaseModel):
# two rows. The capture time is record metadata, not identity.
return hashlib.sha256(identity.encode()).hexdigest()
@property
def written_by_derived_v3_idempotency_key(self) -> str:
"""The last identity that treated the parser's author guess as canonical.
New builds key by protocol role. During an upgrade this key lets a
rescan find and reuse the row already stored by derived-v3 instead of
inserting the same host message under a second Evidence id.
"""
identity = "\0".join(
(
"derived-v3",
self.user_id,
self.source_id,
self.conversation_id,
self.written_by,
self.content_hash,
)
)
return hashlib.sha256(identity.encode()).hexdigest()
@property
def written_by_derived_v3_idempotency_keys(self) -> tuple[str, ...]:
"""Every legacy author guess this protocol role may have carried."""
guesses = {
"user": (
WrittenBy.PERSON,
WrittenBy.AGENT,
WrittenBy.UNKNOWN,
),
"assistant": (WrittenBy.AGENT, WrittenBy.UNKNOWN, WrittenBy.PERSON),
"tool": (WrittenBy.TOOL, WrittenBy.UNKNOWN),
}.get(self.role, tuple(WrittenBy))
ordered = (self.written_by, *guesses)
keys: list[str] = []
for written_by in dict.fromkeys(ordered):
identity = "\0".join(
(
"derived-v3",
self.user_id,
self.source_id,
self.conversation_id,
written_by,
self.content_hash,
)
)
keys.append(hashlib.sha256(identity.encode()).hexdigest())
return tuple(keys)
@property
def legacy_message_idempotency_key(self) -> str | None:
"""The pre-workspace key for a message-shaped input, when one exists.
@@ -212,7 +267,13 @@ class EvidenceInput(BaseModel):
"""
return self.derived_v2_idempotency_key("derived-v2", workspace_agnostic=False)
def derived_v2_idempotency_key(self, label: str, *, workspace_agnostic: bool) -> str:
def derived_v2_idempotency_key(
self,
label: str,
*,
workspace_agnostic: bool,
written_by: WrittenBy | None = None,
) -> str:
identity = "\0".join(
(
label,
@@ -220,7 +281,7 @@ class EvidenceInput(BaseModel):
self.source_id,
*([] if workspace_agnostic else [self.workspace or "\uffffnull"]),
self.conversation_id,
self.written_by,
written_by or self.written_by,
self.created_at.isoformat() if self.created_at else "",
self.source_ordinal or "",
# v2 hashed the cleaned body; v1 predates raw_verbatim entirely,
@@ -230,6 +291,26 @@ class EvidenceInput(BaseModel):
)
return hashlib.sha256(identity.encode()).hexdigest()
@property
def older_written_by_idempotency_keys(self) -> tuple[str, ...]:
keys: list[str] = []
for written_by in WrittenBy:
keys.append(
self.derived_v2_idempotency_key(
"derived-v1",
workspace_agnostic=True,
written_by=written_by,
)
)
keys.append(
self.derived_v2_idempotency_key(
"derived-v2",
workspace_agnostic=False,
written_by=written_by,
)
)
return tuple(keys)
@property
def evidence_id(self) -> str:
return str(uuid5(EVIDENCE_NAMESPACE, self.idempotency_key))
@@ -249,6 +330,7 @@ class EvidenceRecord(BaseModel):
source_ordinal: str | None
content_kind: EvidenceContentKind
role: str
# Retained while old databases and export formats still carry the column.
written_by: WrittenBy
created_at: datetime | None
verbatim: str
@@ -199,7 +199,7 @@ class ImportRunner:
raise ValueError("Import job source does not match adapter source")
job, runner_token = await in_the_background(self.job_store.claim, job_id)
diagnostic_event(
level="info",
level="debug",
component="import.parser",
event="import.parse.started",
outcome="started",
@@ -310,7 +310,7 @@ class ImportRunner:
if event.kind == "progress" and str(event.phase) != last_phase:
phase = str(event.phase)
diagnostic_event(
level="info",
level="debug",
component="import.parser",
event="import.phase_changed",
outcome="changed",
@@ -516,7 +516,7 @@ class ImportRunner:
async def _emit_parse_result(self, job: ImportJob, started: float) -> None:
succeeded = job.status is ImportStatus.COMPLETED
diagnostic_event(
level="info" if succeeded else "warn",
level="debug" if succeeded else "warn",
component="import.parser",
event="import.parse.completed",
outcome="succeeded" if succeeded else str(job.status),
@@ -141,11 +141,18 @@ def parse_codex_record(
if role == "user":
written_by = _user_author(record, payload, session)
# Machine inputs may quote a whole transcript, including user_query tags.
# Session provenance is not proof of who wrote any particular future turn.
if written_by is WrittenBy.PERSON:
# Protocol role is authoritative; host author metadata does not change
# how a user message is cleaned.
if not (
content.startswith(
"The following is the Codex agent history whose request action you are assessing."
)
and ">>> TRANSCRIPT START" in content
and ">>> TRANSCRIPT END" in content
):
content = strip_host_wrappers(content)
if not content:
return ParsedSkip(ScanIssueCode.NO_TEXT)
if not content:
return ParsedSkip(ScanIssueCode.NO_TEXT)
return ParsedMessage(
conversation_id=conversation_from_relative_path(context.relative_path),
message_id=first_string(payload.get("id"), record.get("id")),
@@ -44,6 +44,7 @@ from one_memory.imports.discovery import (
discovered_off_loop,
)
from one_memory.imports.models import (
MAX_VERBATIM_CHARACTERS,
EvidenceContentKind,
EvidenceInput,
EvidenceOrigin,
@@ -65,6 +66,20 @@ from one_memory.imports.sources.parsing import parse_timestamp
_log = logging.getLogger(__name__)
def _clamp_verbatim(text: str) -> str:
"""Keep one pathological record from violating EvidenceInput's limit.
A single pasted dump larger than MAX_VERBATIM_CHARACTERS used to fail
validation, retry, and then wedge the whole history import on the same
record. Keep the head and make the truncation explicit and actionable.
"""
if len(text) <= MAX_VERBATIM_CHARACTERS:
return text
marker = f"\n…[truncated: original {len(text)} chars]"
keep = max(0, MAX_VERBATIM_CHARACTERS - len(marker))
return text[:keep] + marker
#: At most one read heartbeat per second. Faster would be more writes than the
#: counts are worth; slower leaves the job's visible state stale.
_READ_PROGRESS_INTERVAL_SECONDS = 1.0
@@ -355,7 +370,7 @@ class JsonlSourceAdapter:
source_file, checkpoint, header = await asyncio.to_thread(
self._prepare, listed, options
)
_log.info(
_log.debug(
"import_file_read source=%s file=%d/%d file_bytes=%d planned_bytes=%d",
self.descriptor.source_id,
index,
@@ -376,7 +391,7 @@ class JsonlSourceAdapter:
else DEFAULT_IMPORT_MAX_FILE_BYTES
)
if source_file.size > budget:
_log.info(
_log.debug(
"import_file_budgeted source=%s file=%d/%d file_bytes=%d budget=%d path=%s",
self.descriptor.source_id,
index,
@@ -1076,8 +1091,11 @@ def to_evidence(
role=parsed.role,
written_by=parsed.written_by,
created_at=parsed.created_at,
verbatim=parsed.verbatim,
raw_verbatim=parsed.raw_verbatim,
verbatim=_clamp_verbatim(parsed.verbatim),
raw_verbatim=(
None if parsed.raw_verbatim is None
else _clamp_verbatim(parsed.raw_verbatim)
),
origin_meta={
EvidenceOrigin.FILE: OriginMetadata(
source_version=origin.source_version or source_version,
@@ -1102,6 +1120,11 @@ def discovery_failure_code(
) -> ScanIssueCode:
if failure.reason_code is SourceAccessErrorCode.BUDGET_EXCEEDED:
return ScanIssueCode.BUDGET_EXCEEDED
if failure.reason_code is SourceAccessErrorCode.FILE_TOO_LARGE:
# The file is readable and its format is fine; only its size is over
# the access gate. Reporting that as `unsupported_schema` sends the
# user looking for a parser problem that does not exist.
return ScanIssueCode.FILE_TOO_LARGE
return ScanIssueCode.UNSUPPORTED_SCHEMA
@@ -1109,6 +1132,8 @@ def _scan_failure_code(code: SourceAccessErrorCode) -> ScanIssueCode:
"""Map a per-file open failure onto a machine code, never onto free text."""
if code is SourceAccessErrorCode.BUDGET_EXCEEDED:
return ScanIssueCode.BUDGET_EXCEEDED
if code is SourceAccessErrorCode.FILE_TOO_LARGE:
return ScanIssueCode.FILE_TOO_LARGE
if code is SourceAccessErrorCode.SYMLINK_NOT_ALLOWED:
return ScanIssueCode.UNSUPPORTED_SCHEMA
if code in {
@@ -7,7 +7,7 @@ import sqlite3
import stat
from collections.abc import AsyncIterator, Iterator
from dataclasses import dataclass, replace
from datetime import UTC, datetime, timedelta
from datetime import UTC, datetime, timedelta, timezone
from functools import partial
from pydantic import JsonValue
@@ -82,6 +82,11 @@ _CURSOR_HOST_CONTEXT_START = re.compile(
)
_CURSOR_USER_QUERY = re.compile(r"<user_query\b")
_CURSOR_TIMESTAMP_PREFIX = re.compile(r"<timestamp>[\s\S]*?</timestamp>\s*")
_CURSOR_TIMESTAMP_VALUE = re.compile(
r"<timestamp>\s*"
r"([A-Za-z]+,\s+[A-Za-z]{3}\s+\d{1,2},\s+\d{4},\s+\d{1,2}:\d{2}\s+[AP]M)"
r"\s+\(UTC([+-]\d{1,2})(?::(\d{2}))?\)\s*</timestamp>",
)
_CURSOR_IMAGE_FILES_PREFIX = re.compile(r"\[Image\]\s*<image_files>[\s\S]*?</image_files>\s*")
_CURSOR_BARE_IMAGE_PREFIX = re.compile(r"(?:\[Image\]\s*)+")
@@ -123,6 +128,24 @@ def _strip_cursor_timestamp(content: str | None) -> str | None:
return _CURSOR_TIMESTAMP_PREFIX.sub("", content, count=1) if content else None
def _cursor_host_timestamp(content: str | None) -> datetime | None:
"""Parse Cursor's user-visible timestamp envelope into UTC."""
if not content:
return None
match = _CURSOR_TIMESTAMP_VALUE.search(content)
if match is None:
return None
try:
local = datetime.strptime(match[1], "%A, %b %d, %Y, %I:%M %p")
hours = int(match[2])
minutes = int(match[3] or "0")
sign = -1 if hours < 0 else 1
offset = timedelta(hours=hours, minutes=sign * minutes)
return local.replace(tzinfo=timezone(offset)).astimezone(UTC)
except (ValueError, OverflowError):
return None
def _strip_cursor_host_prefix(content: str | None) -> str | None:
if not content:
return content
@@ -398,10 +421,7 @@ class CursorUnifiedSourceAdapter:
if (
isinstance(event, ScannedEvidence)
and event.evidence.conversation_id in transcript_ids
and not (
event.evidence.written_by is WrittenBy.AGENT
and is_host_generated_user_turn(event.evidence.verbatim)
)
and not is_host_generated_user_turn(event.evidence.verbatim)
):
yield ScanSkip(
cursor=event.cursor,
@@ -426,10 +446,7 @@ def _shadowed_by_transcript(event: ScanEvent, transcript_ids: set[str]) -> ScanE
if (
isinstance(event, ScannedEvidence)
and event.evidence.conversation_id in transcript_ids
and not (
event.evidence.written_by is WrittenBy.AGENT
and is_host_generated_user_turn(event.evidence.verbatim)
)
and not is_host_generated_user_turn(event.evidence.verbatim)
):
return ScanSkip(
cursor=event.cursor,
@@ -703,6 +720,7 @@ def _parse_transcript_record(
project_name = path_parts[path_parts.index("projects") + 1] if "projects" in path_parts else ""
session_name = context.relative_path.rsplit("/", 1)[-1].removesuffix(".jsonl")
is_subagent = f"{project_name}\0{session_name}" in subagent_ids
host_timestamp = _cursor_host_timestamp(content) if role == "user" else None
created_at = _transcript_fallback_time(context)
native_message_id = record.get("id") or message.get("id")
message_id = (
@@ -728,7 +746,16 @@ def _parse_transcript_record(
created_at=created_at,
workspace=workspace,
record_type="cursor_transcript",
origin_metadata=OriginMetadata(raw_meta={"time_basis": "file_mtime"}),
origin_metadata=OriginMetadata(
raw_meta={
"time_basis": "file_mtime",
**(
{"turn_event_at": host_timestamp.isoformat()}
if host_timestamp is not None
else {}
),
}
),
),
tool_blocks_evidence(message.get("content"), block_types=TOOL_BLOCKS),
)
@@ -819,6 +846,13 @@ def _read_store_messages(
reason_code=ScanIssueCode.UNKNOWN_RECORD,
)
continue
host_timestamp = (
_cursor_host_timestamp(content) if role == "user" else None
)
fallback_timestamp = datetime.fromtimestamp(
source_file.modified_ns / 1_000_000_000,
tz=UTC,
) + timedelta(microseconds=rowid)
parsed = ParsedMessage(
conversation_id=conversation_id,
message_id=str(record.get("id") or row["id"]),
@@ -829,8 +863,19 @@ def _read_store_messages(
else written_by_user
),
verbatim=cleaned,
created_at=fallback_timestamp,
workspace=_store_workspace(source_file),
record_type="cursor_store_blob",
origin_metadata=OriginMetadata(
raw_meta={
"time_basis": "file_mtime",
**(
{"turn_event_at": host_timestamp.isoformat()}
if host_timestamp is not None
else {}
),
}
),
)
yield ScannedEvidence(
cursor=cursor,
@@ -57,7 +57,8 @@ _UNANSWERED_PREFIXES = (
def _is_visible_user_row(role: str, written_by: str, verbatim: str) -> bool:
if role != "user" or written_by != WrittenBy.PERSON.value:
del written_by # Legacy storage metadata; protocol role is authoritative.
if role != "user":
return False
return not verbatim.lstrip().lower().startswith(_UNANSWERED_PREFIXES)
@@ -468,16 +469,54 @@ class EvidenceStore:
if anchored is not None:
self._write_origins(connection, anchored, evidence.origin_meta, now)
workspace, project = self._resolve_conversation_attribution(connection, evidence)
connection.execute(
"""
UPDATE evidence
SET workspace = COALESCE(workspace, ?),
project = COALESCE(project, ?),
updated_at = ?
WHERE evidence_id = ?
""",
(workspace, project, now, anchored),
)
existing = connection.execute(
"SELECT role,revision FROM evidence WHERE evidence_id=?",
(anchored,),
).fetchone()
if existing is None:
raise RuntimeError("anchored evidence disappeared")
if str(existing["role"]) != evidence.role:
was_revised = True
revision = int(existing["revision"]) + 1
connection.execute(
"""
UPDATE evidence
SET workspace=COALESCE(workspace,?),
project=COALESCE(project,?),
content_kind=?,role=?,created_at=COALESCE(?,created_at),
verbatim=?,content_hash=?,revision=?,updated_at=?
WHERE evidence_id=?
""",
(
workspace,
project,
evidence.content_kind,
evidence.role,
evidence.created_at.isoformat() if evidence.created_at else None,
evidence.verbatim,
evidence.content_hash,
revision,
now,
anchored,
),
)
self._insert_current_version(
connection,
anchored,
revision=revision,
changed_at=now,
)
else:
connection.execute(
"""
UPDATE evidence
SET workspace=COALESCE(workspace,?),
project=COALESCE(project,?),
updated_at=?
WHERE evidence_id=?
""",
(workspace, project, now, anchored),
)
record = self._get_with_connection(connection, anchored)
if record is None:
raise RuntimeError("anchored evidence disappeared")
@@ -485,7 +524,11 @@ class EvidenceStore:
from one_memory import ingest_journal
ingest_journal.record(connection, anchored, cause="import")
return PutEvidenceResult(record=record, was_created=False, was_revised=False)
return PutEvidenceResult(
record=record,
was_created=False,
was_revised=was_revised,
)
insert = connection.execute(
"""
INSERT INTO evidence (
@@ -803,27 +846,20 @@ class EvidenceStore:
and same_scope[0]["idempotency_key"] != evidence.idempotency_key
):
return str(same_scope[0]["evidence_id"])
elif (legacy_id := claim_legacy_key(evidence.legacy_derived_idempotency_key)) is not None:
return legacy_id
if not evidence.message_id and not evidence.correlation_id:
# BUG-03 moved the derived key off the capture time and onto the
# pre-cleaning text. A replay of a row stored under the old scheme
# has to be recognized as the same message.
replayed = connection.execute(
"""
SELECT evidence_id FROM evidence
WHERE idempotency_key = ? AND user_id = ? AND source_id = ?
AND conversation_id = ?
""",
elif (
legacy_id := next(
(
evidence.prior_derived_idempotency_key,
evidence.user_id,
evidence.source_id,
evidence.conversation_id,
anchored
for key in (
*evidence.written_by_derived_v3_idempotency_keys,
*evidence.older_written_by_idempotency_keys,
)
if (anchored := claim_legacy_key(key)) is not None
),
).fetchone()
if replayed is not None:
return str(replayed["evidence_id"])
None,
)
) is not None:
return legacy_id
if (
EvidenceOrigin.HOOK in evidence.origins
and evidence.message_id is None
@@ -838,7 +874,7 @@ class EvidenceStore:
ON hook.evidence_id=e.evidence_id AND hook.origin='hook'
WHERE e.user_id=? AND e.source_id=? AND e.conversation_id=?
AND (e.workspace IS ? OR e.workspace IS NULL OR ? IS NULL)
AND e.role=? AND e.written_by=? AND e.content_hash=?
AND e.role=? AND e.content_hash=?
AND e.message_id IS NOT NULL AND hook.evidence_id IS NULL
AND ABS((julianday(e.created_at)-julianday(?))*86400.0) <= 5.0
ORDER BY ABS((julianday(e.created_at)-julianday(?))*86400.0), e.evidence_id
@@ -851,7 +887,6 @@ class EvidenceStore:
evidence.workspace,
evidence.workspace,
evidence.role,
evidence.written_by,
evidence.content_hash,
evidence.created_at.isoformat(),
evidence.created_at.isoformat(),
@@ -872,7 +907,7 @@ class EvidenceStore:
AND (e.workspace IS ? OR e.workspace IS NULL OR ? IS NULL)
-- Unary plus keeps the coverage index from looking better than
-- the message index, which narrows to this exact text.
AND e.role = ? AND +e.written_by = ? AND e.content_hash = ?
AND e.role = ? AND e.content_hash = ?
AND e.message_id IS NULL
""",
(
@@ -882,7 +917,6 @@ class EvidenceStore:
evidence.workspace,
evidence.workspace,
evidence.role,
evidence.written_by,
evidence.content_hash,
),
).fetchall()
@@ -1247,7 +1281,7 @@ class EvidenceStore:
rows = connection.execute(
"""
SELECT evidence_id FROM evidence
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
WHERE user_id = 'local' AND role = 'user'
ORDER BY imported_at, evidence_id
LIMIT ?
""",
@@ -1301,7 +1335,7 @@ class EvidenceStore:
rows = connection.execute(
f"""
SELECT evidence_id FROM evidence
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
WHERE user_id = 'local' AND role = 'user'
AND ({predicates})
ORDER BY imported_at, evidence_id
LIMIT ?
@@ -1333,7 +1367,7 @@ class EvidenceStore:
rows = connection.execute(
f"""
SELECT evidence_id FROM evidence
WHERE user_id = 'local' AND role = 'user' AND written_by = 'person'
WHERE user_id = 'local' AND role = 'user'
AND workspace IS NOT NULL
AND ({predicates})
ORDER BY imported_at, evidence_id
@@ -1476,7 +1510,7 @@ class EvidenceStore:
)
values: list[object] = [user_id, *(source_ids or ())]
visible_user = """
role='user' AND written_by='person' AND NOT (
role='user' AND NOT (
lower(ltrim(verbatim)) LIKE '<codex_internal_context%' OR
lower(ltrim(verbatim)) LIKE '<environment_context>%' OR
lower(ltrim(verbatim)) LIKE '<heartbeat>%' OR
@@ -1485,7 +1519,7 @@ class EvidenceStore:
)
"""
selected_visible_user = """
e.role='user' AND e.written_by='person' AND NOT (
e.role='user' AND NOT (
lower(ltrim(e.verbatim)) LIKE '<codex_internal_context%' OR
lower(ltrim(e.verbatim)) LIKE '<environment_context>%' OR
lower(ltrim(e.verbatim)) LIKE '<heartbeat>%' OR
@@ -1638,7 +1672,7 @@ class EvidenceStore:
if seed is None:
return ConversationTurnPage((), {}, 0, offset, limit)
visible_user = """
role='user' AND written_by='person' AND NOT (
role='user' AND NOT (
lower(ltrim(verbatim)) LIKE '<codex_internal_context%' OR
lower(ltrim(verbatim)) LIKE '<environment_context>%' OR
lower(ltrim(verbatim)) LIKE '<heartbeat>%' OR
@@ -1731,7 +1765,7 @@ class EvidenceStore:
AND first.conversation_id = evidence.conversation_id
AND first.workspace IS evidence.workspace
AND (
(first.role = 'user' AND first.written_by = 'person')
first.role = 'user'
OR first.content_kind = 'memory_content'
)
ORDER BY julianday(COALESCE(first.created_at, first.imported_at)),
@@ -2125,8 +2159,6 @@ class EvidenceStore:
(
existing["content_kind"] != evidence.content_kind,
existing["role"] != evidence.role,
evidence.written_by is not WrittenBy.UNKNOWN
and existing["written_by"] != evidence.written_by,
existing["content_hash"] != evidence.content_hash,
)
)
@@ -111,7 +111,7 @@ class ImportSupervisor:
)
)
diagnostic_event(
level="info",
level="debug",
component="import.supervisor",
event="import.queued",
outcome="queued",
@@ -198,7 +198,7 @@ class ImportSupervisor:
started = monotonic()
failure_type: str | None = None
diagnostic_event(
level="info",
level="debug",
component="import.supervisor",
event="import.started",
outcome="started",
@@ -85,15 +85,14 @@ def _state(connection: sqlite3.Connection, evidence_id: str) -> dict[str, str |
value = evidence.model_dump(mode="json")
# The stored normalized content_hash is insufficient for exact-content leases.
content = _digest(value["verbatim"])
material = _digest(
{key: value[key] for key in ("verbatim", "role", "written_by", "content_kind")}
)
material = _digest({key: value[key] for key in ("verbatim", "role", "content_kind")})
origins = _digest(value["origin_meta"])
identity = _digest(
{
key: item
for key, item in value.items()
if key not in {"verbatim", "origin_meta", "imported_at", "updated_at"}
if key
not in {"verbatim", "written_by", "origin_meta", "imported_at", "updated_at"}
}
)
return dict(
@@ -3,6 +3,7 @@
import json
import time
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
from typing import TextIO, cast
@@ -104,7 +105,13 @@ def capture_hook(source_id: str, payload: object, database: Path) -> bool:
# Cursor also runs ~/.claude/settings.json hooks; its own hook already files the turn.
if source_id != "cursor" and "cursor_version" in payload:
return False
if not database.is_file() or not _sync_open(database, source_id):
if not database.is_file():
return False
rejection = capture_rejection(database, source_id)
if rejection is not None:
# Silent, and the reason is counted rather than printed: a hook that
# spoke would put this into the host's turn as text.
record_capture_rejection(database, source_id, rejection)
return False
if source_id == "pi" and _is_automated_session(first_string(payload.get("cwd"))):
return False
@@ -139,17 +146,79 @@ def capture_hook(source_id: str, payload: object, database: Path) -> bool:
def _sync_open(database: Path, source_id: str) -> bool:
return capture_rejection(database, source_id) is None
class CaptureRejection(StrEnum):
"""Why a hook's turn was dropped, or nothing at all.
Capture fails open on purpose: a hook that raises or prints would be read
back into the host's turn, so a refusal has to be silent and exit 0. It was
also completely invisible, which is why "持续同步开着但一条都没进" could not be
told apart from "capture 坏了". These values name the reason without any
payload, path, session or message — the only part that may be recorded.
"""
SYNC_DISABLED = "sync_disabled"
GRANT_MISSING = "grant_missing"
GRANT_REVOKED = "grant_revoked"
def capture_rejection(database: Path, source_id: str) -> CaptureRejection | None:
"""None when this turn may be captured, otherwise why it may not."""
from one_memory.memory_server.config_state import ConfigStore
if not ConfigStore(database.parent).sync_enabled(source_id):
return False
return CaptureRejection.SYNC_DISABLED
with main_database(database).read() as connection:
row = connection.execute(
"SELECT 1 FROM source_grants WHERE user_id='local' AND source_id=? "
"AND revoked_at IS NULL",
"SELECT revoked_at FROM source_grants WHERE user_id='local' AND source_id=?",
(source_id,),
).fetchone()
return row is not None
if row is None:
return CaptureRejection.GRANT_MISSING
# Indexed, not named: this connection has no row factory, and a hook is
# not the place to add one.
if row[0] is not None:
return CaptureRejection.GRANT_REVOKED
return None
def record_capture_rejection(
database: Path,
source_id: str,
reason: CaptureRejection,
*,
window_seconds: float = 600.0,
) -> None:
"""Note one dropped turn, at most once per window per source and reason.
Best effort by construction: a hook runs inside somebody's turn, so a sink
that is slow, full or read-only must never surface here. The record is a
count and nothing else — no payload, no path, no session or conversation
id — because this file is meant to be handed to support as is.
One small file per source and reason, rather than a line per turn: capture
runs on every prompt, and a log that grew with the conversation would be a
privacy problem wearing a diagnosis as a disguise.
"""
try:
directory = database.parent / "logs"
directory.mkdir(parents=True, exist_ok=True)
marker = directory / f"capture-rejection-{source_id}-{reason.value}.json"
now = time.time()
payload: dict[str, object] = {"source_id": source_id, "reason": reason.value}
if marker.exists() and now - marker.stat().st_mtime < window_seconds:
payload = json.loads(marker.read_text(encoding="utf8"))
previous = payload.get("count", 1)
payload["count"] = int(previous) + 1 if isinstance(previous, (str, int, float)) else 2
else:
payload["count"] = 1
payload["window_started_at"] = datetime.fromtimestamp(now, UTC).isoformat()
payload["last_at"] = payload["window_started_at"]
marker.write_text(json.dumps(payload, sort_keys=True), encoding="utf8")
except Exception:
return
def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed: bool) -> None:
@@ -167,7 +236,7 @@ def _submit_live_memory(store: EvidenceStore, record: EvidenceRecord, *, changed
):
if (
record in chunk
and any(item.role == "user" and item.written_by is WrittenBy.PERSON for item in chunk)
and any(item.role == "user" for item in chunk)
and any(item.role == "assistant" for item in chunk)
):
intake.submit_records(chunk, priority="live")
@@ -160,7 +160,9 @@ class MatterDiscovery:
task_by_id: dict[str, MemoryTask] = {}
for chunk in chunks:
for record in chunk:
task = await asyncio.to_thread(processor.tasks.task_for_evidence, record.evidence_id)
task = await asyncio.to_thread(
processor.tasks.task_for_evidence, record.evidence_id
)
if task is not None and task.status is MemoryTaskStatus.COMPLETED:
task_by_id[task.task_id] = task
tasks = tuple(sorted(task_by_id.values(), key=lambda task: (task.created_at, task.task_id)))
@@ -270,7 +270,7 @@ async def reconcile_current(
(
r
for r in records
if r.written_by == "person"
if r.role == "user"
and r.created_at
and (r.source_id, r.conversation_id) in related
),
@@ -51,14 +51,19 @@ def validate_claim(claim: Claim, records: Mapping[str, EvidenceRecord | SampleRe
text = record.text if isinstance(record, SampleRecord) else record.verbatim
if not citation.quote.strip() or citation.quote not in text:
raise ValueError("citation quote is not in the source")
authors.add(str(record.written_by))
author = {
"user": "person",
"assistant": "agent",
"tool": "tool",
}.get(record.role, "unknown")
authors.add(author)
created = record.created_at
citations.append(
citation.model_copy(
update={
"revision": record.revision,
"source_id": record.source_id,
"written_by": str(record.written_by),
"written_by": author,
"created_at": created.isoformat() if isinstance(created, datetime) else created,
}
)
@@ -1,36 +1,25 @@
"""Read-only, exact-revision support checks inside the caller's SQLite snapshot.
"""Memory evidence freshness checks are retired.
No proof is legacy/unknown, not stale. One complete fresh proof suffices; all
complete proofs stale means false. Malformed records raise. An exceeded budget
means the memory is too large to verify, not that it is stale: it reads as
current, the same as having no proof, and is reported once in a while as a
content-free diagnostic. One such memory used to fail every COUNT over the
library, so the homepage returned 503 for as long as it existed (D76).
Set ONE_MEMORY_FRESHNESS_BUDGET_RAISES=1 to restore the old raising behavior.
The caller owns BEGIN/deadline/progress-handler and error-to-unavailable mapping.
Read paths used to re-validate ``memory_task_outputs.support_refs`` against
live ``evidence`` rows via ``is_memory_evidence_current``. That design hid
memories when evidence changed, but at high cost (library 503s, per-row SQL,
segment-key breakage). Call sites still import this module; registration is
now a no-op that always admits the row.
Proof rows may still be written by the ingest path; they are ignored on read.
See ``新鲜度_todo.md`` for the remaining teardown steps.
"""
import json
import logging
import os
from __future__ import annotations
import sqlite3
import threading
import time
from collections.abc import Iterator, Sequence
from contextlib import contextmanager
from one_memory.database_access import register_function
from one_memory.diagnostic_log import diagnostic_event
from one_memory.imports.models import EvidenceRecord
from one_memory.memory_tasks import MemoryTaskOutput, matches_evidence_ref
_PROOFS = 64
_REFERENCES = 128
_BODY_BYTES = 65_536
_EVIDENCE_BYTES = 8 * 1024 * 1024
_SCOPE_STEPS = 64
#: Budget refusals: the check could not finish, the data is not malformed.
#: Historical budget error codes. Retained so diagnostics/env docs do not break
#: mid-teardown; nothing raises them on the read path anymore.
BUDGET_ERRORS = frozenset(
{
"memory_freshness_proof_budget",
@@ -39,80 +28,23 @@ BUDGET_ERRORS = frozenset(
"memory_freshness_scope_budget",
}
)
_REPORT_INTERVAL_SECONDS = 60.0
_reported: dict[str, tuple[float, int]] = {}
_reported_lock = threading.Lock()
def budget_errors_raise() -> bool:
"""Opt back into the pre-D76 behavior, where one large memory fails the query."""
return os.environ.get("ONE_MEMORY_FRESHNESS_BUDGET_RAISES", "").strip() == "1"
"""Former opt-in to raise on budget overrun. Always false after retirement."""
return False
def _report_budget(code: str) -> None:
"""Count every budget skip; write at most one line per reason per interval."""
now = time.monotonic()
with _reported_lock:
last, pending = _reported.get(code, (float("-inf"), 0))
pending += 1
if now - last < _REPORT_INTERVAL_SECONDS:
_reported[code] = (last, pending)
return
_reported[code] = (now, 0)
diagnostic_event(
level="warn",
component="memory.freshness",
event="memory_freshness.budget_exceeded",
outcome="skipped",
details={"reason": code, "count": pending},
def has_proof_table(db: sqlite3.Connection) -> bool:
"""Whether ``memory_task_outputs`` exists. Unused by the retired check."""
return (
db.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_task_outputs'"
).fetchone()
is not None
)
def _object(pairs: list[tuple[str, object]]) -> dict[str, object]:
result: dict[str, object] = {}
for key, value in pairs:
if key in result:
raise ValueError("memory_freshness_invalid_proof")
result[key] = value
return result
class _Prefetched:
"""Prefetched rows for the current batch, private to thread and connection.
Scoped, never global: a cache that outlived the transaction would answer
from rows a later write had already replaced.
"""
def __init__(self) -> None:
self._local = threading.local()
def get(self) -> dict[str, list[sqlite3.Row]] | None:
return getattr(self._local, "rows", None)
def evidence(self) -> dict[str, sqlite3.Row] | None:
return getattr(self._local, "evidence", None)
@contextmanager
def using(
self,
rows: dict[str, list[sqlite3.Row]],
evidence: dict[str, sqlite3.Row] | None = None,
) -> Iterator[None]:
previous = getattr(self._local, "rows", None)
previous_evidence = getattr(self._local, "evidence", None)
self._local.rows = rows
self._local.evidence = evidence
try:
yield
finally:
self._local.rows = previous
self._local.evidence = previous_evidence
_prefetched_proofs = _Prefetched()
@contextmanager
def freshness_proofs_prefetched(
db: sqlite3.Connection,
@@ -120,328 +52,16 @@ def freshness_proofs_prefetched(
*,
evidence_ids: Sequence[str] = (),
) -> Iterator[None]:
"""Serve the freshness check's proof reads from one query.
The decisions are unchanged — the rows come from the same transaction —
but the per-memory reads collapse into the one in `proof_rows_for`.
"""
with _prefetched_proofs.using(
proof_rows_for(db, memory_ids),
evidence_rows_for(db, evidence_ids) if evidence_ids else None,
):
yield
def has_proof_table(db: sqlite3.Connection) -> bool:
"""Whether this database keeps proofs at all.
Without the table the registered UDF answers 1 unconditionally, so there
is nothing to prefetch and nothing to read.
"""
return (
db.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_task_outputs'"
).fetchone()
is not None
)
_EVIDENCE_COLUMNS = (
"evidence_id,user_id,source_id,workspace,project,conversation_id,"
"message_id,correlation_id,source_ordinal,content_kind,role,written_by,"
"created_at,verbatim,content_hash,revision,imported_at,updated_at"
)
#: The budget measure the single read applies, kept as one string so the
#: batched read cannot compute it differently.
_EVIDENCE_SIZE_SQL = (
"length(CAST(verbatim AS BLOB)) + "
"length(CAST(json_array(evidence_id,user_id,source_id,workspace,project,"
"conversation_id,message_id,correlation_id,source_ordinal,content_kind,"
"role,written_by,created_at,content_hash,revision,imported_at,updated_at) "
"AS BLOB))"
)
def evidence_rows_for(
db: sqlite3.Connection,
evidence_ids: Sequence[str],
) -> dict[str, sqlite3.Row]:
"""Read the cited evidence rows for a page in one query.
The check reads every citation twice — once for its size, against the
evidence budget, and once for the row it validates against the proof's
reference. Both are per citation, so both grow with the page.
The size comes from the same expression the single read uses, so the
budget arithmetic cannot drift between the two paths.
"""
if not evidence_ids:
return {}
cursor = db.cursor()
cursor.row_factory = sqlite3.Row
by_id: dict[str, sqlite3.Row] = {}
for start in range(0, len(evidence_ids), 900):
batch = list(dict.fromkeys(evidence_ids[start : start + 900]))
placeholders = ",".join("?" for _ in batch)
for row in cursor.execute(
f"SELECT {_EVIDENCE_COLUMNS},{_EVIDENCE_SIZE_SQL} AS measure "
f"FROM evidence WHERE evidence_id IN ({placeholders})",
batch,
):
by_id[str(row["evidence_id"])] = row
return by_id
def proof_rows_for(
db: sqlite3.Connection,
memory_ids: Sequence[str],
) -> dict[str, list[sqlite3.Row]]:
"""Read the proof rows for a whole page in one query.
The freshness check reads these per memory, and that is the read path's
whole cost: measured on the old-user library, 160 memories cost 0.69 s in
the UDF against 0.08 s for the citation subquery, and grew further while
the library was also being written to — enough to cross the five second
interrupt on `/api/v1/memories` and on the map.
Returns only ids that have rows; a caller falling back to a single read
for anything absent keeps the same answer.
"""
if not memory_ids:
return {}
if not has_proof_table(db):
return {}
cursor = db.cursor()
cursor.row_factory = sqlite3.Row
by_id: dict[str, list[sqlite3.Row]] = {}
for start in range(0, len(memory_ids), 400):
batch = list(memory_ids[start : start + 400])
placeholders = ",".join("?" for _ in batch)
for row in cursor.execute(
"SELECT task_id,memory_id,length(CAST(body AS BLOB)) AS size,"
"substr(CAST(body AS BLOB),1,?) AS body FROM memory_task_outputs "
f"WHERE memory_id IN ({placeholders})",
(_BODY_BYTES + 1, *batch),
):
by_id.setdefault(str(row["memory_id"]), []).append(row)
return by_id
def _proof_revision(
cursor: sqlite3.Cursor,
memory_id: str,
revision: int,
user_id: str,
available: set[int],
) -> int:
if revision in available or not available:
return revision
if (
cursor.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='governed_memory_versions'"
).fetchone()
is None
):
return revision
original = revision
for step in range(_SCOPE_STEPS + 1):
if revision in available:
return revision
row = cursor.execute(
"SELECT v.action, typeof(v.content)='text' AND typeof(p.content)='text' "
"AND CAST(v.content AS BLOB)=CAST(p.content AS BLOB) AS same_content, "
"v.status=p.status AS same_status, "
"CAST(v.content AS BLOB)=CAST(m.content AS BLOB) AS current_content "
"FROM governed_memory_versions v "
"JOIN governed_memory_versions p ON p.memory_id=v.memory_id "
"AND p.revision=v.revision-1 "
"JOIN governed_memories m ON m.memory_id=v.memory_id "
"WHERE v.memory_id=? AND v.revision=? AND m.user_id=? AND m.revision=?",
(memory_id, revision, user_id, original),
).fetchone()
if row is None or row[0] != "set_scopes" or not all(row[index] == 1 for index in (1, 2, 3)):
return original
if step == _SCOPE_STEPS:
raise ValueError("memory_freshness_scope_budget")
revision -= 1
raise ValueError("memory_freshness_scope_budget")
"""No-op. Kept so batched read callers need not change in this PR."""
del db, memory_ids, evidence_ids
yield
def register_memory_freshness(db: sqlite3.Connection) -> str:
"""Register UDF; return a predicate for alias m. Does not change row_factory.
"""Register a always-true UDF; return SQL predicate ``1``.
Production initialization supplies memory_task_outputs(memory_id) index.
Old no-index databases depend on the caller's SQL deadline. Per evaluation:
65 rows / 64KiB each, 129 active IDs, 8MiB total Evidence material. No library
bodies are loaded; Evidence objects live only for this evaluation. No cache
persists across transactions (safe when a caller reuses its connection).
Callers that embed the return value in ``WHERE`` keep working. Callers that
invoke ``is_memory_evidence_current`` by name also get current.
"""
if not has_proof_table(db):
register_function(db, "is_memory_evidence_current", 3, lambda *_: 1)
return "1"
raising = budget_errors_raise()
def current(memory_id: str, revision: int, user_id: str) -> int:
try:
return checked(memory_id, revision, user_id)
except ValueError as error:
code = str(error)
if raising or code not in BUDGET_ERRORS:
raise
_report_budget(code)
return 1
except sqlite3.Error as error:
# A freshness proof is an enhancement, never the reason a person
# cannot see their library. Under a concurrent history import the
# 5s read deadline interrupts this UDF mid-statement; the old
# behaviour let that escape, and the caller turned it into
# `memory_library_unavailable`, so the memory list stayed empty for
# as long as the import ran. Answer "current" instead and record
# why, which is the same trade the D76 budget skip already makes.
_report_budget("memory_freshness_sql_error")
logging.getLogger(__name__).warning(
"memory_freshness_udf_failed %s: %s", type(error).__name__, error
)
return 1
def checked(memory_id: str, revision: int, user_id: str) -> int:
if not isinstance(memory_id, str) or type(revision) is not int or revision < 1:
raise ValueError("memory_freshness_invalid_identity")
if not isinstance(user_id, str):
raise ValueError("memory_freshness_invalid_identity")
cursor = db.cursor()
cursor.row_factory = sqlite3.Row
prefetched = _prefetched_proofs.get()
if prefetched is not None:
rows = prefetched.get(memory_id, ())
else:
rows = cursor.execute(
"SELECT task_id,memory_id,length(CAST(body AS BLOB)) AS size,"
"substr(CAST(body AS BLOB),1,?) AS body FROM memory_task_outputs "
"WHERE memory_id=? LIMIT ?",
(_BODY_BYTES + 1, memory_id, _PROOFS + 1),
).fetchall()
rows = list(rows)
if len(rows) > _PROOFS:
raise ValueError("memory_freshness_proof_budget")
decoded = []
for row in rows:
if type(row["size"]) is not int or row["size"] > _BODY_BYTES:
raise ValueError("memory_freshness_proof_budget")
body = json.loads(row["body"], object_pairs_hook=_object)
if not isinstance(body, dict) or type(body.get("memory_revision")) is not int:
raise ValueError("memory_freshness_invalid_proof")
decoded.append((row, body))
revision = _proof_revision(
cursor,
memory_id,
revision,
user_id,
{body["memory_revision"] for _, body in decoded},
)
applicable = []
for row, body in decoded:
if body["memory_revision"] != revision:
continue
raw_refs = body.get("support_refs", [])
if not isinstance(raw_refs, list) or any(
not isinstance(ref, dict)
or set(ref) - {"evidence_id", "revision", "content_hash", "identity_hash"}
for ref in raw_refs
):
raise ValueError("memory_freshness_invalid_reference")
output = MemoryTaskOutput.model_validate_json(row["body"], strict=True)
if output.memory_id != memory_id or output.task_id != row["task_id"]:
raise ValueError("memory_freshness_proof_identity")
if len(output.support_refs) > _REFERENCES:
raise ValueError("memory_freshness_reference_budget")
for ref in output.support_refs:
if (
set(ref) - {"evidence_id", "revision", "content_hash", "identity_hash"}
or not ref["evidence_id"]
or ref["revision"] < 1
or any(
len(ref[key]) != 64
or any(char not in "0123456789abcdef" for char in ref[key])
for key in ("content_hash", "identity_hash")
if key in ref
)
):
raise ValueError("memory_freshness_invalid_reference")
proof_ids = [ref["evidence_id"] for ref in output.support_refs]
if len(proof_ids) != len(set(proof_ids)):
raise ValueError("memory_freshness_duplicate_reference")
applicable.append(output)
if not applicable:
return 1
active = cursor.execute(
"SELECT evidence_id FROM governed_memory_evidence "
"WHERE memory_id=? AND active=1 LIMIT ?",
(memory_id, _REFERENCES + 1),
).fetchall()
if len(active) > _REFERENCES:
raise ValueError("memory_freshness_reference_budget")
ids = {row[0] for row in active}
complete = [
output
for output in applicable
if output.support_refs and {ref["evidence_id"] for ref in output.support_refs} == ids
]
if not complete:
return 1
records: dict[str, EvidenceRecord | None] = {}
used = 0
prefetched_evidence = _prefetched_proofs.evidence()
for identifier in ids:
evidence_row = (
prefetched_evidence.get(identifier) if prefetched_evidence is not None else None
)
if evidence_row is not None:
size = evidence_row["measure"]
if type(size) is not int or size < 0:
raise ValueError("memory_freshness_invalid_evidence")
else:
measured = cursor.execute(
f"SELECT {_EVIDENCE_SIZE_SQL} FROM evidence WHERE evidence_id=?",
(identifier,),
).fetchone()
if measured is None:
records[identifier] = None
continue
if type(measured[0]) is not int or measured[0] < 0:
raise ValueError("memory_freshness_invalid_evidence")
size = measured[0]
used += size
if used > _EVIDENCE_BYTES:
raise ValueError("memory_freshness_evidence_budget")
if evidence_row is None:
evidence_row = cursor.execute(
f"SELECT {_EVIDENCE_COLUMNS} FROM evidence WHERE evidence_id=?",
(identifier,),
).fetchone()
if evidence_row is None:
records[identifier] = None
continue
# evidence_ref explicitly excludes origin_meta/imported_at/updated_at.
# `sqlite3.Row` iterates values, not column names, so pair the
# names with the values explicitly.
fields = {
name: value
for name, value in zip(evidence_row.keys(), tuple(evidence_row), strict=False)
if name != "measure"
}
records[identifier] = EvidenceRecord.model_validate({**fields, "origin_meta": {}})
for output in complete:
if all(
(record := records[ref["evidence_id"]]) is not None
and record.user_id == user_id
and matches_evidence_ref(record, ref)
for ref in output.support_refs
):
return 1
return 0
register_function(db, "is_memory_evidence_current", 3, current)
return "is_memory_evidence_current(m.memory_id,m.revision,m.user_id)"
register_function(db, "is_memory_evidence_current", 3, lambda *_: 1)
return "1"
@@ -290,7 +290,7 @@ def read_memory_library(
COALESCE(
MIN(
CASE WHEN e.role = 'user'
AND e.written_by IN ('person', 'user')
AND e.role = 'user'
THEN e.created_at END
),
MIN(e.created_at)
@@ -45,6 +45,7 @@ from one_memory.hy_memory_bridge import (
HyMemoryOutput,
HyMemoryTurn,
builtin_retrieval_snapshot,
coding_tool_result_max_bytes,
decide_profile_graph_width,
hy_memory_store_exists,
inspect_profile_graph_width,
@@ -52,7 +53,7 @@ from one_memory.hy_memory_bridge import (
from one_memory.hy_runtime_config import hy_memory_version
from one_memory.hy_source_context import HySourceContext, HySupportedMemory
from one_memory.imports.access import opaque_workspace_id
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord, WrittenBy
from one_memory.imports.models import EvidenceContentKind, EvidenceOrigin, EvidenceRecord
from one_memory.imports.runner import live_sync_arrival
from one_memory.imports.storage import EvidenceStore
from one_memory.input_sanitization import sanitize_model_input
@@ -162,7 +163,7 @@ def _report_stale_known_reasons(task_id: str, reasons: Mapping[str, int]) -> Non
"""Emit one schema-valid scalar diagnostic per stale-memory reason."""
for reason, count in sorted(reasons.items()):
diagnostic_event(
level="info",
level="debug",
component="memory.context",
event="known_memory_stale_omitted",
outcome="omitted",
@@ -173,6 +174,21 @@ def _report_stale_known_reasons(task_id: str, reasons: Mapping[str, int]) -> Non
MEMORY_INGEST_POLICY_VERSION = "hy-native-v23"
UNIFIED_HY_AGENT_ID = "one-memory"
#: Deterministic refusals raised before the payload reaches HY, so nothing was
#: sent even though the invocation marker is already recorded by then: the
#: write bindings are built inside `ingest` ahead of the call, and the input
#: window is checked while the request is frozen.
#:
#: They are terminal rather than unknown. `unknown` here means "HY may hold a
#: write we cannot confirm", and that is exactly what did not happen — parking
#: the task on it hid the reason and stopped the queue behind it (issue #44).
#: Kept in step with `durable_queue.PRE_SEND_REFUSAL_CODES`, which the durable
#: write path applies to the same codes.
PRE_SEND_REFUSAL_CODES = frozenset(
{
"hy_memory_input_exceeds_window",
}
)
_BACKFILL_BATCH_SIZE = 8
_BACKFILL_DRAIN_BUDGET_SECONDS = 330.0
_HISTORY_MERGE_TURNS = 8
@@ -280,6 +296,7 @@ class MemoryTaskRunner:
*,
timeout: float | None = None,
before_hy_invocation: Callable[[], Awaitable[None]] | None = None,
input_records: tuple[EvidenceRecord, ...] | None = None,
) -> MemoryTask:
started = monotonic()
diagnostic_event(
@@ -290,13 +307,28 @@ class MemoryTaskRunner:
task_id=task_id,
)
try:
body = (
self._run_without_content_permission(task_id)
if before_hy_invocation is None
else self._run_without_content_permission(
task_id, before_hy_invocation=before_hy_invocation
if input_records is None:
body = (
self._run_without_content_permission(task_id)
if before_hy_invocation is None
else self._run_without_content_permission(
task_id,
before_hy_invocation=before_hy_invocation,
)
)
else:
body = (
self._run_without_content_permission(
task_id,
input_records=input_records,
)
if before_hy_invocation is None
else self._run_without_content_permission(
task_id,
before_hy_invocation=before_hy_invocation,
input_records=input_records,
)
)
)
# The ceiling bounds how long this task holds its harness's slot:
# a harness is serial, so one hung task starves every turn behind
# it. It is insurance against a single bad call, not a way to run
@@ -367,7 +399,7 @@ class MemoryTaskRunner:
@staticmethod
def _log_step(task_id: str, step: int, action: str, started: float) -> None:
diagnostic_event(
level="info",
level="debug",
component="memory.task",
event="memory_task.step_completed",
outcome="succeeded",
@@ -381,6 +413,7 @@ class MemoryTaskRunner:
task_id: str,
*,
before_hy_invocation: Callable[[], Awaitable[None]] | None = None,
input_records: tuple[EvidenceRecord, ...] | None = None,
) -> MemoryTask:
# Every store call here is synchronous, and several are writes that
# wait on the write lock for up to five seconds. Made from the loop,
@@ -434,7 +467,9 @@ class MemoryTaskRunner:
step_started = monotonic()
frozen = await in_the_background(self.tasks.inputs, task.task_id)
chunk: tuple[EvidenceRecord, ...] | None
if frozen:
if input_records is not None:
chunk = input_records
elif frozen:
try:
chunk = await in_the_background(self._frozen_chunk, task)
except HyMemoryBridgeError as error:
@@ -467,8 +502,11 @@ class MemoryTaskRunner:
self._source_context, task, request.source_context
)
request = request.model_copy(update={"source_context": context})
uses_frozen_input = bool(frozen) or input_records is not None
grant_token = (
await in_the_background(self._grant_token, task, chunk[0]) if frozen else None
await in_the_background(self._grant_token, task, chunk[0])
if uses_frozen_input
else None
)
memory_contents = tuple(
sanitize_model_input(record.verbatim)
@@ -511,19 +549,24 @@ class MemoryTaskRunner:
agent_id=request.agent_id,
),
)
if frozen:
await in_the_background(self._frozen_chunk, task)
if uses_frozen_input:
if input_records is None:
await in_the_background(self._frozen_chunk, task)
if grant_token != await in_the_background(self._grant_token, task, chunk[0]):
raise HyMemoryBridgeError("hy_memory_source_changed")
if request.source_context is not None:
await in_the_background(self._check_source_context, request.source_context)
await in_the_background(
self._check_source_context,
request.source_context,
skip_current=input_records is not None,
)
# One thread hop for the whole batch rather than a dozen writes
# made from the loop.
step_started = monotonic()
await in_the_background(
self._record_outputs,
task=task,
chunk=self._original_records(chunk),
chunk=chunk,
outputs=outputs,
request=request,
derivation=derivation,
@@ -533,6 +576,13 @@ class MemoryTaskRunner:
self._log_step(task_id, 6, "governance", step_started)
except HyMemoryBridgeError as error:
code = str(error)
if code == "model_sedimentation_disabled" and not hy_invoked:
return await in_the_background(
self.tasks.release_unstarted,
task_id,
runner_token=runner_token,
error_code=code,
)
model_wait_errors = {
"hy_memory_unavailable",
"hy_memory_rate_limit",
@@ -547,6 +597,16 @@ class MemoryTaskRunner:
"hy_memory_model_timeout",
}
safe_terminal_after_invocation = {"hy_memory_import_no_readable_output"}
if code in PRE_SEND_REFUSAL_CODES:
# Raised before the call, so the marker only says "about to":
# fail with the reason rather than parking on a write HY never
# saw. See the constant and issue #44.
return await in_the_background(
self.tasks.fail,
task_id,
runner_token=runner_token,
error_code=code,
)
if hy_invoked and (
code == "hy_memory_evidence_changed"
or (
@@ -744,8 +804,18 @@ class MemoryTaskRunner:
raise HyMemoryBridgeError("hy_memory_evidence_changed") from error
return context
def _check_source_context(self, context: HySourceContext) -> None:
for original in (*context.current, *context.history, *context.supporting):
def _check_source_context(
self,
context: HySourceContext,
*,
skip_current: bool = False,
) -> None:
checked = (
(*context.history, *context.supporting)
if skip_current
else (*context.current, *context.history, *context.supporting)
)
for original in checked:
record = self.evidence.get(original.evidence_id)
if record is None or not matches_evidence_ref(record, evidence_ref(original)):
raise HyMemoryBridgeError("hy_memory_evidence_changed")
@@ -846,7 +916,7 @@ class MemoryTaskRunner:
# HY's own attribution. An undecided memory counts as the person's:
# HY's prompt defaults that way.
from_agent = output.owner == "agent" or all(
record.written_by is WrittenBy.AGENT
record.role == "assistant"
and record.content_kind is EvidenceContentKind.RAW_CONVERSATION
for record in support
)
@@ -1016,15 +1086,20 @@ class MemoryTaskRunner:
task: MemoryTask,
chunk: tuple[EvidenceRecord, ...],
) -> HyMemoryIngestRequest:
if not chunk:
# A task with no inputs has nothing to name the request after and
# nothing to attribute an answer to. Left alone this reached
# `chunk[0]` below and came out as IndexError, which no caller can
# read: a reconcile pass catches domain errors per task and moves
# on, so one such task ended the whole pass and left every other
# unknown task unreconciled. On this machine 68 of 303 unknown
# tasks had no inputs, and the pass raised on the first of them.
raise HyMemoryBridgeError("hy_memory_task_without_inputs")
has_raw = any(
record.content_kind is EvidenceContentKind.RAW_CONVERSATION for record in chunk
)
source = next(
(
record
for record in chunk
if record.role == "user" and record.written_by is WrittenBy.PERSON
),
(record for record in chunk if record.role == "user"),
chunk[0],
)
request_turns = tuple(
@@ -1033,12 +1108,9 @@ class MemoryTaskRunner:
if (
record.content_kind is EvidenceContentKind.RAW_CONVERSATION
and (
# D98: any `user` message goes, whoever the parser says
# wrote it. A sub agent's question is ordinary
# user-assistant data; the author field cannot tell who
# wrote a message, and filtering on it dropped the sub
# agent's turns. `written_by` still labels the display and
# decides owner — it no longer decides what is sent.
# Protocol role is authoritative. A sub-agent question is
# an ordinary user message regardless of a host parser's
# former author guess.
record.role == "user"
# The assistant's side goes too. Without it 「就按你说的那个来」
# arrives with nothing to resolve it against, and HY's own
@@ -1049,7 +1121,7 @@ class MemoryTaskRunner:
or record.role == "assistant"
# Tool calls go as they happened; whether they make the
# turn a coding turn is HY's call, not ours.
or (record.role == "tool" and record.written_by is WrittenBy.TOOL)
or record.role == "tool"
)
)
or (not has_raw and record.content_kind is EvidenceContentKind.MEMORY_CONTENT)
@@ -1218,6 +1290,7 @@ class MemoryProcessingSupervisor:
max_concurrent_tasks: int | None = None,
source_writes_permitted: Callable[[str], bool] | None = None,
service_levels: ServiceLevelScheduler | None = None,
sedimentation_enabled: Callable[[], bool] | None = None,
) -> None:
self.tasks = tasks
self.profiles = profiles
@@ -1230,6 +1303,12 @@ class MemoryProcessingSupervisor:
#: absent every source is treated as writable, which is what the
#: product did before the switch existed.
self.source_writes_permitted = source_writes_permitted
#: User-level "stop turning conversations into memories" switch. Optional
#: so tests and a Core without a config file keep their current
#: behaviour; when absent sedimentation is on, which is what the product
#: did before the switch existed. Read per decision rather than cached,
#: so stopping takes effect on the next dispatch without a restart.
self.sedimentation_enabled = sedimentation_enabled
self.service_levels = service_levels or ServiceLevelScheduler()
self.extraction_settings = extraction_settings or ExtractionSettingsStore(
data_dir / "one-memory.sqlite3"
@@ -1283,12 +1362,26 @@ class MemoryProcessingSupervisor:
self._orphan_reclaimer: asyncio.Task[None] | None = None
self._diagnostic_idle_reason: str | None = None
def _sedimentation_stopped(self) -> bool:
"""Whether the user stopped model-driven sedimentation.
Stopping must not look like a broken pipeline: nothing is sent to the
model, so the pending count stays as it is and the scheduler reports
why it is idle. Starting again picks the same work up.
"""
if self.sedimentation_enabled is None:
return False
if self.sedimentation_enabled():
return False
self._record_idle("sedimentation_stopped")
return True
def _record_idle(self, reason: str) -> None:
if self._diagnostic_idle_reason == reason:
return
self._diagnostic_idle_reason = reason
diagnostic_event(
level="info",
level="debug",
component="memory.scheduler",
event="memory_scheduler.idle",
outcome="idle",
@@ -1310,7 +1403,8 @@ class MemoryProcessingSupervisor:
) -> tuple[MemoryTask, ...]:
"""Process a bounded, authorized snapshot through the existing HY runner."""
async with self._explicit_processing_lock(records):
records = await in_the_background(self._original_records, records)
if self._sedimentation_stopped():
raise HyMemoryBridgeError("model_sedimentation_disabled")
active = await in_the_background(self.profiles.active, self.secrets)
if active is None or (profile_id is not None and profile_id != active.profile_id):
raise HyMemoryBridgeError("model_profile_not_active")
@@ -1325,13 +1419,6 @@ class MemoryProcessingSupervisor:
raise HyMemoryBridgeError("hy_memory_source_disabled")
conversations: dict[tuple[str, str, str | None], list[EvidenceRecord]] = {}
for record in records:
stored = await in_the_background(self.evidence.get, record.evidence_id)
if (
stored is None
or stored.revision != record.revision
or stored.content_hash != record.content_hash
):
raise ValueError("hy_memory_evidence_changed")
conversations.setdefault(
(record.source_id, record.conversation_id, record.workspace), []
).append(record)
@@ -1348,7 +1435,7 @@ class MemoryProcessingSupervisor:
for conversation in conversations.values()
for chunk in _conversation_chunks(conversation, max_turns=12, completed_only=True)
]
prepared = []
prepared: list[tuple[MemoryTask, tuple[EvidenceRecord, ...]]] = []
for chunk in chunks:
task = await in_the_background(
self._enqueue_chunk,
@@ -1356,12 +1443,12 @@ class MemoryProcessingSupervisor:
active,
freeze_inputs=True,
)
prepared.append(task)
prepared.append((task, chunk))
runner = self.runner_factory()
completed = []
if on_progress:
on_progress(tuple(prepared), 0)
for task in prepared:
on_progress(tuple(task for task, _chunk in prepared), 0)
for task, chunk in prepared:
if task.status is not MemoryTaskStatus.COMPLETED:
# Another importer may already own this task. Never race its claim.
task = await in_the_background(self.tasks.require, task.task_id)
@@ -1390,23 +1477,41 @@ class MemoryProcessingSupervisor:
work_id=task.task_id,
scope_key=chunk[0].source_id,
):
task = (
await runner.run(task.task_id, timeout=timeout)
if before_hy_invocation is None
else await runner.run(
task.task_id,
timeout=timeout,
before_hy_invocation=before_hy_invocation,
if isinstance(runner, MemoryTaskRunner):
task = (
await runner.run(
task.task_id,
timeout=timeout,
input_records=chunk,
)
if before_hy_invocation is None
else await runner.run(
task.task_id,
timeout=timeout,
before_hy_invocation=before_hy_invocation,
input_records=chunk,
)
)
else:
task = (
await runner.run(task.task_id, timeout=timeout)
if before_hy_invocation is None
else await runner.run(
task.task_id,
timeout=timeout,
before_hy_invocation=before_hy_invocation,
)
)
)
finally:
_unmark_direct_task(self.tasks.database_path, task.task_id)
if task.status is not MemoryTaskStatus.COMPLETED:
raise HyMemoryBridgeError(task.last_error_code or "hy_memory_incomplete")
await in_the_background(runner._frozen_chunk, task)
completed.append(task)
if on_progress:
on_progress(tuple(prepared), len(completed))
on_progress(
tuple(prepared_task for prepared_task, _chunk in prepared),
len(completed),
)
expected = {record.evidence_id for record in records if _is_hy_turn_record(record)}
actual = {
row["evidence_id"] for task in completed for row in self.tasks.inputs(task.task_id)
@@ -1417,14 +1522,24 @@ class MemoryProcessingSupervisor:
await self._refresh_read_bridge(completed[-1].profile_snapshot)
return tuple(completed)
async def reconcile_unknown_outputs(self, *, limit: int = 128) -> int:
async def reconcile_unknown_outputs(
self,
*,
limit: int = 128,
exclude_error_codes: tuple[str, ...] = (),
) -> int:
"""Publish HY outputs that can be proven by an unknown task's request id.
This inquiry is read-only on HY and never invokes the model. Rows with
no exact match remain unknown for the person's explicit retry choice.
"""
reconciled = 0
for task in await in_the_background(self.tasks.unknown_tasks, limit=limit):
tasks = await in_the_background(
self.tasks.unknown_tasks,
limit=limit,
exclude_error_codes=exclude_error_codes,
)
for task in tasks:
runner = self.runner_factory()
bridge: MemoryBridgeProtocol | None = None
try:
@@ -1667,6 +1782,8 @@ class MemoryProcessingSupervisor:
def enqueue_evidence(self, record: EvidenceRecord) -> MemoryTask | None:
if self._shutting_down:
return None
if self._sedimentation_stopped():
return None
if (
record.role not in {"user", "assistant"}
and record.content_kind is not EvidenceContentKind.MEMORY_CONTENT
@@ -1699,6 +1816,8 @@ class MemoryProcessingSupervisor:
should_stop: Callable[[], bool] = lambda: False,
) -> list[QueuedWork]:
"""Synchronous compatibility entry; preparation never starts a model."""
if self._sedimentation_stopped():
return []
work = [item for batch in self._prepare_backfill_batches(should_stop) for item in batch]
return [] if should_stop() or self._shutting_down else work
@@ -1707,7 +1826,7 @@ class MemoryProcessingSupervisor:
should_stop: Callable[[], bool] = lambda: False,
) -> Generator[list[QueuedWork], None, None]:
"""Read bodies as needed; yield the first task before the next conversation."""
if should_stop() or self._shutting_down:
if should_stop() or self._shutting_down or self._sedimentation_stopped():
return
active = self.profiles.active(self.secrets)
if active is None:
@@ -1718,7 +1837,7 @@ class MemoryProcessingSupervisor:
visited = 0
seed_ids = self.evidence.list_conversation_seed_ids()
diagnostic_event(
level="info",
level="debug",
component="memory.scheduler",
event="memory_scheduler.scan_inventory",
outcome="observed",
@@ -1788,7 +1907,7 @@ class MemoryProcessingSupervisor:
started = monotonic()
prepared_count = 0
diagnostic_event(
level="info",
level="debug",
component="memory.scheduler",
event="memory_scheduler.scan_started",
outcome="started",
@@ -1800,6 +1919,7 @@ class MemoryProcessingSupervisor:
stop = threading.Event()
batches = self._prepare_backfill_batches(stop.is_set)
exhausted = False
stopped_saturated = False
rounds = 0
scan_error: BaseException | None = None
try:
@@ -1815,20 +1935,21 @@ class MemoryProcessingSupervisor:
>= self._backfill_round_budget_seconds
):
break
while (
if (
cohort_size is not None
and len(self._queue) + len(self._in_flight) >= cohort_size
):
if self._shutting_down:
return
if (
handled_batch
and self._backfill_clock() - round_started
>= self._backfill_round_budget_seconds
):
break
self._record_idle("queue_saturated")
await asyncio.sleep(0.1)
# A bounded scan stops at its ceiling instead of
# waiting for the queue to drain. Waiting here slept
# inside the scan lock, so whoever asked for the
# scan — a switch toggle, over HTTP — waited with
# it, and on a saturated backlog that is not a
# moment but the whole backlog. The caller's
# timeout then reported a failure for work that had
# already been accepted.
stopped_saturated = True
exhausted = True
break
if (
handled_batch
and self._backfill_clock() - round_started
@@ -1847,7 +1968,7 @@ class MemoryProcessingSupervisor:
on_first_scheduled()
if not exhausted and not self._shutting_down:
diagnostic_event(
level="info",
level="debug",
component="memory.scheduler",
event="memory_scheduler.scan_yielded",
outcome="yielded",
@@ -1877,14 +1998,18 @@ class MemoryProcessingSupervisor:
stop.set()
batches.close()
if prepared_count == 0 and not self._shutting_down:
if scan_error is None:
if stopped_saturated:
# Parked behind a full queue, not out of work: the status
# must not read "no_tasks" while hundreds wait.
self._record_idle("queue_saturated")
elif scan_error is None:
self._record_idle("no_tasks")
else:
self._record_idle("scan_failed")
elif prepared_count:
self._diagnostic_idle_reason = None
diagnostic_event(
level="info",
level="debug",
component="memory.scheduler",
event="memory_scheduler.scan_completed",
outcome=(
@@ -2092,6 +2217,8 @@ class MemoryProcessingSupervisor:
Work that falls outside the extraction window is marked deferred so
the pending count stops pretending it is about to run.
"""
if self._sedimentation_stopped():
return []
now = datetime.now(UTC)
window = self._extraction_window() if within_window else None
exempt_all_apps_from_window = True
@@ -2174,7 +2301,7 @@ class MemoryProcessingSupervisor:
if work:
self._diagnostic_idle_reason = None
diagnostic_event(
level="info",
level="debug",
component="memory.scheduler",
event="memory_scheduler.tasks_queued",
outcome="succeeded",
@@ -2239,7 +2366,7 @@ class MemoryProcessingSupervisor:
return self.HARNESS_SERIAL
def _ensure_workers(self) -> None:
if self._shutting_down:
if self._shutting_down or self._sedimentation_stopped():
return
load = self._harness_load()
background_capacity = (
@@ -2317,6 +2444,8 @@ class MemoryProcessingSupervisor:
"""
looked_for_older = False
while not self._shutting_down:
if self._sedimentation_stopped():
return
# The harness cap is recomputed from what is running, so a worker
# that just finished frees its harness's slot immediately.
work = self._queue.take(
@@ -2370,7 +2499,16 @@ class MemoryProcessingSupervisor:
scope_key=scope_key,
),
):
finished = await runner.run(work.task_id, timeout=self._profile_timeout)
async def require_sedimentation() -> None:
if self._sedimentation_stopped():
raise HyMemoryBridgeError("model_sedimentation_disabled")
finished = await runner.run(
work.task_id,
timeout=self._profile_timeout,
before_hy_invocation=require_sedimentation,
)
if finished.status is MemoryTaskStatus.COMPLETED:
self._request_read_bridge_refresh(finished.profile_snapshot)
elif (
@@ -3049,7 +3187,7 @@ class MemoryProcessingSupervisor:
self._read_bridges_ready.add(key)
self._read_bridge_generations[key] = generation
diagnostic_event(
level="info",
level="debug",
component="memory.semantic",
event="semantic_read_bridge.refreshed",
outcome="succeeded",
@@ -3172,9 +3310,7 @@ class MemoryProcessingSupervisor:
if not tail:
return False
last = tail[-1]
return bool(
last.role == "user" and last.written_by is WrittenBy.PERSON and _opens_turn(last)
)
return bool(last.role == "user" and _opens_turn(last))
def _admitted_chunks(
self,
@@ -3519,7 +3655,6 @@ def _is_memory_note(record: EvidenceRecord) -> bool:
meta = record.origin_meta.get(EvidenceOrigin.MANUAL_IMPORT)
return (
record.source_id == "one-memory-manual"
and record.written_by is WrittenBy.PERSON
and meta is not None
and meta.raw_meta.get("input_mode") == "direct_memory_note"
)
@@ -3595,7 +3730,7 @@ _HOST_ENVELOPE = re.compile(
#: before HY chose a path cut long answers mid-way and sent tool-heavy turns as
#: dozens of calls. ONE_MEMORY_SPLIT_TURNS=1 restores the 12,000 / 24 split.
_WHOLE_TURN_CHARACTERS = 60_000
_HY_TOOL_CHARACTERS = 2_048
_HY_TOOL_CHARACTERS = coding_tool_result_max_bytes()
#: Hard ceiling on the UTF-8 bytes one window may hold, whatever HY counts.
#: HY reads the payload in-process, so a turn made of enormous raw tool
#: records is held in memory even though HY will cut each one down. Cut by
@@ -3814,14 +3949,12 @@ def _merge_historical_chunks(
def _chunk_input_ref(chunk: tuple[EvidenceRecord, ...]) -> str:
person_records = tuple(
item for item in chunk if item.role == "user" and item.written_by is WrittenBy.PERSON
)
return person_records[-1].evidence_id if person_records else chunk[-1].evidence_id
user_records = tuple(item for item in chunk if item.role == "user")
return user_records[-1].evidence_id if user_records else chunk[-1].evidence_id
def _opens_turn(record: EvidenceRecord) -> bool:
return record.role == "user" and record.written_by is WrittenBy.PERSON
return record.role == "user"
def _trailing_user_turn_is_open(turn: list[EvidenceRecord]) -> bool:
@@ -3838,10 +3971,8 @@ def _trailing_user_turn_is_open(turn: list[EvidenceRecord]) -> bool:
def _is_hy_turn_record(record: EvidenceRecord) -> bool:
"""What HY receives of a turn: the input turns and the tool calls made in it."""
if record.role == "tool":
return (
record.written_by is WrittenBy.TOOL
and record.content_kind is EvidenceContentKind.RAW_CONVERSATION
and bool(record.verbatim.strip())
return record.content_kind is EvidenceContentKind.RAW_CONVERSATION and bool(
record.verbatim.strip()
)
return _is_hy_input_turn(record)
@@ -3858,11 +3989,8 @@ def _is_hy_input_turn(record: EvidenceRecord) -> bool:
# from a model.
if record.role == "assistant":
return True
# D98: the author no longer decides. A sub agent's question arrives as a
# `user` message marked written by the agent — it is ordinary
# user-assistant data, and dropping it sent Claude Code transcripts to HY
# with every question but the first missing. `written_by` labels the
# display and decides owner; it cannot tell who actually wrote a message.
# A sub-agent question is still a protocol `user` message. Host-specific
# author guesses do not decide whether it reaches HY.
if record.role != "user":
return False
if source.casefold().startswith(_HARNESS_PREFIXES):
@@ -30,6 +30,12 @@ class ConfigDocument(BaseModel):
schema_version: int = Field(default=1, ge=1, le=1)
sources: dict[str, SourceConfig] = Field(default_factory=dict)
#: Whether the configured model may turn imported conversations into
#: memories. Defaulting to on keeps every existing install doing what it
#: did before the switch existed; stopping it is the user's cost control
#: and leaves the model configuration itself untouched, so starting again
#: needs no re-entry of a key.
sedimentation_enabled: StrictBool = True
@field_validator("sources")
@classmethod
@@ -63,6 +69,19 @@ class ConfigStore:
source = self._document.sources.get(source_id)
return bool(source and source.sync_enabled)
def sedimentation_enabled(self) -> bool:
with self._lock:
return bool(self._document.sedimentation_enabled)
def set_sedimentation_enabled(self, enabled: bool) -> ConfigDocument:
if type(enabled) is not bool:
raise ValueError("invalid_sedimentation_config")
with self._lock:
updated = self._document.model_copy(update={"sedimentation_enabled": enabled})
self._write(updated)
self._document = updated
return updated
def set_sync(self, source_id: str, enabled: bool) -> ConfigDocument:
return self.set_sync_many({source_id: enabled})
@@ -23,10 +23,13 @@ import hashlib
import hmac
import os
import uuid
from datetime import datetime
from enum import StrEnum
from typing import Annotated, Final, Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator
from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator
from one_memory.imports.models import EvidenceRecord
#: Bumped when a derivation rule changes, so old and new ids never compare equal.
SCHEMA_VERSION: Final[str] = "ms1"
@@ -166,7 +169,6 @@ class CanonicalMessage(BaseModel):
message_id: str | None = Field(default=None, min_length=1, max_length=512)
ordinal: int = Field(ge=0)
role: str = Field(min_length=1, max_length=128)
written_by: str = Field(min_length=1, max_length=64)
content: str = Field(min_length=1)
workspace: str | None = Field(default=None, max_length=1024)
@@ -213,8 +215,13 @@ class WriteRequest(BaseModel):
source_id: str | None = Field(default=None, min_length=1, max_length=128)
batch_identity: BatchIdentity
evidence_ids: list[str] = Field(min_length=1)
# New jobs freeze the complete input in their own payload. The default is
# solely for reading pre-upgrade rows whose payload only carried ids.
input_records: tuple[EvidenceRecord, ...] = ()
idempotency_key: str = Field(min_length=1, max_length=256)
priority: Literal["live", "normal"] = "normal"
event_at: datetime | None = None
time_quality: Literal["exact", "anchored", "session", "approximate"] = "approximate"
@field_validator("evidence_ids")
@classmethod
@@ -227,6 +234,16 @@ class WriteRequest(BaseModel):
raise ValueError("evidence_ids must be unique")
return value
@field_validator("input_records")
@classmethod
def _input_records_match_ids(
cls, value: tuple[EvidenceRecord, ...], info: ValidationInfo
) -> tuple[EvidenceRecord, ...]:
evidence_ids = info.data.get("evidence_ids")
if value and evidence_ids != [record.evidence_id for record in value]:
raise ValueError("input_records must match evidence_ids in order")
return value
@property
def request_key(self) -> str:
"""Stable dedup key. Same batch + same intent must not enqueue twice."""
@@ -30,7 +30,7 @@ from uuid import uuid4
from one_memory.background_work import in_the_background
from one_memory.database_access import main_database
from one_memory.diagnostic_log import diagnostic_event
from one_memory.imports.models import EvidenceRecord
from one_memory.imports.models import EvidenceOrigin, EvidenceRecord
from one_memory.imports.storage import EvidenceStore
from one_memory.memory_server.contracts import WriteRequest
from one_memory.memory_server.data_safety import MIN_FREE_BYTES # noqa: F401 - re-exported
@@ -38,10 +38,11 @@ from one_memory.memory_server.migrations import memory_write_jobs_sqlite_schema
from one_memory.memory_server.queue import build_reclaim_sqlite_statements
from one_memory.memory_server.registry import WriteRecord
from one_memory.memory_tasks import (
POLICY_STOPPED_CODES as _TASK_POLICY_STOPPED_CODES,
NON_RETRYABLE_POST_WRITE_CODES,
MemoryTaskStatus,
)
from one_memory.memory_tasks import (
MemoryTaskStatus,
POLICY_STOPPED_CODES as _TASK_POLICY_STOPPED_CODES,
)
from one_memory.service_levels import ServiceLevel, service_level_context
@@ -249,6 +250,7 @@ MODEL_NOT_READY_CODES = frozenset(
#: nothing was sent and retrying the same intent changes nothing — a terminal
#: `failed`, not `unknown` (which would wrongly imply a possibly-applied
#: write) and not a retry (which would fail identically).
#:
PRE_SEND_REFUSAL_CODES = frozenset(
{
"hy_memory_input_exceeds_window",
@@ -282,11 +284,6 @@ _VALID_JOB_STATES = frozenset(
#: A row in one of these states can never be moved back to a live one.
_TERMINAL_WRITE_STATES = frozenset({COMPLETED, FAILED, DEAD, CANCELLED})
#: A job in one of these states can never run again, so its payload is dead
#: weight. `unknown` is deliberately absent: it is non-terminal and still needs
#: its intent for reconciliation. `running` is obviously still live.
TERMINAL_PAYLOAD_STATES = (COMPLETED, FAILED, DEAD, CANCELLED)
#: Columns a current schema declares but an older `memory_write_jobs` may lack.
#: `CREATE TABLE IF NOT EXISTS` is a no-op for an existing table, so without
#: this an upgraded install would keep an older queue that every claim/lease
@@ -304,9 +301,12 @@ LEGACY_MEMORY_WRITE_JOB_COLUMNS = (
("lease_expires_at", "TEXT"),
("hy_invoked", "INTEGER NOT NULL DEFAULT 0"),
("last_error", "TEXT"),
("event_at", "TEXT"),
("time_quality", "TEXT NOT NULL DEFAULT 'approximate'"),
)
_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)
EVENT_TIME_WINDOW = timedelta(minutes=10)
@dataclass(frozen=True)
@@ -393,6 +393,42 @@ def _matches_payload_except_priority(stored: str, payload: str) -> bool:
return old == new
def _matches_legacy_reference_replay(stored: str, payload: str) -> bool:
"""Match an old id-only caller to the same job's frozen payload."""
try:
old = json.loads(stored)
new = json.loads(payload)
except (TypeError, ValueError):
return False
if not isinstance(old, dict) or not isinstance(new, dict):
return False
old_inputs = old.get("input_records")
new_inputs = new.get("input_records")
if bool(old_inputs) == bool(new_inputs):
return False
old.pop("input_records", None)
new.pop("input_records", None)
return old == new
def _matches_reaped_legacy_payload(stored: str, payload: str) -> bool:
"""Compare a current frozen request with a pre-snapshot reaped payload."""
try:
old = json.loads(stored)
current = json.loads(payload)
except (TypeError, ValueError):
return False
if (
not isinstance(old, dict)
or not isinstance(current, dict)
or not isinstance(old.get(REAPED_PAYLOAD_MARKER), str)
):
return False
current.pop("input_records", None)
legacy_payload = json.dumps(current, sort_keys=True)
return bool(old[REAPED_PAYLOAD_MARKER] == _payload_digest(legacy_payload))
class DurableQueueSchemaUnavailableError(RuntimeError):
"""The durable queue schema has not been created yet.
@@ -445,6 +481,7 @@ class DurableWriteQueue:
if not initialize:
return
with main_database(database_path).write() as connection, connection:
connection.row_factory = sqlite3.Row
# First create the base table if absent. Then widen an older table
# before replaying indexes that reference the new columns; a bare
# CREATE INDEX would fail on a pre-priority queue.
@@ -455,6 +492,7 @@ class DurableWriteQueue:
for statement in memory_write_jobs_sqlite_schema()[1:]:
connection.execute(statement)
self._backfill_legacy_references(connection)
self._backfill_legacy_event_times(connection)
# A process that died while holding a lease cannot run its own
# recovery. Opening the queue is the first safe point to recover any
# rows it left behind, before any new worker claims them.
@@ -479,8 +517,7 @@ class DurableWriteQueue:
"UPDATE memory_write_jobs SET source_key=session_key WHERE source_key=''"
)
legacy_rows = connection.execute(
"SELECT job_id FROM memory_write_jobs WHERE queue_seq=0 "
"ORDER BY created_at, rowid"
"SELECT job_id FROM memory_write_jobs WHERE queue_seq=0 ORDER BY created_at, rowid"
).fetchall()
next_seq = int(
connection.execute(
@@ -514,6 +551,49 @@ class DurableWriteQueue:
if columns and columns != ["state", "priority", "created_at"]:
connection.execute("DROP INDEX idx_write_jobs_claim")
@staticmethod
def _backfill_legacy_event_times(connection: sqlite3.Connection) -> None:
if (
connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='evidence'"
).fetchone()
is None
):
connection.execute(
"UPDATE memory_write_jobs SET event_at=created_at WHERE event_at IS NULL"
)
return
evidence_columns = {
str(row[1]) for row in connection.execute("PRAGMA table_info(evidence)").fetchall()
}
if not {"created_at", "imported_at", "role"} <= evidence_columns:
connection.execute(
"UPDATE memory_write_jobs SET event_at=created_at WHERE event_at IS NULL"
)
return
connection.execute(
"""
UPDATE memory_write_jobs
SET event_at = COALESCE(
(
SELECT MAX(COALESCE(e.created_at, e.imported_at))
FROM memory_write_job_evidence AS linked
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
WHERE linked.job_id = memory_write_jobs.job_id
AND e.role = 'user'
),
(
SELECT MIN(COALESCE(e.created_at, e.imported_at))
FROM memory_write_job_evidence AS linked
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
WHERE linked.job_id = memory_write_jobs.job_id
),
created_at
)
WHERE event_at IS NULL
"""
)
@staticmethod
def _backfill_legacy_references(connection: sqlite3.Connection) -> None:
"""Populate the reference table after its schema exists."""
@@ -610,8 +690,20 @@ class DurableWriteQueue:
replacing the earlier intent.
"""
self._refuse_if_below_reserve()
payload = json.dumps(request.model_dump(mode="json"), sort_keys=True)
request = self._freeze_request_inputs(request)
payload = json.dumps(
request.model_dump(
mode="json",
exclude={"event_at", "time_quality"},
),
sort_keys=True,
)
now = datetime.now(UTC).isoformat()
resolved_event, resolved_quality = self._event_time_for_request(request)
event = request.event_at or resolved_event or datetime.fromisoformat(now)
if event.tzinfo is None:
event = event.replace(tzinfo=UTC)
event_at = event.astimezone(UTC).isoformat()
key = request.request_key
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
@@ -624,16 +716,22 @@ class DurableWriteQueue:
(key,),
).fetchone()
if existing is not None:
# A replay of work that already finished is a hit, not a
# conflict. Terminal rows store a reaped digest instead of the
# original payload, and the caller's key is derived from the
# evidence's revision, which can move on between replays while
# the payload's evidence ids stay the same. Refusing that pair
# turned every startup replay of one settled job into an
# exception: 236 of them in a single afternoon, all one key.
already_settled = (
str(existing["state"]) == COMPLETED and int(existing["hy_invoked"]) == 1
)
stored_payload = str(existing["payload_json"])
priority_replay = _matches_payload_except_priority(stored_payload, payload)
legacy_reaped_match = bool(
already_settled
and _matches_reaped_legacy_payload(stored_payload, payload)
)
if (
not _matches_stored_payload(stored_payload, payload)
and not priority_replay
and not _matches_legacy_reference_replay(stored_payload, payload)
and not legacy_reaped_match
):
raise ValueError(f"idempotency key {key} already exists with different content")
if already_settled:
return DurableWriteRecord(
job_id=str(existing["job_id"]),
@@ -644,14 +742,6 @@ class DurableWriteQueue:
updated_at=str(existing["updated_at"]),
detail="already completed; replay treated as a hit",
)
priority_replay = _matches_payload_except_priority(
str(existing["payload_json"]), payload
)
if (
not _matches_stored_payload(str(existing["payload_json"]), payload)
and not priority_replay
):
raise ValueError(f"idempotency key {key} already exists with different content")
if priority_replay and request.priority == "live":
connection.execute(
"UPDATE memory_write_jobs SET priority='live', updated_at=? "
@@ -711,11 +801,13 @@ class DurableWriteQueue:
)
connection.execute(
"INSERT INTO memory_write_jobs ("
" job_id, queue_seq, session_key, source_key, batch_id, idempotency_key, payload_json,"
" job_id, queue_seq, session_key, source_key, batch_id,"
" idempotency_key, payload_json,"
" evidence_ids_json,"
" state, priority, attempts, max_attempts, lease_owner,"
" lease_expires_at, hy_invoked, last_error, created_at, updated_at"
") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
" lease_expires_at, hy_invoked, last_error, event_at, time_quality,"
" created_at, updated_at"
") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
(
job_id,
queue_seq,
@@ -733,6 +825,8 @@ class DurableWriteQueue:
None,
0,
None,
event_at,
request.time_quality if request.event_at is not None else resolved_quality,
now,
now,
),
@@ -741,6 +835,18 @@ class DurableWriteQueue:
"INSERT INTO memory_write_job_evidence (job_id, evidence_id) VALUES (?,?)",
((job_id, str(evidence_id)) for evidence_id in request.evidence_ids),
)
diagnostic_event(
level="info",
component="memory.write_queue",
event="memory_write_job.queued",
outcome="queued",
job_id=job_id,
details={
"source_id": request.source_id or request.internal_session_id,
"mode": request.priority,
"count": len(request.evidence_ids),
},
)
return DurableWriteRecord(
job_id=job_id,
request_id=key,
@@ -750,6 +856,45 @@ class DurableWriteQueue:
updated_at=now,
)
def _freeze_request_inputs(self, request: WriteRequest) -> WriteRequest:
if request.input_records or self.evidence is None:
return request
records: list[EvidenceRecord] = []
for evidence_id in request.evidence_ids:
record = self.evidence.get(evidence_id)
if record is None:
return request
records.append(record)
return request.model_copy(update={"input_records": tuple(records)})
def _event_time_for_request(self, request: WriteRequest) -> tuple[datetime | None, str]:
records = list(request.input_records)
if not records and self.evidence is not None:
records = [
record
for evidence_id in request.evidence_ids
if (record := self.evidence.get(evidence_id)) is not None
]
if not records:
return None, "approximate"
opener = next(
(record for record in records if record.role == "user"),
records[0],
)
if opener.created_at is None:
return opener.imported_at, "approximate"
origin = opener.origin_meta.get(EvidenceOrigin.FILE)
basis = origin.raw_meta.get("time_basis") if origin is not None else None
turn_event_at = origin.raw_meta.get("turn_event_at") if origin is not None else None
if isinstance(turn_event_at, str):
try:
return datetime.fromisoformat(turn_event_at), "exact"
except ValueError:
pass
if basis == "file_mtime":
return opener.created_at, "session"
return opener.created_at, "exact"
def repair_pre_send_identity_failures(self, *, limit: int = 128) -> int:
"""Refresh stale Evidence journals and requeue only provably unsent jobs."""
if self.evidence is None:
@@ -859,11 +1004,55 @@ class DurableWriteQueue:
raise ReconcileSchemaUnavailableError(
"evidence table is unavailable; run migrations before reconcile"
)
where = (
"WHERE evidence.imported_at IS NOT NULL "
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence ref "
"WHERE ref.evidence_id=evidence.evidence_id)"
has_versions = (
connection.execute(
"SELECT 1 FROM sqlite_master "
"WHERE type='table' AND name='evidence_versions'"
).fetchone()
is not None
)
evidence_columns = {
str(row[1])
for row in connection.execute("PRAGMA table_info(evidence)").fetchall()
}
if {"revision", "content_hash", "updated_at"} <= evidence_columns:
revision_time = (
"COALESCE((SELECT version.changed_at FROM evidence_versions version "
"WHERE version.evidence_id=evidence.evidence_id "
"AND version.revision=evidence.revision),"
"evidence.updated_at,evidence.imported_at)"
if has_versions
else "COALESCE(evidence.updated_at,evidence.imported_at)"
)
where = (
"WHERE evidence.imported_at IS NOT NULL "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_job_evidence linked "
"JOIN memory_write_jobs job ON job.job_id=linked.job_id, "
"json_each(job.payload_json,'$.input_records') frozen "
"WHERE linked.evidence_id=evidence.evidence_id "
"AND json_extract(frozen.value,'$.evidence_id')=evidence.evidence_id "
"AND json_extract(frozen.value,'$.revision')=evidence.revision "
"AND json_extract(frozen.value,'$.content_hash')=evidence.content_hash"
") "
# Pre-snapshot builds only recorded an Evidence id. Treat
# that as coverage only when its Job was created after this
# exact revision.
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_job_evidence legacy "
"JOIN memory_write_jobs legacy_job ON legacy_job.job_id=legacy.job_id "
"WHERE legacy.evidence_id=evidence.evidence_id "
"AND COALESCE(json_array_length("
"json_extract(legacy_job.payload_json,'$.input_records')),0)=0 "
f"AND legacy_job.created_at>=({revision_time}))"
)
else:
# A genuinely old/minimal schema has no revision identity.
where = (
"WHERE evidence.imported_at IS NOT NULL "
"AND NOT EXISTS (SELECT 1 FROM memory_write_job_evidence legacy "
"WHERE legacy.evidence_id=evidence.evidence_id)"
)
params: tuple[Any, ...] = ()
cursor_clause = ""
if after_cursor is not None:
@@ -1095,6 +1284,7 @@ class DurableWriteQueue:
with main_database(self.database_path).write() as connection, connection:
updated = connection.execute(
"UPDATE memory_write_jobs SET state=?,lease_owner=NULL,"
"attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,"
"lease_expires_at=NULL,updated_at=? WHERE " + where,
(PENDING, now, *params),
)
@@ -1162,45 +1352,10 @@ class DurableWriteQueue:
return state
def clear_terminal_payloads(self, *, limit: int = 512) -> int:
"""Replace terminal rows' request bodies with a content digest.
A completed/failed/dead/cancelled row can never run again, so the full
serialized `WriteRequest` is unbounded dead weight. A digest keeps the
idempotency contract intact: a replay of the same intent still resolves
to the same row, while the same key with different content still
conflicts. `unknown` is left alone because it is non-terminal and its
intent is needed for reconciliation.
Bounded by `limit` so a large backlog drains over several calls instead
of one long write on the single writer.
"""
"""Deprecated compatibility hook; durable inputs are immutable audit data."""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
placeholders = ",".join("?" for _ in TERMINAL_PAYLOAD_STATES)
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
):
rows = connection.execute(
"SELECT job_id, payload_json FROM memory_write_jobs "
f"WHERE state IN ({placeholders}) "
"AND instr(payload_json, ?) = 0 "
"LIMIT ?",
(*TERMINAL_PAYLOAD_STATES, REAPED_PAYLOAD_MARKER, limit),
).fetchall()
if not rows:
return 0
for row in rows:
connection.execute(
"UPDATE memory_write_jobs SET payload_json=? "
f"WHERE job_id=? AND state IN ({placeholders})",
(
_reaped_payload(_payload_digest(str(row["payload_json"]))),
str(row["job_id"]),
*TERMINAL_PAYLOAD_STATES,
),
)
return len(rows)
return 0
def readmit_parked(self, *, states: tuple[str, ...] = PARKED_STATES) -> int:
"""Move parked jobs back to `pending` for a fresh claim.
@@ -1250,11 +1405,12 @@ class DurableWriteQueue:
) -> int:
"""Re-admit `waiting_for_model` jobs whose wait has elapsed.
A model outage clears on its own, so this state is retried on a timer
A model outage clears on its own, so transient waiters retry on a timer
rather than waiting for a grant event like `waiting_for_permission`.
Keyed on `updated_at` so repeated calls while a job is still parked do
not reset its own clock; the attempt consumed by the parking claim is
returned, the same way `readmit_parked` does.
`model_profile_not_active` is excluded: only explicit activation can
change that condition. Keyed on `updated_at` so repeated calls while a
job is still parked do not reset its own clock; the attempt consumed by
the parking claim is returned, the same way `readmit_parked` does.
"""
now = now or datetime.now(UTC)
threshold = (now - after).isoformat()
@@ -1263,8 +1419,33 @@ class DurableWriteQueue:
"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
"lease_expires_at=NULL, attempts=CASE WHEN attempts>0 "
"THEN attempts-1 ELSE 0 END, last_error=NULL, updated_at=? "
"WHERE state=? AND updated_at <= ? AND hy_invoked=0",
(PENDING, now.isoformat(), WAITING_FOR_MODEL, threshold),
"WHERE state=? AND updated_at <= ? AND hy_invoked=0 "
"AND (last_error IS NULL OR last_error<>?)",
(
PENDING,
now.isoformat(),
WAITING_FOR_MODEL,
threshold,
"model_profile_not_active",
),
)
return changed.rowcount
def readmit_model_profile_waiters(self) -> int:
"""Re-admit no-profile work only after model activation changes the condition."""
now = datetime.now(UTC).isoformat()
with main_database(self.database_path).write() as connection, connection:
changed = connection.execute(
"UPDATE memory_write_jobs SET state=?, lease_owner=NULL, "
"lease_expires_at=NULL, attempts=CASE WHEN attempts>0 "
"THEN attempts-1 ELSE 0 END, last_error=NULL, updated_at=? "
"WHERE state=? AND last_error=? AND hy_invoked=0",
(
PENDING,
now,
WAITING_FOR_MODEL,
"model_profile_not_active",
),
)
return changed.rowcount
@@ -1362,9 +1543,13 @@ class DurableWriteQueue:
connection,
):
row = connection.execute(
"SELECT state FROM memory_write_jobs WHERE job_id=?", (job_id,)
"SELECT state,last_error FROM memory_write_jobs WHERE job_id=?", (job_id,)
).fetchone()
if row is None or str(row["state"]) != UNKNOWN:
if (
row is None
or str(row["state"]) != UNKNOWN
or str(row["last_error"]) in NON_RETRYABLE_POST_WRITE_CODES
):
return 0
connection.execute(
"""
@@ -1402,7 +1587,13 @@ class DurableWriteQueue:
cross back to pending, so this cannot duplicate a possibly-sent write.
"""
now = datetime.now(UTC).isoformat()
payload = json.dumps(request.model_dump(mode="json"), sort_keys=True)
payload = json.dumps(
request.model_dump(
mode="json",
exclude={"event_at", "time_quality"},
),
sort_keys=True,
)
with (
main_database(self.database_path).write(row_factory=sqlite3.Row) as connection,
connection,
@@ -1515,8 +1706,15 @@ class DurableWriteQueue:
rows = connection.execute(
"SELECT job_id FROM memory_write_jobs "
"WHERE state=? AND attempts=1 AND attempts < max_attempts "
f"AND COALESCE(last_error, '') NOT IN "
f"({','.join('?' for _ in NON_RETRYABLE_POST_WRITE_CODES)}) "
"AND updated_at <= ? ORDER BY updated_at, job_id LIMIT ?",
(UNKNOWN, older_than.isoformat(), limit),
(
UNKNOWN,
*NON_RETRYABLE_POST_WRITE_CODES,
older_than.isoformat(),
limit,
),
).fetchall()
job_ids = tuple(str(row["job_id"]) for row in rows)
if not job_ids:
@@ -1580,6 +1778,7 @@ class DurableWriteQueue:
*,
limit: int = 128,
after_cursor: tuple[str, str] | None = None,
exclude_error_codes: tuple[str, ...] = (),
) -> tuple[UnknownWriteJob, ...]:
"""List unknown rows that still need an inquiry, in stable order.
@@ -1596,6 +1795,9 @@ class DurableWriteQueue:
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
excluded = tuple(dict.fromkeys(exclude_error_codes))
if not all(isinstance(code, str) and code for code in excluded):
raise ValueError("exclude_error_codes must contain non-empty strings")
cursor_clause = ""
params: tuple[Any, ...] = ()
if after_cursor is not None:
@@ -1608,14 +1810,20 @@ class DurableWriteQueue:
updated_at, job_id = after_cursor
cursor_clause = "AND (updated_at, job_id) > (?, ?)"
params = (updated_at, job_id)
if excluded:
placeholders = ",".join("?" for _ in excluded)
excluded_clause = f"AND COALESCE(last_error, '') NOT IN ({placeholders})"
else:
excluded_clause = ""
with main_database(self.database_path).read(row_factory=sqlite3.Row) as connection:
rows = connection.execute(
"SELECT job_id, idempotency_key, created_at, updated_at, last_error "
"FROM memory_write_jobs "
"WHERE state=? "
f"{cursor_clause} "
f"{excluded_clause} "
"ORDER BY updated_at, job_id LIMIT ?",
(UNKNOWN, *params, limit),
(UNKNOWN, *params, *excluded, limit),
).fetchall()
return tuple(
UnknownWriteJob(
@@ -1741,9 +1949,12 @@ class DurableWriteRegistry:
def evidence_for_request(store: EvidenceStore, request: WriteRequest) -> tuple[EvidenceRecord, ...]:
"""The Evidence a stored write points at, in the request's own order.
A write may only point at Evidence that still exists; a missing id is a
rejection rather than a silent partial send.
Current jobs are self-contained: their immutable records are the model
input even if a Harness later rewinds or edits its live transcript. The
store lookup remains only for payloads written by pre-upgrade builds.
"""
if request.input_records:
return request.input_records
records = []
for evidence_id in request.evidence_ids:
record = store.get(evidence_id)
@@ -1850,7 +2061,14 @@ async def execute_job(
# Never persist the raw message: it may be a provider response or the
# user's own text, and `last_error` is shown over HTTP as `detail`.
code = safe_error_code(error)
if code == "model_content_source_disabled" and not hy_invoked:
if (
code
in {
"model_content_source_disabled",
"model_sedimentation_disabled",
}
and not hy_invoked
):
await in_the_background(queue.release_unstarted, job_id, owner=owner)
return
# The task layer classifies these before any bridge is opened, so HY
@@ -1861,7 +2079,11 @@ async def execute_job(
# - transient model/provider trouble -> waiting_for_model (parked).
# Everything else may have crossed into HY, and HY has no idempotency
# key, so it is `unknown` rather than a blind retry.
if hy_invoked:
if code in PRE_SEND_REFUSAL_CODES:
# The frozen input window is checked before the payload reaches
# HY. This one refusal therefore has a proven pre-send boundary.
state, invoked_marker = FAILED, False
elif hy_invoked:
# Error names do not prove execution phase: permission and
# evidence rechecks also run after HY returns. Once the durable
# marker is set, every failure is possibly-sent and must stay out
@@ -1873,10 +2095,6 @@ async def execute_job(
state, invoked_marker = WAITING_FOR_PERMISSION, False
elif code in MODEL_NOT_READY_CODES:
state, invoked_marker = WAITING_FOR_MODEL, False
elif code in PRE_SEND_REFUSAL_CODES:
# Provably before the bridge: the frozen input is stale or too
# large. Nothing was sent, so this is terminal rather than unknown.
state, invoked_marker = FAILED, False
elif code == "model_content_transport_request_failed":
# The generation or embedding HTTP exchange can fail after HY has
# already persisted a node. Treating that as a pre-send failure
@@ -1901,7 +2119,12 @@ async def execute_job(
# Never `logger.exception` here: it prints the exception's message,
# which may be a provider response or the user's own text. Log the
# already-sanitised code instead, matching `hy-failure-classification`.
logger.error("durable_write_job_failed job_id=%s error_code=%s", job_id, code)
if code == "model_profile_not_active":
logger.debug("durable_write_job_parked job_id=%s error_code=%s", job_id, code)
elif state in PARKED_STATES:
logger.info("durable_write_job_parked job_id=%s error_code=%s", job_id, code)
else:
logger.error("durable_write_job_failed job_id=%s error_code=%s", job_id, code)
return
await in_the_background(queue.settle, job_id, state=COMPLETED, owner=owner)
@@ -1965,7 +2188,6 @@ class DurableWriteDrain:
#: sustained backlog must not keep finished rows' request bodies in the
#: business database indefinitely. Its own gate so neither interval can
#: starve the other.
self._last_terminal_reap: datetime | None = None
#: `waiting_for_model` re-admission is also time-based recoverability:
#: a model outage clears on its own, and a queue that never goes idle
#: must still re-admit the write once the wait has elapsed.
@@ -2016,7 +2238,7 @@ class DurableWriteDrain:
now: datetime | None = None,
budget_ms: int | None = None,
) -> tuple[str, ...]:
"""Claim at most one FIFO head per enabled Harness."""
"""Claim event-time heads, with one running job per enabled Harness."""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
now = now or datetime.now(UTC)
@@ -2058,9 +2280,7 @@ class DurableWriteDrain:
(PENDING,),
).fetchall()
enabled_sources = {
str(row[0])
for row in source_rows
if self.source_writes_permitted(str(row[0]))
str(row[0]) for row in source_rows if self.source_writes_permitted(str(row[0]))
}
if not enabled_sources:
return ()
@@ -2079,38 +2299,142 @@ class DurableWriteDrain:
placeholders = ",".join("?" for _ in enabled_sources)
source_clause = f"AND candidate.source_key IN ({placeholders}) "
source_params = tuple(sorted(enabled_sources))
window = connection.execute(
"SELECT window_anchor,window_end FROM memory_write_schedule_state WHERE singleton=1"
).fetchone()
if window is not None:
anchor_text, end_text = str(window[0]), str(window[1])
active_source_clause = ""
active_source_params: tuple[Any, ...] = ()
if enabled_sources is not None:
placeholders = ",".join("?" for _ in enabled_sources)
active_source_clause = f"AND active.source_key IN ({placeholders}) "
active_source_params = tuple(sorted(enabled_sources))
# The contract retires a window when every *eligible*
# pending/running job in it has settled or parked. A pending row
# that scheduling already holds back (an unexpired lease, a
# busy Harness, or an earlier event in its Session) is not eligible, so it
# must not keep the anchor alive: those blockers usually sit
# outside this ten-minute range and can never be claimed from
# inside it, which pinned the anchor and starved the whole
# queue. Reuse the claim's own predicate so "the window is
# busy" and "the window can still be worked" agree.
active = connection.execute(
"SELECT 1 FROM memory_write_jobs AS active "
"WHERE active.event_at>=? AND active.event_at<? AND ("
"active.state=? OR (active.state=? "
"AND (active.lease_expires_at IS NULL "
"OR active.lease_expires_at < ?) " + active_source_clause + "AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=active.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL "
"AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=active.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<active.event_at OR "
"(earlier.event_at=active.event_at "
"AND earlier.queue_seq<active.queue_seq))) "
")) LIMIT 1",
(
anchor_text,
end_text,
RUNNING,
PENDING,
now.isoformat(),
*active_source_params,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchone()
if active is None:
connection.execute("DELETE FROM memory_write_schedule_state WHERE singleton=1")
window = None
if window is None:
anchor = connection.execute(
"SELECT candidate.event_at "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL "
"OR candidate.lease_expires_at < ?) " + source_clause + "AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL "
"AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.event_at,candidate.queue_seq LIMIT 1",
(
PENDING,
now.isoformat(),
*source_params,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchone()
if anchor is None:
return ()
anchor_at = datetime.fromisoformat(str(anchor[0]))
if anchor_at.tzinfo is None:
anchor_at = anchor_at.replace(tzinfo=UTC)
anchor_text = anchor_at.astimezone(UTC).isoformat()
end_text = (anchor_at + EVENT_TIME_WINDOW).astimezone(UTC).isoformat()
connection.execute(
"INSERT INTO memory_write_schedule_state "
"(singleton,window_anchor,window_end,updated_at) "
"VALUES (1,?,?,?)",
(anchor_text, end_text, now.isoformat()),
)
rows = connection.execute(
"SELECT candidate.job_id,candidate.source_key "
"FROM memory_write_jobs AS candidate "
"WHERE candidate.state=? "
"AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "
+ source_clause +
+ source_clause
+ "AND candidate.event_at>=? AND candidate.event_at<? "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS held "
"WHERE held.source_key=candidate.source_key AND held.state=? "
"AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "
"AND NOT EXISTS ("
"SELECT 1 FROM memory_write_jobs AS earlier "
"WHERE earlier.source_key=candidate.source_key "
"WHERE earlier.session_key=candidate.session_key "
f"AND earlier.state IN ({blocking}) "
"AND earlier.queue_seq<candidate.queue_seq) "
"ORDER BY candidate.queue_seq",
"AND (earlier.event_at<candidate.event_at OR "
"(earlier.event_at=candidate.event_at "
"AND earlier.queue_seq<candidate.queue_seq))) "
"ORDER BY candidate.event_at,candidate.queue_seq",
(
PENDING,
now.isoformat(),
*source_params,
anchor_text,
end_text,
RUNNING,
now.isoformat(),
*_FIFO_BLOCKING_STATES,
),
).fetchall()
selected: list[str] = []
selected_sources: set[str] = set()
for row in rows:
source_key = str(row["source_key"])
if source_key in selected_sources:
continue
if self.source_claim_enabled is not None and not self.source_claim_enabled(
str(row["source_key"])
source_key
):
continue
selected.append(str(row["job_id"]))
selected_sources.add(source_key)
if len(selected) == min(limit, self.max_concurrent_sources):
break
if not selected:
@@ -2339,19 +2663,6 @@ class DurableWriteDrain:
self.queue.reclaim_expired,
operation_name="reclaim_expired",
)
# Terminal-payload reaping is a bounded retention write: a
# finished row's full request body must not sit in the business
# database forever just because the queue never went idle. Its
# own interval keeps it off every poll.
if (
self._last_terminal_reap is None
or tick_started - self._last_terminal_reap >= self._reap_interval
):
self._last_terminal_reap = tick_started
await self._run_idle_maintenance(
self.queue.clear_terminal_payloads,
operation_name="clear_terminal_payloads",
)
# A model outage clears on its own, so `waiting_for_model` jobs
# are re-admitted on a timer. That must not depend on idleness: a
# queue busy for longer than the wait window would otherwise keep
@@ -3,7 +3,7 @@
The result decides whether the `unknown` job state can ever be retired, so it
is probed against the installed SDK rather than written down from memory.
What was verified against `hy-memory==2.0.3`:
What was verified against `hy-memory==2.0.4`:
- `HyMemoryClient.add(..., request_id=...)` accepts a caller-supplied id and
echoes it back. It is described as a request/receipt handle, and the SDK
@@ -151,6 +151,8 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
lease_expires_at TEXT,
hy_invoked INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
event_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
time_quality TEXT NOT NULL DEFAULT 'approximate',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
@@ -164,6 +166,8 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
ON memory_write_jobs(source_key, state, queue_seq);
CREATE INDEX IF NOT EXISTS idx_write_jobs_fifo
ON memory_write_jobs(queue_seq);
CREATE INDEX IF NOT EXISTS idx_write_jobs_event_claim
ON memory_write_jobs(state, event_at, queue_seq);
CREATE INDEX IF NOT EXISTS idx_write_jobs_source_recent
ON memory_write_jobs(source_key, updated_at DESC);
CREATE TABLE IF NOT EXISTS memory_write_job_evidence (
@@ -172,7 +176,13 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
PRIMARY KEY (job_id, evidence_id)
);
CREATE INDEX IF NOT EXISTS idx_write_job_evidence_evidence
ON memory_write_job_evidence(evidence_id)
ON memory_write_job_evidence(evidence_id);
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
window_anchor TEXT NOT NULL,
window_end TEXT NOT NULL,
updated_at TEXT NOT NULL
)
"""
return """
CREATE TABLE IF NOT EXISTS memory_write_jobs (
@@ -192,6 +202,8 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
lease_expires_at DATETIME(6) NULL,
hy_invoked TINYINT(1) NOT NULL DEFAULT 0,
last_error TEXT NULL,
event_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
time_quality VARCHAR(32) NOT NULL DEFAULT 'approximate',
created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6)
ON UPDATE CURRENT_TIMESTAMP(6),
@@ -201,6 +213,7 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
KEY idx_write_jobs_source_claim (state, source_key, priority, created_at),
KEY idx_write_jobs_source_fifo (source_key, state, queue_seq),
KEY idx_write_jobs_fifo (queue_seq),
KEY idx_write_jobs_event_claim (state, event_at, queue_seq),
KEY idx_write_jobs_source_recent (source_key, updated_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
@@ -209,6 +222,13 @@ def memory_write_jobs_ddl(dialect: str = "mysql") -> str:
evidence_id VARCHAR(128) NOT NULL,
PRIMARY KEY (job_id, evidence_id),
KEY idx_write_job_evidence_evidence (evidence_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
singleton INT PRIMARY KEY CHECK (singleton = 1),
window_anchor DATETIME(6) NOT NULL,
window_end DATETIME(6) NOT NULL,
updated_at DATETIME(6) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
@@ -1231,6 +1251,68 @@ CREATE TABLE IF NOT EXISTS model_request_budget_events (
"""
def model_request_audit_ddl(dialect: str = "mysql") -> str:
"""Content-free accounting for physical precipitation model requests."""
if dialect == "sqlite":
return """
CREATE TABLE IF NOT EXISTS model_request_audit (
audit_id TEXT PRIMARY KEY,
request_id TEXT NOT NULL,
profile_id TEXT NOT NULL,
profile_revision INTEGER NOT NULL,
provider_kind TEXT NOT NULL,
model TEXT NOT NULL,
request_kind TEXT NOT NULL,
prompt_stage TEXT NOT NULL DEFAULT 'other',
started_at TEXT NOT NULL,
completed_at TEXT,
outcome TEXT NOT NULL CHECK(outcome IN ('in_flight','succeeded','failed')),
http_status INTEGER,
error_type TEXT,
provider_request_id TEXT,
prompt_tokens INTEGER,
completion_tokens INTEGER,
total_tokens INTEGER,
cached_prompt_tokens INTEGER,
reasoning_tokens INTEGER
);
CREATE INDEX IF NOT EXISTS model_request_audit_request
ON model_request_audit(request_id, started_at);
CREATE INDEX IF NOT EXISTS model_request_audit_started
ON model_request_audit(started_at);
"""
return """
CREATE TABLE IF NOT EXISTS model_request_audit (
audit_id VARCHAR(128) PRIMARY KEY,
request_id VARCHAR(512) NOT NULL,
profile_id VARCHAR(128) NOT NULL,
profile_revision INT NOT NULL,
provider_kind VARCHAR(64) NOT NULL,
model VARCHAR(512) NOT NULL,
request_kind VARCHAR(64) NOT NULL,
prompt_stage VARCHAR(128) NOT NULL DEFAULT 'other',
started_at DATETIME(6) NOT NULL,
completed_at DATETIME(6) NULL,
outcome VARCHAR(32) NOT NULL CHECK(outcome IN ('in_flight','succeeded','failed')),
http_status INT NULL,
error_type VARCHAR(128) NULL,
provider_request_id VARCHAR(512) NULL,
prompt_tokens BIGINT NULL,
completion_tokens BIGINT NULL,
total_tokens BIGINT NULL,
cached_prompt_tokens BIGINT NULL,
reasoning_tokens BIGINT NULL,
KEY model_request_audit_request (request_id, started_at),
KEY model_request_audit_started (started_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
async def _052_model_request_audit(cursor: Any) -> None:
for statement in _split_statements(model_request_audit_ddl(_dialect(cursor))):
await cursor.execute(statement)
def model_probe_operations_ddl(dialect: str = "mysql") -> str:
"""Probe operations; `state` is CHECKed to a known set.
@@ -1414,6 +1496,97 @@ async def _051_drop_hy_node_permissions(cursor: Any) -> None:
await cursor.execute("DROP TABLE IF EXISTS hy_node_permissions")
async def _053_memory_write_event_time(cursor: Any) -> None:
"""Add event-time scheduling without re-keying any existing write."""
dialect = _dialect(cursor)
columns = await cursor.columns("memory_write_jobs")
added_event_at = "event_at" not in columns
if added_event_at:
await cursor.execute(
"ALTER TABLE memory_write_jobs ADD COLUMN event_at "
+ ("TEXT" if dialect == "sqlite" else "DATETIME(6) NULL")
)
if "time_quality" not in columns:
await cursor.execute(
"ALTER TABLE memory_write_jobs ADD COLUMN time_quality "
+ (
"TEXT NOT NULL DEFAULT 'approximate'"
if dialect == "sqlite"
else "VARCHAR(32) NOT NULL DEFAULT 'approximate'"
)
)
await cursor.execute(
"""
UPDATE memory_write_jobs
SET event_at = COALESCE(
(
SELECT MAX(COALESCE(e.created_at, e.imported_at))
FROM memory_write_job_evidence AS linked
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
WHERE linked.job_id = memory_write_jobs.job_id
AND e.role = 'user'
),
(
SELECT MIN(COALESCE(e.created_at, e.imported_at))
FROM memory_write_job_evidence AS linked
JOIN evidence AS e ON e.evidence_id = linked.evidence_id
WHERE linked.job_id = memory_write_jobs.job_id
),
created_at
)
WHERE event_at IS NULL
"""
)
if dialect == "sqlite":
await cursor.execute(
"CREATE INDEX IF NOT EXISTS idx_write_jobs_event_claim "
"ON memory_write_jobs(state, event_at, queue_seq)"
)
await cursor.execute(
"""
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
window_anchor TEXT NOT NULL,
window_end TEXT NOT NULL,
updated_at TEXT NOT NULL
)
"""
)
else:
if added_event_at:
await cursor.execute(
"CREATE INDEX idx_write_jobs_event_claim "
"ON memory_write_jobs(state, event_at, queue_seq)"
)
await cursor.execute(
"""
CREATE TABLE IF NOT EXISTS memory_write_schedule_state (
singleton INT PRIMARY KEY CHECK (singleton = 1),
window_anchor DATETIME(6) NOT NULL,
window_end DATETIME(6) NOT NULL,
updated_at DATETIME(6) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
async def _054_legacy_possibly_sent_tasks(cursor: Any) -> None:
"""Preserve the old sent marker before removing its retired permission table."""
columns = await cursor.columns("memory_task_continuous_permissions")
if "task_id" not in columns:
return
await cursor.execute(
"""
UPDATE memory_tasks
SET hy_invocation_count=CASE
WHEN hy_invocation_count<1 THEN 1 ELSE hy_invocation_count END
WHERE task_id IN (SELECT task_id FROM memory_task_continuous_permissions)
"""
)
await cursor.execute("DROP TABLE memory_task_continuous_permissions")
def model_profiles_ddl(dialect: str = "mysql") -> str:
"""Model profiles and their version history.
@@ -2680,7 +2853,11 @@ MIGRATIONS: tuple[Migration, ...] = (
2,
"memory_write_jobs",
_002_write_jobs,
("memory_write_jobs", "memory_write_job_evidence"),
(
"memory_write_jobs",
"memory_write_job_evidence",
"memory_write_schedule_state",
),
),
Migration(3, "source_file_cursors", _003_scanner_cursors, ("source_file_cursors",)),
Migration(4, "change_outbox", _004_change_outbox, ("change_outbox",)),
@@ -2849,4 +3026,21 @@ MIGRATIONS: tuple[Migration, ...] = (
_050_model_probe_operations_reserved_at,
),
Migration(51, "drop_hy_node_permissions", _051_drop_hy_node_permissions),
Migration(
52,
"model_request_audit",
_052_model_request_audit,
("model_request_audit",),
),
Migration(
53,
"memory_write_event_time",
_053_memory_write_event_time,
("memory_write_schedule_state",),
),
Migration(
54,
"legacy_possibly_sent_tasks",
_054_legacy_possibly_sent_tasks,
),
)
@@ -107,6 +107,8 @@ def build_request(
first_round: int,
last_round: int,
evidence_ids: list[str],
input_records: tuple[EvidenceRecord, ...] = (),
source_id: str | None = None,
priority: Literal["live", "normal"] = "normal",
idempotency_key: str | None = None,
) -> WriteRequest:
@@ -125,6 +127,8 @@ def build_request(
internal_session_id=internal_session_id,
batch_identity=batch,
evidence_ids=evidence_ids,
input_records=input_records,
source_id=source_id,
idempotency_key=idempotency_key or batch.batch_id,
priority=priority,
)
@@ -271,6 +275,8 @@ def build_request_from_evidence(
first_round=first_round,
last_round=last_round,
evidence_ids=list(ordered),
input_records=tuple(ordered.values()),
source_id=seed.source_id,
priority=priority,
idempotency_key=idempotency_key,
)
@@ -18,10 +18,11 @@ from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass
from datetime import datetime
from hashlib import sha256
from typing import Literal
from one_memory.imports.models import EvidenceRecord
from one_memory.imports.models import EvidenceOrigin, EvidenceRecord
from one_memory.memory_server.contracts import (
BatchIdentity,
WriteRequest,
@@ -124,6 +125,12 @@ class UnifiedWriteIntake:
priority: WritePriority,
) -> WriteRequest:
session_id = internal_session_id_for(record.conversation_id, record.source_id)
event_at, time_quality = _job_time((record,))
request_key = (
f"entry:{record.evidence_id}"
if record.revision == 1
else (f"entry:v2:{record.evidence_id}:{record.revision}:{record.content_hash}")
)
return WriteRequest(
internal_session_id=session_id,
source_id=record.source_id,
@@ -133,8 +140,11 @@ class UnifiedWriteIntake:
last_round=last_round,
),
evidence_ids=[record.evidence_id],
idempotency_key=f"entry:{record.evidence_id}",
input_records=(record,),
idempotency_key=request_key,
priority=priority,
event_at=event_at,
time_quality=time_quality,
)
def submit_records(
@@ -161,6 +171,7 @@ class UnifiedWriteIntake:
)
request_id = f"turn:v1:{sha256(material.encode()).hexdigest()}"
session_id = internal_session_id_for(first.conversation_id, first.source_id)
event_at, time_quality = _job_time(records)
durable = self.queue.submit(
WriteRequest(
internal_session_id=session_id,
@@ -171,8 +182,11 @@ class UnifiedWriteIntake:
last_round=1,
),
evidence_ids=[record.evidence_id for record in records],
input_records=records,
idempotency_key=request_id,
priority=priority,
event_at=event_at,
time_quality=time_quality,
)
)
return UnifiedWriteReceipt(
@@ -189,3 +203,29 @@ class SourceWriteDisabledError(RuntimeError):
def __init__(self, evidence_id: str) -> None:
super().__init__("model_content_source_disabled")
self.evidence_id = evidence_id
def _job_time(
records: tuple[EvidenceRecord, ...],
) -> tuple[datetime, Literal["exact", "anchored", "session", "approximate"]]:
"""Use the turn opener's wall clock, with an explicit fallback quality."""
opener = next(
(record for record in records if record.role == "user"),
records[0],
)
if opener.created_at is None:
return opener.imported_at, "approximate"
file_origin = opener.origin_meta.get(EvidenceOrigin.FILE)
basis = file_origin.raw_meta.get("time_basis") if file_origin is not None else None
turn_event_at = file_origin.raw_meta.get("turn_event_at") if file_origin is not None else None
if isinstance(turn_event_at, str):
try:
return datetime.fromisoformat(turn_event_at), "exact"
except ValueError:
pass
if basis == "file_mtime":
return opener.created_at, "session"
if EvidenceOrigin.HOOK in opener.origins:
return opener.created_at, "exact"
return opener.created_at, "exact"
+151 -10
View File
@@ -17,7 +17,7 @@ from pydantic import BaseModel, ConfigDict, Field
from one_memory.database import take_the_write_lock_first
from one_memory.database_access import main_database
from one_memory.imports.models import EvidenceRecord
from one_memory.imports.models import EvidenceRecord, WrittenBy
from one_memory.memory_server.migrations import memory_tasks_ddl
from one_memory.model_profiles import ModelProfile, ModelProfileSnapshot
@@ -67,6 +67,12 @@ POLICY_STOPPED_CODES = (
"hy_memory_evidence_changed",
)
#: HY completed a write, but Core could not prove that the selected mutation
#: target belonged to this task's frozen authority. Reconciliation may inspect
#: HY by request id, but neither automatic nor user-confirmed generic retry may
#: invoke the model again for these codes.
NON_RETRYABLE_POST_WRITE_CODES = ("hy_memory_write_verification_authority_invalid",)
class MemoryTask(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
@@ -116,6 +122,24 @@ PROFILE_STOP_CODES = (
ACTIVE_WINDOW_DEFER_LIMIT = 2048
def _task_only_unknown_clause(connection: sqlite3.Connection) -> str:
"""Exclude tasks whose Evidence is already owned by the durable queue."""
linked_table = connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='memory_write_job_evidence'"
).fetchone()
if linked_table is None:
return ""
return """
AND NOT EXISTS (
SELECT 1
FROM memory_task_inputs AS inputs, json_each(inputs.refs_json) AS ref
JOIN memory_write_job_evidence AS linked
ON linked.evidence_id=json_extract(ref.value, '$.evidence_id')
WHERE inputs.task_id=memory_tasks.task_id
)
"""
@dataclass(frozen=True, slots=True)
class ActiveMemoryTask:
"""One task plus only the evidence fields the scheduler needs."""
@@ -174,29 +198,60 @@ def evidence_ref(record: EvidenceRecord) -> TaskEvidenceRef:
"segment_end": int(marker["end"]),
"segment_hash": str(marker["hash"]),
}
identity = record.model_dump(mode="json", exclude={"origin_meta", "imported_at", "updated_at"})
return {
"evidence_id": record.evidence_id,
"revision": record.revision,
"content_hash": record.content_hash,
"identity_hash": sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest(),
"identity_hash": _evidence_identity_hash(record, include_written_by=False),
}
def _evidence_identity_hash(record: EvidenceRecord, *, include_written_by: bool) -> str:
excluded = {"origin_meta", "imported_at", "updated_at"}
if not include_written_by:
excluded.add("written_by")
identity = record.model_dump(mode="json", exclude=excluded)
return sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest()
def _identity_hash_matches(record: EvidenceRecord, expected: object) -> bool:
candidates = {_evidence_identity_hash(record, include_written_by=False)}
# Tasks created by pre-role-only builds keep their original proof. The
# legacy metadata could also have changed in place without a revision, so
# accept the old hash under every historical author guess.
for written_by in WrittenBy:
candidates.add(
_evidence_identity_hash(
record.model_copy(update={"written_by": written_by}),
include_written_by=True,
)
)
return expected in candidates
def matches_evidence_ref(record: EvidenceRecord, ref: TaskEvidenceRef) -> bool:
if "segment_hash" in ref:
base = evidence_ref(record)
if any(
base.get(key) != ref.get(key)
for key in ("evidence_id", "revision", "content_hash", "identity_hash")
for key in ("evidence_id", "revision", "content_hash")
):
return False
if not _identity_hash_matches(record, ref.get("identity_hash")):
return False
start, end = ref.get("segment_start"), ref.get("segment_end")
if not isinstance(start, int) or not isinstance(end, int) or not (0 <= start < end):
return False
return sha256(record.verbatim[start:end].encode()).hexdigest() == ref["segment_hash"]
current = evidence_ref(record)
return all(current.get(key) == value for key, value in ref.items())
return all(
(
_identity_hash_matches(record, value)
if key == "identity_hash"
else current.get(key) == value
)
for key, value in ref.items()
)
def segment_evidence_record(
@@ -977,6 +1032,44 @@ class MemoryTaskStore:
error_code=error_code,
)
def release_unstarted(
self,
task_id: str,
*,
runner_token: str,
error_code: str,
) -> MemoryTask:
"""Return a claimed task to pending when a user pause wins before HY.
Claim increments ``attempts``. A pause is not an execution failure, so
the attempt is returned as well; repeatedly pausing and resuming can
never exhaust an otherwise healthy task.
"""
now = datetime.now(UTC).isoformat()
with self._connect(writing=True) as connection:
changed = connection.execute(
"""
UPDATE memory_tasks
SET status=?, runner_token=NULL,
attempts=CASE WHEN attempts>0 THEN attempts-1 ELSE 0 END,
last_error_code=?, duration_seconds=NULL, updated_at=?
WHERE task_id=? AND status=? AND runner_token=?
""",
(
MemoryTaskStatus.PENDING,
error_code,
now,
task_id,
MemoryTaskStatus.RUNNING,
runner_token,
),
)
if changed.rowcount != 1:
raise MemoryTaskOwnershipError(
"Memory task runner no longer owns this task"
)
return self.require(task_id)
def park_for_permission(self, task_id: str, *, error_code: str) -> bool:
"""Park unstarted work until a later permission snapshot admits it."""
now = datetime.now(UTC).isoformat()
@@ -1245,7 +1338,7 @@ class MemoryTaskStore:
live_clause = f"task.task_id IN ({live_placeholders})" if live_ids else "0"
not_live_clause = f"task.task_id NOT IN ({live_placeholders})" if live_ids else "1"
excluded_clause = f"task.task_id NOT IN ({excluded_placeholders})" if excluded else "1"
common = f"""
common_base = f"""
task.status IN (?,?,?,?)
AND coalesce(task.last_error_code, '') NOT IN (
'hy_memory_input_exceeds_window', 'model_content_outcome_unknown'
@@ -1264,7 +1357,33 @@ class MemoryTaskStore:
"""
moment = "coalesce(evidence.created_at, evidence.imported_at)"
def common_for(connection: sqlite3.Connection) -> str:
has_durable_queue = connection.execute(
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' "
"AND name IN ('memory_write_jobs','memory_write_job_evidence')"
).fetchone()[0]
if int(has_durable_queue) != 2:
return common_base
return (
common_base
+ """
AND NOT EXISTS (
SELECT 1
FROM memory_task_inputs AS inputs, json_each(inputs.refs_json) AS ref
JOIN memory_write_job_evidence AS linked
ON linked.evidence_id=json_extract(ref.value, '$.evidence_id')
JOIN memory_write_jobs AS job ON job.job_id=linked.job_id
WHERE inputs.task_id=task.task_id
AND job.state IN (
'pending', 'running', 'unknown',
'waiting_for_permission', 'waiting_for_model'
)
)
"""
)
def run(connection: sqlite3.Connection, *, defer: bool) -> list[sqlite3.Row]:
common = common_for(connection)
in_window = (
"1"
if extraction_since is None
@@ -1361,6 +1480,7 @@ class MemoryTaskStore:
def has_defer_candidate(connection: sqlite3.Connection) -> bool:
if extraction_since is None:
return False
common = common_for(connection)
row = connection.execute(
f"""
SELECT 1
@@ -2077,6 +2197,7 @@ class MemoryTaskStore:
*,
limit: int = 128,
after_cursor: tuple[str, str] | None = None,
exclude_error_codes: tuple[str, ...] = (),
) -> tuple[MemoryTask, ...]:
"""List `unknown` tasks that still need an external inquiry.
@@ -2091,6 +2212,9 @@ class MemoryTaskStore:
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be positive")
excluded = tuple(dict.fromkeys(exclude_error_codes))
if not all(isinstance(code, str) and code for code in excluded):
raise ValueError("exclude_error_codes must contain non-empty strings")
if after_cursor is not None:
if (
not isinstance(after_cursor, tuple)
@@ -2104,13 +2228,19 @@ class MemoryTaskStore:
else:
cursor_clause = ""
params = ()
if excluded:
placeholders = ",".join("?" for _ in excluded)
excluded_clause = f"AND COALESCE(last_error_code, '') NOT IN ({placeholders})"
else:
excluded_clause = ""
with self._connect() as connection:
rows = connection.execute(
"SELECT * FROM memory_tasks "
"WHERE status=? "
f"{cursor_clause} "
f"{excluded_clause} "
"ORDER BY updated_at, task_id LIMIT ?",
(MemoryTaskStatus.UNKNOWN, *params, limit),
(MemoryTaskStatus.UNKNOWN, *params, *excluded, limit),
).fetchall()
return tuple(self._to_task(row) for row in rows)
@@ -2142,20 +2272,26 @@ class MemoryTaskStore:
"""
now = datetime.now(UTC).isoformat()
with self._connect(writing=True) as connection:
task_only = _task_only_unknown_clause(connection)
blocked = ",".join("?" for _ in NON_RETRYABLE_POST_WRITE_CODES)
changed = connection.execute(
"""
f"""
UPDATE memory_tasks
SET status=?, runner_token=NULL, attempts=0,
last_error_code='model_content_manual_retry_requested', updated_at=?
WHERE task_id IN (
SELECT task_id FROM memory_tasks
WHERE status=? ORDER BY updated_at, task_id LIMIT ?
WHERE status=?
AND COALESCE(last_error_code, '') NOT IN ({blocked})
{task_only}
ORDER BY updated_at, task_id LIMIT ?
)
""",
(
MemoryTaskStatus.PENDING,
now,
MemoryTaskStatus.UNKNOWN,
*NON_RETRYABLE_POST_WRITE_CODES,
limit,
),
)
@@ -2177,14 +2313,18 @@ class MemoryTaskStore:
raise ValueError("limit must be positive")
now = datetime.now(UTC).isoformat()
with self._connect(writing=True) as connection:
task_only = _task_only_unknown_clause(connection)
blocked = ",".join("?" for _ in NON_RETRYABLE_POST_WRITE_CODES)
changed = connection.execute(
"""
f"""
UPDATE memory_tasks
SET status=?, runner_token=NULL,
last_error_code='model_content_unknown_auto_retry', updated_at=?
WHERE task_id IN (
SELECT task_id FROM memory_tasks
WHERE status=? AND attempts=1 AND updated_at <= ?
AND COALESCE(last_error_code, '') NOT IN ({blocked})
{task_only}
ORDER BY updated_at, task_id LIMIT ?
)
""",
@@ -2193,6 +2333,7 @@ class MemoryTaskStore:
now,
MemoryTaskStatus.UNKNOWN,
older_than.isoformat(),
*NON_RETRYABLE_POST_WRITE_CODES,
limit,
),
)
@@ -200,10 +200,12 @@ def _trigger_sql() -> list[tuple[str, str]]:
CREATE TRIGGER IF NOT EXISTS memory_visibility_dirty_on_link_update
AFTER UPDATE ON governed_memory_evidence
BEGIN
INSERT OR IGNORE INTO memory_visibility_dirty (memory_id)
VALUES (NEW.memory_id);
INSERT OR IGNORE INTO memory_visibility_dirty (memory_id)
VALUES (OLD.memory_id);
INSERT INTO memory_visibility_dirty (memory_id)
VALUES (NEW.memory_id)
ON CONFLICT(memory_id) DO NOTHING;
INSERT INTO memory_visibility_dirty (memory_id)
VALUES (OLD.memory_id)
ON CONFLICT(memory_id) DO NOTHING;
END;""",
),
(
@@ -294,6 +296,10 @@ def install_triggers(db: sqlite3.Connection) -> None:
str(row["name"])
for row in cursor.execute("SELECT name FROM sqlite_master WHERE type='table'")
}
# CREATE TRIGGER IF NOT EXISTS cannot repair the legacy definition. Replace
# this derived-cache trigger transactionally on every startup so old
# libraries get the row-level conflict handling without a schema migration.
cursor.execute("DROP TRIGGER IF EXISTS memory_visibility_dirty_on_link_update")
for table, statement in _trigger_sql():
if table not in present:
continue
@@ -303,11 +309,15 @@ def install_triggers(db: sqlite3.Connection) -> None:
def startup(database_path: Path) -> None:
"""Idempotent: create the derived tables and triggers. No version change.
Runs through the write connection because the triggers are schema. It is
one short transaction and happens once per start.
Runs through the write connection because the triggers are schema. The
trigger replacement is one short transaction and happens once per start.
"""
with main_database(database_path).write() as db:
with main_database(database_path).write() as db, db:
ensure_tables(db)
# ensure_tables uses executescript(), which commits any pending legacy
# transaction. Begin only afterward so replacing the installed trigger
# is one atomic schema change and a failed CREATE restores the old one.
db.execute("BEGIN IMMEDIATE")
install_triggers(db)
@@ -0,0 +1,404 @@
"""Persistent, content-free accounting for precipitation model requests."""
from __future__ import annotations
import asyncio
import json
import logging
from collections.abc import Iterator
from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass
from datetime import UTC, datetime
from importlib import import_module
from pathlib import Path
from typing import TypedDict
from uuid import uuid4
import httpx
from one_memory.database_access import main_database
from one_memory.memory_server.migrations import model_request_audit_ddl
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class ModelRequestAuditContext:
request_id: str
_audit_context: ContextVar[ModelRequestAuditContext | None] = ContextVar(
"one_memory_model_request_audit_context",
default=None,
)
@contextmanager
def audit_precipitation_request(request_id: str) -> Iterator[None]:
"""Enable physical-request accounting for one memory precipitation."""
token = _audit_context.set(ModelRequestAuditContext(request_id=request_id))
try:
yield
finally:
_audit_context.reset(token)
def _current_prompt_stage() -> str:
try:
resolver = getattr( # noqa: B009 - optional SDK version boundary
import_module("hy_memory.utils.model_call_context"),
"current_model_call_stage",
)
except (ImportError, AttributeError):
return "other"
return str(resolver())
class ModelRequestAuditStore:
def __init__(self, database: Path) -> None:
self.database = database
with main_database(database).write() as connection, connection:
for statement in model_request_audit_ddl("sqlite").split(";"):
if statement.strip():
connection.execute(statement)
def start(
self,
*,
request_id: str,
profile_id: str,
profile_revision: int,
provider_kind: str,
model: str,
request_kind: str,
prompt_stage: str,
) -> str:
audit_id = str(uuid4())
with main_database(self.database).write() as connection, connection:
connection.execute(
"""
INSERT INTO model_request_audit (
audit_id,request_id,profile_id,profile_revision,provider_kind,
model,request_kind,prompt_stage,started_at,outcome
) VALUES (?,?,?,?,?,?,?,?,?,?)
""",
(
audit_id,
request_id,
profile_id,
profile_revision,
provider_kind,
model,
request_kind,
prompt_stage,
datetime.now(UTC).isoformat(),
"in_flight",
),
)
return audit_id
def finish(
self,
audit_id: str,
*,
outcome: str,
http_status: int | None,
error_type: str | None = None,
provider_request_id: str | None = None,
prompt_tokens: int | None = None,
completion_tokens: int | None = None,
total_tokens: int | None = None,
cached_prompt_tokens: int | None = None,
reasoning_tokens: int | None = None,
) -> None:
with main_database(self.database).write() as connection, connection:
connection.execute(
"""
UPDATE model_request_audit
SET completed_at=?,outcome=?,http_status=?,error_type=?,
provider_request_id=?,prompt_tokens=?,completion_tokens=?,
total_tokens=?,cached_prompt_tokens=?,reasoning_tokens=?
WHERE audit_id=?
""",
(
datetime.now(UTC).isoformat(),
outcome,
http_status,
error_type,
provider_request_id,
prompt_tokens,
completion_tokens,
total_tokens,
cached_prompt_tokens,
reasoning_tokens,
audit_id,
),
)
def report(
self,
*,
since: str | None = None,
request_id: str | None = None,
limit: int = 200,
) -> dict[str, object]:
clauses: list[str] = []
params: list[object] = []
if since is not None:
clauses.append("started_at>=?")
params.append(since)
if request_id is not None:
clauses.append("request_id=?")
params.append(request_id)
where = f"WHERE {' AND '.join(clauses)}" if clauses else ""
with main_database(self.database).read() as connection:
summary = connection.execute(
f"""
SELECT COUNT(*) AS request_count,
SUM(CASE WHEN outcome='succeeded' THEN 1 ELSE 0 END) AS succeeded,
SUM(CASE WHEN outcome='failed' THEN 1 ELSE 0 END) AS failed,
SUM(CASE WHEN outcome='in_flight' THEN 1 ELSE 0 END) AS in_flight,
SUM(prompt_tokens) AS prompt_tokens,
SUM(completion_tokens) AS completion_tokens,
SUM(total_tokens) AS total_tokens,
SUM(cached_prompt_tokens) AS cached_prompt_tokens,
SUM(reasoning_tokens) AS reasoning_tokens,
AVG(prompt_tokens) AS avg_prompt_tokens,
AVG(completion_tokens) AS avg_completion_tokens
FROM model_request_audit {where}
""",
tuple(params),
).fetchone()
rows = connection.execute(
f"""
SELECT audit_id,request_id,profile_id,profile_revision,provider_kind,
model,request_kind,prompt_stage,started_at,completed_at,outcome,http_status,
error_type,provider_request_id,prompt_tokens,completion_tokens,
total_tokens,cached_prompt_tokens,reasoning_tokens
FROM model_request_audit {where}
ORDER BY started_at DESC,audit_id DESC LIMIT ?
""",
(*params, limit),
).fetchall()
minute_rows = connection.execute(
f"""
SELECT substr(started_at,1,16) || ':00Z' AS minute,
prompt_stage,
COUNT(*) AS request_count,
SUM(prompt_tokens) AS prompt_tokens,
SUM(completion_tokens) AS completion_tokens,
SUM(total_tokens) AS total_tokens,
AVG(prompt_tokens) AS avg_prompt_tokens,
AVG(completion_tokens) AS avg_completion_tokens
FROM model_request_audit {where}
GROUP BY substr(started_at,1,16),prompt_stage
ORDER BY minute,prompt_stage
""",
tuple(params),
).fetchall()
keys = (
"audit_id",
"request_id",
"profile_id",
"profile_revision",
"provider_kind",
"model",
"request_kind",
"prompt_stage",
"started_at",
"completed_at",
"outcome",
"http_status",
"error_type",
"provider_request_id",
"prompt_tokens",
"completion_tokens",
"total_tokens",
"cached_prompt_tokens",
"reasoning_tokens",
)
summary_keys = (
"request_count",
"succeeded",
"failed",
"in_flight",
"prompt_tokens",
"completion_tokens",
"total_tokens",
"cached_prompt_tokens",
"reasoning_tokens",
"avg_prompt_tokens",
"avg_completion_tokens",
)
return {
"summary": {
key: (
round(float(value), 2)
if key.startswith("avg_") and value is not None
else int(value)
if value is not None
else 0
)
for key, value in zip(summary_keys, summary, strict=True)
},
"requests": [
dict(zip(keys, tuple(row), strict=True))
for row in rows
],
"by_minute": [
{
"minute": str(row[0]),
"prompt_stage": str(row[1]),
"request_count": int(row[2]),
"prompt_tokens": int(row[3] or 0),
"completion_tokens": int(row[4] or 0),
"total_tokens": int(row[5] or 0),
"avg_prompt_tokens": round(float(row[6] or 0), 2),
"avg_completion_tokens": round(float(row[7] or 0), 2),
}
for row in minute_rows
],
}
def _non_negative_int(value: object) -> int | None:
return value if type(value) is int and value >= 0 else None
class _ResponseUsage(TypedDict):
provider_request_id: str | None
prompt_tokens: int | None
completion_tokens: int | None
total_tokens: int | None
cached_prompt_tokens: int | None
reasoning_tokens: int | None
def _empty_usage() -> _ResponseUsage:
return {
"provider_request_id": None,
"prompt_tokens": None,
"completion_tokens": None,
"total_tokens": None,
"cached_prompt_tokens": None,
"reasoning_tokens": None,
}
def _response_usage(content: bytes) -> _ResponseUsage:
try:
body = json.loads(content)
except (json.JSONDecodeError, UnicodeDecodeError):
return _empty_usage()
if not isinstance(body, dict):
return _empty_usage()
usage = body.get("usage")
if not isinstance(usage, dict):
usage = {}
prompt_details = usage.get("prompt_tokens_details")
completion_details = usage.get("completion_tokens_details")
cached = (
prompt_details.get("cached_tokens")
if isinstance(prompt_details, dict)
else usage.get("prompt_cache_hit_tokens")
)
reasoning = (
completion_details.get("reasoning_tokens")
if isinstance(completion_details, dict)
else None
)
return {
"provider_request_id": body.get("id") if isinstance(body.get("id"), str) else None,
"prompt_tokens": _non_negative_int(usage.get("prompt_tokens")),
"completion_tokens": _non_negative_int(usage.get("completion_tokens")),
"total_tokens": _non_negative_int(usage.get("total_tokens")),
"cached_prompt_tokens": _non_negative_int(cached),
"reasoning_tokens": _non_negative_int(reasoning),
}
def _request_kind(request: httpx.Request) -> str:
path = request.url.path.rstrip("/")
if path.endswith("/chat/completions"):
return "chat_completion"
if path.endswith("/embeddings"):
return "embedding"
return "model_http"
def _request_model(content: bytes, fallback: str) -> str:
try:
body = json.loads(content)
except (json.JSONDecodeError, UnicodeDecodeError):
return fallback
model = body.get("model") if isinstance(body, dict) else None
return model if isinstance(model, str) and model else fallback
class AuditedModelTransport(httpx.AsyncBaseTransport):
"""Record one row for each physical precipitation HTTP attempt."""
def __init__(
self,
store: ModelRequestAuditStore,
*,
profile_id: str,
profile_revision: int,
provider_kind: str,
model: str,
inner: httpx.AsyncBaseTransport | None = None,
) -> None:
self.store = store
self.profile_id = profile_id
self.profile_revision = profile_revision
self.provider_kind = provider_kind
self.model = model
self.inner = inner or httpx.AsyncHTTPTransport(retries=0, trust_env=False)
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
context = _audit_context.get()
if context is None:
return await self.inner.handle_async_request(request)
content = await request.aread()
audit_id = await asyncio.to_thread(
self.store.start,
request_id=context.request_id,
profile_id=self.profile_id,
profile_revision=self.profile_revision,
provider_kind=self.provider_kind,
model=_request_model(content, self.model),
request_kind=_request_kind(request),
prompt_stage=_current_prompt_stage(),
)
try:
response = await self.inner.handle_async_request(request)
except BaseException as error:
await asyncio.to_thread(
self.store.finish,
audit_id,
outcome="failed",
http_status=None,
error_type=type(error).__name__,
)
raise
content = await response.aread()
usage = _response_usage(content)
try:
await asyncio.to_thread(
self.store.finish,
audit_id,
outcome="succeeded" if response.is_success else "failed",
http_status=response.status_code,
error_type=None if response.is_success else f"http_{response.status_code}",
**usage,
)
except Exception:
# The durable in_flight row is more truthful than hiding the model
# response after it has already incurred cost.
logger.exception("model_request_audit_finish_failed audit_id=%s", audit_id)
return response
async def aclose(self) -> None:
await self.inner.aclose()
@@ -87,7 +87,7 @@ def build_acceptance_pack(
"""
SELECT DISTINCT e.evidence_id
FROM evidence e
WHERE e.role = 'user' AND e.written_by = 'person'
WHERE e.role = 'user'
AND LENGTH(e.verbatim) <= 300
AND (
e.verbatim LIKE '%我是一名%' OR e.verbatim LIKE '%我是一个%'
@@ -118,7 +118,7 @@ def build_acceptance_pack(
for row in connection.execute(
"""
SELECT evidence_id FROM evidence
WHERE role = 'user' AND written_by != 'person'
WHERE role != 'user'
"""
):
strata["attribution_edge"].add(row["evidence_id"])
@@ -138,11 +138,7 @@ def build_acceptance_pack(
quota,
)
if len(selected) < sample_size:
eligible = (
evidence_id
for evidence_id, row in evidence.items()
if row["role"] == "user" and row["written_by"] == "person"
)
eligible = (evidence_id for evidence_id, row in evidence.items() if row["role"] == "user")
_extend_unique(selected, _stable_order(eligible, seed=f"{seed}:fill"), sample_size)
selected = selected[:sample_size]
if len(selected) < sample_size:
@@ -163,7 +159,11 @@ def build_acceptance_pack(
row_id=f"{index:03d}",
evidence_id=evidence_id,
source_id=source["source_id"],
written_by=source["written_by"],
written_by=(
"person"
if source["role"] == "user"
else ("agent" if source["role"] == "assistant" else source["role"])
),
character_count=len(source["verbatim"]),
content_hash=source["content_hash"],
strata=tuple(
@@ -2,7 +2,7 @@
from collections.abc import Sequence
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord, WrittenBy
from one_memory.imports.models import EvidenceContentKind, EvidenceRecord
from one_memory.imports.sources.parsing import is_host_generated_user_turn, strip_host_wrappers
from one_memory.input_sanitization import sanitize_model_input
from one_memory.memory_processing import _is_hy_input_turn
@@ -15,10 +15,7 @@ def identity_content_ready(record: EvidenceRecord) -> bool:
# cleaning requires a source map and remains explicitly uncovered for now.
return (
record.content_kind is EvidenceContentKind.RAW_CONVERSATION
and (
(record.role == "user" and record.written_by is WrittenBy.PERSON)
or (record.role == "assistant" and record.written_by is WrittenBy.AGENT)
)
and record.role in {"user", "assistant"}
and _is_hy_input_turn(record)
and not is_host_generated_user_turn(record.verbatim)
and strip_host_wrappers(record.verbatim) == record.verbatim.strip()
@@ -10,7 +10,6 @@ from one_memory.imports.models import (
MAX_VERBATIM_CHARACTERS,
EvidenceContentKind,
EvidenceRecord,
WrittenBy,
)
from one_memory.imports.sources.parsing import (
is_host_generated_user_turn,
@@ -164,10 +163,10 @@ def prepare_mapped_content(record: EvidenceRecord) -> PreparedMappedContent:
The returned text is transient; only metadata is suitable for an outer manifest.
That manifest must independently bind Evidence identity/revision/hash and grants.
"""
if record.content_kind is not EvidenceContentKind.RAW_CONVERSATION or not (
(record.role == "user" and record.written_by is WrittenBy.PERSON)
or (record.role == "assistant" and record.written_by is WrittenBy.AGENT)
):
if record.content_kind is not EvidenceContentKind.RAW_CONVERSATION or record.role not in {
"user",
"assistant",
}:
raise MappedContentError("mapped_content_ineligible")
original = record.verbatim
if len(original) > MAX_VERBATIM_CHARACTERS:
@@ -11,7 +11,7 @@ from datetime import UTC
from one_memory import ingest_journal as journal
from one_memory.context_raw import _check, _connection
from one_memory.imports.models import EvidenceRecord, WrittenBy
from one_memory.imports.models import EvidenceRecord
from one_memory.imports.storage import EvidenceStore
from one_memory.retrieval.index_store import RetrievalIndexStore
from one_memory.retrieval.models import EvidenceSpan, RawEvidence
@@ -166,7 +166,7 @@ def materialize_raw_candidates(
source_id=record.source_id,
conversation_id=record.conversation_id,
role=record.role,
written_by="person" if record.written_by is WrittenBy.PERSON else "agent",
written_by="person" if record.role == "user" else "agent",
event_time=event_time,
spans=tuple(spans),
matched_by=("semantic",),
@@ -140,8 +140,6 @@ async def search_within(
return (), "failed"
error = pending.exception()
if error is not None:
if str(error) == "hy_memory_store_busy":
return (), "timed_out"
return (), "failed"
found = pending.result()
normalized = normalize_semantic_hits(found)
Loaded 100 of 281 files, more files were not shown because too many files have changed in this diff. Show more