Agent Memory

The problem #

Memory frameworks put memory in opaque databases: you cannot inspect, diff, migrate or correct it, deleting it reliably is hard, and the index and the embedding model are locked in. And memory itself is an attack surface: in one 2026 study persistent-memory poisoning reached an average attack success rate of 50.46% across two agents, and another study used one carefully crafted email to succeed against both file-system and vector memory backends.

Meanwhile, “which memory architecture is best” is nowhere near settled: benchmarks contradict each other. A design that bets on no particular retrieval algorithm is the one that can survive three years of change.

How it works #

• Notes are the truth. Markdown with YAML frontmatter: kind (episodic, semantic, procedural, preference), subject, sources, confidence, valid_from, valid_to, supersedes, origin, reviewed.
• Update, do not overwrite. A new fact supersedes the old one; valid time and record time are kept apart, because facts change and common benchmarks barely test that.
• Indexes are a cache. BM25 (for example SQLite FTS5), vectors (several can coexist) and an optional graph; every derivative carries its recipe and inputs. Changing the embedding model is a background recomputation, and the old index is kept until the new one meets the golden recall tests.
• Two kinds of derivation. Deterministic ones (chunking, BM25, embeddings from a fixed model) are reproducible; generative ones (LLM summaries, extracted facts) can only be “regenerated and compared”, so they are marked generated: true and never serve as the only copy.
• Write path. nas_remember writes a note (with sources) to the current branch, and taint is inherited. A note from a session with external taint goes into quarantine, marked unreviewed: it can be recalled but is not autoloaded and does not enter MEMORY.md.
• Read path. nas_recall = search, candidates, a budgeted view plus evidence pointers, and it shows each note’s source and review state.
• Maintenance jobs (overnight or when idle): deduplicate, consolidate episodes into semantic facts, flag contradictions, retire outdated facts, rebuild indexes. Jobs produce proposals on a branch for a person or an agent to review; they never delete the raw journal; the “regret rate” (the share of proposals that get rolled back) is measured.
• Golden recall tests. For each Home, a set of “question, expected note” pairs that run automatically when the model or the indexer changes.

Start small #

• memory/notes/*.md with frontmatter, and SQLite FTS5 for search.
• Run ten golden questions in CI.
• Vector indexes and maintenance jobs can come later; get “provenance” and “only reviewed notes autoload” right first.

Interface sketch #

Illustrative only. Field names may change before specification v0.3.

---
id: mem_4kq2
kind: semantic
subject: customer/acme
valid_from: 2026-10-02
valid_to: null
confidence: 0.8
origin: agent
reviewed: true
sources:
  - nas://home-box/agents/cleo/journal/2026-10-02.md@9f2c1de#L12-L18
supersedes: [mem_1xa9]
---
Acme renewed for 24 months; the champion moved from Dana to Luis.

What the specification asks for #

• Notes MUST be readable without the Memory layer (they are plain files in the Store).
• Every derivative MUST carry its recipe and inputs; an index MUST be disposable and rebuildable.
• Notes written by a tainted session MUST NOT be autoloaded before they are reviewed.
• A new fact MUST replace an old one with supersedes instead of overwriting.
• Maintenance jobs MUST work on a branch and only propose; they MUST NOT delete the raw journal.
• Every Home SHOULD carry golden recall tests and run them whenever the model or the indexer changes.

Borrowed, and what is new #

What follows is all established thinking, and OpenClaw’s memory architecture already does provenance labels, per-tier quarantine and taint propagation during consolidation. What is new: OpenClaw does it inside one runtime; we propose doing it in shared storage so any runtime benefits, and demoting the index to a disposable cache so the retrieval algorithm can change without losing memory.

• Letta Context Repositories
• Letta: is a filesystem all you need?
• OpenClaw memory architecture
• Long-term memory security survey
• Graphiti
• MemGPT

Honest limits #

• Generative derivatives can never be reproduced bit for bit.
• A maintenance job can “harden” a mistake into a fact, which is why it only proposes and never edits directly.
• Which memory architecture is best remains unsettled.

Open questions #

• When should an agent write memory itself, and when should a maintenance job?
• What is the regression standard across model versions?

Questions #

Related concepts #

• Budgeted Reads Read a file as an outline, a section, a line range or in full, degrading to fit the budget.
• Cascade Forget Delete derivatives along lineage, handle history by retention class, and report what it could not do.
• Trust Labels Two small labels plus session taint enforce the Rule of Two.