We built a textbook memory system for our AI apprentice. It worked. It also felt dead. The fix was admitting that memory was organized around the wrong axis — and every finding that got us there.
A few months ago we measured how well Apprentice remembered conversations, with a simple test: take a public long-term-memory benchmark, sample it, score ourselves honestly.
We scored about 15%.
Not 15% behind the leader. Fifteen percent. Apprentice had been built to learn how you work — your formulas, your voice, your standards — and it did that well. But ask it what you told it last Tuesday and it had nothing, because episodic memory simply wasn't there. Dense-vector search over raw history, no keyword leg, no reranking, no memory objects at all. The benchmark didn't find a weakness. It found an absence.
So we built the thing properly. And then the interesting part happened: the properly-built thing worked, and it still felt dead. This is the story of both halves — the textbook system, and the layer we had to invent when the textbook ran out.
TLDR: The whole layer is now open source — store, retrieval, the self-audit that changed our minds, the tests — at github.com/Reppin123/episodic-memory.
We studied the best open reference we could find — TrueMemory, which posts 93% on LoCoMo — and ported its best ideas (it's AGPL; ideas, never code) into our own lane:
It shipped, it recalled facts across sessions on day one, and real use immediately taught us two lessons the design docs hadn't: the archive button didn't disconnect the session, so consolidation never fired for archived chats — and in streaming voice mode, the assistant's own replies were never captured at all. Half the conversation, silently missing. Both fixed within the day. Capture paths fail quietly; test the exits, not just the happy path.
A third lesson was about boundaries. Our agent already had one memory — the rulebook of how you work, loaded into every prompt. The new facts started leaking into it: someone's favorite basketball team ended up in the always-on system prompt. The fix became a design principle: rules are how you work, always on; episodic memory is what's true in your world, retrieved when relevant. Two different systems, because they answer two different questions.
Months in, the store held 826 facts across 371 entities. Recall worked. Contradictions superseded cleanly. And using it still felt like talking to someone with a filing cabinet instead of a memory.
So we audited it — the real store, not a test fixture — and wrote down what we found, unflattered:
The audit did not find a broken system. It found a system faithfully organized around the wrong axis.
Every memory system we'd studied — ours included — indexes by subject: entities, people, things. But nobody sits down at their desk and asks "who is X?" all day. The question that actually opens a work session is "where were we?" Humans file memory by activity: the fund model, the launch, the bug hunt. The thread, not the noun.
That mismatch was the deadness. Facts without a thread to belong to are trivia. And no reference we could find had this layer — systems organize by entity and scope; the effort axis just isn't there. So this part was our own bet.
We call it the missing middle: three objects between atomic facts and entity profiles.
Workstreams — the threads themselves. Each new episode embeds and lands near its neighbors: close enough attaches (and nudges the workstream's centroid), an ambiguous middle band goes to a small-model judge, genuinely new work spawns a fresh workstream. A workstream starts as a hypothesis and is only confirmed when a second session attaches — one conversation is a topic; two is a thread.
Episodes — a one-to-three-sentence summary of what each session actually was, so a workstream reads like a story, not a heap.
Open loops — the unfinished things. Detection is deliberately strict: most sessions produce zero to two, and an empty list is a correct answer. Resolution is stricter — when unsure, a loop stays open. False continuity is worse than none.
Feeding them is a salience router with a rule we consider the layer's constitution: re-level, never delete. Every fact is routed durable, episodic (demoted into its episode's detail — kept, just not top-shelf), or dropped — and dropping is reserved for pure filler. Calibrated against the 879 real facts in the store at the time: 87% durable, 9% demoted, 4% dropped — and the "currently on slide 2" class of noise finally stopped being a first-class memory.
The payoff is one block, once per session, at the top of the first turn: [Ongoing work] — your two most active confirmed workstreams and their open loops. Hypotheses never surface. The agent opens with "last time you were reworking the fund model — the sensitivity table is still open" instead of "how can I help you today?" That single block is the felt difference between a database and a colleague.
Open-sourcing a system means shipping its skeletons. Two are worth confessing. Our own audit caught a live scoring bug: the recency term was multiplied by the fusion score, so recency could never lift a fresh-but-unfashionable fact — the exact opposite of its documented job. It's fixed in the open-source release, with the audit trail in the comments. And "forget everything" originally wiped facts but left workstreams standing — a privacy bug in the feature whose entire job is deletion. Deletion paths deserve the same paranoia as capture paths.
The numbers that run the system, for the curious: duplicate threshold 0.88, supersede band 0.72, RRF k=60, recency half-life 30 days at weight 0.15, workstream attach 0.55 with a 0.38–0.55 judge band, recurrence bonus +0.10 per repeat capped at +0.40. None of them are sacred. All of them are honest.
One honesty note, same as last time: these findings come from one real store on one founder's Mac, not a public benchmark run. The benchmark harness that produced the original 15% exists, and running this full system against it — properly, publishable — is on the list.
The entity graph still has no edges. Salience weights are hand-set priors, not learned. The vector search is brute-force cosine with a validated sqlite-vec path measured 12–14× faster and bit-identical, waiting to be defaulted on. Goals and a self-model are designed and unbuilt. And somewhere past all of that is the version that consolidates memory the way people do: asleep, between sessions, connecting threads on its own.
Facts are what your agent knows. Workstreams are what you're actually doing. An agent that holds the first without the second knows everything and remembers nothing.
That's the bet. That's the missing middle.
Code: github.com/Reppin123/episodic-memory · Product: withapprentice.com · Companion piece: Thought search: memory as a filesystem, minus the cloud · The reference that started it: TrueMemory