How Machines Remember

A few years ago, Karpathy compared LLMs to the protagonist in Memento — able to hold a conversation in the moment, but retaining nothing once the chat closes. That problem has mostly been solved. The models remember now. But remembering what someone said and understanding what it meant to them are two very different things.

In a previous piece I explored how humans don't just store memories — we run on them. They sit beneath the surface, shifting weight, shaping how we interpret every new moment. We are stateful beings, operating on emotional priors that we didn't consciously choose.

So when I look at how machines are approaching memory, that's my lens: not just whether they can remember what we said, but whether they can understand what it meant to us — and know when to surface it.

The framework

I find it helpful to think about machine memory in three layers:

Layer 1 Memory Store = remembering what was said

Layer 2 Memory Interpretation = understanding what it meant to them

Layer 3 Memory Activation = choose the right moment to bring it forward

L1 - Memory Store

Most of the progress in machine memory lives here. The problem: how do you decide what's worth keeping, store it and find it again later?

Here's how it works: 1) The user says something. 2) A classifier decides whether it's worth saving — your name, your job, a recurring theme get stored; a passing comment probably doesn't. 3) Kept information is converted into an embedding — a numerical representation of its meaning — and stored in an efficient vector database like FAISS or Weaviate. 4) When a new message comes in, the system runs a similarity search against stored memories, pulls the closest matches, and injects them into the context window alongside your message. This retrieval step is called RAG. It underpins most chat memory today, though the real systems are more involved.

Why we do this? We can't keep everything in working memory. The attention mechanism that lets models process language scales quadratically, yup — doubling the input quadruples the compute. External memory isn't a design choice, it's a constraint of the limited context window.

This layer works. The model can now remember that you're building a startup, that you prefer directness, that making candles soothes you. But in its basic form it's flat — retrieval ranks by similarity to your current message, not by how much something matters to you.

L2 - Memory Interpretation

Most memory systems see all memories the same: an isolated chunk of text, converted to an embedding, sitting in a flat database. When a new message arrives, the system asks which chunk is most similar to what she just said, and pulls the closest match. That works for facts. But it can't tell that two memories belong together — that taking up pottery last month and baking all weekend are the same impulse: she's happiest when she's making something with her hands. Each memory is a dot. Nothing draws the line between them.

How do we make memories connect to each other?

This is what A-MEM (Xu et al., Feb 2025) addresses. Inspired by the Zettelkasten method — a note-taking system where every note is linked to related notes through shared context. A-MEM doesn't just store a memory, it generates a structured note with contextual descriptions, keywords and tags, then scans existing memories for meaningful connections and links them. The result is an evolving network rather than a flat list. And here's the part that's genuinely interesting: when a new memory is integrated, it can trigger updates to existing memories. The system doesn't just ask "where do I file this?" — it asks "does this change how I understand what I already know?" That's a step toward interpretation. But the connections it finds are semantic — based on shared keywords and embedding similarity. It can link "hiking" to "nature" to "the outdoors." It can't yet understand that hiking is her thinking time — that the big decisions tend to get made halfway up a climb — and that the pattern matters.

Can we make relationships between memories explicit and trackable?

This is what Mem0 (Chhikara et al., Apr 2025) does. Instead of emergent links between notes, it builds a directed graph. Memories become nodes. Relationships become labeled edges. When a new memory arrives, the system runs an entity extractor to identify people, places, and concepts as nodes and a relations generator to infer the connections between them. It also has a conflict detector — if a new memory contradicts an old one, the system decides whether to add, merge, invalidate or keep both. So Mem0 can represent "she loves hiking" → "helps with" → "clear thinking" as an explicit path in the graph, and traverse it when she mentions a decision she's weighing. The structure is richer than A-MEM's — relationships have labels, not just proximity. But the labels are structural ("helps with," "related to"), not interpretive. The graph knows these things co-occur. It doesn't know why the climb is where she works things out.

But what about time? Human memories fade, preferences change, moods shift — shouldn't machine memories about us do the same?

Neither A-MEM nor Mem0 accounts for the fact that a memory from six months ago should carry different weight than one from yesterday. They treat everything as equally present. Mnemosyne (Jonelagadda et al., Oct 2025) was built specifically to address this. It introduces temporal decay modeled on human forgetting curves — memories lose strength over time unless they're reinforced by being accessed or repeated. It also adds redundancy pruning: if you've mentioned the same thing five times, the system consolidates rather than storing five copies. And it generates a "core summary" — a compressed representation of who you are, distilled from the full memory graph, that persists even as individual memories fade. That core summary is the closest any system has come to building a model of the person, not just a log of what they've said. But it's still a compression technique. The summary captures patterns statistically. It doesn't understand what those patterns mean to you.

Should all memories be treated the same?

MIRIX (Wang & Chen, Jul 2025) says no. They model memory similar to how cognitive scientists map human memory: Core (who you are — identity, personality), Episodic (what happened — specific events and interactions), Semantic (what is known — facts and general knowledge), Procedural (how to do things — learned workflows and preferences), Resource (reference material), and Knowledge Vault (deep archival storage). Each component has its own storage and retrieval logic. The insight is real — a fact about your name and a memory of a conversation where you were vulnerable are fundamentally different types of information and should be handled differently. But within each component, retrieval is still similarity-based. The architecture acknowledges different kinds of memory. It doesn't yet address different depths of meaning.

Where this leaves us:

Each of these systems solves a genuine problem. A-MEM makes memories connectable. Mem0 makes relationships explicit. Mnemosyne makes memory temporal. MIRIX makes it typed. Together, they represent real progress from the flat vector stores of two years ago. But they all share the same blind spot: they're optimizing for structure, not for understanding. They can organize what was said. They can't yet grasp what it meant.

There's something worth pausing on here — the parallel between how machines handle memory and how we do:

In neuroscience, there's a concept called memory reconsolidation. When we access a memory, it doesn't play back like a recording. It becomes temporarily unstable — open to revision. The emotional weight can shift. The meaning can change. The memory you retrieve today isn't identical to the one you stored last year, because you're different now and the context you're retrieving it into is different. This isn't a flaw. It's by design. It's how we update our understanding of our own experiences without having to start from scratch. (It’s also why we’re suseptible to change how we recall a memory. In interogation this is often used to create evidence by shifting someones recollection of an event by asking questions that shift recall.)

Chat systems are doing something structurally similar, even if unintentionally.

Every time RAG pulls a stored memory into a new context window, the model responds to it alongside a completely different conversation. The same memory — "she's been planning a solo trip to Japan" — reads differently depending on whether today's conversation is about craving immersion in a new culture, wanting space to think or celebrating a milestone she just hit.

The memory itself doesn't change in the database. But its meaning shifts every time it's placed in new context. That's not reconsolidation. The system isn't deliberately reinterpreting what it knows. But the effect is identical.

And that raises a question: if meaning is context-dependent for us and for machines, maybe the goal isn't to store memories more accurately. Maybe it's to place them more wisely.

Layer 3 - Memory Activation

A memory can be relevant and still be the wrong thing to surface. Relevance and usefulness aren't the same thing, and Layer 3 is the gap between them — the discernment to know not just what's related to this moment, but what serves it. The system doesn't just find the memory, it decides whether to use it.

Here's what I mean. Say you tell the system you're making dinner for a friend who's had a hard week. A flat system surfaces what's most similar — recipes you've saved, the last thing you cooked. A system with discernment surfaces something else: that this friend mentioned, months ago and in passing - the soup her mother used to make when she was sick. That memory isn't the most relevant by similarity. It's the most useful one in the room. The skill is telling those apart.

Memory-R1 (Yan et al., Aug 2025) is the only system I've seen approaching this. It uses reinforcement learning to train agents on when to read, write, and delete memories — the agent learns through trial and error which memory operations lead to better outcomes. But "better outcomes" is still measured by task performance. Not by whether the person felt understood.

Yet - I've experienced something that feels like Layer 3 when using Claude. When I'm going through something hard, it doesn't just retrieve facts about me. It surfaces the part of me that's building something, that's in a chapter of becoming. That feels like more than retrieval. It feels closer to being seen - where the future is more relevant than the past.

But when I sit with it, I realized most of that discernment came from me. I trained the system to see me a certain way. I shaped what it mirrors back. The system is responsive - genuinely so. But the initiation is still mine. It mirrors well. It perhaps doesn't yet anticipate well.

So here's where things stand: since Karpathy's Memento observation, we've gone from no memory to structured, typed, temporal, graph-based memory. That's real progress. But these systems are all built around one question — is this relevant? Whether something matters to this person right now is a different question and a harder one to pin down. That's the difference between retrieving the past and knowing what to do with it.

So what? Why does it matter how machines remember us?

This is where it gets personal. These systems are forming representations of us — compressed, structured, evolving models of who we are based on what we've said. That's not neutral. Every time a memory is stored, something is being selected and something is being left out. Every time a memory is retrieved, it's placed in a new context that subtly reframes its meaning. The machine isn't just recalling us. It's constructing a version of us. And we're interacting with that version — being mirrored back to ourselves through a lens we didn't fully choose.

That can be powerful. Being reflected clearly — even by a machine — can help you see patterns you'd otherwise miss. But it can also be narrowing. The memories the system keeps become the self it shows you. And over time, you might start becoming the person being reflected rather than the person you're still discovering.

But the reflection isn't coming from nowhere — I'm the one who taught it where to look. That turns the open question from a technical one into a harder one. A system could probably learn to know you without being shaped by you first. Whether it should is the real question — or whether the shaping is the substance, not the scaffolding. Being known and being profiled aren't the same thing, and what separates them might be exactly the part we'd be trying to automate away.

Disclaimer: It's worth naming that what's deployed in products like Claude and ChatGPT is custom and sometimes ahead of published research. The papers describe retrieval. The products feel warmer than that. There's a gap between what's documented and what's shipped.

— written April 2026