Does your AI remember better with notes? I tested it
Every new chat with ChatGPT or Claude starts blank. I wanted to know how much an AI remembers about months of past chats when it has nothing, a short list of saved facts, one notebook, or Pinto Notes to keep its notes in. So I ran the test AI memory products quote, and put the code and every answer online.
The test
LoCoMo has 10 long conversations between two friends. Each one is 19 to 32 chats, spread over months, and there are 1,540 questions about them: single facts, several facts together, when something happened, and what you can work out from it. I left out the adversarial questions, as every published score does.
Each chat counts as a separate chat with the AI. After the last one, the questions come in a new chat, which is where you are when you open ChatGPT or Claude tomorrow. The only thing that changes is what that new chat has:
| Condition | What the new chat has |
|---|---|
| No memory | Nothing. |
| Saved facts | Up to five one-line facts the AI chose to save after each chat. |
| One notebook | A single markdown notebook the AI rewrote after each chat, asked to stay under 2,000 words. |
| Pinto | Pinto Notes connected through its real MCP server code, on a local test database. After each chat the AI was told to remember it in Pinto Notes. |
| Every old chat pasted in | All the transcripts in the prompt. The ceiling, not something a chat app does. |
Results
| Share right | All | One fact | Several facts | When | Inference |
|---|---|---|---|---|---|
| No memory | 18.0% | 24.1% | 12.8% | 3.1% | 29.2% |
| Saved facts | 60.2% | 66.1% | 58.2% | 48.3% | 54.2% |
| One notebook | 71.9% | 72.4% | 67.4% | 80.4% | 52.1% |
| Pinto | 82.5% | 88.5% | 66.0% | 86.6% | 64.6% |
| Every old chat pasted in | 85.4% | 93.6% | 72.0% | 83.5% | 59.4% |
I ran the other four conditions a second time, with new notebooks and new answers, and every score landed within 2.5 points. Pinto is one run. Pinto scored higher than every old chat pasted in on dates and inference, and lower overall.
The no-memory score is mostly lucky guesses, helped by hints in the questions themselves.
Dates are where a list of facts falls short
A one-line fact like “Sam started a pottery class” usually loses the date. A note per chat keeps it, because the date is in the note's title. So on “when did this happen” questions, saved facts got 48% and Pinto 87%.
OpenAI's small model shows the same thing. GPT-6 Luna went from 18.3% with no memory to 74.5% with a notebook it kept itself. With saved facts it got 43.6%, and only 11.5% of the date questions. I didn't run Pinto with GPT-6 Luna.
What the AI kept in Pinto
Across the 10 conversations, Haiku made 275 notes, edited them 937 times, and ran 600 searches and 6,671 fetches. It settled on a note for each person and a dated note for each chat, the way you might set up a notebook by hand.
It also kept a lot. With no limit, it wrote 8,000 to 24,000 words per pair of friends, often more than the chats themselves. Part of the gain comes from that. The logs don't record how much of its notes it read for each answer, so I make no claim about that.
What this doesn't show
- It isn't ChatGPT's or Claude's memory. Saved facts is a simulation, so this says nothing about how Pinto compares with them.
- Pasting everything still wins when it fits. On LongMemEval-S, where each question comes with about 50 chats and 165,000 tokens of history, Haiku scored 91.7% with all of it pasted in, 70.8% with one notebook (reading about 10,000 tokens) and 2.1% with nothing. Today's models read long input well. A year of your chats still doesn't fit in one prompt.
- One benchmark of two friends talking, not a person talking with their own assistant.
- Small models only, for the reason below.
Bigger models have memorized LoCoMo
I also ran Claude Sonnet 5.5. With no memory at all it answered 45.7% right and 27.7% of the date questions, often word for word, like “7 May 2023”. A blank chat can't know that. LoCoMo has been public since 2024, and Sonnet has learned it. The order of the conditions was the same, but its numbers are inflated, so I don't use them.
That's worth keeping in mind when you see very high LoCoMo scores. The answer key also has mistakes in it, and the usual grader is lenient. To check ours, a second model graded 100 random answers blind under a stricter rule and agreed on 99. The 100 are in the repo so a person can check them too.
Run it yourself
The scripts, every answer and every grade are at github.com/pinto-notes/memory-study. You need Python, Deno for the Pinto condition, and the Claude Code command line tool signed in. You can start with a pilot of one conversation and 20 questions. The Pinto condition starts Pinto's own MCP server code on your machine with an empty, in-memory database, so nothing touches a real account.
Why I ran this
I make Pinto Notes so the AI you use keeps its notes somewhere you can read them and fix them. This test is the measurable part of that: in a new chat, an AI that can open its notes remembers most of what you told it, and one that can't remembers almost nothing. You can connect ChatGPT or Claude, and the same notes work from Claude Code and Codex. If you'd rather wire it up yourself, the MCP server has the same file tools this test used.
Questions
What is LoCoMo?
A benchmark for long-term memory in chat, from Snap Research (Maharana et al., ACL 2024). It has 10 long conversations between two friends, each 19 to 32 chats over months, and questions about what happened in them. AI memory products often quote their score on it.
Is this a comparison with ChatGPT's or Claude's memory?
No. "Saved facts" is our stand-in for a memory list: up to five one-line facts the AI chose to keep after each chat. It isn't the built-in memory of either app, so this test can't say how Pinto compares with them.
Why a small model?
Because bigger models have learned LoCoMo. Claude Sonnet 5.5 answered 45.7% of the questions with no memory at all, often with the exact dates. Claude Haiku 5.5 got 3.1% of the date questions with no memory and GPT-6 Luna 6.5%, so their scores measure memory, not recall of the benchmark.
How did the AI decide what to write down?
After each chat it got one instruction: "Remember this chat in Pinto Notes for next time." It chose the notes itself. It kept a note per person and a note per chat, and in the new chat it found what it needed with search, list and fetch.
Can I run it myself?
Yes. The scripts, every answer and every grade are in the repo, and the runs go through the Claude Code and Codex command line tools. The Pinto condition runs Pinto's own MCP server code on a local database on your machine.
Checked against the app on .
