How to Anchor an AI's Memory
Let's talk about how language model memory works.
The Amnesiac
When you first start a new chat, the AI's memory is like a character in a
movie who just got bonked on the head and has amnesia. Somehow
they can speak their language and know how the world works, but they
don't recognize anyone or know what is going on.
This is your AI model — the amnesiac. Yes, it understands the world
least, up until its training data cut-off date), but it has to infer and
assume everything based on what you tell it.
(at
best then, to treat the AI like a new employee on day 1. You've got
to spell things out for it. Things that you know about yourself, your role,
your business — information you use so often that you do so without
thinking — the AI knows none of it.
It's
Memory Within a Chat
As your chat thread progresses, the AI will seem to remember what has
happened so far in the chat. It does have memory, and typically this
memory is limited to the chat session itself. (Caveat: Bing Chat may try
to learn about user preferences and keep track of some things, but
don't assume it remembers much of use between chat sessions.)
The memory within
chat is running.
a
chat works two ways, depending on how long the
Short Term Memory
Conversational AIs have something called a context window which is the
maximum amount of information they can process at once. This is
usually measured in tokens where each token represents a word or
part of a word.
,
Here are some common token limits:
ChatGPT
3.5:
4,000 tokens,
GPT-4: 8,000 tokens,
or
Claude+: 8,000 tokens,
or
2,700 words
5,400 words
or
5,400 words
So, if your conversation is short enough, the AI may be able to see the
entire conversation verbatim. However, even before you reach the
limit, as the conversation gets longer, the AI will start to have a harder
and harder time putting ideas together.
Long Term Memory
As
conversation extends outward, the AI will begin to log information
in chunks inside a separate type of memory, usually a vector database
Now, instead of being able to see the information verbatim, the AI uses
a memory process.
a
.
Explained
in
plain language,
it
looks
like this:
looks at your prompt and recent conversation and identifies
generally what it is about
AI
uses the main concepts present
memory database
AI
to
run
a
search against
its
The memory database returns a certain amount
snippets which are given to the AI as context
relevant
of
This memory system works very well in some cases, but not so well
others. There are things we can do to improve it, however.
in
Using Memory Anchors
As said above, the long-term memory looks at how related ideas are
to each other. And easy way to guarantee that certain information
stays in the memory is to anchor it with specific words.
I
Imagine
in
my
initial
prompt define some aspects
I
of
the
AI's
behavior,
like so:
ASSISTANT: You are WisdomBot. For every challenge, you deconstruct
the challenge into its sub-challenges, and then analyze them for
opportunities and risks.
Above, I've named the AI something specific: WisdomBot. Right after
that, I've given an instruction.
Now, any time want to remind the AI of its core instructions, just
need to use its name. For example, even miles into a conversation,
could say:
I
WisdomBot, what do you think about
during a period of high inflation?
I
a
company trying
to
expand
...and just by referencing its name, the associated instructions are
pulled from the database and included in the context.
I
An anchor can be any concept. It doesn't have to have a unique name.
It just needs to be something you've included that you want to be able
to repeatedly reference.
you're struggling with an AI asking questions like, "Now analyze that
information and tell me what you think?" consider the word choice in
your prompt. Have you previously given instructions on analysis? Is it
clear what that information is referring to?
If
For long conversations, always
instructions.
try to
use recurring themes, words, and