LLMs don't remember — and the system prompt is just message[0].
It FEELS like it remembers...
How does it work?
messages = [
]
System
Instructs
"You are a helpful coding assistant."
User
Asks
"How do I sort a list in Python?"
Assistant
Answers
"Use the sorted() function..."
API Call #1
messages = [
]
LLM
Your messages array now:
messages = [
]
↑ You append the response to YOUR array
LLM
No memory!
Already forgot everything about Call #1
API Call #2
messages = [
]
Every previous message re-sent
LLM
Sees it all fresh — zero memory of Call #1
Token cost per API call
You pay per token, and every call re-sends the whole chat.
So long chats cost more with every single turn.
Illustrative token counts. Prices differ by model and change over time.
Every model has a maximum context window
Rough examples of context sizes over time (bars not to scale; check your model's docs):
However big it is, it is always a limit, and a fuller window means a bigger bill per call.
Strategy 1: Drop Old Messages
Dropped 6 messages — room freed up!
Strategy 2: Summarize
Before (8 messages)
2,400 tokens
After (3 messages)
"User is planning a Japan trip. Discussed Tokyo (3 days), food recs (ramen, sushi), hotels in Shinjuku, and Kyoto temples."
380 tokens (saved 84%!)
illustrative numbers