📖 Token Saving Series (4 parts in total): Part 1 (this post) · Part 2: If an Abacus Works, Don’t Boot Up a Supercomputer · Part 3: Organizing a Clean Desk for AI · Part 4: I Installed a Budget Dashboard for AI

You’ve definitely had this experience: you chat with ChatGPT or Claude for an afternoon, and the longer it goes, the more sluggish it gets. It replies slower, suddenly “forgets” what you said earlier, or even starts talking in circles.

Many people think their internet is slow, or the AI has crashed. It hasn’t. Behind this is a truth that most people don’t know, but once you do, it will immediately change how you use it.

Truth 1: AI actually has no “memory”

We intuitively think that AI is like a human, that it “remembers” the conversation as we chat.

It doesn’t.

Every time you hit send, the AI rereads your entire conversation from the very first sentence to the present, all over again, and only then replies with the next sentence. It doesn’t just pick up where it left off; it rereads the whole conversation every single time.

So you can imagine: the longer the conversation, the more it has to reread before answering each sentence. This is why it gets slower the more you chat.

Truth 2: You’re actually constantly paying, you just don’t see the bill

The unit AI uses to process text is called a token (roughly a character or half a word). Every token you input, and every token it outputs, is being billed behind the scenes.

The easiest analogy to remember is: a token is AI’s taxi meter. The longer the distance (word count), the higher the fare.

You don’t see this bill in a subscription-based App, but it shows up in front of you with a different face: that “you have reached your message limit for today” prompt, and the “why is it getting slower” feeling. Behind the limits and the lag is the computational load of tokens.

And this bill doesn’t add up linearly. When the conversation length doubles, the computational load you pay for isn’t double, it’s closer to quadruple (this is a mathematical property of the AI’s internal mechanics). No wonder the lag in long conversations feels like it’s exploding.

Truth 3: The more you cram in, the dumber it gets

This is the most counterintuitive point, but the most useful one.

AI’s “attention” is limited, and all of its attention added up always equals one whole. The more content you cram in, the thinner the attention allocated to each key point gets diluted. Extra fluff will steal focus that should have gone to the critical questions.

I’m not just making this up. A famous study, Lost in the Middle (Liu et al., 2024), found a U-shaped curve: AI remembers information best when it’s placed at the beginning or end of the conversation; but for key points buried in the middle, the probability of remembering them drops to only about 20%. No wonder it constantly forgets the things you said in the middle.

Another study by Chroma in 2025 tested 18 mainstream models and found that all of them degraded in performance as the input grew longer. This phenomenon is called context rot.

So the conclusion is simple: less is more. Give AI just enough information, and it won’t just be cheaper and faster, it will answer more accurately too.

What can I do today? Three tricks, zero cost, immediate results

Trick 1: When changing topics, start a new chat

This is the most effective and most easily overlooked trick. After you finish chatting with it about a travel itinerary, and then want it to help you edit your resume, don’t continue in the same chat. Start a new one.

The old conversation is not only unhelpful for the new topic, but it will also force the tone and assumptions from the previous topic in as interference. Starting a new chat is like wiping a whiteboard full of old equations clean before writing again; the AI will actually be smarter.

Many people worry: if I start a new chat, will my old data disappear? It won’t. The old conversation is still sitting quietly in your sidebar, where you can find it anytime. Starting a new chat just makes it so that “the AI in this new chat” can’t see the old ones, it doesn’t delete them.

Trick 2: Don’t mindlessly paste in entire PDFs or long articles

Every word you paste in counts toward the bill, and it gets counted again in every subsequent round of conversation. Pasting a long PDF when you really only need one paragraph from it is like dragging the weight of the entire document along with every following sentence.

Just paste the specific paragraph you actually want to ask about.

Trick 3: State your questions clearly and concisely

This doesn’t just save money, it also makes the answers more accurate (remember Truth 3?). Instead of giving it a massive chunk of background and vaguely asking “what do you think?”, it’s better to just state exactly what you want clearly.

Why do I care about this?

I’m a physiatrist, and I normally use AI heavily: organizing notes, looking up literature, automating a bunch of life chores. The deeper I use it, the more I realize that “saving tokens” is basically training myself to “speak clearly and organize my work cleanly.”

Interestingly, as I later moved from using AI in phone chat Apps all the way to more advanced tools on my computer, I found that every money-saving action I take in the advanced tools is essentially exactly the same as these three tricks above, just with different names.

It’s the same underlying principle that applies everywhere from phones to terminals. In the next post, I’ll talk about a critical dividing line: when should you tell AI to use its brain, and when should you not use AI at all?

—

➡️ Next post: If an Abacus Works, Don’t Boot Up a Supercomputer: When Should You Tell AI to Use Its Brain?

References

  • Liu, N. F., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173. arXiv:2307.03172
  • Chroma Research (2025). Context Rot: How Increasing Input Tokens Impacts LLM Performance.