How LLMs Actually Work

AI language models (LLMs) — like ChatGPT or Claude — are word predictors. Every response they generate is actually the result of picking one word at a time, over and over, based on what word seems most likely to come next given everything before it. They don't "think" or "understand" in the way humans do — they've learned statistical patterns from billions of pages of human writing.

How text gets split up (Tokens)

The AI doesn't read word by word — it reads in small chunks called tokens (roughly ¾ of a word each). "ChatGPT" becomes 2 tokens. "Hello" is 1. This matters because AI services charge per token, and there's a limit on how many tokens can fit in one conversation.

The AI's memory window (Context Window)

The AI can only see what's in the current conversation — there's no memory between sessions unless you build it in. Everything the AI needs to answer your question must be in the prompt you send. Think of it like talking to someone who forgets everything the moment the call ends.

Creativity dial (Temperature)

You can control how predictable or creative the AI's answers are. At temperature = 0, it always picks the most likely next word — same input, same output every time. At temperature = 1 or higher, it picks less predictably, producing more varied or creative responses. Use low temperature (0–0.3) for factual tasks, higher (0.7–1) for creative work.

How the AI connects words (Attention)

Inside the AI, every word in your prompt gets compared to every other word to figure out which ones are related. This is why the AI understands that "bank" in "river bank" means water, not money. Instructions placed toward the end of a prompt often get more weight than those at the start.

From text predictor to helpful assistant (RLHF)

A base AI model just predicts text. Models like Claude and ChatGPT went through an extra training step called RLHF (Reinforcement Learning from Human Feedback): thousands of human raters chose which AI responses were helpful, safe, and honest. The model learned to prefer those patterns. This is what makes it feel like an assistant rather than a random text generator.

Words with similar meanings are stored close together

The AI organizes words and concepts in a kind of invisible map — words that mean similar things end up "near" each other in this map. That's why the AI can handle synonyms and analogies without being explicitly told about them.

Practical takeaway: You're communicating with a system that learned from human writing. The best prompts are the ones a knowledgeable human would find clear, complete, and easy to act on.

Detailed Analysis

The core loop is simple: take all the text so far → predict the single most likely next word → add it → repeat. That loop is what generates an entire paragraph.

When the AI processes your prompt, it converts every word chunk into a number (a vector). It then runs an "attention" calculation that scores how related every word chunk is to every other word chunk. This is what lets it track that the word "it" in "Maria told Sofia that she liked her" refers back correctly.

Pre-training on internet-scale text gives the AI its broad knowledge. But the RLHF training step is what made it helpful — it learned that answers humans rate highly tend to be accurate, organized, and directly responsive to the question asked.

The memory limit (context window) is a hard technical boundary. GPT-4 can handle roughly 128,000 word chunks at once; Claude can handle about 200,000. Once you exceed that limit, the AI literally cannot see what came before it. Memory tricks — like summarizing earlier parts of a conversation — must be built on top.

Take-Home Points

• The AI predicts the next word probabilistically — it doesn't "understand" the way humans do
• Temperature controls the creativity/predictability tradeoff — use low values for factual tasks
• The context window is everything the AI can "see" — fill it with what matters
• Word relationships (attention) is what gives the AI its contextual reasoning ability
• RLHF is what turns a text predictor into an assistant that follows instructions well

Conclusion

Understanding how the AI works at a basic level is the foundation for everything that follows. When you know the AI is a word predictor with a fixed memory window that weighs word relationships to understand context, you can immediately see why the structure, position, and specificity of your prompt all matter. You're not talking to an oracle — you're steering a very sophisticated autocomplete system. This mental model will guide every prompting decision you make.