How LLMs Work (The PM Perspective)

The Story: When the PM Understood the Machine

Priya Kapoor had been a product manager at a B2B SaaS company for six years. She'd shipped dozens of features, managed backlogs the size of novels, and negotiated with stakeholders who spoke in acronyms she pretended to understand. But when the CEO walked into the all-hands in January 2024 and declared that every product team would "integrate AI into their roadmap by Q3," Priya felt something she hadn't felt since her first day on the job: she felt completely unprepared.

The engineering team was already buzzing. They wanted to add an "AI assistant" to the platform. The design team was sketching chatbot interfaces. Marketing was drafting press releases about "intelligent automation." And Priya -- the person who was supposed to own the product vision -- couldn't answer a basic question from her VP: "What can this actually do for our users?"

She spent that weekend doing what any good PM would do: research. She read blog posts, watched YouTube explainers, skimmed academic papers. Most of them fell into two categories: either painfully oversimplified ("It's like autocomplete!") or impossibly technical ("The attention mechanism computes scaled dot-product attention across query, key, and value matrices...").

Neither helped her do her job.

What Priya needed wasn't a computer science degree. She needed a product-grade mental model -- one that helped her evaluate feature requests, set realistic expectations with stakeholders, and ask the right questions in engineering reviews. She needed to understand LLMs the way a PM understands databases: not by writing SQL, but by knowing what's possible, what's expensive, and where things break.

Over the next three months, Priya built that mental model. She shipped an AI-powered feature that reduced customer onboarding time by 40%. She killed two AI feature proposals that would have wasted engineering cycles. And she became the person other PMs came to when they needed to separate AI hype from AI reality.

This lesson gives you the same mental model Priya built -- but in an afternoon instead of three months.

---

The Concept: Understanding LLMs Through a Product Lens

What Is an LLM, Really?

Let's start with what matters to you as a PM. A Large Language Model is a piece of software that predicts what text should come next, given some input text. That's it. Every magical-seeming behavior -- writing code, answering questions, summarizing documents, translating languages -- emerges from that single capability: next-token prediction.

Think of it this way: if you've ever used your phone's keyboard suggestions, you've used a tiny language model. You type "Looking forward to" and it suggests "seeing," "hearing," or "meeting." An LLM does the same thing, except it was trained on hundreds of billions of words and can sustain coherent, contextual predictions across thousands of tokens.

The Training Process (What You Need to Know)

You don't need to understand gradient descent, but you need to understand the three phases of how an LLM gets built, because each phase explains different product capabilities and limitations.

Phase 1: Pre-training (The Knowledge Phase)

The model reads vast amounts of text from the internet -- Wikipedia, books, code repositories, forums, news articles, academic papers. It learns patterns: grammar, facts, reasoning patterns, code syntax, writing styles. This is where the model gets its "general knowledge."

Product implication: The model knows things that were common in its training data. It's strong on well-documented topics and weak on niche, proprietary, or very recent information. If your users need AI to answer questions about your specific product, the base model won't know those answers out of the box.

Phase 2: Fine-tuning (The Behavior Phase)

After pre-training, the model is further trained on curated examples of helpful conversations. Human trainers write both the questions and ideal answers. This is where the model learns to be helpful, to follow instructions, and to format responses appropriately.

Product implication: This is why the model follows instructions, answers in bullet points when asked, and generally tries to be useful. But it also means the model is optimized to seem helpful -- sometimes at the cost of accuracy. It will confidently generate plausible-sounding but incorrect information rather than saying "I don't know."

Phase 3: Reinforcement Learning from Human Feedback -- RLHF (The Refinement Phase)

Human evaluators rank different model responses, and the model is optimized to produce responses that humans prefer. This shapes the model's personality, safety behavior, and response quality.

Product implication: RLHF is why different models feel different to use. It's also why models sometimes refuse requests, over-qualify their answers, or add unnecessary caveats. When you're building AI features, the RLHF layer is what determines whether the AI feels helpful or annoying to your users.

The Five Properties Every PM Must Understand

Here's where we move from theory to practical product thinking. Every LLM has five properties that directly affect your product decisions:

1. Probabilistic, Not Deterministic

Ask the same question twice and you'll get different answers. This isn't a bug -- it's fundamental to how the technology works. The model samples from a probability distribution over possible next tokens.

What this means for your product: You cannot guarantee identical outputs for identical inputs. If your feature requires exact reproducibility (like a calculator or a lookup table), an LLM is the wrong tool. If your feature benefits from variety (like generating marketing copy or brainstorming ideas), this property is actually an advantage.

2. Context-Dependent

LLMs don't have persistent memory. Each conversation starts fresh. The model's behavior depends entirely on what's in the current prompt and conversation history -- its "context window."

What this means for your product: If your AI feature needs to "remember" things about a user across sessions, you need to build that memory layer yourself. This might mean storing conversation history in a database, passing user profile data into each prompt, or using retrieval-augmented generation (RAG) to pull in relevant documents. The model won't remember anything on its own.

3. Confidently Wrong (Hallucination)

LLMs generate text that is statistically likely, not text that is factually verified. They will fabricate citations, invent statistics, and state falsehoods with the same confidence as truths. This is called "hallucination."

What this means for your product: Every AI feature that surfaces LLM-generated information to users needs a reliability strategy. Options include: grounding responses in your own data (RAG), adding citations and source links, including confidence indicators, having human review for high-stakes outputs, or clearly labeling content as "AI-generated." The right approach depends on the risk tolerance of your use case.

4. Instruction-Following but Not Understanding

LLMs follow instructions remarkably well, but they don't "understand" what they're doing in the way humans do. They pattern-match against similar instructions in their training data. Complex, multi-step, or ambiguous instructions may be interpreted differently than you expect.

What this means for your product: Prompt engineering is real engineering work, not an afterthought. The quality of your AI feature is often determined by the quality of the system prompt -- the hidden instructions that shape the model's behavior. You'll need to invest in prompt iteration, testing, and monitoring just like you invest in any other core technology.

5. Capability Boundaries Are Fuzzy

Unlike traditional software where capabilities are binary (the feature either works or it doesn't), LLM capabilities exist on a spectrum. The model might correctly summarize 90% of documents, handle 85% of customer queries, or generate code that works 70% of the time. These percentages shift based on the complexity of the input.

What this means for your product: You can't spec AI features the way you spec traditional features. Instead of "the system shall correctly process all inputs," you need to think in terms of accuracy rates, error handling, and graceful degradation. This requires a different approach to acceptance criteria, testing, and launch decisions.

The Architecture Stack: What PMs Should Know

When your engineering team talks about building AI features, they'll mention several components. Here's what each one means for your product:

The Model (Foundation) This is the LLM itself -- GPT-4, Claude, Gemini, Llama, etc. Your choice of model affects cost, speed, accuracy, and capability. Think of it like choosing a database: PostgreSQL vs. MongoDB isn't about which is "better," it's about which fits your use case.

The Prompt (Instructions) The system prompt is the hidden text that tells the model how to behave in your product. "You are a helpful customer support agent for Acme Corp. You only answer questions about Acme products. You always respond in a professional tone." This is where your product requirements become AI behavior.

The Context (Data) RAG (Retrieval-Augmented Generation) is the pattern of pulling relevant data from your own systems and injecting it into the prompt before the model generates a response. "Here are the five most relevant help articles about this customer's question. Use these to answer." This is how you make the model accurate about your specific product.

The Guardrails (Safety) Input filters, output validators, content policies, and fallback mechanisms that prevent the model from going off the rails. "If the user asks about competitor products, redirect them to our comparison page." This is your product's insurance policy.

The Integration (Orchestration) The code that connects the model to your product's data, UI, and business logic. This handles things like: when to call the model, how to display streaming responses, how to handle errors, and how to log interactions for monitoring.

Evaluating AI Feature Requests: The PM Framework

Now that you have the mental model, here's how to apply it. Every AI feature request that comes across your desk should go through five questions:

Question 1: Is this a pattern-matching problem? LLMs excel at tasks that can be framed as "given this input text, produce this output text." Classification, summarization, translation, extraction, generation -- these are pattern-matching tasks. If the task requires real-time data, mathematical precision, or accessing external systems, the LLM can only be part of the solution.

Question 2: What does "good enough" look like? Since LLM outputs are probabilistic, you need to define your accuracy threshold. For brainstorming assistance, 70% quality might be fine -- the user is going to edit anyway. For medical advice, 99.9% might not be enough. Define your threshold before building.

Question 3: What happens when it's wrong? This is the most important question. Model your failure modes. If the AI summarizes a customer call incorrectly, does the sales rep lose a deal? If the AI generates the wrong code, does a developer spend two hours debugging? The cost of failure determines your investment in guardrails.

Question 4: Do we have the right data? If the feature needs to be accurate about your product, your customers, or your domain, you need data to ground the model's responses. Do you have that data in a format the model can use? Is it clean, current, and comprehensive? Data quality is the ceiling on AI feature quality.

Question 5: Can we measure success? How will you know if the AI feature is working? User satisfaction surveys? Task completion rates? Time savings? Error rates? If you can't measure it, you can't iterate on it, and AI features that don't iterate get worse over time as the world changes and the model stays the same.

Model Selection: The PM's Guide

You'll likely be involved in (or should be involved in) model selection decisions. Here's how to think about the tradeoffs:

PM decision framework: Start with the most capable model to validate the feature idea. Then optimize for cost and speed once you've proven value. Don't pre-optimize -- many teams waste months building infrastructure for open-source models before validating that the feature is even useful.

---

Apply: Building Your PM Mental Model

Exercise 1: Map Your Product's AI Opportunities

Take your product's core user workflows and categorize each step:

Fill this out for your own product. Aim for at least 10 workflow steps.

Exercise 2: Attend an Engineering Discussion Armed with Questions

The next time your engineering team discusses an AI feature, bring these questions:

• "What model are we planning to use and why?"
• "Walk me through the system prompt -- what instructions are we giving the model?"
• "How are we grounding the model's responses in our data?"
• "What happens when the model hallucinates? How does the user know?"
• "What's our latency target? Will users be waiting for a response?"
• "How are we logging interactions so we can measure quality?"

You don't need to understand every technical detail of the answers. You need to confirm that the team has answers.

Exercise 3: Test Model Boundaries

Pick any LLM you have access to (ChatGPT, Claude, Gemini -- any of them) and deliberately test its boundaries:

1. Ask it a factual question about something very recent (this week's news). Watch it either refuse or hallucinate.
2. Ask it to do arithmetic on a 10-step word problem. Watch it get some steps wrong.
3. Ask it the same creative writing prompt three times. Notice how the outputs differ.
4. Give it a long, complex instruction with five constraints. See how many it actually follows.
5. Ask it about your company's specific product. See what it knows vs. what it makes up.

Each test teaches you something about the technology's boundaries -- and every boundary is a product design decision.

---

Reflect: What This Means for Your PM Practice

Understanding how LLMs work isn't about becoming technical. It's about becoming a better product decision-maker in a world where AI is reshaping what's possible.

Priya didn't learn to build models. She learned to ask the right questions. She learned that "add AI" isn't a feature spec -- it's a starting point for product work. She learned that the gap between a demo and a production feature is where most AI projects die. And she learned that the PM who understands AI's capabilities and limitations becomes the most valuable person in the room -- because they're the bridge between what engineering can build and what users actually need.

Here's what to carry forward:

For your next sprint planning: When an AI feature comes up, ask "What happens when this is wrong?" before "How long will this take?"

For your next stakeholder conversation: You can now explain why "just add AI" isn't simple, using concrete examples of the tradeoffs involved.

For your next product strategy discussion: You can identify which parts of your product are genuinely ripe for AI augmentation and which are being proposed because AI is trendy.