How LLMs Actually Work (For Engineers)

It was 2:47 AM and Marcus was staring at his terminal. He'd been prompting Claude to refactor a gnarly state machine in his payment processing service for the past hour. Sometimes the AI nailed it. Sometimes it produced code that compiled but subtly changed the business logic. The inconsistency drove him crazy.

"Why does it sometimes get it perfectly right and other times completely miss?" he asked his tech lead the next morning.

"Because you're thinking of it as a compiler," she said. "It's not. It's a probabilistic text predictor with emergent reasoning abilities. Once you understand the mechanics, you can predict when it'll succeed and when it'll fail."

That conversation changed how Marcus worked with AI forever. And that's exactly what we're going to cover in this lesson -- not the hand-wavy "AI is like a brain" analogy, but the actual engineering that makes LLMs work, explained for people who think in systems, architectures, and data structures.

The Transformer Architecture: What You Actually Need to Know

Every modern LLM -- GPT-4, Claude, Gemini, Llama -- is built on the transformer architecture introduced in the 2017 paper "Attention Is All You Need." As an engineer, you don't need to derive the math, but you do need to understand the data flow, because it directly explains the tool's capabilities and limitations.

The High-Level Pipeline

Here's what happens when you send a prompt to an LLM:

Input Text → Tokenizer → Embedding Layer → N x Transformer Blocks → Output Probabilities → Token Selection → Detokenizer → Output Text

Each of these stages has engineering implications:

1. Tokenization -- Your text is split into tokens, not words. This matters more than most engineers realize.

# Using tiktoken (OpenAI's tokenizer) to see how code gets tokenized
import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")

code = "const handlePayment = async (req: Request) => {"
tokens = enc.encode(code)
print(f"Token count: {len(tokens)}")
# Token count: 14

# Let's see the actual tokens
for token_id in tokens:
    print(f"  {token_id} → '{enc.decode([token_id])}'" )
# 1040 → 'const'
# 3790 → ' handle'
# 20711 → 'Payment'
# 284 → ' ='
# 3393 → ' async'
# 320 → ' ('
# ...

Key insight: common programming patterns tokenize efficiently (fewer tokens), while unusual variable names, obscure APIs, or novel syntax burn through tokens faster. This directly affects what fits in your context window and how well the model "understands" your code.

# Common pattern - efficient tokenization
common = "function fetchUsers() { return fetch('/api/users').then(r => r.json()); }"
print(f"Common pattern: {len(enc.encode(common))} tokens")
# Common pattern: 23 tokens

# Unusual naming - less efficient
unusual = "fn xqz_proc_blk(ctx: &mut Zygomorphism<T>) -> Result<(), XqzError> {"
print(f"Unusual pattern: {len(enc.encode(unusual))} tokens")
# Unusual pattern: 32 tokens

2. Embeddings -- Each token is converted into a high-dimensional vector (typically 4,096 to 12,288 dimensions for large models). These vectors encode semantic meaning. The word "function" in JavaScript, the word "function" in Python, and the word "function" in mathematics all get similar but distinct embeddings based on surrounding context.

3. Transformer Blocks -- This is where the magic happens. Let's dig in.

Self-Attention: The Core Mechanism

The transformer's key innovation is self-attention -- the mechanism that lets each token "look at" every other token in the sequence and decide which ones are relevant.

Here's the simplified version of what happens in a single attention head:

# Pseudocode for self-attention (simplified)
import numpy as np

def self_attention(tokens_embedded, W_q, W_k, W_v):
    """
    tokens_embedded: (seq_len, d_model) - the input token embeddings
    W_q, W_k, W_v: learned weight matrices for Query, Key, Value
    """
    # Each token generates three vectors
    Q = tokens_embedded @ W_q   # "What am I looking for?"
    K = tokens_embedded @ W_k   # "What do I contain?"
    V = tokens_embedded @ W_v   # "What information do I provide?"

    # Compute attention scores - how relevant is each token to every other?
    d_k = K.shape[-1]
    scores = (Q @ K.T) / np.sqrt(d_k)  # (seq_len, seq_len)

    # Softmax to get probabilities
    attention_weights = softmax(scores, axis=-1)

    # Weighted sum of values
    output = attention_weights @ V
    return output

Why does this matter for you as an AI-assisted engineer?

Self-attention is why context matters so much. When the model processes your prompt, every token attends to every other token. If you give it a function signature at the beginning of a 2,000-token prompt, the model can still "see" that signature when generating code at the end. But attention is diluted across all tokens -- the more irrelevant noise in your context, the weaker the signal from the important parts.

This is the engineering reason why focused, well-structured prompts outperform massive context dumps.

Multi-Head Attention: Parallel Pattern Recognition

Modern LLMs don't use a single attention mechanism -- they use multiple attention heads running in parallel (typically 32-128 heads). Each head learns to attend to different types of relationships:

• Head A might learn syntax relationships (matching brackets, indentation patterns)
• Head B might learn type relationships (variable declarations and their usage sites)
• Head C might learn semantic relationships (function name implies return type)
• Head D might learn positional relationships (nearby code is often related)

Multi-Head Attention = Concat(head_1, head_2, ..., head_n) @ W_output

where head_i = Attention(Q_i, K_i, V_i)

This is why LLMs can simultaneously understand your code's syntax, semantics, types, and conventions -- they're running dozens of parallel attention computations, each specializing in different aspects.

Layer Stacking: From Tokens to Understanding

A typical large model stacks 80-120 transformer blocks sequentially. Research shows these layers develop a hierarchy:

This layered processing explains why LLMs handle syntactically similar tasks well (early layers activate) but sometimes struggle with novel architectural decisions (requires deep late-layer reasoning with less training signal).

Context Windows: The Engineering Constraints

The context window is the total number of tokens the model can process in a single forward pass. Here's the current landscape:

But raw context size is misleading. Here's what engineers need to know:

The "Lost in the Middle" Problem

Research from Stanford and UC Berkeley demonstrated that LLMs perform best on information placed at the beginning and end of the context window, with degraded performance on content in the middle. This has direct implications for how you structure prompts:

┌─────────────────────────────────────────────────┐
│ HIGH ATTENTION: System prompt, key instructions │  ← Put critical context here
├─────────────────────────────────────────────────┤
│                                                 │
│ LOWER ATTENTION: Supporting context, examples   │  ← OK for reference material
│                                                 │
├─────────────────────────────────────────────────┤
│ HIGH ATTENTION: Current task, question           │  ← Put the actual ask here
└─────────────────────────────────────────────────┘

KV Cache and Why Long Conversations Slow Down

When you chat iteratively with an AI coding tool, each message includes the entire conversation history in the context. The model maintains a Key-Value cache (KV cache) to avoid recomputing attention for previous tokens, but this cache grows linearly with conversation length.

Request 1: [System prompt + Your code + Question]          = 5K tokens
Request 2: [System prompt + Your code + Q + A + Question]  = 12K tokens
Request 3: [All of above + more Q&A]                       = 25K tokens
...
Request 10: Context is now 80K tokens, responses slow down

This is why long coding sessions with AI tools feel progressively slower and less accurate. The practical engineering response: start fresh conversations when you shift to a new task or when quality degrades.

Temperature, Top-p, and Sampling: Controlling Output

When the model processes your prompt through all transformer layers, the final output is a probability distribution over the entire vocabulary for the next token. How the model selects from this distribution is controlled by sampling parameters.

Temperature

Temperature scales the logits (raw scores) before the softmax function:

def sample_with_temperature(logits, temperature):
    """
    temperature = 0.0: Always pick the highest probability token (greedy/argmax)
    temperature = 0.7: Moderate randomness - good for code
    temperature = 1.0: Standard sampling - more creative
    temperature = 1.5: High randomness - unusual combinations
    """
    scaled_logits = logits / max(temperature, 1e-8)
    probabilities = softmax(scaled_logits)
    return np.random.choice(len(probabilities), p=probabilities)

For code generation, most tools use temperature 0 to 0.3. Here's why:

# At temperature 0, given: "def fibonacci(n):\n    if n <= 1:\n        return"
# The model strongly predicts: " n" (probability ~0.92)
# This is correct and deterministic.

# At temperature 1.0, the same context might produce:
# " n" (probability ~0.45)
# " 1" (probability ~0.20)
# " n if" (probability ~0.08)
# " fibonacci" (probability ~0.03)
# More randomness → more chance of incorrect completions

Top-p (Nucleus Sampling)

Top-p sampling is an alternative to temperature that's often more intuitive. Instead of scaling probabilities, it truncates the distribution to the smallest set of tokens whose cumulative probability exceeds p:

def top_p_sampling(logits, p=0.9):
    """Only consider tokens in the top 90% cumulative probability"""
    probs = softmax(logits)
    sorted_indices = np.argsort(probs)[::-1]
    cumulative = np.cumsum(probs[sorted_indices])

    # Find cutoff
    cutoff_idx = np.searchsorted(cumulative, p) + 1
    allowed = sorted_indices[:cutoff_idx]

    # Renormalize and sample
    allowed_probs = probs[allowed] / probs[allowed].sum()
    return np.random.choice(allowed, p=allowed_probs)

How Training Data Shapes Code Generation

LLMs are trained on massive datasets that include public code repositories, documentation, Stack Overflow, technical books, and more. This training process has direct implications for what the model can and cannot do:

What's Well-Represented (Model Excels)

// React components - millions of training examples
function UserProfile({ user }) {
  return (
    <div className="profile">
      <h1>{user.name}</h1>
      <p>{user.email}</p>
    </div>
  );
}

// Express.js REST endpoints - extremely common pattern
app.get('/api/users/:id', async (req, res) => {
  try {
    const user = await User.findById(req.params.id);
    if (!user) return res.status(404).json({ error: 'Not found' });
    res.json(user);
  } catch (err) {
    res.status(500).json({ error: 'Internal server error' });
  }
});

What's Under-Represented (Model Struggles)

// Niche Rust crate with 200 GitHub stars
use obscure_crate::ZygomorphicFunctor;

impl<T: Coalgebra> ZygomorphicFunctor for MyType<T> {
    // Model likely hallucinates API here because it has
    // very few (or zero) training examples
    fn zygo_fold(&self) -> T::Carrier {
        // ...
    }
}

The Knowledge Cutoff Problem

LLMs have a training data cutoff date. Code patterns, library APIs, and framework conventions that emerged after this date may produce hallucinated or outdated suggestions:

// If the model was trained before Next.js 15 released:
// It might generate Next.js 14 patterns that are deprecated

// Model might generate (outdated):
export async function getServerSideProps(context) {
  // ...this API was deprecated in favor of server components
}

// When you actually need (current):
export default async function Page() {
  const data = await fetch('...'); // Direct server component data fetching
  return <div>{/* ... */}</div>;
}

Autoregressive Generation: Why Order Matters

LLMs generate text one token at a time, left to right. Each new token is conditioned on all previous tokens (both your prompt and the model's own prior output). This autoregressive nature has important implications:

The Snowball Effect

If the model makes a slightly wrong choice early in its output, all subsequent tokens are conditioned on that mistake. This can snowball:

# You ask: "Write a function to calculate compound interest"

# Token 1-5: "def calculate_compound_interest("
# Token 6-10: "principal, rate, time):"     ← Missed a parameter (periods per year)
# Token 11+: The entire function is now built on a 3-parameter signature
#            The model will implement simple interest or incorrect compound interest
#            because it's committed to the wrong function signature

This is why plan-then-implement prompting works so well -- it forces the model to lay out the correct structure before committing to code:

Prompt: "First, list the parameters needed for compound interest calculation,
then implement the function."

Model output:
"Parameters needed:
- principal (P): initial investment amount
- annual_rate (r): annual interest rate as decimal
- periods_per_year (n): compounding frequency
- years (t): time in years

def calculate_compound_interest(principal, annual_rate, periods_per_year, years):
    return principal * (1 + annual_rate / periods_per_year) ** (periods_per_year * years)"

Beam Search vs. Greedy Decoding

Most production systems use variations of greedy decoding (always pick the most likely next token) or sampling with low temperature. Some systems use beam search, which maintains multiple candidate sequences in parallel:

Greedy:  Pick best token → pick best token → pick best token → done
         (Fast, but can miss globally better sequences)

Beam-4:  Maintain 4 candidates at each step
         Select top-4 combined scores
         (Slower, but can find better overall sequences)

For code generation, greedy decoding at temperature 0 is usually preferred because code has strong syntactic constraints that naturally guide the model toward correct sequences.

Putting It All Together: A Mental Model for Engineers

Here's the mental model that will serve you throughout this course:

┌────────────────────────────────────────────────┐
│              YOUR PROMPT                        │
│  ┌──────────────────────────────────────────┐  │
│  │ System context + Instructions             │  │
│  │ Relevant code files                       │  │
│  │ Specific task description                 │  │
│  └──────────────────────────────────────────┘  │
│                    ↓                            │
│           TOKENIZATION                          │
│     (Text → integer sequences)                  │
│                    ↓                            │
│          EMBEDDING LAYER                        │
│     (Integers → semantic vectors)               │
│                    ↓                            │
│     80-120 TRANSFORMER BLOCKS                   │
│  ┌──────────────────────────────────────────┐  │
│  │  Multi-Head Self-Attention                │  │
│  │  (Every token attends to every other)     │  │
│  │  Feed-Forward Network                     │  │
│  │  Layer Normalization                      │  │
│  │  Residual Connections                     │  │
│  └──────────────────────────────────────────┘  │
│                    ↓                            │
│     PROBABILITY DISTRIBUTION                    │
│  (Scores for every token in vocabulary)         │
│                    ↓                            │
│     SAMPLING (temperature, top-p)               │
│  (Select next token from distribution)          │
│                    ↓                            │
│     REPEAT until stop token or max length       │
│                    ↓                            │
│          DETOKENIZATION                         │
│     (Integer sequences → text)                  │
│                    ↓                            │
│              MODEL OUTPUT                       │
│       (Code, explanation, etc.)                 │
└────────────────────────────────────────────────┘

Every capability and limitation of AI coding tools traces back to this pipeline:

• Why does it know React so well? → Massive training data with React examples
• Why does it hallucinate APIs? → Probability-based generation, not lookup-based retrieval
• Why does more context sometimes hurt? → Attention is diluted across more tokens
• Why does it fail on novel algorithms? → Fewer training examples in late-layer abstract reasoning
• Why is it better at writing code than debugging? → Generation is the core training task; debugging requires multi-step reasoning that's harder to learn from text

Why This Matters for Your Daily Work

Understanding transformer architecture isn't academic -- it directly informs practical decisions:

1. Prompt Structure Follows Attention Patterns

Place your most important instructions at the beginning and end of prompts. Put supporting context in the middle. This aligns with how attention mechanisms actually weight information.

2. Token Budget is Your Resource Constraint

Just like memory or CPU, context window tokens are a finite resource. You need to make deliberate choices about what to include. A 200-line relevant file beats a 2,000-line entire module dump.

3. Predict Success Based on Training Data

Before asking the AI to help with a task, ask yourself: "How many examples of this pattern exist in public code?" If the answer is "millions" (React components, REST APIs, SQL queries), expect excellent results. If it's "dozens" (your custom DSL, a niche library), expect more hallucinations.

4. Use Temperature Intentionally

Don't just accept default settings. If you're generating boilerplate, use temperature 0. If you're exploring architectural options, bump it up. If you can control the temperature (API calls, some tools), treat it like any other engineering parameter.

5. Fresh Conversations for Fresh Context

Long conversation threads accumulate tokens and dilute attention. When you notice degrading quality, start a new conversation with clean, focused context.