The AI Engineering Toolkit Landscape

You type:     function validateEm
Tool suggests: ail(email: string): boolean {
                 const regex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
                 return regex.test(email);
               }

// What the tool sends to the model (simplified)
interface CompletionRequest {
  prefix: string;      // Code before cursor
  suffix: string;      // Code after cursor (fill-in-the-middle)
  language: string;
  filePath: string;
  neighbors: string[]; // Content of related open files
}

// What comes back
interface CompletionResponse {
  completions: Array<{
    text: string;
    confidence: number;
    stopReason: 'endOfLine' | 'endOfBlock' | 'maxTokens';
  }>;
}

Traditional:  [prefix] → generate →
FIM:          [prefix] → generate → [suffix must fit]

# Simplified agent loop
while not task_complete:
    # Model receives: conversation history + available tools
    response = model.generate(
        messages=conversation_history,
        tools=[
            {"name": "read_file", "params": {"path": "string"}},
            {"name": "write_file", "params": {"path": "string", "content": "string"}},
            {"name": "run_command", "params": {"command": "string"}},
            {"name": "search_codebase", "params": {"query": "string"}},
            {"name": "list_directory", "params": {"path": "string"}},
        ]
    )

    if response.has_tool_calls:
        for tool_call in response.tool_calls:
            result = execute_tool(tool_call)
            conversation_history.append({"role": "tool", "content": result})
    else:
        # Model is done, present result to user
        task_complete = True

GitHub Copilot:
  ✓ Current file
  ✓ Open tabs (limited)
  ✗ Full codebase search
  ✗ Terminal/runtime context

Cursor:
  ✓ Current file
  ✓ Full codebase index
  ✓ @-mentions for specific files
  ✓ Documentation crawling
  ✗ Terminal command execution (in chat; agent mode can)

Claude Code:
  ✓ Full filesystem access
  ✓ Terminal command execution
  ✓ Git history awareness
  ✓ CLAUDE.md project context
  ✓ Can read/search entire codebase

GitHub Copilot:
  - Approach: Had to manually create middleware file, Copilot completed function bodies
  - Time: ~25 minutes of interactive coding
  - Result: Working auth middleware, but I did all the architecture
  - Quality: Good function-level completions, missed some edge cases

Cursor (Agent Mode):
  - Approach: Described task, Cursor created middleware and modified route files
  - Time: ~8 minutes including review
  - Result: Complete solution across all 4 files
  - Quality: Good overall structure, needed manual fixes for refresh token rotation
  - Tokens used: ~45K input, ~12K output

Claude Code:
  - Approach: Described task, Claude Code explored codebase, planned, then implemented
  - Time: ~12 minutes including review
  - Result: Complete solution with tests, updated package.json
  - Quality: Most thorough -- added error handling, logging, and migration notes
  - Tokens used: ~80K input, ~20K output (more exploration)

Aider:
  - Approach: Added relevant files to context, described task
  - Time: ~10 minutes including review
  - Result: Clean implementation, modified specified files
  - Quality: Good code, but needed manual prompting for tests

# .cursorrules

## Project Context
This is a Next.js 14 application using the App Router, TypeScript strict mode,
and Tailwind CSS. We use Supabase for the backend.

## Code Style
- Use functional components with hooks, never class components
- Prefer named exports over default exports
- Use `type` for object shapes, `interface` for extendable contracts
- All API routes must validate input with Zod
- Error handling: use Result<T, E> pattern, never throw in business logic

## File Naming
- Components: PascalCase (UserProfile.tsx)
- Utilities: camelCase (formatDate.ts)
- API routes: route.ts in appropriate app/api directory

## Testing
- Unit tests: Vitest with React Testing Library
- E2E tests: Playwright
- Test files: colocated as ComponentName.test.tsx

## Common Patterns
When creating a new API route, always include:
1. Zod input validation
2. Auth middleware check
3. Error response formatting with proper status codes
4. Rate limiting headers

Good Agent Mode prompts:
✓ "Add pagination to the /api/users endpoint. Use cursor-based pagination
   with a default page size of 20. Update both the API route and the
   UserList component that calls it."

✓ "Refactor the authentication flow to use refresh token rotation. Currently
   tokens are in auth/tokens.ts and middleware is in middleware.ts."

Bad Agent Mode prompts:
✗ "Fix my code" (too vague -- which code? what's broken?)
✗ "Build me an e-commerce site" (too large -- Agent mode works best on
   focused, well-defined tasks)

# Starting Claude Code
$ claude

# It has full access to your filesystem and can run commands
> Look at the project structure and tell me about the architecture

# Claude Code will:
# 1. Run `find` or `ls` to explore directories
# 2. Read key files (package.json, README, config files)
# 3. Search for patterns with grep
# 4. Synthesize an architectural overview

# CLAUDE.md

## Project Overview
Payment processing microservice handling Stripe integration
for our SaaS platform. Processes ~50K transactions/day.

## Commands
npm run dev      # Start with hot reload
npm run test     # Jest unit tests
npm run test:e2e # Playwright E2E tests
npm run lint     # ESLint + Prettier

## Architecture
- Express.js API with TypeScript
- PostgreSQL via Prisma ORM
- Redis for idempotency keys and rate limiting
- Bull queues for async webhook processing

## Key Conventions
- All monetary values stored as integers (cents)
- Webhook handlers must be idempotent
- Every endpoint needs integration tests
- Use structured logging (pino) -- never console.log

Claude Code excels at:
✓ Large-scale refactors across many files
✓ Tasks requiring shell commands (running tests, checking git status)
✓ Exploratory work ("understand this codebase and suggest improvements")
✓ Complex multi-step tasks (create feature branch, implement, test, commit)
✓ Projects where you want AI that integrates with existing terminal workflow

Claude Code struggles with:
✗ Real-time inline completions (it's not designed for this)
✗ Visual feedback (no diff view, no syntax highlighting in output)
✗ Quick micro-tasks (overhead of terminal interaction)
✗ Large binary files or non-text content

Layer 1: Inline Completions (always on)
  → GitHub Copilot or Cursor Tab
  → Handles: boilerplate, completions, obvious patterns
  → Effort: zero -- it just works in the background

Layer 2: Chat/Agent for focused tasks
  → Cursor Agent Mode or IDE chat
  → Handles: multi-file changes, refactors, feature implementation
  → Effort: write a clear prompt, review diffs

Layer 3: Terminal Agent for complex operations
  → Claude Code or Aider
  → Handles: large refactors, codebase exploration, CI/CD tasks
  → Effort: describe the task, monitor execution, review results

class AIToolEvaluation:
    """Framework for evaluating AI coding tools"""

    def evaluate(self, tool_name: str) -> dict:
        return {
            "context_quality": self._assess_context(tool_name),
            "output_accuracy": self._assess_accuracy(tool_name),
            "workflow_integration": self._assess_integration(tool_name),
            "cost_efficiency": self._assess_cost(tool_name),
            "learning_curve": self._assess_learning(tool_name),
        }

    def _assess_context(self, tool) -> str:
        """How well does the tool understand your codebase?"""
        # Does it index your repo?
        # Can it find relevant files automatically?
        # Does it understand cross-file relationships?
        # Can you manually specify context?
        pass

    def _assess_accuracy(self, tool) -> str:
        """How often is the generated code correct?"""
        # Test with: known patterns, novel code, edge cases
        # Measure: compiles, passes tests, handles errors
        pass

    def _assess_integration(self, tool) -> str:
        """How well does it fit your existing workflow?"""
        # IDE compatibility, terminal integration
        # Git workflow support
        # Team collaboration features
        pass

    def _assess_cost(self, tool) -> str:
        """What's the total cost of ownership?"""
        # Subscription + API costs
        # Time saved vs. time reviewing AI output
        # Context switching overhead
        pass

    def _assess_learning(self, tool) -> str:
        """How quickly can your team become productive?"""
        # Initial setup complexity
        # Days to proficiency
        # Advanced features discovery
        pass