An AI Systems Architect's Year-in-Review - Inspired by Karpathy's format, written for the systems layer.
Introduction
Every year, someone writes the review that names what you've been feeling. In 2025, Karpathy did it for LLM research - RLVR, ghosts not animals, Software 3.0. Millions shared it because it made implicit shifts explicit.
But nobody wrote the equivalent for the systems layer - the architecture that actually makes AI run in production. This is that essay.
These are the six paradigm shifts that redefined how we design, deploy, and maintain AI systems in 2025-2026. Not the model breakthroughs. Not the product launches. The architectural shifts that changed what it means to be a systems builder.
Shift 1: Context Window Economics Replaced Token Limits as the Real Constraint
What changed
In 2024, the conversation was about token limits. "Can your model handle 128K tokens?" In 2025, the question became: "What does it cost to fill that window, and does performance actually improve when you do?"
The dirty secret of 2025 was that larger context windows often made agents worse, not better. Attention dilutes across longer sequences. The model that could "handle" 1M tokens often performed worse on tasks at position 800K than a model with a 32K window where the relevant information sat in the first 8K tokens.
The systems architecture implication
The winning architecture in 2026 isn't "stuff everything into context." It's context engineering - the discipline of ensuring the right information appears in the right position in the context window at the right time.
This created an entirely new systems layer:
- Context compressors: Systems that summarize prior conversation history into dense, token-efficient representations
- Priority injection: Architectures that dynamically reorder context based on the current subtask
- Sliding window managers: Infrastructure that maintains a "working memory" within a fixed budget
Traditional Architecture (2024):
User Query → Retrieve Everything → Stuff into Context → Generate
Modern Architecture (2026):
User Query → Context Router → [Compress | Retrieve | Inject | Reorder] → Optimal Window → Generate
Why this matters now
If you're building AI systems and your architecture is "retrieve everything, dump it in," you're already behind. Context window economics is the new systems bottleneck, and the engineers who understand it will build the next generation of reliable AI products.
Shift 2: Agentic Search Killed Classical RAG for Most Production Use Cases
What changed
For three years, RAG (Retrieval-Augmented Generation) was the default architecture for grounding AI in external knowledge. Vector databases became a $2B+ market. Every AI system had an embedding pipeline.
Then Boris Cherny and the Claude Code team made a decision that shocked the industry: they killed RAG internally and replaced it with agentic search - giving the model tools like grep, glob, and read, and letting it search the codebase at runtime.
Their reasoning was devastating in its simplicity:
- Accuracy: Exact text matching via
grep is more reliable than semantic similarity via embeddings
- Freshness: No stale index. The agent reads the actual current file state
- Security: No vector database means no embedded data to leak
- Simplicity: Fewer moving parts, fewer failure modes
The systems architecture implication
This didn't just change one product. It challenged a $2B ecosystem. The question every systems architect must now answer:
"Does my RAG pipeline actually outperform letting the agent search directly?"
For most codebases and structured knowledge bases: the answer is increasingly no. Agentic search - where the model iteratively searches, reads, and refines - handles the same tasks with higher accuracy and lower infrastructure cost.
Where RAG still wins
RAG isn't dead everywhere. It remains superior for:
- Unstructured, large-scale corpora where grep patterns don't apply
- Multilingual semantic search across languages
- Real-time similarity matching at scale (recommendation systems)
But for the majority of production AI systems - code assistants, document Q&A, internal knowledge bots - agentic search is the new default.
Shift 3: RLVR Extended Beyond Code and Math Into Every Domain
What changed
Reinforcement Learning with Verifiable Rewards (RLVR) was 2025's single biggest training breakthrough. Unlike RLHF (which uses subjective human preferences), RLVR trains against objective, automatically checkable environments.
In 2025, this was limited to domains with clear verification: math proofs, coding challenges, formal logic. By early 2026, researchers began extending RLVR into:
- Biology: Protein folding verification against known structures
- Chemistry: Reaction pathway validation against thermodynamic constraints
- Legal: Contract clause verification against regulatory databases
- Finance: Trading strategy validation against historical market data
The systems architecture implication
RLVR changes what you build around the model. Every RLVR deployment requires:
- A verifier: An environment or oracle that can check the model's output programmatically
- A reward signal: A scalar that tells the model how well it performed
- A training loop: Infrastructure to continuously train on new verified examples
This means AI systems architects now need to think about verification infrastructure as a first-class concern. Building the verifier is often harder than building the model itself.
RLVR System Architecture:
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Model │────▶│ Verifier │────▶│ Reward │
│ (generates)│ │ (checks) │ │ (scores) │
└──────────────┘ └──────────────┘ └──────────────┘
▲ │
│ │
└─────────────── Training Loop ───────────┘
Why this matters now
If you're building AI products in 2026 and your training pipeline doesn't include a verifier, you're missing the biggest lever for model improvement. The bottleneck isn't model capability - it's building high-quality verification infrastructure.
Shift 4: Localhost-First Agent Design Replaced Cloud-First Defaults
What changed
In 2024, the default architecture for AI agents was cloud-first: API call → cloud processing → response. Agents lived on servers.
Karpathy, Anthropic (with Claude Code), and the broader community shifted this in 2025. The new default: agents run on localhost.
Why?
- Latency: Local agents respond in milliseconds, not seconds
- Privacy: Code never leaves the developer's machine
- Context: A local agent has access to the full filesystem, running processes, git history - context no cloud agent can match
- Cost: No per-request API charges for agent reasoning loops
The systems architecture implication
Localhost-first agent design inverts the traditional architecture:
Cloud-First (2024):
User → Cloud API → Model → Response → User
Localhost-First (2026):
User → Local Agent (model runs locally or makes targeted API calls) → Full filesystem access → Response
This requires fundamentally different infrastructure:
- Local model serving: Tools like Ollama, LM Studio make it possible to run capable models on consumer hardware
- Hybrid routing: The agent decides when to use a local model vs. when to call a frontier API
- File system integration: Agents that can read, write, and modify local files as a native capability
- Session management: Persistent local state across agent sessions
Why this matters now
The agents that win in 2026 are the ones that feel like they live on your computer. They know your project. They remember your preferences. They don't need you to upload anything. If you're still building cloud-first agents for developer tools, you're building for 2024.
Shift 5: MCP Became the De Facto Standard for Tool Integration
What changed
Model Context Protocol (MCP) emerged in late 2024 as Anthropic's protocol for connecting AI models to external tools. By mid-2025, it became the de facto standard for tool integration across the entire AI ecosystem.
MCP provides a standardized way for AI agents to:
- Discover available tools
- Understand tool capabilities and schemas
- Execute tools with structured inputs
- Receive structured outputs
The systems architecture implication
Before MCP, every AI system had its own tool integration layer. Custom function calling formats, proprietary tool definitions, bespoke error handling. Integration was expensive and fragile.
MCP standardized this into a protocol - similar to how HTTP standardized web communication. This created:
- Tool marketplaces: Build a tool once, deploy it across any MCP-compatible agent
- Composable agent architectures: Agents can discover and use tools they weren't specifically designed for
- Vendor-agnostic tooling: Tools work across Claude, GPT, Gemini, and open-source models
Before MCP:
Agent A → Custom Tool Format A → Tool 1
Agent B → Custom Tool Format B → Tool 1 (rebuild integration)
Agent C → Custom Tool Format C → Tool 1 (rebuild again)
After MCP:
Agent A ─┐
Agent B ──┼── MCP Protocol ── Tool 1 (one integration)
Agent C ─┘
Why this matters now
If you're building AI tools in 2026 and you're not MCP-compatible, you're invisible to the growing ecosystem of agents that expect MCP as the default protocol. MCP is to AI agents what REST was to web APIs.
Shift 6: Inference-Time Scaling Changed the Economics of Everything
What changed
The traditional AI economics were: spend millions training, pennies on inference. In 2025, this flipped.
Inference-time scaling - allowing models to "think longer" before answering - proved that spending more compute at inference could match or exceed the performance gains of more expensive training. Models like o1, o3, and their open-source equivalents showed that a smaller model + more inference compute could outperform a larger model + standard inference.
The systems architecture implication
This changed the cost model for every AI system:
- Pricing models: Per-token pricing doesn't capture the value of inference-time reasoning. Systems need to account for "thinking tokens" that aren't visible in the output
- Latency budgets: You can have fast or accurate, and the systems architect decides where to allocate the inference budget
- Cascading architectures: Use a fast, cheap model for easy queries; escalate to slow, expensive reasoning for hard ones
Inference-Time Scaling Architecture:
User Query → Difficulty Estimator →
├── Easy → Fast Model (7B, 100ms, $0.001) → Response
├── Medium → Standard Model (70B, 2s, $0.01) → Response
└── Hard → Reasoning Model (70B + CoT, 30s, $0.10) → Response
Why this matters now
Every AI system in 2026 needs an inference budget - a conscious allocation of compute, latency, and cost across the query distribution. "Same model, same inference for every request" is the new anti-pattern.
The Meta-Lesson
The pattern across all six shifts is the same: the systems layer is where the real innovation is happening in 2026.
Models are converging. Architecture matters more than ever. The engineers who understand context window economics, agentic search patterns, RLVR verification infrastructure, localhost-first design, MCP integration, and inference-time scaling economics will build the next generation of AI products.
The model research gets the headlines. The systems architecture gets the results.
This essay is part of a series on AI systems architecture. Follow for weekly deep-dives on the patterns and trade-offs that define production AI in 2026.