The AI Stack: How Modern AI Systems Are Built
A working AI product is never just “a model.” It’s a stack — hardware, data pipelines, a foundation model, a layer that wires that model to tools and knowledge, and the application on top. The naive “data → model → app” picture is long gone; the real architecture is five layers deep, and most of the hard engineering now lives in the middle. Knowing the layers tells you where any given tool fits and what you still have to build around it.
The five layers
| Layer | What it does | Representative components |
|---|---|---|
| 1. Infrastructure | Compute, tuned for always-on inference | GPUs/ASICs (NVIDIA, Google TPUs), cloud and edge, high-speed networking |
| 2. Data & development | Ingest, process, and manage data across the lifecycle | Pipelines (Airflow), vector stores (Pinecone), MLOps, experiment tracking |
| 3. Foundation models | The core reasoning and generative engine | LLMs and multimodal models, adapted via RAG and fine-tuning |
| 4. Serving & orchestration | Wires models to data and tools; runs multi-step behavior | Model gateways, RAG services, tool-calling runtimes, agent frameworks |
| 5. Applications & agents | Delivers value inside real workflows | Copilots, embedded features, multi-agent systems, process automation |
1. Infrastructure
The raw compute. The center of gravity has moved from occasional, massive training runs to cheap, constant inference — the cost of serving a model around the clock. That means specialized serving chips (GPUs, TPUs, custom ASICs), a deployment mix of public cloud for scale, private cloud for control, and edge for latency and privacy, and fast interconnects to move data between them.
2. Data and development
Where data is prepared and the model lifecycle is managed. Three pieces recur: pipelines that ingest, clean, and transform structured and unstructured data; vector databases (Pinecone, Weaviate) that store embeddings for semantic retrieval; and MLOps tooling for versioning, experiment tracking, deployment, and production monitoring. The vector store is the piece most teams underestimate — it’s what makes retrieval work. See Embeddings & Vector Databases.
3. Foundation models
The intelligence layer — large, pre-trained models used as a general base rather than built per project. It’s a competitive market of proprietary models (OpenAI’s o-series, Claude 3, Gemini 2.0) and strong open-source options (Llama 3, Mistral). Models are rarely used raw; they’re specialized three ways:
- RAG — connect the model to an external knowledge base at query time.
- Fine-tuning — continue training on a smaller, domain-specific dataset.
- Adapters — lightweight modules that shift behavior without touching core weights.
4. Serving and orchestration
The layer that turns a model into a system — the runtime for agentic behavior. It translates a high-level goal into executable steps and holds everything together: gateways that host models and route requests; tool-calling runtimes that let a model use APIs, databases, and code interpreters; orchestration frameworks that manage state, memory, and the plan-reflect-correct loop of a multi-step task; and interoperability protocols such as the Model Context Protocol (MCP) that standardize how context passes between models and enterprise systems. This is where most current innovation concentrates.
5. Applications and agents
The top, where AI reaches users: software with embedded intelligent features, interactive copilots that assist inside existing workflows, and autonomous multi-agent systems where specialized agents collaborate on complex problems like logistics or research.
Two shifts worth understanding
Inference now drives the economics. In the early 2020s the headline cost was training a foundation model. Today the dominant cost is inference — running it 24/7 to serve traffic. That reframes engineering around quantization and distillation (smaller, faster models), smart routing and caching, and real-time capacity planning. “How we run the model” has become as consequential as “which model we picked.”
The runtime went agentic. Prompt-in, response-out is giving way to goal-driven agents, and layer 4 has grown a dedicated agentic runtime to support them — planning that decomposes a goal into steps, tool use that reaches external systems, and persistent memory that holds context across a long task. For the pattern in depth, see the agentic systems section.
Put together: building with AI today is a systems problem, not a model-selection problem. The model is one layer of five, and the value increasingly comes from the four around it.
