LLM and System Architectures: From Transformers to Agents
AI architecture lives at two layers, and confusing them is a common source of bad design decisions.
- Model architecture is the neural network itself — the design that gives a Large Language Model its raw capability. The Transformer is the industry standard.
- System architecture is the pattern you build around the model to make it useful: direct prompting, Retrieval-Augmented Generation (RAG), fine-tuning, or an agentic loop.
You rarely change the first. You always choose the second. This reference covers both.
The Transformer
The Transformer, introduced in the 2017 paper Attention Is All You Need, underpins virtually every modern LLM — GPT, Gemini, Claude, and the open-weight families alike. Its defining move is the self-attention mechanism: for each token, the model weighs how much every other token in the input should influence its meaning. That is what lets it track context across a long passage instead of reading words in isolation.
| Component | Role | What it does |
|---|---|---|
| Embeddings | Representation | Converts tokens (words or sub-words) into numerical vectors that encode meaning. |
| Positional Encoding | Order | Injects each token’s position, since self-attention has no inherent sense of sequence. |
| Self-Attention | Contextual weighting | Scores how much every other token matters when interpreting a given token. The core mechanism. |
| Multi-Head Attention | Parallel focus | Runs self-attention many times in parallel — one head may track grammar, another long-range references. |
| Encoder / Decoder | Transform & generate | The encoder builds a representation of the input; the decoder generates output token by token. Most current LLMs are decoder-only. |
| Feed-Forward Network | Processing | A standard layer applied after attention to transform its outputs further. |
Four system patterns
The Transformer is the engine; the system pattern is the vehicle. The right choice depends on the task, the data it needs, and the trade-offs you can accept.
| Pattern | How it works | Best for | Trade-off |
|---|---|---|---|
| Zero / Few-Shot Prompting | The model is used directly with a well-crafted prompt — no external data, no training. | Quick tasks, drafting, classification, brainstorming. | Limited to pre-trained knowledge; no private data; more prone to hallucination. |
| RAG | Relevant documents are retrieved (usually from a vector database) and passed to the model as context, grounding the answer in real sources. | Q&A over private or current documents, support bots, internal search. | Added latency and retrieval complexity; only as good as the knowledge base. |
| Fine-Tuning | A pre-trained model’s weights are further trained on a smaller, domain-specific dataset to shift its behavior or style. | Consistent brand voice, niche terminology, reliable structured output. | Costly dataset and training work; risks eroding general ability (“catastrophic forgetting”). |
| Agentic | The model acts as a reasoning engine that plans, calls tools, and loops until a goal is met. | Multi-step automation, research, workflows that touch the outside world. | Highest complexity and cost; harder to debug and constrain. |
The model’s job shifts with the pattern. In prompting it is the entire system. In RAG it is a synthesizer that reasons over supplied context. In fine-tuning it is a specialist reshaped for a narrow task. In an agentic system it is the orchestrator that plans and delegates.
Choosing a pattern
Balance capability against cost, complexity, and control.
| Consideration | Prompting | RAG | Fine-Tuning | Agentic |
|---|---|---|---|---|
| External / live data | None | Yes | No (baked in) | Yes |
| Development complexity | Low | Medium | High | Very high |
| Development cost | Low | Medium | High | Very high |
| Inference cost | Low | Medium (larger context) | Low (if tuned small model) | High (multiple calls) |
| Latency | Low | Higher | Low | High |
| Controllability | Low | Medium | High (on-task) | Medium (can drift) |
| Up-to-date knowledge | No | Yes | No | Yes |
Heuristic: start with prompting. If it fails on knowledge gaps, add RAG. If it fails on reasoning or style, consider fine-tuning. If the task needs multi-step action in the world, build an agent. Escalate only when the simpler pattern demonstrably falls short.
Mixture of Experts
The most significant recent shift in model architecture is the Mixture of Experts (MoE) design. Instead of one dense network where every parameter fires for every token, an MoE model splits into many smaller expert networks plus a router that sends each token to the most relevant few. Only a fraction of the model activates per token.
That decoupling of total size from per-token compute buys three things: faster inference, cheaper training relative to a dense model of equal size, and far higher total parameter counts kept affordable. Mistral’s Mixtral 8x7B and Google’s Gemini family use MoE architectures.
Key takeaways
- Architecture is two layers: the internal model (the Transformer) and the external system (prompting, RAG, fine-tuning, agents).
- Self-attention is what gives the Transformer its grasp of context and sequence.
- Match the system pattern to the need — RAG for knowledge, fine-tuning for skill and style, agents for autonomous action.
- Choosing is a trade-off among capability, cost, complexity, and control. Start simple; escalate only when forced.
- MoE decouples total parameters from per-token compute, pushing performance and efficiency at once.
