Building a 100% Local MCP Client with LlamaIndex and Ollama

An MCP client is the part of an AI application that opens standardized connections to external tools and data. This walkthrough builds one that runs entirely on local hardware — no data leaves the machine, and there are no API bills — by pairing a LlamaIndex agent with an Ollama-served model and a small SQLite MCP server.

The stack

• LlamaIndex — builds the agent and wraps discovered MCP tools as native callable functions.
• Ollama — serves the local model (Deepseek-R1 in this example).
• SQLite — a small local database, exposed through an MCP server.
• LightningAI — optional, for development and hosting.

How it runs

The loop is deliberately simple, and every step stays local:

1. The user sends a query to the agent.
2. The agent connects to the local MCP server and discovers the available tools.
3. Based on the query, it invokes the right tool — for example, querying the SQLite database.
4. The tool returns context to the agent.
5. The agent uses that context to generate and return its answer.

Building it

1. Build a SQLite MCP server. A minimal server with two tools — add_data and fetch_data — keeps the example focused. The client architecture is unchanged whether the server exposes two tools or two hundred.

2. Serve a local model with Ollama. Run a local instance of Deepseek-R1 through Ollama. Because the model runs on the machine, no prompt or result is sent to an external API.

3. Write the system prompt. Instruct the agent to reach for its tools to gather context before answering, rather than relying on the model’s parametric memory.

4. Define the LlamaIndex agent. Build a FunctionAgent and pass it the tools discovered from the MCP server; LlamaIndex exposes each one to the model as a native callable function.

5. Handle the interaction. A small controller manages the conversation: it passes user messages to the FunctionAgent, holds a shared context for memory, streams tool calls as they happen, and returns the final reply.

6. Wire it together. The main script launches the MCP client, loads its tools, wraps them for LlamaIndex, and initializes the agent with those tools and the context manager. The agent is then ready to run.

With that in place, the agent answers questions against the SQLite MCP server’s tools — a complete agentic loop with nothing leaving the local machine.