Architecting a Private, Multi-User Agentic RAG System

This is a blueprint for a self-hosted platform that gives every user their own agentic chatbot — one that answers only from documents the user is permitted to see, with no data leaving your infrastructure and no per-token API cost. Local models handle both embedding and reasoning; permissions are enforced at the data layer, not bolted on after retrieval.

Everything hangs off three operational flows served by six loosely coupled components.

Components

1. Application server (Python) — the core: exposes the front-end API and talks to the message queue.
2. Front-end (Streamlit or Angular) — auth, file management, and chat. Streamlit for a fast prototype; Angular for production.
3. Blob storage (Minio) — S3-compatible object storage for the original uploaded files.
4. Vector database (PostgreSQL + pgvector) — one database for both relational data (users, metadata, access groups) and embeddings. Because retrieval is a SQL join, you can filter vector search by permission in the same query — the key to multi-tenant isolation.
5. LLM host (Ollama) — runs open-source models locally, typically two: a lightweight one for embeddings and a stronger one for chat and reasoning.
6. Message queue (RabbitMQ) — decouples the front-end from heavy background work. Uploads return immediately while workers embed files asynchronously, which also lets you scale workers independently.

Flow 1: File submission

1. The user authenticates and receives a token.
2. They upload a file and assign it to one or more access groups.
3. The API validates the token and stores the file in Minio.
4. The file’s metadata — storage location and access groups — is written to Postgres.
5. A message carrying the file_id is published to RabbitMQ.
6. The API returns success immediately; embedding proceeds in the background.

Flow 2: Asynchronous embedding

1. A worker consumes a message from the queue.
2. Using the file_id, it fetches the file’s metadata and location.
3. It downloads the file from Minio.
4. It extracts the content, splits it into chunks, and embeds each chunk via the local Ollama embedding model.
5. It writes the chunks and their vectors to Postgres, linked to the file’s access-control data.

Flow 3: Agentic chat and secure retrieval

1. The user sends a prompt to their private agent.
2. The system loads recent conversation history for context.
3. A LangGraph workflow runs the agent’s reasoning.
4. The LLM decides whether it needs more information — that is, whether to call a tool.
5. If so, it runs vector-search. The query is filtered at the database level to only the chunks this user may access.
6. Retrieved context feeds back into the LLM’s reasoning.
7. The final, grounded answer streams back to the user.

What makes it work

• Permission-aware retrieval in one store. Putting metadata and vectors in the same Postgres/pgvector database means access control is a transaction-safe join, not a fragile post-filter. This is the security backbone.
• LangGraph for agent control. Modeling the agent as a state machine makes “when to call a tool” explicit and inspectable.
• Local agents are viable. With Ollama, a fully private, self-hosted agentic system runs on commodity hardware.
• Loose coupling scales. The message queue and separate services let you scale the slow part — embedding workers — without touching the rest.

Keep going

• Vector Databases
• Building an AI Agent with Dual Memory
• Designing Effective Agent Tools

Privacy here isn’t a policy — it’s an architecture. Local models plus permission-scoped retrieval mean sensitive documents are answerable without ever being sent anywhere.