Architecture for a Private, Multi-User Agentic Assistant
The hard part of a multi-user assistant isn’t the chat — it’s isolation. Many people share one deployment, yet each must only ever retrieve from documents they’re allowed to see, and nothing should leak to an outside API. This architecture delivers that with a fully self-hosted, permission-scoped RAG stack.
Four principles shape it:
- Privacy-first — every component, including the models, runs locally. No external dependencies, no data egress.
- Multi-tenant — one platform, many users, each with a private agent over isolated data.
- Agentic — the chatbot is an agent with tools (notably vector search) it decides when to use, not a fixed retrieval pipeline.
- Scalable — a message queue decouples heavy work, so embedding load scales across workers independently of the API.
Components
| Component | Technology | Role |
|---|---|---|
| App | Python (e.g. FastAPI) | Core logic; exposes APIs and dispatches background jobs. |
| Front-end | Streamlit (prototype) | Auth, file upload/delete, and chat. |
| Blob storage | Minio | S3-compatible store for raw uploaded files. |
| Database | Postgres + pgvector |
Relational data (users, groups, metadata) and vector embeddings in one system, enabling permission-scoped queries. |
| Model host | Ollama | Serves local models for both embedding and chat. |
| Queue | RabbitMQ | Decouples the API from embedding work so uploads return instantly. |
Keeping relational data and vectors in the same Postgres instance is the key choice: it lets a single query join “closest chunks” with “chunks this user may access,” so access control is a WHERE clause, not a bolted-on afterthought.
Three workflows
1. File management. The user authenticates and receives a JWT. They upload a file and assign it to one or more access groups. The app stores the raw file in Minio, records its metadata and permissions in Postgres, publishes the file_id to the queue, and returns success immediately — the UI never waits on processing.
2. Asynchronous embedding. A worker pulls a file_id from the queue, fetches the metadata, downloads the file from Minio, extracts and chunks the text, embeds each chunk with the local model, and writes the chunks and vectors back to pgvector — linked to the file’s metadata and its access-control groups. Because this runs in the background, large files never block anyone.
3. Agentic chat. The user sends a message; the token is validated and recent history loaded for context. The LangGraph agent is invoked. The model reasons about the request and decides whether it needs a tool. If it calls vector search, the query runs against Postgres filtered to chunks the current user is permitted to access — this is where tenant isolation is enforced. The agent folds the retrieved passages into its reasoning and streams the answer back.
The agent loop (LangGraph)
The agent is a small graph: the model reasons, and if it decides a tool is needed, the graph routes to a tool node (e.g. vector search). The tool’s output returns to the model, which either answers or calls another tool. When no further tools are needed, it produces the final response. This loop is what makes retrieval a decision the agent makes, rather than a step every query is forced through.
Alternative: serverless self-hosting
This design self-hosts by running containers you manage. A serverless alternative pushes the same idea onto managed cloud primitives — running the agent on Cloudflare Workers with sandboxed execution and object storage — trading container management for platform lock-in. See Moltworker.
Where to take it next
- Incremental re-embedding for documents that change over time (e.g. syncing a live vault).
- Citations — return file, page, and chunk so users can verify answers.
- More tools — structured summarizers, SQL access, knowledge-graph queries.
- A production front-end to replace the prototype UI.
