Advanced Multimodal RAG

Standard multimodal retrieval-augmented generation (RAG) tends to retrieve the wrong images, tables, and figures. The reason is simple: it captions each image in isolation, and a caption written from pixels alone loses the narrative context that made the image relevant in the first place. This pipeline fixes that with two changes — one at ingestion, one at generation.

Why standard multimodal RAG misses

The standard approach assumes an image’s own content is enough to describe it. In real documents it isn’t. Two visually near-identical tables labeled “working capital” might differ only in the sentence above them — one covers farmers, the other processors. A pixel-only caption erases exactly the distinction a query depends on. Relevance lives in the text around the image, not just the image.

Fix 1: context-aware image summaries (ingestion)

Instead of captioning from the image alone, build a richer summary from the surrounding document text.

• Process: for each image, capture roughly 200 characters of text before it and 200 after.
• Result: that combined text becomes the image’s embedded summary. It naturally folds in the author’s caption and the narrative context, so the stored representation reflects what the image means in the document — not just what it depicts.

Fix 2: answer-guided image selection (generation)

Rather than matching the user’s short, often ambiguous query against image embeddings directly, retrieve in two stages:

1. Generate the text answer first. Retrieve the top text chunks for the query and have the LLM write the full answer.
2. Retrieve images for that answer. Use the complete generated answer as the query vector against the context-aware image summaries.

The images then match the answer actually being given, not the vague question that started it.

Implementation sequence

1. Parse. Extract text, images, and their positions with a robust tool such as the Adobe PDF Extract API.
2. Caption in context. For each image, locate its position in the document structure and concatenate the surrounding text into its context-aware summary.
3. Embed. Create embeddings for every text chunk and every context-aware image summary; store them in a vector database such as FAISS.
4. Retrieve and generate.
5. Retrieve text chunks for the user query.
6. Generate the final text answer.
7. Use that answer to retrieve the top N images.
8. Optionally, generate a clean display caption for each selected image before showing it.