What managers should know about RAG: A complement to AI models

– Prof. Radul De & Minarva Mallik
IIM Bangalore

Retrieval-Augmented Generation shifts the enterprise AI question from “which model should we use?” to “how do we connect AI models to what we already know?”

Imagine hiring the world’s most intelligent analyst in your team — someone who can synthesise information instantly, reason across complex topics, and communicate with remarkable clarity. Now imagine that analyst has never seen a single internal document from your company. That is precisely the limitation facing enterprises that deploy AI today.

Large Language Models are extraordinarily capable — but they are generalists. They have no knowledge of your customer research, compliance documents, or last quarter’s support tickets. When asked, they either confess ignorance or generate a plausible-sounding but fabricated answer — a phenomenon known as hallucination. Retrieval-Augmented Generation, or RAG, attempts to solve this problem. It is one of the most significant architectural developments in applied AI, and yet it remains poorly understood outside specialist circles.

RAG is a method that combines two capabilities: searching a private knowledge base, and generating intelligent responses grounded in what is found. Instead of relying on what a model memorised during training, RAG first retrieves the most relevant information from your own documents — then uses that to generate an accurate, sourced answer. Think of it as giving your AI five minutes to review the relevant files before responding — every time.

How RAG Works: A Two-Phase Process

Phase 1 — Building the Index (done once). Company documents, such as interview transcripts, policy manuals, legal briefs and support tickets, are broken into meaningful segments. Each segment is converted into a numerical representation called an embedding vector, which captures its semantic meaning. Think of it as tearing a book into topic-wise pages and filing each one in a smart cabinet that understands context, not just keywords.

Phase 2 — Answering a Question (every time). When a user submits a prompt or asks a question, it is converted into the same numerical format and matched against the index based on semantic similarity rather than keywords, which is the norm in traditional search methods. The most relevant segments are retrieved and passed to the AI model, which uses the source material to generate a grounded response, often citing where the information came from.

“Find the most relevant information, show it to the AI, and let the AI explain it in plain English.”

RAG in Practice: A Product Management Use Case

Let us now consider how RAG works in practice. Imagine a product manager working on an expense management application who has accumulated months of research, including interview transcripts, support tickets, app reviews, user testing documents, and records of previous outputs—amounting to thousands of pages. Finding specific insights often means manually searching through the material, reading multiple documents, missing hidden connections, and spending hours synthesising information.

With a RAG system built on the same document repository, answers can be obtained in seconds. A query such as “reimbursement complaints” retrieves passages where users may have described “getting paid back” through semantic matching rather than exact keywords—something traditional keyword searches would likely miss. The AI then generates a grounded response, citing the source documents and providing a confidence rating, giving product managers the traceability and reliability they need.

RAG Applications Across Industries

The same RAG approach can be applied wherever organisations manage large volumes of domain-specific information that employees need to search, interpret and synthesise. Its applications are rapidly expanding across industries.

In banking and financial services, compliance teams can query internal policy repositories to instantly retrieve relevant regulations, reducing both manual effort and compliance risk. In healthcare, clinicians can access patient records and treatment protocols while ensuring sensitive data remains within secure enterprise systems. Legal professionals can search an organisation’s case history by describing a legal scenario in plain English, rather than relying on case names or keywords. In manufacturing, engineers can quickly retrieve repair procedures and equipment documentation to diagnose faults and resolve issues in real time.

According to Menlo Ventures, enterprise adoption of RAG grew from 31 per cent to 51 per cent in 2024 alone, with organisations now using it in 30–60 per cent of their AI applications where accuracy and proprietary data are critical. The report also estimates that the average RAG project generates a return of $3.70 for every dollar invested. Supporting this growth is the rapid expansion of the vector database market—the specialised software used to store the embedding vectors created during the indexing phase—which is projected to reach nearly $9.86 billion by 2030.

A growing ecosystem of vector database providers caters to different organisational needs. ChromaDB and Qdrant are well suited for prototyping and smaller deployments, while Pinecone offers a fully managed, cloud-native solution for enterprise-scale applications. Weaviate combines vector and keyword search within a single query, and Milvus is designed to handle GPU-accelerated deployments across hundreds of millions of documents. Choosing the right vector database has significant implications for cost, latency and retrieval accuracy, making it just as important as selecting the AI model itself.

The Challenges

RAG offers immense promise, but success in a prototype does not always translate into reliable performance in production. Three challenges, in particular, determine whether a RAG implementation succeeds at scale.

Scale can affect accuracy. A system that performs well across a few hundred documents may begin retrieving less relevant information when scaled to tens of thousands of documents. If the wrong context is retrieved, the AI model can generate confident but inaccurate responses—the very problem RAG is designed to prevent.

Chunking and embedding quality matter. The way documents are segmented and the embedding model used to convert them into vectors have a direct impact on retrieval accuracy. Decisions made during the prototype stage can become costly to fix once the system is deployed across the enterprise.

Maintenance is a continuous process. As organisations generate new documents and update existing ones, the knowledge index must be refreshed regularly. Access permissions need to be enforced, and response quality should be monitored through systematic evaluation rather than occasional spot checks.

Consider the experience of a technology manager at a multinational corporation who deployed a RAG system across more than 30,000 internal documents. While the prototype performed well, the production system became slow and unreliable as it scaled, eventually forcing the team to withdraw the deployment.

These challenges are solvable, but they require careful planning, robust engineering and ongoing investment. Organisations that treat RAG as a long-term capability rather than a plug-and-play solution are the ones most likely to realise its full potential.

Conclusion for enterprises

RAG shifts the enterprise AI question from “which model should we use?” to “how do we connect AI models to what we already know?” The organisations that gain the most will not necessarily have the most powerful models; they will be those that most effectively ground AI in their own institutional knowledge.

RAG is not a finished product. It is a framework — one that scales from a prototype built by a single product manager to a deployment serving thousands across the, possibly global, organisation. What remains constant is the underlying principle: the AI is only as good as what you feed it. RAG ensures you feed it the right thing.

Mallik is an IIM Bangalore Alumni; De’ (Retired), IIM Bangalore and Memoric AI.

Source: Hindu Business Line