Skip to content
AI Development

How to Build a RAG Chatbot over Your Company Documents

Written by
Developer at Appyweb
8 min read Updated November 6, 2026

The short answer

To build a RAG chatbot over company documents, clean up and scope the sources, split them along their structure, index them for hybrid search with permissions stored as metadata, retrieve and rerank the best passages for each question, and have the model answer only from those passages, with citations. Test retrieval and answers on real questions before launch.

To build a RAG chatbot, you set it up to answer questions from your company documents by pulling the most relevant passages at question time and having a language model answer only from those passages, with citations. Getting one to hold up in production comes down to five things: clean sources, chunks that follow each document’s structure, hybrid search filtered by permissions, a prompt that keeps answers grounded, and a set of real questions you rerun before every change.

What does a RAG chatbot’s architecture look like?

A RAG chatbot runs two pipelines: one that prepares your documents ahead of time, and one that runs on every question. It’s the same structure behind our RAG and LLM development work. If you want the concept before the build, read what RAG is first.

The ingestion pipeline (runs when documents change)

  1. Step 1

    Connect to sources

    Shared drives, SharePoint, Confluence, a help desk or a database.

  2. Step 2

    Parse into clean text

    Keep headings, tables and links; drop menus and boilerplate.

  3. Step 3

    Chunk along structure

    Sections, FAQ entries, clauses, help articles.

  4. Step 4

    Attach metadata

    Title, section path, URL, owner, last-updated date and access groups.

  5. Step 5

    Embed and index

    Write chunks to an index that supports both vector and keyword search.

Then, for every question:

The query pipeline User question → Hybrid search filtered by permissions → Rerank and keep the best passages → Grounded prompt with sources → Answer with citations, or a handoff
  1. User question
  2. Hybrid search filtered by permissions
  3. Rerank and keep the best passages
  4. Grounded prompt with sources
  5. Answer with citations, or a handoff

Here’s that query pipeline in a sample scenario: a resident at a fictional Denver property management company asks about pets, and every sentence in the answer points back to the policy it came from.

Illustrative example of an AI assistant that answers from company documents for a fictional Denver property management company. A resident asks whether dogs are allowed in the Elm Street units and whether there is a pet deposit. The assistant searches the knowledge base, picks two relevant passages, from the pet policy and the Elm Street lease addendum, and writes an answer where every sentence cites its source: dogs up to 50 lb are allowed, with a one-time refundable $300 deposit plus $35 a month in pet rent, and service and support animals are not treated as pets. Questions the documents don’t cover go to the leasing team.

Illustrative example Fictional business, people and data. Not a real client, conversation or result.

The key decisions at a glance

Decision Options How to choose
Chunking Fixed size, by structure, parent-child Follow the document’s structure; use parent-child when answers need the surrounding context
Embedding model Hosted embedding APIs or open models you run yourself Test on your own questions; check cross-language support if you need English and Spanish
Index Postgres with pgvector, a dedicated vector database (Pinecone, Qdrant, Weaviate) or a search engine with vectors (Elasticsearch, OpenSearch) Use what you already run unless scale or filtering needs justify another system
Retrieval Vector, keyword, hybrid, plus reranking Hybrid with reranking is the safest default
Generation model Models from several providers Choose based on your evaluation set: faithfulness, latency, cost
Evaluation Manual review, automated checks, or both Both, against a fixed test set with expected sources

Retrieval versus training the model on your content is a separate question, and our comparison of RAG vs fine-tuning covers it. For company documents that change, retrieval almost always wins.

Step 1: Scope the sources and their owners

Decide which documents the bot is allowed to answer from, and who owns each set. A chatbot pointed at “everything on the shared drive” will happily quote a 2019 draft of a policy that’s changed twice since. For each source, write down its owner, how often it changes, who’s allowed to see it and whether it’s the official version. Leave out drafts, duplicates and archives, or tag them so retrieval can filter them out. Stale and conflicting documents are one of the most common causes of wrong answers we see, and no model can fix them.

Step 2: Parse documents without losing structure

Turning files into text is where quality slips without anyone noticing. Watch for PDFs whose columns, headers and footers get mixed into the body text; tables flattened into a stream of numbers (convert them to Markdown, or keep each row with its column headers); and web pages full of menus and cookie banners. Scanned documents need OCR first, and if you need specific fields from them rather than passages, that’s a document processing problem. See IDP vs OCR.

Keep the headings. They’re your best signal for chunking and for readable citations.

Step 3: Chunk by structure, and keep context

A chunk is the unit you retrieve. Too small and it loses context; too big and it dilutes the match and eats up the model’s context window. Here’s what works for us:

  • Split along structure, and use size limits only as a fallback for long sections.
  • Overlap a little between back-to-back chunks of the same section so sentences don’t get cut in half.
  • Prefix each chunk with its path, like “Employee Handbook › Leave › Parental leave.” It improves retrieval and makes citations readable.
  • Use parent-child retrieval for long documents: search the small chunks, then send the model the section they belong to.

There’s no universal chunk size. Try two or three settings against your evaluation set and keep the one that pulls the right passage most often.

Step 4: Embeddings and the index

Embeddings turn text into vectors so that similar meanings land close together. Pick an embedding model by testing it on your own questions, not by its leaderboard ranking, and if users will ask in Spanish about English documents, test that specifically. Store the model name and version with each chunk, because switching embedding models means re-embedding everything.

Use the database you already run if it supports vectors. Postgres with pgvector handles plenty of company knowledge bases. Move to a dedicated vector database when volume, latency or filtering needs make another system worth running.

Step 5: Retrieval that finds the right passage

Pure vector search is good at meaning and bad at exact strings. Keyword search is the opposite. Use both:

  • Hybrid search: run keyword (BM25) and vector search, then merge the results.
  • Metadata filters: permissions first, then product, region, document type or date.
  • Reranking: a reranking model reorders the top candidates by how relevant they are to the question. It often matters more than the embedding model.
  • Query rewriting for follow-ups, so “and in Texas?” becomes a complete question before the search runs.

Step 6: Generation with citations

The prompt sets the rules. Ours always say: answer only from the passages provided; cite the source of each claim by title and section, linked to the original; if the passages don’t contain the answer, say so and offer the handoff; ignore instructions that show up inside documents; and reply in the user’s language.

Keep the temperature low and the passages clearly separated. Show citations as links in the interface so users can check a source in one click. That’s what gets people to trust the answers.

Step 7: Permissions and security

Also plan for prompt injection hidden in documents (treat retrieved text as data, and keep write tools away from a Q&A bot unless it truly needs them), personal data in your sources (decide whether HR files, customer records or health information belong in the index at all), and the logs themselves, since questions can contain sensitive information too.

Step 8: Evaluate before you launch

Build a test set of real questions, ideally collected from the people who’ll use the bot: the questions they ask coworkers, the tickets they get. For each one, write down the expected answer and the document it should come from, and include questions the bot should refuse.

Metric The question it answers
Retrieval hit Was the right source among the passages retrieved?
Answer correctness Is the answer right, according to the document owner?
Citation correctness Does the cited passage actually back up the answer?
Correct refusal Did the bot say “I don’t know” when it should have?

Have document owners review a sample, and rerun the whole set after every change to chunking, prompts, models or sources.

Step 9: Launch and keep it in sync

After launch

Should you build a RAG chatbot or use an off-the-shelf tool?

Built-in assistant vs custom RAG chatbot

Assistant built into your suite

Sources
Usually just the suite’s own files
Permission filtering
The suite’s own model, if it respects it
Citations and format
Set by the vendor
Customer-facing use
Rarely a good fit
Inside your own product
No
Effort to get started
Low

Custom RAG chatbot

Sources
Any mix: drives, wikis, help desk, databases
Permission filtering
Designed around your identity provider
Citations and format
Your format, linked to the original
Customer-facing use
Built with your guardrails and handoff
Inside your own product
Yes, through your API
Effort to get started
A real project: ingestion, evaluation, upkeep

If your documents already live in one suite and its assistant respects your permissions, try that first. Go custom when you need several sources, strict permission filtering, your own citation format, a customer-facing bot, or the chatbot built into your product. For support use cases, see our AI customer support agents. To plug the bot into an existing application, read how to integrate AI into existing software.

Tell us what your documents look like and who should be able to ask what, and we’ll propose an architecture for a first version.

Frequently asked questions

How many documents do I need for a RAG chatbot?

There’s no minimum. What matters is that the documents cover the questions people actually ask and that they’re up to date. A small, well-kept set beats a big one that contradicts itself.

Do I need a vector database?

You need an index that supports vector search, ideally combined with keyword search. If you already run Postgres, the pgvector extension is often enough. A dedicated vector database makes sense at larger scale or when your filtering needs are demanding.

Can a RAG chatbot answer in Spanish from English documents?

Yes. Multilingual embedding models can match questions and documents across languages, and the model can answer in the user’s language. Test it specifically with your own questions, because quality varies from model to model.

How do I stop the chatbot from making things up?

Make it answer only from retrieved passages, require citations, let it say it doesn’t know, and measure faithfulness on a test set. Most made-up answers happen when retrieval failed and the model filled in the gap.

What does it cost to run a RAG chatbot?

Running costs come from embedding documents, storing the index and the model calls for each question, plus the time someone spends reviewing feedback and keeping sources current. Measure cost per question during the pilot and multiply by expected usage before you scale up.

More on this topic

Keep reading

All topics
AI Development

How to Build an AI Agent: A Step-by-Step Guide for Business Use

To build an AI agent for business, define one job and what "done" looks like, gather the knowledge it answers from, give it a few narrow tools through your systems' APIs, write its instructions, test it on real cases, add guardrails and approvals, and roll it out gradually with logs and a weekly review.

Germán Gutiérrez
· 7 min read
AI Automation

n8n vs Make vs Zapier for AI Workflows: Which One Fits Your Business?

Pick Zapier when non-technical teams need to hook up popular SaaS apps fast, Make when your workflows branch a lot and reshape data, and n8n when a technical team wants complex AI agent logic, custom code or self-hosting. For high volume or unusual systems, custom code can beat all three.

Germán Gutiérrez
· 8 min read
AI Automation

AI PPC Management: How AI Agents Automate Paid Media in 2026

AI PPC management means connecting AI agents to Google Ads, Microsoft Ads, Meta, GA4 and your CRM so they watch the accounts, investigate what changed and draft fixes, while people approve anything that moves money. Automate monitoring, search term triage, pacing checks and reporting first; keep strategy, budgets, conversion goals and brand claims human.

Raúl Aránega Segura
· 21 min read

Want help putting this into practice?

Tell us what you want to automate. We’ll give you a straight answer on what’s possible and what it would take.

  1. 01 We read what you sent before we talk.
  2. 02 A free call to map the workflows worth automating.
  3. 03 A written proposal with scope and a fixed price.

We need an email or a phone number to get back to you.