How to Train an AI on Your Company Data Without a PhD

How to Train an AI on Your Company Data Without a PhD

One of the most powerful things you can do with AI in your business is make it work with your specific knowledge - your documentation, your processes, your historical data, your institutional memory. The question is: how do you actually do that?

The answer depends on which approach is right for your use case. There are three main approaches, and choosing the wrong one will cost you significant time and money. This post explains all three in plain English, tells you when each is appropriate, and walks through the implementation path for the most commonly relevant approach.

The Three Main Approaches

Approach 1: Prompt Engineering with Context

What it is: You put your data directly into the prompt. The LLM reads it, reasons over it, and responds.

When it works: When your data is small enough to fit in a context window and you do not need real-time updates. For example, if your knowledge base is 20 FAQ pages, you can paste them into the system prompt and the LLM will use them when answering questions.

Limitations: Context windows are large but not unlimited. GPT-4o can handle around 128k tokens (~100,000 words). Claude can handle up to 200k tokens. This sounds like a lot, but a mid-size company's knowledge base often exceeds this. More importantly, stuffing everything into the prompt is expensive (you pay for all those tokens on every request) and the LLM does not always prioritise the most relevant content well when there is too much of it.

Best for: Simple prototypes, small knowledge bases (under 50 pages), one-off document analysis tasks.

Approach 2: Retrieval-Augmented Generation (RAG)

What it is: Instead of putting all your data in the prompt, you build a system that retrieves only the relevant pieces of data when a question is asked, then injects those pieces into the prompt for the LLM to reason over.

When it works: This is the right approach for most business data use cases. It scales to millions of documents, updates as your data changes, is far more cost-efficient than stuffing full documents into every prompt, and generally produces better answers because the LLM is working with focused, relevant context rather than drowning in everything.

Limitations: Requires more engineering to set up. If your data is very complex or requires understanding relationships across many documents simultaneously, retrieval may not capture the right context. Quality depends heavily on how well you chunk and index your data.

Best for: Company knowledge bases, document Q&A systems, internal search tools, customer support bots with large FAQ libraries, any system where your data exceeds the context window or changes frequently.

Approach 3: Fine-Tuning

What it is: You take a pre-trained model and continue training it on your specific data, adjusting the model's weights so it learns patterns specific to your domain.

When it works: When you need the model to produce outputs in a very specific format or style consistently (and prompt engineering cannot achieve this), when you are dealing with a highly specialised domain where the base model lacks knowledge, or when you have thousands of labelled examples and need consistent behaviour across them.

Limitations: Fine-tuning is expensive (compute cost for training runs), requires significant data preparation and labelling, needs ML expertise to do correctly, does not update automatically as your data changes (you need to retrain), and often underperforms a well-engineered RAG system for knowledge retrieval tasks. Many businesses think they need fine-tuning but actually need RAG.

Best for: Highly stylised output (the model needs to write in a very specific format), domain-specific classification, situations where you have thousands of labelled training examples, and use cases where RAG has provably failed.

Why RAG Is Usually the Right Choice for Business Data

Most of the "we want to train AI on our data" requests from businesses translate to: "we want AI to answer questions accurately using our internal knowledge." That is a RAG use case, not a fine-tuning use case.

Here is why RAG wins for this:

• Always up to date: Add a document to your knowledge base and it is available for retrieval immediately. Fine-tuned models require expensive retraining to add new knowledge.
• Cheaper: RAG's incremental cost is embedding and retrieval. Fine-tuning costs thousands of dollars in compute for a training run, plus the engineering time to manage it.
• Explainable: RAG can show you which document chunks it retrieved to generate an answer. This is critical for trust - you can verify that the AI answered based on something real in your knowledge base.
• Separates concerns cleanly: Your data and your AI reasoning are separate systems. You can update your data without touching the AI, and you can swap AI models without touching your data.

How RAG Works: A Non-Technical Explanation

RAG has four stages:

Stage 1: Chunking

Your documents are split into chunks - pieces of text small enough to be individually searchable but large enough to contain complete ideas. A naive chunking approach splits every 500 characters. A good chunking approach respects document structure: paragraphs stay together, section headers are preserved, lists are not split mid-item.

Poor chunking is one of the most common reasons RAG systems produce bad answers. If a policy document's relevant clause is split across two chunks and only one chunk is retrieved, the answer will be incomplete.

Stage 2: Embedding

Each chunk is converted to an embedding - a numerical representation that captures semantic meaning. Two chunks that mean similar things will have similar embeddings, even if they use different words. This is what enables semantic search rather than keyword search.

Embedding models (like OpenAI's text-embedding-3-small or open-source alternatives like BGE) convert text to vectors of numbers, typically 768-3072 dimensions.

Stage 3: Vector Search

When a user asks a question, that question is also converted to an embedding. The system then searches the vector database for chunks whose embeddings are closest (most similar) to the question's embedding. This retrieves the most semantically relevant chunks.

The result: if someone asks "what is our policy for remote work expenses?" the system retrieves the chunks about expense policies, remote work allowances, and reimbursement procedures - even if those chunks do not contain the exact words the user typed.

Stage 4: Retrieval and Generation

The retrieved chunks are inserted into the LLM's prompt along with the user's question. The LLM generates an answer using only the retrieved context. Crucially, a well-designed prompt instructs the LLM to answer based only on the retrieved context and to say "I do not know" if the answer is not there, rather than hallucinating.

What Data You Can Use

RAG works with almost any text-based data:

• Internal documentation: Process docs, runbooks, policy manuals, SOPs, training materials
• PDFs: Contracts, reports, research papers, product manuals
• Databases: Any structured data can be converted to natural language and indexed, or queried via tools at retrieval time
• Emails and Slack messages: With appropriate processing and privacy controls
• Websites: Your help centre, knowledge base, or product documentation pages
• Spreadsheets: Row-by-row data can be converted to text chunks

What does not work well:

• Scanned PDFs without OCR: Images of text are not readable without an OCR preprocessing step
• Audio and video: Need transcription first
• Complex relational data: Data that requires joins and calculations is better handled by letting the LLM query a database via tools rather than RAG

Data Preparation Requirements

Before ingesting data into a RAG system, it needs to be:

1. Clean: Formatting artefacts from PDF conversion (page numbers mid-sentence, header/footer text, corrupted encoding) degrade retrieval quality.
2. Deduped: Multiple versions of the same document mean retrieval may return outdated information. Decide which version is authoritative.
3. Structured for chunking: Headings and section markers help the chunking algorithm preserve context. Documents with no structure are harder to chunk well.
4. Privacy-reviewed: If your data includes personal information about employees, customers, or patients, you need to decide what goes into the AI system and ensure it is handled under your privacy policies.

Data preparation is typically 20-40% of total project time for RAG systems. Teams consistently underestimate this.

Cost and Timeline

For a typical business RAG system (company knowledge base, 100-1,000 documents):

Ongoing costs:

• Embedding API (infrequent, as new documents are added): $5-50/month
• Vector database: $0-300/month depending on data size and provider
• LLM API (query costs): volume-dependent

Tools for Building RAG Without Deep ML Knowledge

LlamaIndex

The most user-friendly framework for building RAG systems. It handles data loading from dozens of source types (PDF, Word, Notion, websites), chunking, embedding, and vector index management with relatively little boilerplate code.

A functional RAG system can be built with LlamaIndex in Python in a few hundred lines of code. The documentation is good and the community is active. See our LangChain vs LlamaIndex comparison for more on when to use each.

LangChain

More flexible than LlamaIndex for complex retrieval strategies but has a steeper learning curve. Often used in combination with LlamaIndex for the retrieval layer.

Pinecone

The most popular managed vector database. No server management required - you create an index, push vectors, and query. Starts free, scales with usage.

pgvector

If you already use PostgreSQL, pgvector is an extension that adds vector search. This avoids adding a new database to your infrastructure and works well for small-to-medium datasets (under a few million vectors).

Chroma

Open-source, self-hosted vector database. Good for development and for production at medium scale. Zero vendor cost but requires management.

What You Cannot Do Without Fine-Tuning

To be complete: there are legitimate use cases where RAG is not enough and fine-tuning is necessary.

Consistent format and style: If you need an LLM to consistently produce output in a specific format (eg. your company's proprietary report structure with specific fields), and prompt engineering cannot reliably enforce this, fine-tuning on examples teaches the model that format deeply.

Highly specialised domain knowledge: If your domain uses terminology, concepts, or reasoning patterns that the base model has limited exposure to (very niche scientific, legal, or technical domains), fine-tuning on domain-specific text can improve baseline competence.

Latency-sensitive, high-volume classification: If you need to classify thousands of items per second and cannot afford the latency of complex retrieval + generation, a fine-tuned smaller model may be the only viable approach.

For everything else - and for 80-90% of "train AI on our data" requests - RAG is the faster, cheaper, more maintainable path.

Getting Started

The right starting point is a data audit. Before any technology decision, inventory your company data:

• What documents and knowledge exist?
• In what formats?
• How much of it is current and accurate?
• What questions do you most want the AI to be able to answer?

The answers to these questions determine the right chunking strategy, retrieval approach, and evaluation criteria before a line of code is written.

If you want to build a RAG system or any AI system that works with your company data, our LLM Integration service covers the full pipeline. You can also use our AI Feasibility Checker to quickly assess how well suited your use case is for an AI knowledge base.

Get a free quote to discuss your specific data and requirements.