The Real Cost of Building an AI Feature Into Your Product

The Real Cost of Building an AI Feature Into Your Product

The most common mistake founders and product managers make when budgeting AI features is treating the AI API cost as the only cost. "GPT-4 charges per token, I will use maybe 10 million tokens a month, that is $30, done."

The actual cost of a production-ready AI feature is usually 5-20x the API cost alone. This post breaks down what you are actually paying for, gives realistic cost models for common feature types, and helps you decide when to build custom versus buy off-the-shelf.

The Four Components of AI Feature Cost

1. API Usage

This is the cost of calling the underlying LLM. Pricing models vary:

• OpenAI (GPT-4o): ~$2.50 per 1M input tokens, ~$10 per 1M output tokens
• Anthropic (Claude Sonnet 3.5): ~$3 per 1M input tokens, ~$15 per 1M output tokens
• Google (Gemini 1.5 Pro): ~$3.50 per 1M input tokens (above 128k context)

For a chatbot handling 1,000 conversations per day, each with an average context of 2,000 tokens and generating 500 tokens of response, you are looking at roughly 2.5M tokens/day in input and 500k tokens/day in output - in the range of $400-700/month in API costs alone for a moderately busy feature.

But this is not the full API cost picture. RAG systems (retrieval-augmented generation) also require embedding API calls and vector database queries. Embedding 1M tokens with OpenAI's text-embedding-3-small costs around $0.02 - cheap for the data pipeline, but it adds up at scale.

2. Engineering Time

This is typically the largest single cost and the most underestimated. Consider what engineering time actually covers for an AI feature:

• Architecture design: Deciding the retrieval strategy, chunking approach, prompt design, memory handling
• Prompt engineering and testing: This is more time-consuming than most developers expect. A good production prompt is not the first thing you try - it is the result of 20-50 iterations against real data
• Integration: Connecting the LLM API to your existing data sources, user authentication, and UI
• Evaluation: Building the testing framework to know whether the AI is giving good answers
• Edge case handling: What happens when the AI refuses to answer? When it gives a wrong answer? When the API is down?
• Latency optimisation: Streaming responses, caching common queries, async processing

A chatbot that looks like a two-day project in demo form is typically a 6-10 week project in production form. The difference is entirely in evaluation, edge cases, and reliability.

3. Infrastructure

AI features often require infrastructure that your existing product does not have:

• Vector database: For any RAG system, you need somewhere to store embeddings. Pinecone, Weaviate, Chroma (self-hosted), or pgvector (if you use PostgreSQL) all have costs.
• Compute for document processing: If you ingest user-uploaded PDFs or documents, you need compute to parse, chunk, and embed them.
• Increased API gateway / backend costs: AI features typically generate significantly more backend traffic than equivalent non-AI features.
• Monitoring: LLM outputs are not deterministic - you need logging to debug issues. Tools like LangSmith, LangFuse, or custom logging add infrastructure.

Typical additional infrastructure cost for a mid-size AI feature: $200-800/month on top of your existing hosting costs.

4. Ongoing Maintenance

AI features are not set-and-forget. Budget for:

• Prompt updates: As the product evolves, prompts need updating. As model providers update models, prompts that worked may degrade.
• Model updates: When providers deprecate a model version, migration requires testing.
• Evaluation monitoring: Tracking answer quality over time as your data changes.
• Cost monitoring: API costs can spike unexpectedly if a feature is used differently than anticipated.

Budget 5-15% of initial build cost per year for maintenance - more if the feature involves complex prompts or rapidly changing data.

Cost Breakdown by Feature Type

Chatbot (General Customer Support)

Semantic Search

Personalised Recommendations

Document Analysis (Q&A over Uploaded Documents)

Image Analysis

These ranges are wide because they depend heavily on scale (how many users and queries), complexity (how many edge cases must be handled), and team rate (offshore vs local engineering).

API Cost Modelling: Thinking About Token Usage at Scale

The token-based pricing model requires explicit modelling before you commit to a feature. Here is how to estimate:

Step 1: Define a typical interaction

• How many tokens in the system prompt? (Often 500-2,000 for a well-engineered prompt)
• How much user input per request? (Maybe 50-200 tokens for a user question)
• How much retrieved context (for RAG)? (500-3,000 tokens of relevant chunks)
• How long is the typical response? (200-1,000 tokens)

Step 2: Estimate daily interactions

• How many users? At what usage frequency?

Step 3: Calculate

• Total input tokens per day = (system prompt + user input + retrieved context) x daily interactions
• Total output tokens per day = average response length x daily interactions
• Apply per-million pricing

Example: A document Q&A tool with 500 queries/day:

• System prompt: 800 tokens
• Average user question: 80 tokens
• Retrieved context: 1,500 tokens
• Average response: 400 tokens
• Input per query: 2,380 tokens x 500 = 1.19M tokens/day
• Output per query: 400 tokens x 500 = 200k tokens/day
• Using Claude Sonnet 3.5: (1.19M x $3 + 0.2M x $15) / 1M = $3.57 + $3.00 = $6.57/day = ~$200/month

At 5,000 queries/day, that is $2,000/month in API costs. Model your specific volume before you commit.

The Vibe-Coded MVP vs Production-Ready Gap

There is a pattern that causes real pain for product teams: the "vibe-coded MVP" that demos beautifully but fails in production.

An MVP built over a weekend to impress stakeholders might have:

• A prompt that works 80% of the time on the demo dataset
• No error handling when the API returns unexpected output
• No streaming (so users wait 15 seconds for a response)
• No fallback when the API is rate-limited
• No evaluation metrics (how do you know if it is getting worse?)
• No cost monitoring (surprise $5,000 API bill after a traffic spike)

Moving from MVP to production typically requires as much or more engineering as building the MVP. Budget for this honestly. A chatbot MVP takes 1-2 weeks. A production chatbot takes 8-12 weeks. The gap is evaluation, reliability, monitoring, and edge case handling.

Off-the-Shelf AI Tools vs Building Custom

For many use cases, you should not build from scratch. The decision tree:

Use an off-the-shelf tool if:

• A tool exists that covers 80%+ of your requirement with acceptable customisation
• The feature is not a core differentiator (support chatbots, document summarisation, FAQ answering)
• Your timeline is under 3 months and speed matters more than perfect fit
• You have limited engineering capacity

Build custom if:

• The feature is a core product differentiator (your AI IS the product, not a feature of it)
• Your data requirements are non-standard (proprietary formats, real-time data, unusual domain)
• You need specific integration with your data model that generic tools cannot provide
• Your volume is high enough that per-seat SaaS costs exceed custom build costs within 18 months

Cost Comparison: Offshore vs Local AI Engineering

Offshore AI engineering from a team like ours versus local UK/US engineering:

The offshore cost range for a comparable skill level is typically 70-85% lower than equivalent UK or US rates. For a 10-week project that would cost £75,000 locally, an offshore team costs £15,000-25,000.

The trade-offs to manage: timezone overlap (we work with UK and US clients across overlapping hours), communication clarity, and code quality standards. These risks are manageable with the right team structure.

Worked Example: Document Q&A Feature

Scenario: A legal technology startup wants to add a feature allowing lawyers to ask natural language questions about uploaded contracts and receive answers with citations to specific clauses.

Requirements: PDF upload, chunking and indexing, semantic search, LLM-generated answers with source references, 100-500 queries/day initially.

Cost Model

Engineering (10 weeks, offshore team):

• PDF ingestion and parsing: 1 week
• Chunking strategy and embedding pipeline: 1 week
• Vector database setup (Pinecone): 0.5 weeks
• Retrieval and prompt engineering: 2 weeks
• API integration and UI: 1.5 weeks
• Evaluation framework and testing: 2 weeks
• Deployment and monitoring: 1 week
• Buffer and bug fixing: 1 week
• Total: ~10 weeks at £180/day average = £12,600

Infrastructure (setup): £2,000-4,000

Monthly ongoing:

• LLM API (250 queries/day average, as modelled above): ~£160/month
• Pinecone: $70/month (~£55/month)
• Additional compute: £50/month
• Total monthly: ~£265/month

Year 1 total: £14,600-16,600 upfront + £3,180 ongoing = approximately £18,000-20,000

The equivalent built with a local UK senior engineer at £750/day: £37,500 for engineering alone, £42,000-45,000 total year 1.

Before You Start: Questions to Answer

1. What is the success metric for this feature? (Not "it works" - what specific user behaviour changes?)
2. At what usage volume does the API cost become a significant line item?
3. What happens when the AI gives a wrong answer? (Who is responsible, what is the fallback?)
4. How will you evaluate quality over time?
5. Is this a core differentiator or a commodity feature you should buy?

Getting clear on these before starting will save significant rework later.

If you want to discuss the right architecture for your AI feature with engineers who have built production systems, see our LLM Integration service or estimate your project to get a ballpark cost for your specific requirements.