What is RAG

The Problem RAG Solves

Large Language Models (LLM) learn from a massive but frozen snapshot of the world. Once training ends, the model’s knowledge is sealed. It cannot read your internal documentation, does not know what changed last quarter, and has never seen your proprietary data.

The result: when you ask an LLM about anything outside its training data, it fabricates a plausible-sounding answer. This is called hallucination — and it is not a bug that will be fixed. It is a fundamental property of how language models work.

Three strategies exist to close this gap:

RAG was introduced by Meta AI researchers in 2020 (Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”).

How RAG Works

RAG combines two systems: a retriever that finds relevant information, and a generator (the LLM) that formulates an answer using that information. Neither system works well alone — the retriever cannot answer questions, and the LLM without retrieval will hallucinate.

There are exactly two phases:

• Ingestion (offline): Documents are split into chunks, each chunk is converted into a vector (a list of numbers that captures its meaning), and stored in a vector database. This runs once — or on a schedule when documents change.
• Retrieval (online): When a user asks a question, the question is also converted into a vector. The vector database finds the chunks whose vectors are most similar — these are the chunks most likely to contain the answer. They are injected into the LLM prompt alongside the original question.

The Three Core Components

Embedding model — Transforms text into a high-dimensional vector where semantically similar texts end up geometrically close. “car” and “automobile” will be near each other. “car” and “quarterly earnings” will be far apart. This is how the retriever finds relevant chunks without keyword matching.

Vector store — A specialized database optimized for similarity search. Unlike a SQL database that matches exact values, a vector store finds the k vectors most similar to a query vector using Approximate Nearest Neighbor (ANN) algorithms. It also stores the original text so retrieved chunks can be read by the LLM.

LLM (generator) — Takes the user’s question plus the retrieved chunks and produces an answer. The critical difference from a standard LLM call: the model is explicitly instructed to answer only from the provided context. If the answer is not in the chunks, it should say so — not invent one.

What RAG Is Not

The #1 failure mode in production RAG is not the LLM — it’s the retriever silently returning irrelevant chunks. The LLM then confidently answers from wrong context. Always instrument retrieval separately from generation so you can tell them apart when quality degrades.

Where Can RAG Run?

RAG is not tied to any specific platform. The three components — embedding model, vector store, LLM — can run anywhere that can execute Python and make HTTP calls. The platform choice is an infrastructure decision, not an ML decision.

Deployment Options Compared

Platform Decision Tree

Questions to Ask Before Building Anything

Before writing a single line of code, run a discovery session with stakeholders. These questions determine whether you need RAG at all, what stack fits, and where the hard constraints live. Skipping this step is the most common reason RAG projects are rebuilt from scratch after three months.

Use Case & Users

Goal: understand what the system must do and who will use it.

Question 4 is the most important. If a wrong answer has legal, financial, or safety consequences, you need a human-in-the-loop review step and a confidence threshold — not just better retrieval.

Data & Knowledge Base

Goal: understand the corpus — its size, format, freshness, and quality.

Ask to see 10–20 sample documents before the discovery session ends. Written answers about “well-structured PDFs” often mean scanned images with inconsistent formatting. Eyes on the data beats any description.

Security & Compliance

Goal: identify hard constraints that eliminate options before any design work starts.

Question 18 is a binary gate. If the answer is “no, data cannot leave the VNet”, Azure OpenAI with Managed Private Endpoints is the minimum — and fully self-hosted on AKS may be required.

Question 20 is frequently forgotten. A user asking “what is the salary band for a senior engineer?” should not receive chunks from an HR document they have no permission to view — even if those chunks are the most relevant. Document-level access control in the vector store is a hard requirement for multi-tenant or role-separated knowledge bases.

Infrastructure & Operations

Goal: understand the existing environment and the team’s capacity to operate new components.

Question 28 is the one that changes minds. Teams that choose self-hosted for cost reasons often switch to managed after the first 2am Qdrant OOM incident.

Cost & Budget

Goal: establish financial guardrails before stack selection.

Model cost at three scenarios: current volume, 10× growth, and 50× growth. The stack choice that is cheapest today may not be cheapest at scale.

Quality & Evaluation

Goal: define what “good” looks like before building, so you have a way to know when you are done.

If the answer to question 35 is “no, we don’t have any example Q&A pairs”, stop the discovery session and make building that dataset the first deliverable. Without a golden set, you cannot measure whether the RAG system is better than what the team already has.

Discovery Output — Requirements Card

At the end of the session, fill out this card. It summarizes the decisions that flow from the answers above.

┌─────────────────────────────────────────────────────────────┐
│  RAG Requirements Card                                      │
├──────────────────────────┬──────────────────────────────────┤
│ Corpus size              │                                  │
│ Update frequency         │                                  │
│ Peak queries / day       │                                  │
│ Acceptable latency       │                                  │
│ Data sovereignty         │ VNet-only / Regional / Any       │
│ Compliance               │ HIPAA / PCI / GDPR / None        │
│ PII in corpus            │ Yes / No / Unknown               │
│ Document-level ACL       │ Required / Not required          │
│ MLOps capacity           │ High / Medium / None             │
│ Existing AKS + GPU       │ Yes (Lab 1) / No                 │
│ Monthly budget           │ $                                │
│ Golden eval dataset      │ Yes / No → build first           │
│ Recommended stack        │ Self-Hosted / Azure Managed      │
├──────────────────────────┴──────────────────────────────────┤
│ Hard blockers (must resolve before design):                 │
│                                                             │
│ Open questions:                                             │
│                                                             │
└─────────────────────────────────────────────────────────────┘

When RAG Makes Sense — The Formula

Before deciding on a solution, quantify the problem. These formulas help you choose the right approach using numbers from your discovery session, not intuition.

Context Window Break-Even

The first gate: can you skip RAG entirely with context stuffing?

corpus_tokens = total_documents × avg_tokens_per_document
if corpus_tokens < context_window_limit:
    → use context stuffing
else:
    → RAG is justified on size alone

Example:

500 documents × 800 tokens/doc = 400,000 tokens
GPT-4o context window = 128,000 tokens
400,000 > 128,000  →  context stuffing won't fit  →  RAG justified

Estimate token count with: characters / 4 ≈ tokens. A 10-page Word doc is ~5,000 words ~= 6,500 tokens. A 100-doc corpus is usually under 1M tokens — always measure before assuming you need RAG.

Cost Break-Even — Self-Hosted vs Azure Managed vs Context Stuffing

Three competing costs at different query volumes. Find where they cross.

Context Stuffing cost per query:

cost_stuffing = (corpus_tokens / 1000) × price_per_1K_input_tokens
Example (GPT-4o at $0.005/1K input tokens, 400K token corpus):
cost_stuffing = (400,000 / 1000) × $0.005 = $2.00 per query

Azure Managed RAG cost per query:

cost_azure = cost_embed_query + cost_vector_search + cost_generation
cost_embed_query    = (query_tokens / 1000) × embed_price
                    = (50 / 1000) × $0.00002 = $0.000001
cost_vector_search  = ai_search_monthly / monthly_queries
                    = $75 / 300,000 = $0.00025
cost_generation     = ((top_k × chunk_tokens + query_tokens) / 1000) × input_price
                    + (output_tokens / 1000) × output_price
                    = ((5 × 512 + 50) / 1000) × $0.00015
                    + (300 / 1000) × $0.0006
                    = $0.000413 + $0.00018 = $0.000593
cost_azure ≈ $0.000001 + $0.00025 + $0.000593 ≈ $0.00085 / query

Self-Hosted RAG cost per query:

cost_selfhosted = (infra_monthly + vector_store_monthly) / monthly_queries
infra_monthly       = gpu_cost_per_hr × active_hours_per_day × 30
                    + cpu_cost_per_hr × 24 × 30
                    = $0.53 × 4 × 30 + $1.20 × 24 × 30
                    = $63.60 + $864 = ~$164/mo  
vector_store_monthly = $20   ← Premium SSD P10
cost_selfhosted = ($164 + $20) / monthly_queries

Break-even between Azure Managed and Self-Hosted:

selfhosted_monthly  = gpu_monthly + cpu_monthly + storage_monthly
                    = $63.60 + $100 + $20 = $183.60 ≈ $184  (fixed, regardless of volume)
azure_monthly       = azure_fixed + azure_variable × monthly_queries
                    = $105 + ($0.00085 × q_per_day × 30)
# Set equal and solve:
$184 = $105 + ($0.00085 × q_per_day × 30)
$79  = $0.0255 × q_per_day
q_per_day = 79 / 0.0255 ≈ 3,098 queries/day

⚠️ A common mistake: dividing self-hosted cost by the per-query Azure price. That ignores Azure’s fixed costs ($105/mo for AI Search + AKS), which shifts the break-even significantly. Always subtract fixed costs from both sides before solving.

Monthly cost at key volumes:

Line = Self-Hosted (flat $184/mo) · Bars = Azure Managed (grows with volume) · They cross at ~3,100 queries/day

Latency Budget Formula

RAG adds latency at every step. Validate the total fits within the user’s SLA before committing to the architecture.

latency_total = t_embed_query + t_vector_search + t_prompt_build + t_llm_ttft + t_llm_generation
Self-Hosted (in-cluster):
  t_embed_query    ≈  10ms   (bge-base, CPU)
  t_vector_search  ≈  15ms   (Qdrant, 100K vectors)
  t_prompt_build   ≈   2ms
  t_llm_ttft       ≈ 400ms   (Phi-4 Mini, vLLM, warm GPU)
  t_llm_generation ≈ 800ms   (300 output tokens @ ~375 tok/s)
  ─────────────────────────
  total            ≈ 1,227ms ✅ under 2s
Azure Managed:
  t_embed_query    ≈  35ms   (AOAI text-embedding-3-small)
  t_vector_search  ≈  25ms   (AI Search Basic)
  t_prompt_build   ≈   2ms
  t_llm_ttft       ≈ 450ms   (GPT-4o-mini)
  t_llm_generation ≈ 600ms   (300 output tokens)
  ─────────────────────────
  total            ≈ 1,112ms ✅ under 2s

TTFT (Time To First Token) and generation time are the dominant terms — embedding and vector search are cheap. If you need to cut latency, reduce top_k (fewer chunks = shorter prompt = faster generation) or switch to a smaller/faster model. Do not optimize embedding latency first.

Add a streaming response if generation time exceeds 1s. Users perceive streaming as fast even when total latency is 3–4s. Both vLLM and Azure OpenAI support Server-Sent Events (SSE) streaming — LangChain handles it with stream=True.

Quality Threshold Formula

Use this to decide when RAG quality is good enough to ship.

RAGAS scores (0.0 – 1.0, higher is better):
faithfulness        = answers grounded in retrieved context (target: > 0.85)
answer_relevancy    = answer addresses the question         (target: > 0.80)
context_recall      = correct chunks retrieved              (target: > 0.75)
context_precision   = retrieved chunks are relevant         (target: > 0.70)
composite_score = mean(faithfulness, answer_relevancy, context_recall, context_precision)
Ship when: composite_score > 0.78 AND faithfulness > 0.85
           (faithfulness is non-negotiable — low faithfulness = hallucination)

Run RAGAS against your golden dataset (discovery question 35) before and after any chunking or model change. A change that improves answer_relevancy but drops faithfulness below 0.85 is a regression — do not ship it.

When NOT to Use RAG

With the formulas above answered, check whether a simpler solution already solves the problem. RAG adds infrastructure (vector store + embedding model), operational overhead, and latency. It is only the right choice when no simpler alternative fits.

The Alternatives in Detail

Context Stuffing

If your entire knowledge base fits in the model’s context window, skip RAG entirely. Just load the documents directly into the prompt.

# No vector DB. No embeddings. Just a prompt.
with open("knowledge_base.md") as f:
    context = f.read()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": f"Answer using this knowledge:\n\n{context}"},
        {"role": "user",   "content": question},
    ]
)

Modern models handle 128K–1M token windows. A 300-page technical manual is ~150K tokens — it fits in a single GPT-4o or Claude prompt. Measure first before building a retrieval pipeline.

Context stuffing has a cost: every query pays for the full knowledge base in tokens. At 100K tokens × $0.005/1K = $0.50/query. Fine for low-volume internal tools. Expensive at scale. That’s the break-even point where RAG starts saving money.

NL-to-SQL

If your data is already structured (tables, schemas), teach the LLM to write SQL instead of searching documents.

NL-to-SQL breaks when schemas are complex or ambiguous. Provide the LLM with schema descriptions and a few example queries (few-shot). LangChain’s SQLDatabaseChain and Azure AI Studio’s Text-to-SQL feature handle this pattern.

Always run LLM-generated SQL against a read-only connection. Never give the LLM a connection string with write permissions.

Function Calling / Tool Use

If the data is live (stock prices, IoT sensor readings, incident tickets), a static index will always be stale. Give the LLM tools to query the source directly at runtime.

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_incident_status",
            "description": "Returns the current status of an incident by ID",
            "parameters": {
                "type": "object",
                "properties": {"incident_id": {"type": "string"}},
                "required": ["incident_id"],
            },
        },
    }
]
# LLM decides when to call the tool and with what arguments
response = client.chat.completions.create(model="gpt-4o", messages=messages, tools=tools)

Tool use and RAG are not mutually exclusive. A common production pattern is RAG + tools: RAG handles static documentation, tools handle live data. The LLM decides which to use based on the question.

Each tool call is a network hop. Add timeout handling and fallback responses. Log every tool invocation — tool calls are the hardest part of LLM pipelines to debug in production.

Fine-Tuning

Fine-tuning bakes knowledge into model weights. It is the right choice for:

• Enforcing a specific output format (JSON schema, structured reports)
• Adopting domain-specific terminology or writing style
• Reducing prompt length by removing repeated instructions

It is the wrong choice for:

• Keeping facts up to date — weights are frozen after training
• Citing sources — fine-tuned models can still hallucinate
• One-off Q&A over a document corpus — RAG is cheaper and updatable

A common mistake: fine-tuning a model to “know” a company’s internal docs. When docs change, you retrain. Use RAG for facts, fine-tuning for behavior.

Fine-tuning on Azure OpenAI costs ~$0.008/1K training tokens + the fine-tuned model hosting fee (~$1.70/hr for GPT-3.5-turbo). Only justified if it eliminates significant prompt engineering overhead or dramatically reduces inference token count.

Decision Summary

Start with the simplest option and add complexity only when you hit a concrete limit. Most internal knowledge-base chatbots can be built with context stuffing for the first 6 months. Add RAG when the corpus outgrows the context window or per-query token costs become a concern.

Why This Lab Uses AKS

This document uses AKS for the self-hosted path for one specific reason: the AKS cluster from Lab 1 (LLM Inference on AKS) already exists. Adding RAG components to an existing cluster has near-zero additional infrastructure cost — the GPU nodes, NAP, KEDA, APIM, and Workload Identity are already in place.

If you are starting from scratch without an existing AKS cluster, weigh the options above first. AKS has the highest operational ceiling but also the highest setup cost. For a team building their first RAG pipeline, Azure Container Apps or Azure AI Foundry will reach production faster.

Data Preparation for RAG — ETL for AI

Before a single embedding is computed, the source data must be extracted, cleaned, and structured. This phase is called the RAG data pipeline or, more formally, the Knowledge Ingestion Pipeline. It is the most underestimated part of a RAG project — and the most common source of poor retrieval quality.

Retrieval quality is bounded by data quality. A perfect embedding model on dirty data will always underperform a simple model on clean, well-structured data.

The ingestion pipeline is a data engineering problem, not an ML problem. Treat it like any ETL pipeline: idempotent runs, schema validation, lineage tracking, and alerting on failures.

The Full Data Pipeline

Parsing by File Format

Not all files are equal. Each format requires a different extraction strategy.

Chunking Strategies

Chunking is not just splitting text — it directly controls what the LLM sees as context. The wrong strategy is the #1 cause of poor answers.

Always include chunk overlap. A key sentence that straddles two chunk boundaries will be lost in both — overlap ensures it appears in at least one chunk. 64 tokens is a good starting point; increase to 128 for dense technical content.

Metadata Enrichment

Every chunk must carry metadata. Metadata enables filtered retrieval (only search within a date range, a specific document, a topic) and source citation in the final answer.

# Minimum viable metadata schema
chunk.metadata = {
    "doc_id":    "sha256-of-original-document",  # dedup key
    "source":    "blob://corpus/aks-lab.md",
    "title":     "Running LLM Inference on AKS",
    "section":   "Cost Break-Even Analysis",
    "author":    "ricardo.trentin",
    "date":      "2025-03-15",
    "language":  "en",
    "chunk_id":  42,
    "chunk_total": 87,
}

Rich metadata unlocks hybrid retrieval: filter by date > 2025-01-01 then run vector search on the filtered subset. This is far more precise than pure ANN search on the full index.

Store the doc_id (a hash of the source document) in every chunk. This is your key for incremental updates — when a document changes, delete all chunks with that doc_id and re-ingest. Without it, you’ll have stale and fresh versions of the same document coexisting in your index.

Incremental Ingestion

A one-time bulk load is not enough. Documents change. New ones arrive. The index must stay current.

Trigger incremental ingestion via Azure Event Grid on BlobCreated / BlobModified events. This keeps ingestion latency under a few minutes for document updates. For Azure AI Search, use the built-in indexer + change detection policy — it handles this natively without custom code.

RAG Poisoning

RAG poisoning (also called indirect prompt injection or knowledge base poisoning) is when an attacker embeds malicious instructions inside a document that gets indexed. When a user asks a question, the poisoned chunk is retrieved and injected into the LLM prompt — causing the model to follow the attacker’s instructions instead of answering correctly.

Defense layers:

Injection scanning is a heuristic — sophisticated attacks use Unicode lookalikes, base64 encoding, or fragmented instructions across multiple chunks. Treat scanning as one layer, not the only layer. The most robust defense is restricting who can write to the document source (Blob Storage RBAC).

Use Azure AI Content Safety for production-grade content scanning. It detects prompt injection, jailbreak attempts, and harmful content via a managed API — no pattern maintenance required. Pair it with Defender for Storage which scans uploaded blobs for malware before they ever reach the ingestion pipeline.

Log every quarantined document to a dedicated Log Analytics table. Alert when quarantine rate exceeds a baseline (e.g., > 1% of daily ingestions) — this is an early signal of an active attack or a misconfigured source system.

The RAG Pipeline

There are two phases: ingestion (offline, run once or on schedule) and retrieval (online, runs per query).

Ingestion Phase

Chunk size is the most impactful tuning knob. Too small → chunks lack context. Too large → irrelevant content dilutes the signal. 512 tokens with 64 token overlap is a good starting point for technical docs. Use semantic chunking (split on paragraphs/sections) when document structure allows it.

Ingestion is a batch job. Run it as a Kubernetes Job — CPU only, spins down after completion. No GPU billing during ingestion unless you’re using a large embedding model. Schedule it off-peak with a CronJob to avoid contention with live inference workloads.

Retrieval Phase (Query Time)

The embedding model used at query time must be the same as the one used during ingestion. The vector space is model-specific — mixing models breaks retrieval completely.

top-k controls how many chunks are injected into the prompt. Higher k = more context = higher token cost per request. Watch this in your APIM cost tracking dashboard. For Azure OpenAI, each extra chunk at 512 tokens = ~$0.000005 added to every query — small per call, large at scale.

Architecture Comparison

Self-Hosted Stack (on your AKS cluster)

Stack summary:

Qdrant runs as a StatefulSet with a PersistentVolumeClaim. Use Azure Premium SSD (P10, 128 GiB) — ANN search is disk-I/O sensitive, ZRS disks add cross-zone redundancy. Backup the PVC to Blob Storage nightly via a CronJob. Define a recovery time objective (RTO): if the Qdrant pod crashes and PVC is lost, how long to re-ingest? That number drives whether you need a Qdrant cluster or a single replica with fast restore.

Qdrant has no built-in auth in OSS mode. Use Kubernetes NetworkPolicy to restrict access to the Qdrant pod to only the RAG app’s service account. Never expose Qdrant on a LoadBalancer service.

bge-base-en-v1.5 (768-dim, 110M params) is a strong open-source embedding model. It fits on CPU for low-throughput workloads. Move it to GPU if ingestion time becomes a bottleneck or if you’re serving embeddings at query time with latency SLAs.

Azure Managed Stack

Stack summary:

Azure AI Search (Standard tier and above) offers 99.9% SLA for read operations and 99.9% SLA for indexing. It handles replication and failover transparently. No StatefulSets, no PVCs, no Qdrant upgrade runbooks. For a team without MLOps capacity, this is the reliability-correct default.

Use Managed Private Endpoint to keep all traffic inside the VNet — Search ↔ AOAI ↔ AKS never traverses the public internet. Authenticate with Workload Identity (OIDC federation) rather than API keys stored in Kubernetes Secrets. Rotate keys with Azure Key Vault if you must use keys.

text-embedding-3-small (1536-dim) consistently outperforms ada-002 on MTEB benchmarks. Use text-embedding-3-large if retrieval quality is critical — at 3× the cost. Azure AI Search also supports hybrid search (keyword + vector) out of the box, which improves recall on technical queries with specific terminology.

Same Code, Two Configs

The key insight is that both stacks run the same LangChain pipeline — only the config changes. This makes A/B testing retrieval quality between stacks straightforward.

Shared RAG Pipeline

# rag_pipeline.py
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
PROMPT_TEMPLATE = """You are a helpful assistant. Use only the context below to answer.
If the answer is not in the context, say "I don't know."
Context:
{context}
Question: {question}
Answer:"""
def build_rag_chain(llm, retriever):
    prompt = PromptTemplate(
        template=PROMPT_TEMPLATE,
        input_variables=["context", "question"]
    )
    return RetrievalQA.from_chain_type(
        llm=llm,
        retriever=retriever,
        chain_type_kwargs={"prompt": prompt},
        return_source_documents=True
    )

Self-Hosted Config

# config_selfhosted.py
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_qdrant import QdrantVectorStore
from qdrant_client import QdrantClient
# Reuse your vLLM endpoint from Lab 1
LLM_BASE_URL    = "http://vllm-service.inference.svc.cluster.local:8000/v1"
EMBED_BASE_URL  = "http://embed-service.inference.svc.cluster.local:8001/v1"
QDRANT_URL      = "http://qdrant.vectorstore.svc.cluster.local:6333"
def get_llm():
    return ChatOpenAI(
        base_url=LLM_BASE_URL,
        api_key="placeholder",          # vLLM doesn't enforce auth inside cluster
        model="phi-4-mini",
        temperature=0,
    )
def get_embeddings():
    return OpenAIEmbeddings(
        base_url=EMBED_BASE_URL,
        api_key="placeholder",
        model="bge-base-en-v1.5",
    )
def get_retriever():
    client = QdrantClient(url=QDRANT_URL)
    store  = QdrantVectorStore(client=client, collection_name="docs", embedding=get_embeddings())
    return store.as_retriever(search_kwargs={"k": 5})

Azure Managed Config

# config_azure.py
from langchain_openai import AzureOpenAIEmbeddings, AzureChatOpenAI
from langchain_community.vectorstores.azuresearch import AzureSearch
import os
def get_llm():
    return AzureChatOpenAI(
        azure_deployment="gpt-4o-mini",
        azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
        api_key=os.environ["AZURE_OPENAI_API_KEY"],
        api_version="2024-08-01-preview",
        temperature=0,
    )
def get_embeddings():
    return AzureOpenAIEmbeddings(
        azure_deployment="text-embedding-3-small",
        azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
        api_key=os.environ["AZURE_OPENAI_API_KEY"],
    )
def get_retriever():
    store = AzureSearch(
        azure_search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],
        azure_search_key=os.environ["AZURE_SEARCH_KEY"],
        index_name="rag-docs",
        embedding_function=get_embeddings().embed_query,
    )
    return store.as_retriever(search_type="hybrid", search_kwargs={"k": 5})

Entry Point

# query.py
import sys
from rag_pipeline import build_rag_chain
STACK = sys.argv[1] if len(sys.argv) > 1 else "selfhosted"
if STACK == "azure":
    from config_azure import get_llm, get_retriever
else:
    from config_selfhosted import get_llm, get_retriever
chain = build_rag_chain(get_llm(), get_retriever())
while True:
    question = input("\nQuestion: ").strip()
    if not question:
        break
    result = chain.invoke({"query": question})
    print(f"\nAnswer: {result['result']}")
    print(f"\nSources:")
    for doc in result["source_documents"]:
        print(f"  - {doc.metadata.get('source', 'unknown')} (chunk {doc.metadata.get('chunk_id', '?')})")

Ingestion Script

# ingest.py
import sys
from pathlib import Path
from langchain_community.document_loaders import DirectoryLoader, UnstructuredMarkdownLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
STACK = sys.argv[1] if len(sys.argv) > 1 else "selfhosted"
if STACK == "azure":
    from config_azure import get_embeddings, get_retriever
    from langchain_community.vectorstores.azuresearch import AzureSearch
    import os
else:
    from config_selfhosted import get_embeddings
    from langchain_qdrant import QdrantVectorStore
    from qdrant_client import QdrantClient
# [DS] Tune chunk_size and chunk_overlap for your document type.
# Technical docs with dense information → smaller chunks (256-512)
# Narrative content → larger chunks (512-1024)
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ".", " "],
)
loader = DirectoryLoader("./docs", glob="**/*.md", loader_cls=UnstructuredMarkdownLoader)
docs   = loader.load()
chunks = splitter.split_documents(docs)
# Tag each chunk with source metadata
for i, chunk in enumerate(chunks):
    chunk.metadata["chunk_id"] = i
print(f"Loaded {len(docs)} documents → {len(chunks)} chunks")
embeddings = get_embeddings()
if STACK == "azure":
    import os
    store = AzureSearch(
        azure_search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],
        azure_search_key=os.environ["AZURE_SEARCH_KEY"],
        index_name="rag-docs",
        embedding_function=embeddings.embed_query,
    )
    store.add_documents(chunks)
else:
    from qdrant_client import QdrantClient
    from qdrant_client.models import Distance, VectorParams
    client = QdrantClient(url="http://localhost:6333")
    client.recreate_collection(
        collection_name="docs",
        vectors_config=VectorParams(size=768, distance=Distance.COSINE),
    )
    QdrantVectorStore.from_documents(chunks, embeddings, client=client, collection_name="docs")
print("Ingestion complete.")

Stack Comparison

For retrieval quality benchmarking, use RAGAS (Retrieval-Augmented Generation Assessment). Measure faithfulness, answer relevancy, and context recall on a small eval set before committing to either stack.

Azure AI Search’s hybrid search (BM25 + vector) consistently beats pure vector search on technical queries. If you go self-hosted, consider adding a BM25 layer (Qdrant’s sparse vector support) to match it. For latency-sensitive workloads, pin Azure AI Search to the same region as your AKS cluster to minimize cross-region RTT.

Enable diagnostic logs on Azure AI Search and stream them to a Log Analytics workspace. Alert on SearchLatencyMs > 500 and ThrottledSearchQueriesPercentage > 5 — these are your early warning signals before users notice degradation.

Reliability

Define your RTO and RPO before choosing a stack. If you can tolerate 2 hours to re-ingest after a Qdrant failure, self-hosted is fine. If you need near-zero RTO, either invest in a Qdrant cluster or use Azure AI Search.

Security

Neither stack is secure by default. The self-hosted stack requires you to write NetworkPolicy manifests and configure Workload Identity correctly. The managed stack requires you to set up Private Endpoints — without them, AOAI and Search are reachable from the internet. Security is an active choice in both cases.

Cost Optimization

Apply Cost Management budgets and alerts on the resource group. Tag all RAG resources with workload=rag and stack=selfhosted|managed for showback. Use Azure Reservations on the CPU node pool (always-on) — 1-year reserved saves ~40% vs pay-as-you-go.

Performance Efficiency

Cold start on GPU node provisioning (~2 min via NAP) is acceptable for internal tools but not for customer-facing products. Mitigate with KEDA minReplicaCount: 1 during business hours and scale to zero overnight — keeps one warm GPU pod available while cutting ~75% of off-peak GPU cost.

Operational Excellence

Instrument your RAG app with OpenTelemetry regardless of stack. Trace the full request: embed → search → prompt build → LLM. The most common production issue is silent retrieval degradation — the LLM returns an answer, but from wrong chunks. Only distributed tracing catches this. Use langchain-opentelemetry or add spans manually around each step.

References

Academic Papers

Azure Documentation

Open Source Libraries & Tools

Models Referenced