How to Build a Self-Hosted RAG Pipeline Using Milvus + Ollama on Bare Metal

Every added document, support ticket, or internal wiki page sent to a cloud LLM API is another line item on next month's bill — and another copy of your company's data sitting on someone else's infrastructure. For teams working with proprietary documentation, customer records, or regulated data, that trade-off is becoming harder to justify.

A self-hosted RAG pipeline solves both problems at once. Retrieval-Augmented Generation (RAG) is a technique that lets a language model answer questions using your own documents as source material, rather than relying only on what it learned during training — it retrieves relevant chunks of your content first, then generates an answer grounded in that context.

This tutorial walks through building a complete, private RAG stack using Milvus as the vector database and Ollama to run the language model locally, connected with LangChain, all running on a single bare metal server you control. No API keys, no per-token billing, and no documents leaving your hardware.

What You'll Learn

The Architecture of Our Private AI

Before writing any code, it helps to see how the pieces connect. The pipeline has two phases: an indexing phase (done once, or whenever documents change) and a query phase (done every time a user asks a question).

• Indexing: Document → split into chunks → converted to embeddings (Nomic) → stored in Milvus

• Querying: User question → Milvus similarity search retrieves relevant chunks → chunks + question sent to Ollama (Llama 3) → generated answer

Every component in that chain — the embedding model, the vector database, and the LLM — runs as a local process. Nothing in this flow requires an outbound API call.

Prerequisites & Server Requirements

This pipeline is not resource-intensive for small-to-medium document sets, but running two model-serving processes (embeddings and generation) alongside Milvus benefits from headroom. A reasonable baseline:

• CPU: 8 cores or more (a GPU is optional but significantly speeds up token generation for larger models)

• RAM: 32GB or more

• Storage: NVMe SSD, for fast vector index reads and writes

• OS: Ubuntu 22.04 LTS or later

Important: This guide assumes Milvus is already running on your server. If you haven't set up Milvus yet, stop here and follow our guide on How to Host Milvus Vector Database on a Dedicated Server first — this tutorial picks up with Milvus already listening on localhost:19530.

Step 1: Installing Ollama on Your Dedicated Server

Ollama handles model serving for both the LLM and the embedding model, exposing a simple local API on port 11434. Install it with the official script:

curl -fsSL https://ollama.com/install.sh | sh

Verify it's running:

systemctl status ollama

Running Ollama on bare metal rather than a shared VPS matters here for a practical reason: model inference is CPU/GPU and memory-bandwidth intensive. On a shared or resource-limited VPS, "noisy neighbor" workloads can throttle throughput unpredictably. On a dedicated server, every core and every gigabyte of memory bandwidth is available to your model exclusively, which translates directly into faster, more consistent token generation — especially noticeable under concurrent queries.

Step 2: Pulling the LLM and Embedding Models

With Ollama installed, pull the two models this pipeline needs.

The generation model — this produces the final natural-language answer:

ollama pull llama3

(Mistral is a viable alternative: ollama pull mistral.)

The embedding model — this converts text chunks into numerical vectors that Milvus can index and search:

ollama pull nomic-embed-text

nomic-embed-text is purpose-built for embedding tasks and is significantly smaller and faster than using a general-purpose LLM for vectorization, which is why it's kept separate from the generation model.

Step 3: Setting Up the Python Environment

Create a project directory and install the required libraries:

mkdir rag-pipeline && cd rag-pipeline
python3 -m venv venv
source venv/bin/activate
pip install pymilvus langchain langchain-community langchain-milvus bs4

LangChain acts as the connective layer here — it provides consistent interfaces for loading documents, calling Ollama for embeddings and generation, and querying Milvus, so we don't have to hand-write the glue code between each service.

Step 4: Writing the RAG Pipeline Script

Note: Create a single file named rag_app.py and paste all three of the following Python blocks into it sequentially. The script breaks down into three logical stages: loading/chunking data, embedding and storing it, then retrieving and generating answers.

1. Loading and Chunking the Data

Large documents need to be split into smaller chunks before embedding — this keeps each vector focused on a specific, retrievable idea rather than diluting it across an entire document.

from langchain_community.document_loaders import WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load a source document (swap for TextLoader, PyPDFLoader, etc. as needed)
loader = WebBaseLoader("https://example.com/your-internal-doc")
documents = loader.load()

# Split into overlapping chunks so context isn't lost at chunk boundaries
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150
)
chunks = text_splitter.split_documents(documents)

print(f"Document split into {len(chunks)} chunks")

2. Generating Embeddings and Storing in Milvus

Each chunk is converted into a vector using the local nomic-embed-text model, then written into the Milvus collection running on the same server.

from langchain_community.embeddings import OllamaEmbeddings
from langchain_milvus import Milvus

# Local embedding model — no external API call
embeddings = OllamaEmbeddings(model="nomic-embed-text")

vector_store = Milvus.from_documents(
    documents=chunks,
    embedding=embeddings,
    connection_args={"host": "localhost", "port": "19530"},
    collection_name="rag_documents"
)

print("Chunks embedded and stored in Milvus")

3. Querying and Generating the Answer

With chunks indexed, set up a retriever backed by Milvus and a generation model backed by Ollama, then chain them together so a question triggers a retrieval step followed by a generation step.

Note: older RAG tutorials often use RetrievalQA.from_chain_type() for this step. That class has been deprecated by LangChain (since v0.1.17) in favor of the create_retrieval_chain + create_stuff_documents_chain pattern below, which is the current recommended approach and won't throw deprecation warnings.

from langchain_community.llms import Ollama
from langchain_core.prompts import ChatPromptTemplate
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains import create_retrieval_chain

# Local generation model
llm = Ollama(model="llama3")

# Retriever pulls the most relevant chunks for a given question
retriever = vector_store.as_retriever(search_kwargs={"k": 4})

# Prompt template that tells the model to answer strictly from retrieved context
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer the question using only the following context. "
                "If the answer isn't in the context, say you don't know.\n\n"
                "Context: {context}"),
    ("human", "{input}")
])

# Combines retrieved chunks into the prompt, then generates an answer
combine_docs_chain = create_stuff_documents_chain(llm, prompt)

# Full RAG chain: retrieve -> stuff context into prompt -> generate
rag_chain = create_retrieval_chain(retriever, combine_docs_chain)

question = "What does this document say about deployment requirements?"
result = rag_chain.invoke({"input": question})

print("\nAnswer:", result["answer"])

Step 5: Testing Your Self-Hosted RAG System

Run the script from your terminal:

python3 rag_app.py

If everything is wired correctly, you'll see console output confirming the chunk count and embedding step, followed by a generated answer grounded strictly in the source document — for example:

Document split into 14 chunks
Chunks embedded and stored in Milvus

Answer: According to the document, deployment requires a minimum
of 8 CPU cores, 32GB RAM, and NVMe storage for the vector index.

Because the model is answering from retrieved context rather than general training knowledge, you can validate correctness directly against the source document — a useful check for catching retrieval issues (wrong chunks returned) versus generation issues (model ignoring context).

Why Run RAG on Bare Metal Instead of the Cloud?

Once the pipeline is working, the case for bare metal over managed cloud AI services comes down to three practical factors:

• Zero API costs. There's no per-1K-token billing for embeddings or generation. Query volume can scale without a corresponding line item on a usage invoice.

• Absolute data privacy. Documents, embeddings, and generated answers never leave the server. For legal, healthcare, financial, or internal engineering documentation, this removes an entire category of data-handling risk.

• Zero network latency between components. Milvus and Ollama communicate over localhost rather than across the internet, eliminating the round-trip latency inherent to calling external vector database or LLM APIs.

Conclusion

You've now built a complete, self-hosted RAG pipeline: documents are chunked, embedded locally with nomic-embed-text, indexed in Milvus, and queried through Llama 3 — all without a single external API call. It's a scalable foundation you can extend with more document sources, larger models, or a web front end, and it stays entirely under your control as it grows.

Running local embedding and generation models well — especially as document volume or concurrent users increase — takes real, dedicated compute. Deploy your private AI on high-performance bare metal servers today with BytesRack.