RAG Step 4: Ingest Knowledge into the Vector Database
Now you'll load the Clark College news articles and support documents into pgvector.
Step 1: Understand the Sample Data
The file advanced/rag/sample_data.py contains ~30 documents about Clark College:
- News articles: CTE Showcase, Geomatics competition, Sakura Festival, etc.
- Support services: Tutoring, Financial Aid, Penguin Pantry, Counseling, etc.
Each document looks like:
{
"title": "Tutoring Services at Clark College",
"content": "Clark College offers free, friendly, and supportive tutoring...",
"source": "https://www.clark.edu/campus-life/student-support/",
"category": "support",
}
These are sample documents — you should replace them with your own data.
Step 2: Chunk the Documents
Documents are too long to embed as one piece. We split them into chunks:
Original: "Clark College offers free tutoring in business, English,
math, science, and general education classes. Online
eTutoring is also available..."
Chunk 1: "Clark College offers free tutoring in business, English, math,"
Chunk 2: "math, science, and general education classes. Online eTutoring"
Chunk 3: "Online eTutoring is also available to currently enrolled..."
Each chunk is ~300 words with 50 words of overlap so no context is lost at boundaries.
Step 3: Create the Ingestion Script
Create advanced/rag/knowledge_base.py:
from sentence_transformers import SentenceTransformer
from sample_data import DOCUMENTS
CHUNK_SIZE = 300
CHUNK_OVERLAP = 50
embedder = SentenceTransformer("all-MiniLM-L6-v2")
def chunk_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
words = text.split()
chunks = []
start = 0
while start < len(words):
end = start + chunk_size
chunk = " ".join(words[start:end])
chunks.append(chunk)
start += chunk_size - overlap
return chunks
def get_all_chunks(documents=None):
if documents is None:
documents = DOCUMENTS
all_chunks = []
for doc in documents:
chunks = chunk_text(doc["content"])
for i, chunk in enumerate(chunks):
all_chunks.append({
"title": doc["title"],
"content": chunk,
"source": doc["source"],
"category": doc.get("category", "general"),
"chunk_index": i,
})
return all_chunks
def index_documents(db, DocumentModel, documents=None):
chunks = get_all_chunks(documents)
for chunk in chunks:
vec = embedder.encode(chunk["content"]).tolist()
doc = DocumentModel(
title=chunk["title"],
content=chunk["content"],
source=chunk["source"],
category=chunk["category"],
embedding=vec,
)
db.session.add(doc)
db.session.commit()
return len(chunks)
if __name__ == "__main__":
from app import create_app, db
from app.models import Document
app = create_app()
with app.app_context():
db.create_all()
count = index_documents(db, Document)
print(f"Indexed {count} chunks from {len(DOCUMENTS)} documents")
Step 4: Run the Ingestion
Make sure PostgreSQL + pgvector is running:
cd aiproject/advanced
docker compose -f docker-compose.rag.yml up -d db
Then run the ingestion:
cd aiproject/backend
DATABASE_URL="postgresql://aiproject:aiproject@localhost:5432/aiproject" \
python -m advanced.rag.knowledge_base
Expected output:
Indexed 87 chunks from 30 documents
Step 5: Verify the Data
from app import create_app, db
from app.models import Document
app = create_app()
with app.app_context():
docs = Document.query.limit(3).all()
for d in docs:
print(f"[{d.category}] {d.title}: {d.content[:80]}...")
Expected:
[support] Tutoring Services at Clark College: Clark College offers free tutoring...
[news] CTE Showcase 2026: Clark College welcomed 20 high schools...
[support] Disability Access Center: The Disability Access Center assists...
How to Add Your Own Data
Option 1: Edit sample_data.py
Add new entries to the DOCUMENTS list:
{
"title": "Your Document Title",
"content": "Full text of your document...",
"source": "https://example.com",
"category": "your-category",
}
Then re-run knowledge_base.py to index the new data.
Option 2: Load from a JSON file
import json
with open("my_documents.json") as f:
documents = json.load(f)
The JSON should be a list of objects with title, content, source, and category fields.
Option 3: Web Scraper (Optional)
Use requests + beautifulsoup4 to scrape web pages. See advanced/rag/challenges.md for details.
Write Tests for Embeddings
Test that the embedding model produces the right output:
Create advanced/rag/tests/test_rag_engine.py:
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
from rag_engine import embed_text, build_rag_context
def test_embed_text_returns_384_dimensions():
vec = embed_text("Clark College offers free tutoring")
assert len(vec) == 384
assert all(isinstance(v, float) for v in vec)
def test_embed_text_consistent_output():
vec1 = embed_text("tutoring services")
vec2 = embed_text("tutoring services")
assert vec1 == vec2 # Same input → same vector
def test_build_rag_context_formats_correctly():
results = [{"title": "Test", "content": "Hello world"}]
context = build_rag_context(results)
assert "[Test]" in context
assert "Hello world" in context
Run them:
cd aiproject
python -m pytest advanced/rag/tests/ -v
These tests verify that: - Embeddings are the right size (384 dimensions) - Embeddings are deterministic (same input → same vector) - The context builder formats results correctly
Test Your Work (Manual)
from sentence_transformers import SentenceTransformer
from app.models import Document
from app import create_app, db
model = SentenceTransformer("all-MiniLM-L6-v2")
app = create_app()
with app.app_context():
query = "Where can I get free tutoring?"
query_vec = model.encode(query).tolist()
results = (
db.session.query(Document)
.order_by(Document.embedding.cosine_distance(query_vec))
.limit(3)
.all()
)
for r in results:
sim = 1 - r.embedding.cosine_distance(query_vec)
print(f"[{sim:.2f}] {r.title}")
print(f" {r.content[:100]}...")
Expected: tutoring-related documents should rank highest.