Skip to main content
AI / Machine Learning

RAG System Implementation

10 Modules
Chapter 9: Learning Objectives -- RAG System Implementation90%

Learning Objectives -- RAG System Implementation

Workshop Overview

Build a complete Retrieval-Augmented Generation system that answers questions from your own documents, with source citations and quality evaluation. By the end of this 3-hour afternoon session, you will have a working knowledge base you can continue to develop.

Primary Learning Outcomes

By the end of this workshop, you will be able to:

1. RAG Architecture and Concepts

You will be able to:

  • Explain the RAG pipeline: ingestion, embedding, retrieval, generation
  • Identify when RAG is the right approach (vs fine-tuning, prompt engineering, or web search)
  • Choose appropriate vector databases for different use cases
  • Describe how embeddings represent semantic meaning as vectors
  • Understand the trade-offs between local and cloud embedding models

Success Criteria:

  • Can draw and explain the RAG pipeline from memory
  • Can justify technology choices for a given scenario
  • Can explain cosine similarity and distance scores

2. Document Processing and Chunking

You will be able to:

  • Load documents from multiple formats (markdown, text, PDF)
  • Apply recursive character splitting with configurable chunk size and overlap
  • Choose chunk parameters appropriate to your document type
  • Preserve source metadata through the chunking pipeline

Success Criteria:

  • Successfully loaded and chunked at least 3 documents
  • Can explain why chunk size and overlap matter
  • Can describe when to use small vs large chunks

3. Vector Store Setup and Management

You will be able to:

  • Create and configure a ChromaDB persistent collection
  • Add documents with metadata to a vector store
  • Run semantic similarity searches with distance scores
  • Incrementally add new documents without rebuilding the entire index

Success Criteria:

  • ChromaDB collection created and queryable
  • Search returns relevant results for test queries
  • New documents added successfully to existing collection

4. Embedding Model Selection and Usage

You will be able to:

  • Use local embedding models (Sentence Transformers, Ollama)
  • Optionally configure cloud embedding models (OpenAI)
  • Compare embedding model quality, speed, and privacy characteristics
  • Choose the right embedding model for your deployment context

Success Criteria:

  • At least one embedding model producing vectors
  • Can articulate the privacy implications of cloud vs local embeddings
  • Understand dimension count and its relationship to quality

5. Query Pipeline Construction

You will be able to:

  • Build a complete retrieve-then-generate pipeline
  • Construct effective prompts that constrain the LLM to retrieved context
  • Implement citation tracking with source attribution
  • Create an interactive question-answering interface

Success Criteria:

  • Pipeline produces grounded answers from retrieved context
  • Answers include [Reference N] style citations
  • Interactive chat mode works for ad-hoc queries

6. Advanced Retrieval Techniques

You will be able to:

  • Apply metadata filtering to scope searches
  • Implement hybrid search (vector + keyword)
  • Use query expansion to improve recall
  • Apply cross-encoder re-ranking for improved precision

Success Criteria:

  • Metadata filtering narrows results correctly
  • Hybrid search handles exact-term queries better than vector-only
  • Re-ranking demonstrably improves result ordering

7. Quality Evaluation

You will be able to:

  • Write test cases with expected facts and sources
  • Verify that retrieved chunks come from the correct documents
  • Check that generated answers are grounded in context (not hallucinated)
  • Identify when chunking or embedding choices need adjustment

Success Criteria:

  • Evaluation script runs and produces pass/fail results
  • At least 80% of test cases pass
  • Can diagnose and explain test failures

Skill Levels

Beginner Level (First 45 Minutes)

Knowledge:

  • Understand what RAG is and why it exists
  • Know the difference between vector search and keyword search
  • Recognise the main components: embeddings, vector store, LLM

Skills:

  • Set up a Python environment with RAG libraries
  • Load and chunk documents
  • Create a ChromaDB collection
  • Run basic similarity searches

Intermediate Level (45-120 Minutes)

Knowledge:

  • Understand chunking trade-offs (size, overlap, strategy)
  • Know how to evaluate retrieval quality
  • Comprehend the role of prompting in RAG quality

Skills:

  • Build a complete RAG pipeline end to end
  • Add citation tracking to generated answers
  • Apply metadata filtering
  • Implement hybrid search

Advanced Level (120-180 Minutes)

Knowledge:

  • Understand re-ranking and when it helps
  • Know production deployment considerations
  • Recognise limitations and failure modes

Skills:

  • Apply cross-encoder re-ranking
  • Implement query expansion
  • Build a domain-specific knowledge base
  • Evaluate and iterate on system quality

Profession-Specific Outcomes

For Researchers and Academics

  • Build a literature search assistant over your paper collection
  • Cross-reference findings across multiple studies
  • Generate cited summaries of research topics

For Business Professionals

  • Create a policy and compliance advisor
  • Build a project knowledge hub across meeting notes and reports
  • Enable quick retrieval of decisions and action items

For Technical Teams

  • Index internal documentation for instant search
  • Build a troubleshooting assistant from support tickets
  • Create an onboarding knowledge base for new team members

For Creative and Content Professionals

  • Search and retrieve from content archives
  • Find and cite previous work for consistency
  • Build a brand voice reference system

Assessment Criteria

Knowledge Assessment (10 points)

  • RAG fundamentals and architecture (2 points)
  • Embedding and vector concepts (2 points)
  • Chunking strategy reasoning (2 points)
  • Search and retrieval methods (2 points)
  • Production considerations (2 points)

Practical Skills (15 points)

  • Environment setup and verification (2 points)
  • Document loading and chunking (3 points)
  • Vector store creation and querying (3 points)
  • Complete RAG pipeline with citations (4 points)
  • Advanced technique implementation (3 points)

Application and Reflection (5 points)

  • Domain-specific project completion (3 points)
  • Quality evaluation and iteration (2 points)

Passing Score: 24/30 (80%)

Success Indicators

Immediate (End of Workshop)

  • Working RAG pipeline on your laptop
  • At least 5 documents indexed and searchable
  • Citation-backed answers to domain queries
  • Understanding of how to extend and improve the system

Short-term (1 Week)

  • Added your own real documents to the knowledge base
  • Tried at least one alternative embedding model
  • Shared the system with a colleague
  • Identified 3+ use cases for your organisation

Long-term (1 Month)

  • Production-ready knowledge base in daily use
  • Evaluation pipeline catching quality regressions
  • Team members contributing documents
  • Measurable time savings in information retrieval

Begin Workshop --> | Check Prerequisites | Back to Workshop Overview