Systematically Improving RAG Applications

4.7

(23 ratings)

·

6 Weeks

·

Cohort-based Course

Follow a repeatable process to continually evaluate and improve your RAG application

Instructors' Experience

Stitch Fix
Google
Straive
Meta
Weights & Biases

Course overview

Learn and implement the playbook for success with RAG

You're working on a new RAG application in a chaotic environment. Competing priorities and limited resources make progress challenging, with each day feeling like a roll of the dice. Success hinges on finding "the right path" - a sequence of actions for maximum growth in minimal time.


Beyond technical skills, effective AI development requires strategic decision-making and resource allocation.

Build strong fundamentals in building searching systems and improve your RAG systems. I've refined these over 8 years of machine learning engineering and AI consulting. This course will equip you with the skills to:


- Implement robust processes for identifying system bottlenecks

- Significantly enhance your probabilistic systems

- Apply advanced RAG techniques for real-world applications

- Seize this opportunity to transform your RAG capabilities and stay ahead in the rapidly evolving field of applied AI


Week 1: RAG System Foundations


- Create robust evaluation pipelines, develop scalable datasets, and apply the RAG System Inference Flywheel concept

- Recognize and mitigate intervention bias and absence blindness in your RAG systems

- Implement fast, iterative improvement cycles using precision and recall metrics

- Distinguish between leading and lagging metrics to set actionable goals

- Master techniques for continuous improvement and optimization of RAG foundations


Week 2: Query Processing and Intent Analysis


- Implement classification systems for query segmentation and conduct thorough bottleneck analysis

- Apply data-driven approaches to detect concept drift and adapt systems dynamically

- Understand inventory vs capability segments and develop strategies for unsupervised topic discovery

- Create comprehensive dashboards for visualizing query patterns and system performance

- Learn advanced techniques for enhancing query understanding and intent analysis


Week 3: Multi-Index Search and Intelligent Routing


- Design and implement search strategies for diverse content types (images, documents, tables, text-to-SQL)

- Develop metrics to measure and improve search quality across multiple indices

- Build intelligent routers and implement sophisticated query understanding techniques

- Construct and optimize multi-hop agent systems utilizing multiple search indices

- Evaluate and balance trade-offs between search architectures for latency, cost, and accuracy


Week 4: RAG Product Design and User Experience


- Design RAG products that effectively collect user feedback and implement streaming strategies

- Create intuitive UI components for citations and user interaction, enhancing overall UX

- Develop strategies for handling negative examples and continuously improving performance

- Implement validators and monologue techniques to significantly enhance response quality

- Align product features with business objectives while maximizing user satisfaction and engagement


What You'll Learn


After completing this comprehensive RAG course, you'll be equipped to:


- Implement rapid, data-driven iteration cycles using synthetic evaluations and user feedback to identify and address system bottlenecks

- Design intuitive UIs that simultaneously enhance and measure customer satisfaction, optimizing the balance between user experience and data collection

- Develop intelligent query routing and search architectures that maximize performance within resource constraints

- Create compelling, data-backed arguments for system improvements and resource allocation

- Effectively communicate RAG system enhancements and their impact to both technical and non-technical stakeholders

This comprehensive playbook will enable you to deliver consultant-level value, leading your team to results through structured experimentation. We guarantee meaningful improvements in processes in 5 weeks or we'll personally refund you and do a free consultation to help figure out your needs.


Our consulting has achieved:


- 80%+ satisfaction in previously unsuccessful query segments

- 4x increase in feedback volume and 10% boost in CSAT

- Finetune embedding models that outperform closed models by 20-30%

- 2-3x increase in weekly experiments using synthetic data benchmarks

- Real-time monitoring systems revealing new product positioning insights


Don't do it alone - be part of a small cohort of other teams shipping real applications.


To ensure everyone can be successful, the course requires a short application. You pay only after your application is approved.


Get these free bonuses (over $1500 in value):

$500 Cohere credits (Jason uses Cohere rerankers in every single RAG product he's build or adviced)

$200 LanceDB credits and free access to Lance Cloud

$500 in Modal Labs credits (useful for experimenting with embedding fine-tuning)

6 months free Notion AI Plus (get experience with more RAG products)

3 months Braintrust access ($250 value)

This Course Is For You If You Are

01

An Engineering or Product leader looking to improve an existing RAG system MVP

02

Solving problems like poor retrieval, unreliable outputs or unhappy customers with your existing application

03

Ready to lead your team in building a data flywheel so you can leverage feedback

What you’ll get out of this course

Join a community of operators that are also shipping production applications

Identify and test improvements over time by developing a experimentation platform


Specialize your search backends to serve the questions that matter to your business


Identify differences in query satisfaction across segments, so you focus on key improvement areas


Apply techniques including rerankers and ColBERT to improve search quality while managing system latency


Learn to handle both data idiosyncrasies and organizational challenges from guest instructors who have deployed successful RAG applications

Experiment with credits, tools, and talks from others building in the space

  • Talks from Cohere, Vespa, LanceDB, Zapier, Braintrust, Answer.AI on how to improve systems, not just vendor talks.
  • Credits from Modal to explore fine tuning embedding models and rerankers
  • Talk from New Computer on how they improved memory
  • and more!

This course includes

16 interactive live sessions

Lifetime access to course materials

32 in-depth lessons

Direct access to instructor

Projects to apply learnings

Guided feedback & reflection

Private community of peers

Course certificate upon completion

Maven Satisfaction Guarantee

This course is backed by Maven’s guarantee. You can receive a full refund within 14 days after the course ends, provided you meet the completion criteria in our refund policy.

Course syllabus

Expand all modules
  • Week 1

    Feb 4—Feb 9

    Events

    • Feb

      4

      Intro to the Playbook + RAG Evaluation

      Tue, Feb 4, 6:00 PM - 7:00 PM UTC

    • Feb

      5

      Breakout + Office Hours

      Wed, Feb 5, 6:00 PM - 7:00 PM UTC

    • Feb

      6

      Guest Speaker session

      Thu, Feb 6, 6:00 PM - 7:00 PM UTC

  • Week 2

    Feb 10—Feb 16

    Events

    • Feb

      11

      Identifying Areas of Improvement

      Tue, Feb 11, 6:00 PM - 7:00 PM UTC

    • Feb

      12

      Breakout + Office Hours

      Wed, Feb 12, 6:00 PM - 7:00 PM UTC

    • Feb

      13

      Guest speaker session

      Thu, Feb 13, 6:00 PM - 7:00 PM UTC

  • Week 3

    Feb 17—Feb 23

    Events

    • Feb

      18

      IR Keys + Non-Text Data

      Tue, Feb 18, 6:00 PM - 7:00 PM UTC

    • Feb

      19

      Breakout + Office Hours

      Wed, Feb 19, 6:00 PM - 7:00 PM UTC

    • Feb

      20

      Guest Speaker Session

      Thu, Feb 20, 6:00 PM - 7:00 PM UTC

  • Week 4

    Feb 24—Mar 2

    Events

    • Feb

      25

      Routing Queries

      Tue, Feb 25, 6:00 PM - 7:00 PM UTC

    • Feb

      26

      Breakout + Office Hours

      Wed, Feb 26, 6:00 PM - 7:00 PM UTC

    • Feb

      27

      Guest Speaker Session

      Thu, Feb 27, 6:00 PM - 7:00 PM UTC

  • Week 5

    Mar 3—Mar 9

    Events

    • Mar

      4

      Representations

      Tue, Mar 4, 6:00 PM - 7:00 PM UTC

    • Mar

      5

      Breakout + Office Hours

      Wed, Mar 5, 6:00 PM - 7:00 PM UTC

    • Mar

      6

      Guest Speaker Session

      Thu, Mar 6, 6:00 PM - 7:00 PM UTC

  • Week 6

    Mar 10—Mar 13

    Events

    • Mar

      12

      Breakout + Office Hours

      Wed, Mar 12, 5:00 PM - 6:00 PM UTC

  • Post-Course

    Modules

    • Product Design

    • Rejecting work

    • Intro To The Playbook

    • RAG Evaluation

    • Synthetic Data

    • Identifying Areas of Improvement

    • Production Monitoring and Analysis

    • Improving Retrieval

    • Tables and Non-Text Data

    • Routing Queries

    • Representations

    • Synthetic Text Chunks

4.7

(23 ratings)

What students are saying

Meet your instructor

Jason Liu

Jason Liu

Jason has built search and recommendation systems for the past 6 years. He has consulted and advised a dozens startups in the last year to improve their RAG systems. He is the creator of the Instructor Python library.

Dan Becker

Dan Becker

Dan has worked in AI since 2011, when he finished 2nd (out of 1350+ teams) in a Kaggle competition with a $500k prize. He contributed code to TensorFlow as a data scientist at Google and he has taught online deep learning courses to over 250k people. Dan has advised AI projects for 6 companies in the Fortune 100.

A pattern of wavy dots

Join an upcoming cohort

Systematically Improving RAG Applications

Cohort 2

$1,650

Dates

Feb 4—Mar 13, 2025

Application Deadline

Feb 1, 2025
Get reimbursed

Bulk purchases

Course Schedule Each Week

  • Tuesday: Workshops

    1:00 - 2:00PM ET

    Workshops covering each step of the playbook and helping you build process improvements in your RAG application

  • Wednesday: Office Hours + Breakout Sessions

    1:00 - 2:00PM ET

    The first half hour will be interactive breakout sessions, and the closing half-hour each week is Q&A

  • Thursday: Guest Speakers

    1:00 - 2:00PM ET

    Guest instructors covering key topics in both innovative theory and practical applications in RAG system development.

Learning is better with cohorts

Learning is better with cohorts

Active hands-on learning

This course builds on live workshops and hands-on projects

Interactive and project-based

You’ll be interacting with other learners through breakout rooms and project teams

Learn with a cohort of peers

Join a community of like-minded people who want to learn and grow alongside you

Frequently Asked Questions

What happens if I can’t make a live session?

I work full-time, what is the expected time commitment?

What’s the refund policy?

Stay in the loop

Sign up to be the first to know about course updates.

A pattern of wavy dots

Join an upcoming cohort

Systematically Improving RAG Applications

Cohort 2

$1,650

Dates

Feb 4—Mar 13, 2025

Application Deadline

Feb 1, 2025
Get reimbursed

Bulk purchases

$1,650

4.7

·

6 Weeks