LLMs and RAG Explained: A Practical Guide to Building AI Applications

A large language model (LLM) is an AI model trained on huge amounts of text to generate language. Retrieval-Augmented Generation (RAG) is a technique that gives an LLM relevant documents when it answers, so it can respond from your data and not only from what it learned in training.

Ask a general-purpose chatbot about your company's leave policy and it will either decline or make something up. It has never seen the policy. RAG solves this by finding the right paragraph first and handing it to the model along with the question. That idea sits behind many practical AI products, including internal assistants and customer support tools.

This guide explains how the parts fit together for someone who knows a little programming. It includes a small Python script you can run, with no API key or libraries needed.

What is a large language model?

An LLM is a neural network that learns statistical patterns in text and uses them to predict what should come next. Repeating that prediction, one small piece of text at a time, produces whole answers.

Modern LLMs build on the Transformer architecture introduced in the 2017 paper "Attention Is All You Need". You do not need the mathematics to build with them, but four practical facts matter:

• Models work on tokens. Text is split into small units, and long inputs and outputs cost more. There is also a limit on how much fits in one request.
• Knowledge stops at the training data. A model does not know about your private documents or recent events unless you supply them.
• Fluent does not mean correct. A model can state something false with total confidence. OWASP lists misinformation among its top risks for LLM applications.
• Output can vary. The same prompt can produce different responses, which changes how you have to test.

Prompt engineering: where everyone starts

Prompt engineering is the practice of writing and testing instructions so that a model reliably produces what you need. It works best when you define what a good answer looks like before you start tweaking wording.

Anthropic's prompt engineering guide assumes that you already have clear success criteria and a way to test against them. OpenAI's guide is another useful reference. A few habits go a long way:

• Be specific about the task, the audience and the output format.
• Provide examples of good output for anything subtle.
• Separate instructions from data with clear markers, so that the two do not blur.
• Tell the model what to do when it does not know. "Say you don't know" is one of the most useful lines you can write.

What are embeddings?

Embeddings are lists of numbers that represent the meaning of a piece of text. Texts with related meanings get numerically similar embeddings, which lets a program search by meaning instead of exact words.

That is why a search for "my laptop won't turn on" can match a document titled "Troubleshooting a device that fails to boot". OpenAI's embeddings guide describes the distance between two vectors as a measure of how related they are, and recommends cosine similarity for comparing them.

How does RAG work?

RAG finds the passages most relevant to a question, inserts them into the prompt, and asks the model to answer from them. The technique was named in a 2020 paper by Lewis and colleagues.

A RAG system has two stages. The LangChain RAG tutorial walks through both in code.

Indexing (done ahead of time)

1. Load your source documents.
2. Split them into chunks, as large documents are hard to search precisely. The LangChain tutorial uses chunks of 1,000 characters with a 200-character overlap, which is a reasonable first experiment and not a rule.
3. Embed each chunk.
4. Store the chunks and their embeddings in a vector store.

Answering (done for each question)

1. Embed the question.
2. Retrieve the most similar chunks.
3. Build a prompt containing those chunks and the question.
4. Generate an answer, ideally with references to its sources.

Build the retrieval step in 30 lines of Python

This script runs on the Python standard library alone. Real systems use an embedding model, whereas this one counts words to keep the idea visible. Calling a real model would replace only the final print.

import math
import re
from collections import Counter

CHUNKS = [
    "The Playwright course runs for 8 weeks and covers TypeScript and CI/CD.",
    "The AI course runs for 10 weeks and covers LLMs, LangChain and RAG.",
    "New batches begin on the 1st of every month.",
]

def vectorize(text):
    return Counter(re.findall(r"[a-z0-9]+", text.lower()))

def cosine(a, b):
    dot = sum(a[word] * b[word] for word in a)
    size = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
    return dot / size if size else 0.0

def retrieve(question, k=1):
    query = vectorize(question)
    return sorted(CHUNKS, key=lambda chunk: cosine(query, vectorize(chunk)), reverse=True)[:k]

question = "How many weeks is the AI course?"
context = "\n".join(retrieve(question))

prompt = f"""Answer using only the context. If the answer is not there, say you don't know.

Context:
{context}

Question: {question}"""

print(prompt)

When we ran it, it selected the correct chunk:

Context:
The AI course runs for 10 weeks and covers LLMs, LangChain and RAG.

Question: How many weeks is the AI course?

Now try asking "How long is the training?". When we tried it, the script picked the chunk about batch dates, because the question shares almost no words with the right one. That failure shows exactly why real systems use embeddings, which match on meaning. It is also a good first experiment to run once you have an embedding model available.

Do you need RAG, or something else?

Developers often reach for RAG too soon. In our opinion, this table is a sensible way to decide:

Where does LangChain fit?

LangChain is an open-source framework for building applications on top of LLMs. It provides common interfaces for models, loaders, text splitters, vector stores and agents, so you can swap components without rewriting your application.

We suggest building the simple version by hand first, as above. You will understand what a framework is doing for you, and you will be able to debug it when retrieval quietly returns the wrong chunk.

How do you know if it works?

Evaluate retrieval and generation separately, using a fixed set of real questions with known good answers. Trying a few questions by hand will not reveal problems reliably.

• Was the right chunk retrieved? If it was not, no model can answer correctly. Most RAG problems begin here.
• Did the answer stick to the context? Check for claims that do not appear in the retrieved text.
• Did it refuse when it should? Include questions your documents cannot answer.

Rerun the set after every change to chunking, models or prompts. Anthropic and OpenAI both publish guidance on building evaluations. If you come from software testing, this should feel familiar: it is regression testing for a system whose output can vary.

Security risks you should know

The OWASP Top 10 for LLM Applications (2025) is the best starting list. Three risks are especially relevant to RAG:

• Prompt injection. A retrieved document can contain text such as "ignore your instructions". Treat retrieved content as untrusted data, not as commands.
• Sensitive information disclosure. Retrieval must respect permissions. If a user cannot open a document, its chunks must not reach their prompt.
• Excessive agency. Give an AI system only the tools and permissions it needs, and require human approval for risky actions.

A learning roadmap

1. Learn Python basics: functions, data structures, working with files and calling APIs.
2. Call an LLM API and experiment with prompts and output formats.
3. Understand embeddings by comparing sentences yourself.
4. Build a small RAG app over documents you know well, so you can judge its answers.
5. Add evaluation, then improve chunking and retrieval against your test set.
6. Deploy it. Our cloud computing guide covers the fundamentals, and a MERN or MEAN front end gives your app a real interface.

Right & Bright's 10-week, intermediate-level AI Course with LLMs covers Python, LLMs, prompt engineering, LangChain, RAG and hands-on projects.

Frequently asked questions

Do I need machine learning experience to build with LLMs?

No. Building applications on existing models is closer to software engineering than to training models. Python and an understanding of APIs matter most at first. Machine learning knowledge becomes more valuable if you later move into model training.

Does RAG stop hallucinations?

It reduces them, but it does not eliminate them. The right document may not be retrieved, the document may be wrong, or the model may misread it. Evaluate your system, and show sources so that people can check answers.

Do I need a vector database?

Not at first. A small project can keep embeddings in memory or in a database it already uses. Consider a dedicated vector store when scale or filtering needs justify one.

What is the difference between RAG and fine-tuning?

RAG supplies information at question time, so answers can reflect new documents immediately. Fine-tuning adjusts a model's behaviour through training, and is better suited to style or format than to fast-changing facts.

Which LLM should I learn with?

Any major model is fine for learning, and the concepts carry over. Models and prices change often, so check current documentation, and keep your code easy to switch.

Sources and further reading

• Vaswani et al.: Attention Is All You Need
• Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• LangChain: Build a RAG agent
• OpenAI: Embeddings guide
• Anthropic: Prompt engineering overview
• OWASP: Top 10 for LLM Applications