RAG with Spring AI: Documents → Chunks → Embeddings → Vector Store → Grounded Answers

July 11, 2026 · Spring AI for Enterprise Java (part 6)

▶ Watch on YouTube & subscribe to The Stack Underflow

Ask a production ChatClient about your internal roadmap, your compliance policy, or last quarter’s numbers, and it will answer. Confidently. And it will be wrong — not because the model is broken, but because it was never trained on your documents. Nobody ships a fine-tuned model every time a policy PDF changes, so retraining isn’t the fix.

The fix is retrieval: at query time, find the handful of paragraphs in your own documents that are actually relevant, and hand them to the model as context before it answers. That’s Retrieval-Augmented Generation, and in Spring AI 2.0.0 it costs about six lines of Java for the simple case — plus a modular, production-grade version once you outgrow the simple case.

This tutorial walks the full pipeline end to end: documents in, grounded answer out, with the exact 2.0.0 API surface (no 1.0.3-course leftovers, no deprecated constructors).

The one-sentence version: RAG turns your documents into a searchable vector index at build time, and at query time retrieves the nearest-meaning chunks and stuffs them into the prompt — so the model answers from your text instead of guessing.

Before any of the code below: Spring AI 2.0.0 requires Java 21 to build and runs on Spring Boot 4.0 / Spring Framework 7.0. If you’re pulling in a chat model, the starter coordinate is org.springframework.ai:spring-ai-starter-model-openai:2.0.0 (not the pre-1.0 spring-ai-openai-spring-boot-starter name), and model selection now goes through the flattened spring.ai.openai.chat.model property rather than a nested options.model path.

The pipeline nobody draws all the way through

Everyone talks about “RAG” as one idea. It’s actually five distinct stages, and knowing where each one lives is what makes the Spring AI abstraction click:

YOUR DOCS
   │
   ▼  JsoupDocumentReader / PagePdfDocumentReader
List<Document>
   │
   ▼  TokenTextSplitter.builder().build().apply(documents)
chunks (many smaller Document objects)
   │
   ▼  EmbeddingModel.embed(...)
vectors — each chunk is now a point in vector space
   │
   ▼  VectorStore.add(chunks)
SimpleVectorStore (dev)  or  PgVectorStore / RedisVectorStore (prod)
   │
   ▼  similarity search at query time
top-K nearest chunks
   │
   ▼  QuestionAnswerAdvisor  or  RetrievalAugmentationAdvisor
grounded answer, cited from YOUR text

Every stage below is a real Spring AI 2.0.0 class. Nothing here is hand-waved.

Stage 1 & 2: read the documents, then chunk them

Document readers turn a web page or a PDF into Spring AI’s Document type — same shape regardless of source:

List<Document> wikiDocs = new JsoupDocumentReader(wikipediaUrl).get();
List<Document> pdfDocs  = new PagePdfDocumentReader(wefReportResource).get();

List<Document> allDocs = new ArrayList<>(wikiDocs);
allDocs.addAll(pdfDocs);

A single Document covering an entire web page or report is a bad retrieval unit — it’s too broad to be a precise match and too large to fit cheaply into a prompt. TokenTextSplitter cuts it into smaller, token-sized, embeddable pieces:

List<Document> chunks = TokenTextSplitter.builder()
        .build()
        .apply(allDocs);

Note the builder form. In 2.0.0, TokenTextSplitter’s constructors are deprecated in favor of .builder().build() — if you’re following older Spring AI material (or the 1.0.3 course line), new TokenTextSplitter() still compiles but is no longer the documented path.

Stage 3: embeddings and the vector store

Each chunk gets converted into a numeric vector — a fingerprint of its meaning — via the EmbeddingModel abstraction:

float[] vector = embeddingModel.embed(chunk.getText());
int dimensions = embeddingModel.dimensions();

“Relevant” isn’t a keyword match here. It’s distance. A user’s question is embedded through the exact same model, and the chunks whose vectors sit closest to the question’s vector are the closest in meaning — not the closest in shared words. Spring AI even exposes the math directly: SimpleVectorStore.EmbeddingMath.cosineSimilarity(vectorX, vectorY).

Vectors have to live somewhere searchable, and this is a real dev/prod decision, not a Spring AI limitation:

@Bean
VectorStore vectorStore(EmbeddingModel embeddingModel) {
    // dev only — in-memory, lost on restart
    return SimpleVectorStore.builder(embeddingModel).build();
}

Index the chunks once and the store handles embedding + storage together:

vectorStore.add(chunks);

Swap the bean for an auto-configured PgVectorStore or RedisVectorStore and every line downstream — the advisor, the ChatClient call — stays identical. That’s the entire point of the VectorStore interface.

Stage 4: the one-line grounded answer

This is the payoff. QuestionAnswerAdvisor searches the vector store, stuffs the closest matches into the prompt, and lets the model answer from your text:

QuestionAnswerAdvisor qaAdvisor = QuestionAnswerAdvisor.builder(vectorStore).build();

String answer = chatClient.prompt()
        .advisors(qaAdvisor)
        .user(question)
        .call()
        .content();

Same as TokenTextSplitter, the constructor form (new QuestionAnswerAdvisor(vectorStore)) is what older material shows — 2.0.0 documents the static .builder(VectorStore) factory as the path forward.

A well-grounded system also has to handle the case where nothing relevant was indexed. Ask something outside the knowledge base and a correctly grounded advisor doesn’t fabricate — it says it doesn’t know. That’s the system working, not failing. Treat “I don’t have enough information to answer that” as a grounded success, never as an error.

Stage 5: the enterprise upgrade — RetrievalAugmentationAdvisor

QuestionAnswerAdvisor is one call that does retrieval, augmentation, and prompting together. That’s great for a demo and painful to test or tune in production — you can’t independently swap the retriever, adjust the similarity threshold, or add a post-processing step without touching the whole thing.

RetrievalAugmentationAdvisor (package org.springframework.ai.rag.advisor, new relative to the 1.0.3 course line) breaks that single call into named, individually swappable stages implementing what Spring AI calls the “Modular RAG Architecture”:

RetrievalAugmentationAdvisor ragAdvisor = RetrievalAugmentationAdvisor.builder()
        .documentRetriever(VectorStoreDocumentRetriever.builder()
                .vectorStore(vectorStore)
                .similarityThreshold(0.73)
                .topK(5)
                .filterExpression(filterExpression)
                .build())
        .build();

The builder also accepts .queryTransformers(...), .documentJoiner(...), .documentPostProcessors(...), .queryAugmenter(...), .taskExecutor(...), .scheduler(...), and .order(...) — each one a Spring bean you can test, mock, and replace on its own. VectorStoreDocumentRetriever is where the tuning knobs live: similarityThreshold (how close a match has to be), topK (how many chunks to retrieve), and filterExpression (metadata filtering — tenant isolation, document type, date range). Wire the resulting advisor into ChatClient exactly like QuestionAnswerAdvisor — via .advisors(ragAdvisor).

How to apply this right now

1. Start with SimpleVectorStore locally. Don’t reach for Postgres or Redis until you’re past the demo stage — SimpleVectorStore.builder(embeddingModel).build() is the fastest way to see the pipeline work end to end.
2. Use .builder() for every 2.0.0 class that offers one. TokenTextSplitter, QuestionAnswerAdvisor, and SimpleVectorStore all deprecated their constructors in 2.0.0 — don’t copy constructor-based snippets from older tutorials or the 1.0.3 course material.
3. Chunk before you embed. Never call EmbeddingModel.embed(...) on a whole document; run it through TokenTextSplitter first so each retrieval unit is small and specific.
4. Move to PgVectorStore or RedisVectorStore before production. SimpleVectorStore has no persistence — a restart silently empties your knowledge base.
5. Graduate to RetrievalAugmentationAdvisor once you need tuning. If you find yourself wanting a different similarity threshold per tenant, metadata filtering, or a testable retrieval stage in isolation, that’s the signal you’ve outgrown QuestionAnswerAdvisor.
6. Design for “I don’t know.” Test your grounded system with an out-of-scope question and confirm it declines instead of guessing — that’s the behavior that actually earns trust in an enterprise deployment.

Common misconceptions

“RAG means fine-tuning the model on your docs.” No retraining happens at all. RAG retrieves relevant text at query time and injects it into the prompt; the model’s weights never change. That’s exactly why it’s cheap to keep current — reindex when a document changes, don’t retrain.

“SimpleVectorStore is fine for production if traffic is low.” Traffic has nothing to do with it. SimpleVectorStore is in-memory — every restart, deploy, or scale-out event loses the index. It’s a dev tool by design, not a small-scale production option.

“More retrieved chunks means a better answer.” Stuffing in every marginally-similar chunk dilutes the prompt and can push the model toward irrelevant text. VectorStoreDocumentRetriever’s topK and similarityThreshold exist specifically to keep retrieval precise, not exhaustive.

“A model that says ‘I don’t know’ is a failure case.” For a grounded system, it’s the opposite — it means the retrieval step correctly found no strong match and the model declined to guess instead of fabricating an answer. That’s the entire risk RAG is built to close.

Frequently asked questions

What’s the difference between QuestionAnswerAdvisor and RetrievalAugmentationAdvisor? QuestionAnswerAdvisor.builder(vectorStore).build() is a one-line advisor that combines retrieval and prompt augmentation in a single step — ideal for getting RAG working quickly. RetrievalAugmentationAdvisor breaks that same behavior into composable, individually configurable stages (documentRetriever, documentJoiner, documentPostProcessors, queryAugmenter, and more) for cases where you need tuning, testing, or governance per stage.

Do I need a persistent vector store for a proof of concept? No. SimpleVectorStore.builder(embeddingModel).build() runs entirely in memory and is the fastest way to validate a RAG pipeline locally. Move to PgVectorStore or RedisVectorStore only once the index needs to survive a restart.

How does Spring AI decide which chunks are “relevant”? Through vector similarity, not keyword matching. Each chunk and each incoming question are converted to vectors by the same EmbeddingModel, and the chunks whose vectors are closest — typically by cosine similarity, exposed directly via SimpleVectorStore.EmbeddingMath.cosineSimilarity(...) — are retrieved.

What does TokenTextSplitter actually do, and why not skip it? It splits large documents into smaller, token-bounded chunks before embedding. Skipping it means embedding an entire document as one vector, which blurs its meaning and makes precise retrieval effectively impossible. Use TokenTextSplitter.builder().build().apply(documents) — the constructor form is deprecated in 2.0.0.

Can I filter retrieval by tenant or document type? Yes — that’s what filterExpression on VectorStoreDocumentRetriever.builder() is for, used inside a RetrievalAugmentationAdvisor. QuestionAnswerAdvisor doesn’t expose this level of control, which is one of the concrete reasons to graduate to the modular advisor in a multi-tenant system.

Does RAG replace the need for a fine-tuned model? For grounding a model in current, private, or frequently-changing data — yes, almost always. Fine-tuning changes how a model behaves; RAG changes what it knows at answer time. Most enterprise “the model doesn’t know our data” problems are retrieval problems, not training problems.

Where this fits in the series

This is Episode 6 of Spring AI for Enterprise Java. The previous episode covered typed, structured output from ChatClient — see Structured Output. The next episode moves from retrieving your documents to letting the model call your Java services directly — see Tool Calling.

Runnable code for this episode lives in the 06-rag-vector-stores module of the spring-ai-enterprise-examples repo.

The companion video for this episode isn’t published yet — subscribe to @TheStackUnderflow on YouTube so it lands in your feed the moment it’s live.