Spring AI Evaluation & Guardrails: RelevancyEvaluator, FactCheckingEvaluator, and CI Release Gates

July 11, 2026 · Spring AI for Enterprise Java (part 10)

▶ Watch on YouTube & subscribe to The Stack Underflow

Somewhere in your CI pipeline right now there is probably a test that looks like this: call the endpoint, assert the status is 200, done. No assertion on the body. No check that the data returned is even correct — just proof that something answered. Nobody would accept that as coverage for a payments API. Yet that is exactly how most teams ship AI features today: the model responded, so it shipped.

By Episode 9 of this series you’ve built a RAG pipeline that retrieves context and generates an answer, wired tool calling and MCP servers into it, and instrumented the whole thing with OpenTelemetry traces. All of that proves the pipeline runs. None of it proves the pipeline is right. A retriever can pull the wrong documents, a model can politely ignore the context it was handed and hallucinate an answer anyway, and every trace in your observability dashboard will look perfectly healthy while it happens.

Spring AI closes that gap with an Evaluator interface you can call from inside a @Test, the same way you’d call assertTrue on any other assertion. This tutorial wires it in as a real release gate — not a demo trick, not a log line nobody reads.

The one-sentence version: Spring AI’s Evaluator interface takes the question, the retrieved context, and the model’s answer, and returns a scored pass/fail you can assert on in CI — so a bad answer fails your build instead of reaching a customer.

Why “it responded” isn’t a test

RAG (Episode 6), tool calling (Episode 7), MCP (Episode 8), and observability (Episode 9) each get you closer to a pipeline that works. None of them check whether the answer that came back is actually correct. That’s a specific, nameable gap: relevancy and factual grounding are properties of the content of the response, not the mechanics of producing it. A trace showing 200ms latency and zero errors tells you the call succeeded. It says nothing about whether the model just confidently invented a Spring Boot version number that doesn’t exist.

This is the same distinction as testing an API. assertEquals(200, response.getStatus()) isn’t a test — it’s a liveness check. The actual test asserts on the body: assertEquals(expectedTotal, response.getBody().getTotal()). Spring AI’s evaluation classes let you write that second kind of assertion for AI output, using a judge model instead of a hardcoded expected value.

The Evaluator interface: one seam, two shipped implementations

Everything in Spring AI’s evaluation support plugs into a single functional interface:

public interface Evaluator {
    EvaluationResponse evaluate(EvaluationRequest evaluationRequest);
}

One method in, one scored response out. Spring AI 2.0.0 ships two implementations of it, and they ask different questions of the same answer:

Both take a ChatClient.Builder because both work the same way under the hood: an AI grading an AI. It’s a second, independent model call — not the call that produced the original answer — asked one narrow question about that answer. RelevancyEvaluator’s direct constructor is still valid in 2.0.0 (it was not deprecated), though RelevancyEvaluator.builder() is the more idiomatic construction path if you’re writing new code.

The request the judge model works from is deliberately small:

EvaluationRequest evaluationRequest = new EvaluationRequest(
    question, retrievedDocuments, response);

Three arguments — the original question, the documents the RAG pipeline actually retrieved, and the text the model returned — and nothing else. The judge model never sees your whole conversation history, just the narrow slice it needs to answer one question.

Wiring RelevancyEvaluator into a real @Test

Assume your RAG pipeline already looks like this — built with ChatClient’s fluent API and Spring AI 2.0.0’s builder form for QuestionAnswerAdvisor (the direct new QuestionAnswerAdvisor(vectorStore) constructor from the 1.0.3 course is deprecated in 2.0.0 in favor of the static factory; if you’re following older course code, this is the line to change):

ChatClient ragChatClient = ChatClient.builder(chatModel)
    .defaultAdvisors(QuestionAnswerAdvisor.builder(vectorStore).build())
    .build();

You’ll need the OpenAI starter on the classpath, at the 2.0.0 coordinates, and Java 21 plus Spring Boot 4.0 as the baseline:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
    <version>2.0.0</version>
</dependency>

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        model: gpt-4o-mini

That spring.ai.openai.chat.model path is the flattened 2.0.0 property — it replaces the older nested options.model form, so double-check any config you’re carrying over from a 1.0.3 project.

Now the test. Capture the question, the retrieved documents (pulled from the response metadata under QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS), and the response text, package them into an EvaluationRequest, and assert on isPass():

@Test
void ragAnswerIsRelevant() {
    String question = "What is the latest version of the Spring Framework?";
    ChatResponse chatResponse = ragService.queryWithResponse(question);

    List<Document> retrievedDocuments = chatResponse.getMetadata()
        .get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);
    String response = chatResponse.getResult().getOutput().getText();

    EvaluationRequest evaluationRequest = new EvaluationRequest(
        question, retrievedDocuments, response);

    RelevancyEvaluator relevancyEvaluator =
        new RelevancyEvaluator(ChatClient.builder(chatModel));

    EvaluationResponse evaluationResponse =
        relevancyEvaluator.evaluate(evaluationRequest);

    assertTrue(evaluationResponse.isPass());
}

isPass() true, the test passes, exactly like any other assertion. isPass() false, the build breaks. EvaluationResponse also exposes getScore(), getFeedback(), and getMetadata() if you want more than a boolean — useful the moment you want to log why something failed instead of just that it did.

Swapping in FactCheckingEvaluator for factual grounding

RelevancyEvaluator answers “was this on topic.” It does not answer “was this true.” A model can stay perfectly on topic and still state something the retrieved context never actually said. That’s where FactCheckingEvaluator comes in — same interface, same evaluate(EvaluationRequest) call shape, stricter question:

FactCheckingEvaluator factCheckingEvaluator =
    FactCheckingEvaluator.builder(chatClientBuilder).build();

EvaluationResponse factCheckResponse =
    factCheckingEvaluator.evaluate(evaluationRequest);

assertTrue(factCheckResponse.isPass());

Reusing the same EvaluationRequest you built for the relevancy check is the point — one assembled request, two different judgments run against it. Spring AI 2.0.0 also ships FactCheckingEvaluator.forBespokeMinicheck(ChatClient.Builder), a named-model factory for teams standardizing on the Bespoke Minicheck model specifically for fact-checking. If you’ve been hand-rolling a raw prompt that asks the model “is this true, answer true or false” — a pattern common in earlier Spring AI course material — this is the purpose-built replacement for it, not a rename of the same thing.

Guardrail policy: pass, fail, and human review

Not every score is a clean pass or fail. Treating evaluation as strictly binary either lets too much through or blocks too much for a human to ever review. The workable pattern is a three-way split on getScore(), with the middle band routed to a person instead of a machine:

EvaluationResponse.isPass() / getScore()
              │
   ┌──────────┼───────────────┐
   ▼          ▼               ▼
 ✓ PASS    ⚠ REVIEW         ✕ FAIL
 (high      (borderline,     (low score,
 score,     human-in-the-     dashed —
 ships)     loop queue,       build breaks)
            not auto-ship)

A clean pass ships. A clean fail breaks the build via assertTrue, same as Scene 5’s gate. A borderline score doesn’t auto-ship either way — it goes into a review queue for a human before release. That middle band is what keeps the gate from being either too strict to use or too loose to trust.

The last piece is making this auditable. Log every eval run — isPass(), getScore(), getFeedback() — as a CI artifact next to the request traces you already wired up in Episode 9’s observability tutorial. That pairing is what actually lets compliance sign off on a release: a scored, logged number sitting beside the trace for the same request, not a verbal “looks fine to me.”

How to apply this right now

1. Add org.springframework.ai:spring-ai-starter-model-openai:2.0.0 to your build, and confirm your project is on Java 21 and Spring Boot 4.0 — Spring AI 2.0.0 requires both.
2. If your RAG pipeline still constructs QuestionAnswerAdvisor with new QuestionAnswerAdvisor(vectorStore), switch it to QuestionAnswerAdvisor.builder(vectorStore).build() — the direct constructor is deprecated in 2.0.0.
3. Capture the three evaluation inputs from your existing ChatResponse: the question you sent, the retrieved documents from QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS metadata, and the response text.
4. Package them into an EvaluationRequest and run it through RelevancyEvaluator; wire isPass() into assertTrue() inside a real @Test.
5. Add FactCheckingEvaluator as a second gate against the same EvaluationRequest for factual grounding, not just topical relevance.
6. Define pass/fail/review thresholds against getScore() instead of only isPass(), and route the middle band to a human reviewer, not to production.
7. Log every eval run — score, pass/fail, feedback — as a CI artifact alongside your Episode 9 observability traces so a release decision has a paper trail.

Common misconceptions

“If the model responded, the pipeline works.” A response proves the call succeeded, not that the content is correct. Latency and error-rate dashboards from your observability setup will look perfectly green while the model confidently hallucinates. Relevancy and factual grounding are separate properties that only an evaluator — not a trace — can check.

“RelevancyEvaluator and FactCheckingEvaluator do the same job.” They implement the same Evaluator interface but ask different questions. RelevancyEvaluator checks topical relevance — did this address what was asked. FactCheckingEvaluator checks factual grounding — is this actually supported by the retrieved context. A response can pass one and fail the other.

“Evaluation is a manual QA step, not something that belongs in CI.” EvaluationResponse.isPass() is designed to feed straight into assertTrue() inside a @Test. Treated as a release gate, it runs on every build the same way your other test coverage does — not as a periodic manual spot-check.

“A borderline score should just auto-pass or auto-fail.” Forcing every score into a strict binary either lets shaky answers through or blocks answers that were actually fine. The workable pattern is a third state — route mid-range scores to a human reviewer before release, rather than trusting the judge model’s binary call at the margins.

Frequently asked questions

What is the Evaluator interface in Spring AI 2.0.0? It’s a functional interface, EvaluationResponse evaluate(EvaluationRequest evaluationRequest), that’s the single seam for all evaluation logic in Spring AI. RelevancyEvaluator and FactCheckingEvaluator are the two shipped implementations, both verified against org.springframework.ai.evaluation.Evaluator and org.springframework.ai.chat.evaluation.* in 2.0.0.

How does RelevancyEvaluator differ from FactCheckingEvaluator? RelevancyEvaluator checks whether the answer is on topic relative to the question. FactCheckingEvaluator checks whether the answer is factually supported by the retrieved context. Both take an EvaluationRequest built from the same question/context/response triple, so you can run both checks against one assembled request.

Do I need a full RAG pipeline to use Evaluator? No — EvaluationRequest’s three-argument constructor takes documents as a List<Document>, but nothing forces those documents to come from a vector store retrieval. You can evaluate any chat response as long as you can supply the context it should be grounded against.

Is RelevancyEvaluator’s constructor deprecated in Spring AI 2.0.0? No. Unlike QuestionAnswerAdvisor, RelevancyEvaluator’s direct constructor — new RelevancyEvaluator(ChatClient.Builder) — remains valid and unchanged in 2.0.0. RelevancyEvaluator.builder() was added as an additional, more idiomatic path, not a replacement.

How do I set pass/fail thresholds using getScore()? EvaluationResponse.getScore() gives you a numeric score in addition to the boolean isPass(). Define your own thresholds — for example, high scores pass automatically, low scores fail the build, and a middle band routes to ⚠ REVIEW for a human reviewer rather than an automated ship/kill decision.

Can I log evaluation results alongside my observability traces? Yes, and you should. Log isPass(), getScore(), and getFeedback() from each eval run as a CI artifact next to the request traces from Episode 9’s OpenTelemetry setup. That pairing — a scored eval result next to the trace for the same request — is what gives a compliance sign-off something concrete to point to.

Where this fits in the series

This is Episode 10 of Spring AI for Enterprise Java. The previous episode covered instrumenting your pipeline with OpenTelemetry (Observability); this one covers proving what those instrumented calls actually returned is correct. The next episode locks down who can call this thing at all — Security & Compliance Architecture.

Full runnable code for this episode lives in the spring-ai-enterprise-examples repo: github.com/thestackunderflow/spring-ai-enterprise-examples/tree/main/10-evaluation-guardrails.

Spring AI moves fast — this series is pinned to 2.0.0, and update notes live at thestackunderflow.com/spring-ai. If you want the video walkthrough when it lands, subscribe to @TheStackUnderflow on YouTube.