1 Overview: Is the Answer Actually Good?

Tutorial 22 introduced evaluation infrastructure — a golden dataset, metrics, regression testing. Tutorial 25 covered the operational metrics a dashboard tracks automatically — cost, latency, error rate. Neither answers the question this tutorial takes on directly: is what the model actually said good? A response can be fast, cheap, and error-free while still missing the point of the question, confidently stating something false, or contradicting itself between two similar questions. This tutorial is about judging response quality itself, the dimension no amount of operational monitoring can substitute for.

This is an intermediate-level, practical tutorial covering: evaluating response relevance (does the answer address the actual question); detecting hallucinations (is it making things up); consistency checks (does it answer the same question the same way twice); prompt refinement based on outputs (using what you observe to improve the prompt, not just the code); basic qualitative evaluation metrics (structured but non-automated judgment); and comparing multiple outputs (choosing between candidates, whether from different prompts, different models, or repeated runs). These are the techniques that let you say, with evidence, 'this answer is good' rather than 'this answer arrived quickly.'

2 Learning Objectives

• Evaluate response relevance — whether an answer actually addresses what was asked, independent of factual correctness.
• Detect hallucinations, including groundedness checks against source context for RAG systems.
• Run consistency checks to reveal how stable a model's answers are across repeated or logically equivalent questions.
• Refine prompts based on observed output patterns, closing the loop from evaluation back to prompt design.
• Apply basic qualitative evaluation metrics using rubrics and LLM-as-judge techniques where fully automated metrics fall short.
• Compare multiple candidate outputs systematically, including pairwise comparison, to choose the best one.

3 Prerequisites

• Tutorial 22's evaluation foundation: golden datasets, evaluation metrics, and regression testing.
• Tutorial 15's RAG concepts, particularly grounding and citations, which this tutorial's hallucination detection builds on directly.
• Comfort writing and iterating on system prompts (tutorials 11, 15, 17) — prompt refinement assumes you can already write a first-draft prompt.
• No new frameworks are introduced; this tutorial is technique and practice, applicable to any system built in this course.

4 Key Concepts: Quality Has Dimensions

'Is this answer good?' is not one question — it's several distinct questions that can each be true or false independently. An answer can be relevant but hallucinated (directly addresses the question, with a fabricated fact). It can be grounded but incomplete (everything stated is true, but it misses part of what was asked). It can be excellent once and wildly different the next time on the same question (inconsistent). Evaluating response quality means checking each dimension deliberately rather than forming one vague overall impression.

These six feed into each other as a loop: you evaluate relevance, groundedness, and consistency on real outputs; that evaluation reveals specific patterns of failure; prompt refinement acts on those patterns; and comparing outputs (old prompt versus new) tells you whether the refinement actually helped, closing the loop back to evaluation. Qualitative metrics and rubrics are the tool that makes each of these judgments structured and repeatable rather than ad hoc.

5 Deep Dive 1: Evaluating Response Relevance and Detecting Hallucinations

Response relevance asks a narrower question than 'is this a good answer': does it actually address what was asked, at all? A fluent, well-written, entirely accurate response to a question the user didn't ask is still a failure — a common pattern is a RAG system retrieving the wrong passage and confidently answering the question that passage happens to address, rather than the user's actual question. Evaluating relevance means checking, explicitly, whether the answer's content maps onto the question's content, independent of whether that content is true.

Hallucination is a distinct failure: fluent, confident output that is factually wrong or unsupported. For grounded (RAG) systems specifically, a groundedness check verifies every claim in the answer traces back to the retrieved source passages — an answer can be relevant (it's clearly trying to answer the right question) and still hallucinated (it states something the sources never said). Citation accuracy is a specific, checkable form of this: if an answer cites [2] for a claim, does source [2] actually support that claim, or was the citation just plausibly placed near the right area of the text?

// Illustrative structure: an LLM-as-judge call checking each claim against sources.
async Task<GroundednessResult> CheckGroundednessAsync(
    string answer, IReadOnlyList<string> sourcePassages, CancellationToken ct)
{
    string judgePrompt = $"""
        Sources:
        {string.Join("\n---\n", sourcePassages)}

        Answer to check: {answer}

        For each factual claim in the answer, state whether it is directly
        supported by the sources (SUPPORTED), contradicted (CONTRADICTED),
        or not mentioned at all (UNSUPPORTED). List each claim and its verdict.
        """;

    var judgeCompletion = await judgeClient.CompleteChatAsync(
        new UserChatMessage(judgePrompt), cancellationToken: ct);
    return ParseGroundednessVerdicts(judgeCompletion.Value.Content[0].Text);
}

6 Deep Dive 2: Consistency Checks and Prompt Refinement

Consistency checks test whether a model gives the same, or compatibly equivalent, answer across repeated runs of the same question or across logically equivalent rephrasings. Self-consistency — running the identical prompt several times and comparing outputs — reveals output variance: at low temperature, a well-designed prompt should produce largely stable answers on factual questions; wide variance on a factual question (not a creative one) is a signal the prompt is underspecified or the question is more ambiguous than intended. A related check: asking the same underlying question two different ways ('what's the refund window?' versus 'how many days do I have to return something?') should produce compatible answers — a contradiction between them reveals a gap in the system's actual knowledge or grounding, not just prompt phrasing sensitivity.

async Task<double> MeasureSelfConsistencyAsync(
    string question, int runs, CancellationToken ct)
{
    var answers = new List<string>();
    for (int i = 0; i < runs; i++)
    {
        var completion = await chat.CompleteChatAsync(
            new UserChatMessage(question),
            new ChatCompletionOptions { Temperature = 0.2f }, ct);
        answers.Add(completion.Value.Content[0].Text);
    }

    // Illustrative: an LLM-as-judge call comparing all answers for compatible meaning,
    // returning a 0-1 consistency score rather than requiring exact string match.
    return await JudgeSemanticConsistencyAsync(answers, ct);
}

Prompt refinement based on outputs is the deliberate response to what relevance, groundedness, and consistency checks reveal: a specific, observed failure pattern (not a vague 'the answer wasn't great') points at a specific prompt change. If relevance checks reveal the model frequently answers an adjacent-but-wrong question, the system prompt may need an explicit instruction to restate the question before answering. If groundedness checks reveal frequent unsupported claims, the grounding instruction (tutorial 15) may need strengthening or the retrieval step improved rather than the generation prompt. If consistency checks reveal high variance, lowering temperature or adding more explicit constraints to the prompt is the likely fix.

7 Deep Dive 3: Basic Qualitative Evaluation Metrics

Some quality dimensions resist a clean automated pass/fail check — tone appropriateness, whether an explanation is pitched at the right level, whether a summary captures what actually matters. Qualitative evaluation judges these with human or model-assisted judgment against explicit criteria, rather than leaving 'is it good' as an unstructured impression. A rubric makes this structured: a defined set of criteria (relevance, completeness, tone, groundedness) each with a scoring scale (e.g. 1–5) and concrete guidance for what each score level means, so different evaluators — or the same evaluator on different days — apply the same standard.

public sealed record RubricScore(
    int Relevance,      // 1-5: does it address the actual question?
    int Completeness,   // 1-5: does it cover everything asked?
    int Groundedness,   // 1-5: is every claim supported by sources?
    string Notes);

// LLM-as-judge applying the rubric consistently across many outputs --
// scalable where a human reviewing every single output isn't practical.
async Task<RubricScore> ScoreWithRubricAsync(
    string question, string answer, string sources, CancellationToken ct)
{
    string rubricPrompt = $"""
        Score the answer below on a 1-5 scale for each criterion.
        Relevance: does it directly address the question asked?
        Completeness: does it cover everything the question requires?
        Groundedness: is every claim supported by the provided sources?

        Question: {question}\nAnswer: {answer}\nSources: {sources}

        Respond as: Relevance=<n>, Completeness=<n>, Groundedness=<n>, Notes=<text>
        """;
    var completion = await judgeClient.CompleteChatAsync(
        new UserChatMessage(rubricPrompt), cancellationToken: ct);
    return ParseRubricScore(completion.Value.Content[0].Text);
}

LLM-as-judge — using a model to apply the rubric — makes qualitative evaluation scalable across hundreds of outputs where a human reviewing every single one isn't practical, and tutorial 22's golden dataset is exactly what a rubric-based LLM-judge evaluation runs against as regression testing. The tradeoff: the judge model can share biases or blind spots with the model being judged, so periodic human spot-checks of judge scores (especially when a rubric is new, or scores cluster suspiciously high) remain part of the practice, not an optional extra.

8 Deep Dive 4: Comparing Multiple Outputs

Comparing multiple outputs is the technique behind nearly every improvement decision in this tutorial: is the new prompt better than the old one, is model A better than model B for this task, is this repeated run better than that one? Every comparison needs explicit evaluation criteria stated up front — relevance, groundedness, completeness — so 'better' means something concrete rather than an unstated preference. Scoring each output independently (as the rubric in deep-dive 3 does) works, but pairwise comparison — presenting two candidate outputs side by side and asking which is better, and why — is often more reliable for close calls, because judging 'is this a 4 or a 5' in isolation is harder and less consistent than judging 'is A or B better' directly.

async Task<string> ComparePairwiseAsync(
    string question, string answerA, string answerB, CancellationToken ct)
{
    string comparePrompt = $"""
        Question: {question}
        Answer A: {answerA}
        Answer B: {answerB}

        Which answer better addresses the question, considering relevance,
        groundedness, and completeness? Respond with 'A', 'B', or 'TIE',
        followed by a one-sentence justification.
        """;
    var completion = await judgeClient.CompleteChatAsync(
        new UserChatMessage(comparePrompt), cancellationToken: ct);
    return completion.Value.Content[0].Text;
}

This pairwise approach is also how a real A/B test of prompts works at the level of live traffic, matching tutorial 23's canary rollout practice: a portion of real requests gets the new prompt, the rest the old, and outcomes (evaluation scores, user feedback, downstream metrics) are compared between the two groups rather than just between a handful of manually-reviewed examples. For repeated-run consistency (deep-dive 2) specifically, pairwise or rubric scoring across several runs of the same question is what actually quantifies 'how consistent' rather than leaving it as an impression from reading a few outputs.

9 Ecosystem and Tools

This tutorial adds specific, response-quality-focused techniques on top of tutorial 22's general evaluation infrastructure and tutorial 16's platform tooling — it answers what to measure and how to judge it, while those earlier tutorials provide where that measurement runs and how it's automated at scale.

10 Use Cases

• A RAG support assistant running a groundedness check on every golden-dataset answer before each release, blocking a release if unsupported-claim rate exceeds a threshold.
• A team discovering, via relevance evaluation, that a chatbot frequently answers a related-but-different question, and refining the system prompt to require restating the question first.
• A consistency check revealing a factual-question prompt has high output variance, leading to a temperature reduction and a more explicit constraint in the prompt.
• A prompt refinement cycle: observe hallucination patterns, strengthen the grounding instruction, re-run evaluation, confirm the hallucination rate dropped without a relevance regression.
• A rubric-based qualitative evaluation of tone and completeness for a customer-facing assistant, scored by an LLM judge and spot-checked monthly by a human reviewer.
• A pairwise comparison between two candidate models for a summarization task, run across the golden dataset, informing a model-tiering decision (tutorial 25) with actual quality evidence.
• An A/B test comparing an old and new grounding prompt on live traffic, following tutorial 23's canary practice, before fully committing to the new version.

11 Code Examples

These examples combine relevance scoring, groundedness checking, and pairwise comparison into one evaluation run against a golden dataset.

public sealed record EvalResult(string Question, int Relevance, int Groundedness, string Notes);

async Task<List<EvalResult>> RunQualityEvaluationAsync(
    IReadOnlyList<GoldenCase> goldenSet, IChatService candidate, CancellationToken ct)
{
    var results = new List<EvalResult>();
    foreach (var testCase in goldenSet)
    {
        var answer = await candidate.AskGroundedAsync(testCase.Question, ct);
        var score = await ScoreWithRubricAsync(
            testCase.Question, answer.Text, string.Join("\n", answer.Sources.Select(s => s.Content)), ct);
        results.Add(new EvalResult(testCase.Question, score.Relevance, score.Groundedness, score.Notes));
    }
    return results;
}

var lowScoring = results.Where(r => r.Relevance < 3 || r.Groundedness < 3).ToList();

logger.LogWarning("{Count} of {Total} evaluated answers scored below threshold and need human review",
    lowScoring.Count, results.Count);

foreach (var flagged in lowScoring)
    logger.LogWarning("Flagged: '{Question}' -- Relevance={Relevance} Groundedness={Groundedness} Notes={Notes}",
        flagged.Question, flagged.Relevance, flagged.Groundedness, flagged.Notes);

int newWins = 0, oldWins = 0, ties = 0;
foreach (var testCase in goldenSet)
{
    var oldAnswer = await oldPromptService.AskAsync(testCase.Question, ct);
    var newAnswer = await newPromptService.AskAsync(testCase.Question, ct);
    string verdict = await ComparePairwiseAsync(testCase.Question, oldAnswer, newAnswer, ct);

    if (verdict.StartsWith("B")) newWins++;        // B = new answer in this ordering
    else if (verdict.StartsWith("A")) oldWins++;
    else ties++;
}
logger.LogInformation("New prompt: {NewWins} wins, {OldWins} losses, {Ties} ties out of {Total}",
    newWins, oldWins, ties, goldenSet.Count);

12 Step by Step: Evaluating and Refining a RAG Assistant's Responses

This walkthrough runs a full evaluate-refine-compare cycle against the tutorial-15 RAG assistant, using every technique from this tutorial.

1. Take 10-15 real or representative questions for your RAG assistant (a subset of, or the full, tutorial-22 golden dataset) and generate answers with the current system prompt.
2. Manually review each answer for relevance: does it address the actual question, independent of correctness? Note any that answer an adjacent-but-wrong question.
3. Run a groundedness check (deep-dive 1's pattern) on each answer against its retrieved sources, flagging any unsupported or contradicted claims.
4. Run a self-consistency check (deep-dive 2) on 2-3 of the factual questions, generating 3-5 repeated answers each and comparing them for compatible meaning.
5. Identify the single most common failure pattern across all three checks — likely candidates: answering an adjacent question, unsupported claims, or high variance on factual questions.
6. Make ONE targeted prompt refinement addressing that specific pattern (deep-dive 2's discipline — one change at a time).
7. Re-run the same 10-15 questions with the refined prompt and repeat the relevance, groundedness, and consistency checks.
8. Run a pairwise comparison (Example 3) between the old and new prompt's answers across the same question set, and compute the aggregate win rate.
9. Build a simple rubric (deep-dive 3) scoring relevance, completeness, and groundedness 1-5, and apply it via LLM-as-judge to the refined prompt's answers as a baseline for future comparisons.
10. Document the finding: what failure pattern was observed, what change was made, and what the pairwise win rate showed — this is the artifact that justifies (or rules out) shipping the refinement.

13 Limitations and Caveats

• LLM-as-judge is a scalable heuristic, not ground truth — it can share blind spots or biases with the model it's judging, and periodic human spot-checks remain necessary, especially for a newly-built rubric.
• Relevance and groundedness are related but genuinely independent; a system optimized for one metric alone can silently regress on the other if only one is tracked.
• Self-consistency checks assume low temperature should produce stable answers on factual questions; for genuinely creative or open-ended tasks, high variance is expected and not a defect.
• Rubric quality determines evaluation quality — a vague rubric ('rate 1-5 for quality') produces inconsistent, low-value scores regardless of how carefully it's applied.
• Pairwise comparison can be sensitive to answer ordering (which answer is presented as 'A' versus 'B') in some models; consider running comparisons in both orders and checking for order bias.
• This tutorial's code examples (rubric parsing, pairwise judging) are illustrative structures; production evaluation pipelines should use more robust parsing and, ideally, established evaluation frameworks rather than ad hoc string parsing of judge output.
• Evaluating response quality takes real time and (for LLM-as-judge) real token cost; scope the frequency and depth of evaluation to the stakes of the feature, not uniformly to the maximum for every change.
• None of these techniques catch every failure mode; they are the practical, learnable core of a much larger evaluation discipline that professional evaluation frameworks and research continue to develop.

14 Best Practices

• Evaluate relevance and groundedness as separate, explicit checks — never assume one implies the other.
• Treat hallucination detection as mandatory for any system that generates factual claims, not just RAG systems specifically.
• Run consistency checks on factual/deterministic-expected tasks, and expect (don't penalize) higher variance on genuinely creative tasks.
• Refine one dimension of a prompt at a time and re-evaluate before making a second change, so cause and effect stay traceable.
• Write specific, checkable rubric criteria rather than a single vague 'quality' score.
• Spot-check LLM-as-judge verdicts against human review periodically, especially for a new or recently-changed rubric.
• Use pairwise comparison for close calls between two candidates; use rubric scoring when you need an absolute, comparable score across many outputs over time.
• Document every evaluation finding and refinement decision — the failure pattern observed, the change made, the measured effect — as the evidence trail for why a prompt looks the way it does.

20 Summary

• Response relevance and groundedness are independent quality dimensions: an answer can be relevant-but-hallucinated or grounded-but-irrelevant, so both need explicit, separate checks.
• Hallucination detection via groundedness checks — decomposing an answer into claims and verifying each against source passages — is how confident-but-wrong output gets caught before users see it.
• Consistency checks reveal output variance, interpreted differently for factual tasks (stability expected) versus creative tasks (variance expected and desirable).
• Prompt refinement based on outputs means changing one targeted thing at a time in response to an observed, recurring failure pattern, then re-evaluating before the next change.
• Qualitative evaluation via specific, checkable rubrics — applied by LLM-as-judge for scale, with periodic human spot-checks for reliability — turns 'is this good' into a structured, repeatable judgment.
• Comparing multiple outputs, especially via pairwise comparison, is how a prompt refinement, model choice, or repeated-run consistency gets validated with evidence rather than assumed from a few manual examples.

You now have the substantive judgment techniques that make tutorial 22's evaluation infrastructure meaningful: what to check about a response's quality, and how to check it in a way that's structured, scalable, and honest about its own limitations. Combined with tutorial 25's operational monitoring, you can now say, with real evidence, not just that a system is fast and affordable, but that it's actually giving good answers — the complete picture a production AI system needs. With every technical, architectural, operational, security, and quality discipline of this course now in place, the final tutorial brings everything together into one complete, real-world application.

21 Next Steps

Next tutorial: Building a Real-World GenAI-Powered .NET Application (building-real-world-ai-dotnet-app). This is the course's capstone: a complete application synthesizing every tutorial's techniques — from tutorial 10's model deployment through this tutorial's evaluation practice — into one cohesive, production-shaped GenAI .NET application.

• Practice: complete the step-by-step evaluate-refine-compare cycle on a RAG or agent system you've built earlier in the course, documenting the failure pattern found and the measured improvement.
• Practice: build a rubric for a system of your own with at least three specific, checkable criteria, and apply it via LLM-as-judge to ten real outputs.
• Practice: run a self-consistency check on both a factual question and a creative task from the same system, and confirm you can correctly interpret the different variance expectations.
• Practice: implement pairwise comparison between two prompt variants and compute an aggregate win rate across a small golden dataset.
• Read: any current guidance on LLM evaluation frameworks, groundedness/faithfulness metrics, and Azure AI Foundry's built-in evaluation capabilities (tutorial 16).