1 Overview: From Building Blocks to Working Systems

Tutorials 17 through 21 gave you every building block a multi-agent system needs: agents in two frameworks, planners, orchestration patterns, agent-first architecture, governance, and MCP for shareable tools. This tutorial assembles them into complete, production-oriented systems. Rather than introducing new mechanisms, it names the recurring shapes those mechanisms combine into — multi-agent design patterns — and addresses the questions that only arise once a system is real: how do you orchestrate a whole workflow end to end, how does it scale under load, and how do you know it actually works?

This advanced tutorial covers multi-agent design patterns (pipeline, specialist, critic, hierarchical); workflow orchestration for coordinating a multi-step process reliably; scaling multi-agent systems as load grows; evaluation and testing of agent systems, since 'it worked in the demo' is not evidence of production readiness; and two worked syntheses — a multi-agent document-processing example and a collaborative analysis pipeline — that combine every subtopic into runnable designs.

2 Learning Objectives

• Recognize and apply the four common multi-agent design patterns: pipeline, specialist, critic, and hierarchical.
• Design workflow orchestration for a multi-agent process, including state tracking and failure handling.
• Identify scaling considerations for multi-agent systems and address bottlenecks with horizontal scaling.
• Build an evaluation and testing strategy for agent systems using golden datasets, metrics, and regression testing.
• Design a multi-agent document-processing system and a collaborative analysis pipeline end to end.

3 Prerequisites

• Tutorials 17–19: agents, planners, and orchestration in Semantic Kernel and AutoGen.
• Tutorial 20: agent-first architecture, communication models, and governance (capability scoping, policy engines, audit trails).
• Tutorial 21: MCP servers, for tool-sharing across the agents in a multi-agent system.
• General familiarity with testing practices (unit/integration tests) and basic queueing/scaling concepts from traditional distributed systems.

4 Key Concepts: Naming the Recurring Shapes

A multi-agent system is several specialized agents coordinating — via a communication model (tutorial 20) and an orchestration pattern (tutorial 18) — toward work no single agent does as well alone. Across many real systems, the same handful of design patterns recur: pipeline (fixed sequence), specialist (route to the right expert), critic (produce then review), and hierarchical (a supervisor delegates). Recognizing which pattern a problem fits is faster than re-deriving an orchestration from scratch every time.

The two worked examples — document processing and collaborative analysis — are not new topics; they are the previous four concepts applied to concrete, common enterprise scenarios, showing how a design pattern, an orchestration, scaling considerations, and an evaluation strategy come together in one real system.

5 Deep Dive 1: Multi-Agent Design Patterns

Four patterns cover most multi-agent systems in practice, each a named combination of the orchestration patterns and communication models from tutorials 18 and 20. The pipeline pattern runs agents in a fixed sequence, each consuming the previous one's output — sequential orchestration with direct handoff, well suited when the stages are known and always run in the same order (research, then draft, then edit). The specialist pattern routes each incoming request to whichever specialized agent best matches it — handoff orchestration, well suited when requests vary in kind and each kind has a clear expert (billing questions to a billing agent, technical questions to a technical agent).

The critic pattern has one agent produce work and a second agent review and critique it before acceptance — group chat orchestration between exactly two roles, well suited whenever quality control benefits from an independent second look (a Developer/Reviewer pair from tutorial 19). The hierarchical pattern has a supervising agent decompose a goal into sub-tasks and delegate them to subordinate agents, then combine their results — closest to a planner (tutorial 18) driving several specialist agents, well suited for genuinely complex goals that don't map to one fixed sequence.

6 Deep Dive 2: Workflow Orchestration

Workflow orchestration is the practical machinery that runs a multi-step, possibly multi-agent process reliably — tracking which step is active, what state has accumulated, and what happens when a step fails. Where tutorial 18's termination strategies decided when an agent conversation should stop, workflow orchestration decides how an entire business process (of which agent turns may be only one part) moves from start to finish, including steps that aren't agent calls at all — saving state, sending a notification, waiting for a human approval.

public enum WorkflowStatus { Pending, Extracting, Reviewing, Approved, Failed }
public sealed record WorkflowState(string Id, WorkflowStatus Status, string? Result, string? Error);

async Task RunDocumentWorkflowAsync(string documentId, IWorkflowStore store)
{
    var state = new WorkflowState(documentId, WorkflowStatus.Extracting, null, null);
    await store.SaveAsync(state);   // checkpoint before risky work -- durable execution

    try
    {
        string extracted = await extractionAgent.InvokeAsync(documentId);
        state = state with { Status = WorkflowStatus.Reviewing, Result = extracted };
        await store.SaveAsync(state);   // checkpoint again

        string reviewed = await reviewAgent.InvokeAsync(extracted);
        state = state with { Status = WorkflowStatus.Approved, Result = reviewed };
    }
    catch (Exception ex)
    {
        state = state with { Status = WorkflowStatus.Failed, Error = ex.Message };
    }
    await store.SaveAsync(state);
}

Checkpointing state after each meaningful step, as above, is what gives durable execution: if the process crashes after extraction but before review, a restart can resume from 'Reviewing' with the extracted text already in hand, rather than re-running (and re-billing) the extraction step. This matters more for multi-agent workflows than single-call systems because they take longer, involve more steps, and each step is a real cost (a model call, a tool invocation) worth not repeating unnecessarily.

7 Deep Dive 3: Scaling Multi-Agent Systems

A multi-agent system that works for ten requests a day faces new questions at ten thousand: which agent is the bottleneck, and how much can horizontal scaling — running more instances of an agent in parallel — help. Because each agent typically wraps a model call, the honest first bottleneck is almost always the model deployment's own rate limit (tutorial 11's tokens-per-minute quota), not the agent code itself; scaling agent instances beyond what the underlying model deployment can serve just produces more requests waiting in a queue, not more completed work.

Beyond model-level limits, throughput improves through familiar distributed-systems practice applied to agents: run stateless agent instances behind a queue so bursts don't overwhelm downstream systems; batch independent work where the pattern allows it (concurrent orchestration, tutorial 18); cache results for repeated or near-duplicate requests (especially useful for retrieval steps); and apply tutorial 13's resilience patterns (retry, circuit breaker) so one overloaded dependency degrades gracefully instead of cascading. None of this is agent-specific — it's the same scaling discipline any high-throughput service needs, applied to a system whose 'service calls' happen to be model invocations.

8 Deep Dive 4: Evaluation and Testing of Agent Systems

Tutorial 18 taught testing an individual workflow's termination behavior (does it converge, does the backstop fire); evaluation and testing of agent systems generalizes this to the whole system's ongoing correctness. The foundation is a golden dataset: a curated set of representative inputs with known-correct expected outputs, built from real or realistic cases, covering both the common path and known edge cases. Against it, define evaluation metrics appropriate to the system — accuracy of extracted data, groundedness of a generated answer (tutorial 15), whether a routing decision picked the right specialist, plus operational metrics like latency and cost per request.

public sealed record GoldenCase(string Input, string ExpectedCategory);

async Task<EvaluationReport> EvaluateAsync(
    IReadOnlyList<GoldenCase> goldenSet, ISpecialistRouter router)
{
    int correct = 0;
    var failures = new List<string>();

    foreach (var testCase in goldenSet)
    {
        string actualCategory = await router.RouteAsync(testCase.Input);
        if (actualCategory == testCase.ExpectedCategory) correct++;
        else failures.Add($"'{testCase.Input}' routed to {actualCategory}, expected {testCase.ExpectedCategory}");
    }

    double accuracy = (double)correct / goldenSet.Count;
    return new EvaluationReport(accuracy, failures);
}

Run this evaluation as regression testing — automatically, on every change to prompts, agent instructions, tools, or the underlying model — because agent behavior can shift in subtle ways that ordinary unit tests over deterministic code would never catch. Tutorial 18's two-sided termination test (does the intended path converge, does the backstop actually fire) generalizes here too: a complete agent-system test suite includes the happy path, known edge cases, adversarial inputs (tutorial 20's manipulation concern), and the failure/backstop paths (what happens when a tool errors, when a model call times out).

9 Ecosystem and Tools

Every row in this table is something tutorials 12 through 21 already introduced individually. This tutorial's contribution is showing how they combine into named, recognizable system shapes and closing the two gaps (scale, evaluation) that only appear once a system moves from demo to production.

10 Use Cases

• Contract review pipeline: extraction agent, then a clause-classification agent, then a risk-summary agent — a pipeline pattern with workflow orchestration tracking each document's progress.
• Customer support routing: a specialist pattern with a router agent directing billing, technical, and account questions to dedicated expert agents.
• Content generation with quality gates: a critic pattern where a writer agent drafts and a reviewer agent enforces house style before publication.
• Complex research tasks: a hierarchical pattern where a supervisor agent decomposes 'evaluate these three vendors' into sub-tasks delegated to specialist research agents, then synthesizes their findings.
• Invoice processing at scale: a multi-agent document-processing system handling thousands of invoices a day, requiring horizontal scaling and durable workflow execution to survive restarts mid-batch.
• Incident investigation: a collaborative analysis pipeline where log, metrics, and configuration-review agents each examine the same incident independently, synthesized into one root-cause report.
• Any agent system heading to production: evaluation and testing against a golden dataset before launch, and as regression testing on every subsequent change.

11 Code Examples: A Multi-Agent Document-Processing System

This example works through the tutorial's first synthesis subtopic: a document-processing pipeline combining the pipeline pattern, workflow orchestration, and governance from tutorial 20.

public sealed record DocumentResult(string ExtractedText, string Category, bool IsValid, string? ValidationError);

async Task<DocumentResult> ProcessDocumentAsync(string documentId, IWorkflowStore store)
{
    // Step 1: extraction agent -- pulls text/fields from the raw document.
    string extracted = await extractionAgent.InvokeAsync(documentId);
    await store.CheckpointAsync(documentId, "Extracted", extracted);

    // Step 2: classification agent -- decides document type (invoice, contract, etc.).
    string category = await classificationAgent.InvokeAsync(extracted);
    await store.CheckpointAsync(documentId, "Classified", category);

    // Step 3: validation agent -- checks required fields are present for this category.
    (bool isValid, string? error) = await validationAgent.ValidateAsync(extracted, category);
    await store.CheckpointAsync(documentId, isValid ? "Validated" : "ValidationFailed", error);

    return new DocumentResult(extracted, category, isValid, error);
}

// Following tutorial 20's pattern: consequential actions get a policy check.
if (result.IsValid && result.Category == "invoice")
{
    var decision = policyEngine.Evaluate(
        new AgentAction("InvoiceAgent", "schedulePayment", ExtractAmount(result), documentId));
    if (decision.RequiresHumanApproval)
        await humanApproval.RequestAsync("Invoice payment needs approval", result);
    else
        await SchedulePaymentAsync(result);
}

// A queue decouples document arrival from processing capacity --
// N worker instances pull from the same queue, scaling horizontally.
await documentQueue.EnqueueAsync(new DocumentJob(documentId));

// Each worker instance runs the same loop; add more instances to raise throughput,
// up to whatever the model deployment's rate limit allows.
await foreach (DocumentJob job in documentQueue.ConsumeAsync(ct))
{
    await ProcessDocumentAsync(job.DocumentId, store);
}

12 Step by Step: A Collaborative Analysis Pipeline

This walkthrough builds the tutorial's second synthesis subtopic: a collaborative analysis pipeline where several specialist agents examine the same input independently and a synthesis step combines their findings — extending tutorial 20's blackboard example into a complete, evaluated system.

1. Define the scenario: analyzing a customer-support incident using three specialist agents — a log-analysis agent, a metrics agent, and a customer-history agent — each examining the same incident id independently.
2. Implement each specialist as an agent (Semantic Kernel or AutoGen) with a narrow tool set: the log agent queries log storage, the metrics agent queries a metrics API, the history agent queries customer records — via MCP servers if these are shared enterprise systems (tutorial 21).
3. Run the three specialists concurrently (concurrent orchestration, tutorial 18) since their findings are independent and don't depend on each other's output.
4. Implement a synthesis step: a dedicated agent (or a single well-grounded prompt function) that receives all three findings and produces one coherent root-cause hypothesis, citing which specialist findings support it.
5. Wrap the whole thing in workflow orchestration: checkpoint after the specialists complete and after synthesis, so a failure doesn't require re-running specialists that already succeeded.
6. Add evaluation: build a golden dataset of past incidents with known root causes, run the pipeline against each, and measure how often the synthesized hypothesis matches the known cause.
7. Add scaling: since the three specialists run concurrently per incident, and multiple incidents may be analyzed at once, confirm each specialist's backing system (log storage, metrics API) can handle the resulting concurrent load, not just the orchestration code.
8. Test the two-sided termination-adjacent case: what happens when one specialist's tool times out or errors — confirm the synthesis step still produces a (clearly caveated) result from the remaining findings rather than failing the whole pipeline.
9. Run the evaluation suite as a regression test after any change to a specialist's instructions or tools, and track accuracy over time.
10. Reflect on pattern identity: this system is a hybrid — concurrent specialists (not quite the 'critic' pattern) feeding a hierarchical-style synthesis step — a reminder that real systems often combine named patterns rather than matching one exactly.

13 Limitations and Caveats

• The four design patterns are common shapes, not an exhaustive taxonomy — real systems often combine or deviate from them, as the collaborative analysis walkthrough's hybrid shape shows.
• Workflow orchestration and durable execution add real complexity (a state store, checkpoint logic) that a short-lived, low-stakes workflow may not need — apply it where restart cost or duration genuinely justifies it.
• Scaling agent instances is capped by the underlying model deployment's rate limits; teams that scale infrastructure without also addressing model quota will see queuing, not more completed work.
• Multi-agent systems can be substantially more expensive per request than a single model call (a hierarchical or critic system may issue several calls per logical request) — throughput and cost planning must account for this multiplier.
• A golden dataset is only as good as its coverage; a dataset that only reflects the common path will not catch regressions in edge-case or adversarial handling.
• Evaluation metrics like 'accuracy' can be harder to define precisely for open-ended agent outputs than for classification tasks — some systems need human-graded evaluation or a model-as-judge approach, each with its own caveats.
• This tutorial's code examples (workflow state stores, evaluation harnesses, queues) are illustrative shapes, not endorsements of a specific product; real systems typically adopt an existing workflow engine, queue, or evaluation platform rather than hand-rolling these from scratch.
• Collaborative analysis and document-processing examples here are illustrative scenarios; a real deployment needs the full governance stack from tutorial 20 (capability scoping, authorization, audit) that space here only partially shows.

14 Best Practices

• Name the design pattern for any multi-agent system you build (pipeline, specialist, critic, hierarchical, or an explicit hybrid) — it clarifies the orchestration and communication model choices immediately.
• Checkpoint workflow state after every meaningful step in a long-running or multi-agent process, so failures cost minutes of re-work, not hours.
• Diagnose the real bottleneck (usually the model deployment's rate limit or a downstream tool) before scaling agent instances horizontally — more instances against the same constraint just queues more work.
• Build a golden dataset before launch, not after an incident, and treat evaluation as regression testing that runs on every change to prompts, tools, or instructions.
• Test both the intended path and the failure/backstop paths for every multi-agent workflow, extending tutorial 18's two-sided termination testing to the whole system.
• Attach governance (capability scoping, policy checks, audit logging) at workflow steps, not as an afterthought bolted onto the finished system.
• Log which agent or specialist contributed to a final synthesized or routed outcome, so evaluation failures can be traced to a specific component.
• Expect and design for hybrid patterns in real systems rather than forcing a design into one named pattern that doesn't quite fit.

20 Summary

• Four multi-agent design patterns — pipeline, specialist, critic, hierarchical — name recurring combinations of tutorials 18/20's orchestration patterns and communication models; real systems often combine them into deliberate hybrids.
• Workflow orchestration coordinates an entire multi-step process reliably, including non-agent steps, with durable execution (checkpointing) so failures don't force costly re-work.
• Scaling multi-agent systems means diagnosing the real bottleneck — usually the model deployment's rate limit — before adding agent instances horizontally, since scaling past the true constraint just adds queuing.
• Evaluation and testing of agent systems requires a golden dataset, meaningful metrics, and regression testing on every prompt/instruction/model change, because agent behavior shifts in ways deterministic-code tests can't catch.
• The document-processing pipeline and collaborative analysis pipeline syntheses show every prior concept — design pattern, orchestration, governance, scaling, evaluation — working together in one concrete system.
• This tutorial introduced no new mechanisms: it named recurring shapes and closed the scale and evaluation gaps that only appear once a system built from tutorials 17–21's building blocks needs to run in production.

You now have a complete, production-oriented view of multi-agent systems: recognizable design patterns, reliable workflow orchestration, a scaling discipline that finds the real bottleneck instead of guessing, and an evaluation practice that provides evidence instead of impressions. Every piece traces back to tutorials 12 through 21 — nothing here was new mechanism, only synthesis and the operational maturity production demands. With a working, evaluated, scalable multi-agent system in hand, the course now turns to the final step: actually deploying these GenAI applications reliably.

21 Next Steps

Next tutorial: Deploying GenAI Applications (deploying-genai-applications). You've now built, architected, governed, and evaluated multi-agent systems — the next tutorial addresses getting them safely into production: deployment strategies, environment configuration, and the operational practices that keep a GenAI application running reliably once it's live.

• Practice: identify which of the four design patterns (or hybrid) best fits a real system you're building or have built, and justify the choice in writing.
• Practice: add checkpointing to an existing multi-step agent workflow and simulate a crash between steps, confirming a restart resumes correctly without re-running completed work.
• Practice: build a golden dataset of 20-30 cases for a system you've built in this course, run an evaluation, and identify at least one edge case your manual testing had missed.
• Practice: complete the collaborative analysis pipeline walkthrough, including the partial-failure test (one specialist times out), and confirm the synthesis step degrades gracefully.
• Read: general distributed-systems material on horizontal scaling and bottleneck analysis, plus any current guidance on evaluation frameworks for LLM-based and agentic systems.