1 Overview: Running an AI System Day to Day

Tutorial 23 established the deployment and monitoring foundation for a GenAI application; tutorial 24 covered securing it. This tutorial goes deeper into the discipline of actually operating one day to day — LLMOps, the operational practices specific to running large language model applications: watching what they cost, keeping that cost under control, deciding what's safe and useful to log, tracking the performance users actually feel, surviving the model provider's rate limits gracefully, and building the dashboards that make all of this visible at a glance.

This is where tutorial 23's telemetry becomes an operational practice rather than a one-time setup: token-usage tracking as the foundation for cost visibility; cost-optimization techniques that reduce spend without gutting quality; logging prompts and responses safely, extending tutorial 24's data-privacy rules to the specific question of what's worth capturing and how; performance monitoring focused on what latency actually means for an AI call; rate limits and throttling as an ongoing operational reality, not just a one-time resilience pattern; and operational dashboards that tie all of it into a daily operating picture.

2 Learning Objectives

• Track token usage per request, feature, and user as the foundation for cost visibility.
• Apply cost-optimization techniques — model tiering, prompt caching, prompt size reduction — without unacceptably harming quality.
• Log prompts and responses safely, balancing debugging value against tutorial 24's data-privacy requirements.
• Monitor performance metrics meaningful for AI calls, including time to first token and p95 latency, not just averages.
• Handle rate limits and throttling as an ongoing operational reality, including HTTP 429 responses and quota management.
• Build operational dashboards that surface token spend, latency, error rate, and quota headroom for daily operations.

3 Prerequisites

• Tutorial 23's monitoring and telemetry foundation: Application Insights, custom metrics, distributed tracing.
• Tutorial 24's data-privacy rules, especially the 'never log bodies by default' principle this tutorial refines rather than overturns.
• Tutorial 13's resilience patterns (retry, backoff, circuit breaker), which this tutorial's rate-limiting section extends operationally.
• Basic familiarity with reading a metrics dashboard and interpreting percentile-based latency figures is helpful.

4 Key Concepts: Six Daily Operational Questions

LLMOps, as a discipline, answers six recurring operational questions that a deployed AI system raises continuously, not just at launch. Each subtopic in this tutorial is one of these questions, and together they form the daily operating loop of running an AI system responsibly and sustainably.

These questions are interconnected: token-usage tracking feeds cost optimization and dashboards; performance monitoring and rate-limit handling both show up on the same dashboards; and logging decisions balance against the same privacy rules from tutorial 24 throughout. Treat this tutorial as building one coherent operational practice, not six separate checklists.

5 Deep Dive 1: Token-Usage Tracking and Cost-Optimization Techniques

Token-usage tracking is the foundation everything else in this tutorial builds on: recording prompt and completion token counts for every call, tagged by feature, endpoint, and ideally user or tenant, so spend can be broken down meaningfully rather than seen only as one aggregate number. The resulting cost per request, tracked per feature, is the key unit-economics figure that turns raw token counts into a number a team can actually reason about and budget against. Tutorial 13's structured logging already captures this per call; the operational discipline is aggregating it into per-feature and per-user views and watching trends over time, not just logging it and moving on.

// Extending tutorial 13's telemetry with a feature tag for breakdown by area.
_telemetryClient.GetMetric("AI.TokensUsed", "Feature").TrackValue(
    completion.Usage.InputTokenCount + completion.Usage.OutputTokenCount,
    metricNamespace: "SupportAssistant");

_logger.LogInformation(
    "AI call {Feature} used {PromptTokens}+{CompletionTokens} tokens for user {UserId}",
    "SupportAssistant", completion.Usage.InputTokenCount,
    completion.Usage.OutputTokenCount, userId);

Cost-optimization techniques act on what token tracking reveals. Model tiering routes easier tasks (classification, simple extraction) to a cheaper, faster model and reserves an expensive flagship model for tasks that genuinely need its capability — a classification task rarely needs the same model as an open-ended reasoning task. Prompt caching reuses a previously computed result for an identical or near-identical prompt rather than paying for a fresh call, valuable for common questions or repeated sub-tasks. Reducing prompt size — trimming unnecessary context, summarizing long conversation history instead of resending it in full, tightening system prompts — directly cuts the input-token side of every call. And a token budget, whether advisory or enforced, gives a feature or team a concrete target to design against rather than an open-ended 'use what you need.'

6 Deep Dive 2: Logging Prompts and Responses Safely

Tutorial 24 established 'never log prompt or response bodies by default.' Prompt and response logging is the operational refinement of that rule: sometimes you genuinely need body content for debugging a specific incident, and the question is how to capture it without recreating the exposure risk the default rule exists to prevent. The answer is layered: keep metadata-only logging as the default for all traffic; provide an explicit, off-by-default debug flag that captures full bodies only when deliberately enabled (for a specific user, a specific time window, or a specific investigation); apply redaction to anything that is captured, stripping or masking obvious PII patterns even in debug captures; and set a short, deliberate retention period for any body-level logs, since the exposure risk compounds the longer sensitive content sits in a log store.

// Default: metadata only, always on.
_logger.LogInformation("AI call {Feature} finished in {ElapsedMs} ms", feature, elapsed);

// Debug capture: off by default, enabled deliberately and narrowly (e.g. per
// investigation, with an expiry), and redacted before it's ever written.
if (debugLoggingOptions.IsEnabledFor(userId, feature))
{
    string redactedPrompt = Redactor.Redact(promptText);
    string redactedResponse = Redactor.Redact(responseText);
    _logger.LogDebug("DEBUG CAPTURE {Feature} prompt={Prompt} response={Response}",
        feature, redactedPrompt, redactedResponse);
}

Access control on any body-level log store should be stricter than for metadata logs — fewer people with access, an audit trail of who viewed what, and ideally the capture disabled again automatically after the investigation window closes rather than left on indefinitely. The operational discipline is treating body-level logging as a deliberate, temporary, access-controlled exception process, not a standing feature anyone can flip on casually.

7 Deep Dive 3: Performance Monitoring

Performance monitoring for an AI call needs metrics beyond a simple average response time, because averages hide exactly the experience most users have. p95 latency — the response time below which 95% of requests complete — is the standard way to describe typical worst-case experience while ignoring rare extreme outliers that would otherwise distort an average. For a streaming response (tutorial 14), time to first token matters more than total completion time for perceived responsiveness: a user sees the first word appear quickly even if the full answer takes several more seconds to finish streaming, and that first-token delay is what actually shapes their sense of whether the system is responsive.

Break performance metrics down by the same dimensions as cost — per feature, per model deployment — since a slow RAG retrieval step and a slow model generation step have completely different fixes, and aggregating them into one 'AI call latency' number obscures which one to act on. Set explicit SLOs (service level objectives) for the metrics that matter most — 'p95 time to first token under 2 seconds for the chat assistant' — so performance has a concrete target rather than an implicit 'seems fine' standard.

8 Deep Dive 4: Rate Limits, Throttling, and Operational Dashboards

Rate limits and throttling are an ongoing operational reality, not a one-time resilience pattern implemented once and forgotten. A model deployment's quota — its allowed requests or tokens per time window — is a hard ceiling; approaching it causes throttling, and exceeding it returns HTTP 429. Tutorial 13 taught the code-level response (exponential backoff with jitter, a hard retry cap); the operational practice is watching quota utilization continuously so a team requests more quota, redistributes load across deployments, or optimizes usage before hitting the ceiling in production, rather than discovering the limit only when 429s start appearing.

// Beyond retry logic (tutorial 13), track HOW OFTEN throttling occurs --
// a rising trend is an early warning before it becomes a user-facing incident.
catch (ClientResultException ex) when (ex.Status == 429)
{
    _telemetryClient.GetMetric("AI.ThrottledRequests").TrackValue(1);
    _logger.LogWarning("AI call {Feature} throttled (429); retry {Attempt}", feature, attempt);
    // ... existing backoff/retry logic from tutorial 13 ...
}

Operational dashboards tie every prior subtopic into one daily operating picture. A useful dashboard surfaces, at minimum: token spend and trend (per feature, per day); p95 latency and time to first token; error rate and 429/throttling frequency as a leading indicator of quota pressure; and quota headroom — how close current usage sits to the deployment's limit. Alerts on each of these (a cost spike, a latency regression, rising 429 frequency) turn passive dashboards into active operational signals, and anomaly detection can catch deviations from a metric's normal pattern that a fixed threshold might miss.

9 Ecosystem and Tools

This is largely the same tooling introduced in tutorial 23, used with more depth and a specific operational cadence — checking dashboards, reviewing cost trends, and revisiting rate-limit headroom on a recurring schedule rather than as a one-time setup task.

10 Use Cases

• A support assistant routing simple FAQ-style questions to a cheap model and complex multi-turn troubleshooting to a flagship model, tracked and validated by feature-level token metrics.
• A team enforcing a per-feature token budget during design review, catching a prompt that would have used 3x the intended tokens before it ever shipped.
• An incident investigation where a scoped, time-limited debug-logging flag captures redacted prompt/response bodies for one affected user, then automatically expires.
• A dashboard showing p95 time to first token creeping upward over two weeks, prompting investigation before it becomes a user complaint.
• A team monitoring 429 frequency trending upward days before a growth spike would have caused a hard rate-limit incident, requesting additional quota proactively.
• A monthly cost review comparing token spend per feature against the budget set at design time, feeding back into future feature scoping.
• An anomaly-detection alert firing on an unusual overnight token-spend spike, revealing a runaway retry loop that manual threshold alerts hadn't caught.

11 Code Examples

These examples combine model tiering, quota-aware throttling telemetry, and a dashboard-ready metrics summary into one operational picture.

public sealed class TieredChatService : IChatService
{
    private readonly ChatClient _cheapModel;   // e.g. a smaller, faster deployment
    private readonly ChatClient _flagshipModel;

    public async Task<string> ClassifyAsync(string text, CancellationToken ct)
    {
        // Classification doesn't need the flagship model.
        var completion = await _cheapModel.CompleteChatAsync(
            new UserChatMessage($"Classify this ticket: {text}"), cancellationToken: ct);
        return completion.Value.Content[0].Text;
    }

    public async Task<string> TroubleshootAsync(string conversation, CancellationToken ct)
    {
        // Multi-step reasoning over a full conversation warrants the flagship model.
        var completion = await _flagshipModel.CompleteChatAsync(
            new UserChatMessage(conversation), cancellationToken: ct);
        return completion.Value.Content[0].Text;
    }
}

var stopwatch = Stopwatch.StartNew();
bool firstTokenRecorded = false;

await foreach (var update in chat.CompleteChatStreamingAsync(messages, cancellationToken: ct))
{
    if (!firstTokenRecorded && update.ContentUpdate.Count > 0)
    {
        _telemetryClient.GetMetric("AI.TimeToFirstTokenMs").TrackValue(stopwatch.ElapsedMilliseconds);
        firstTokenRecorded = true;
    }
    // ... stream the fragment to the client (tutorial 14) ...
}

_telemetryClient.GetMetric("AI.TotalCompletionMs").TrackValue(stopwatch.ElapsedMilliseconds);

// Conceptual Application Insights query (Kusto/KQL) an operational dashboard
// tile might run -- illustrating what the tracked telemetry from this tutorial enables.
// customMetrics
// | where name in ('AI.TokensUsed', 'AI.TimeToFirstTokenMs', 'AI.ThrottledRequests')
// | summarize
//     TotalTokens = sumif(value, name == 'AI.TokensUsed'),
//     P95Ttft = percentileif(value, 95, name == 'AI.TimeToFirstTokenMs'),
//     ThrottleCount = countif(name == 'AI.ThrottledRequests')
//   by bin(timestamp, 1h), tostring(customDimensions.Feature)

12 Step by Step: Building an LLMOps Dashboard for a Deployed Assistant

This walkthrough builds the full operational picture for the tutorial-23 deployed assistant: token tracking, a cost optimization, safe debug logging, performance metrics, and a dashboard tying it together.

1. Add feature-tagged token-usage telemetry (Example/deep-dive 1) to every AI call path in the deployed assistant, distinguishing at least two features if the app has more than one.
2. Identify one genuine cost-optimization opportunity — a task currently using the flagship model that a cheaper model could handle — and implement model tiering (Example 1), validating quality against tutorial 22's golden dataset before and after.
3. Add the scoped debug-logging mechanism from deep-dive 2: off by default, enabled per-user/per-window, with redaction applied, and confirm it does not capture anything when disabled.
4. Instrument time to first token separately from total completion time for any streaming endpoint (Example 2), and confirm both appear as distinct metrics.
5. Add throttling telemetry (deep-dive 4) that records every 429 response, distinct from the retry logic itself, so throttling frequency is visible as a trend.
6. Build an operational dashboard with four tiles: token spend by feature, p95 time to first token, error/429 rate, and quota headroom (estimated from request volume against known limits if the exact quota isn't queryable directly).
7. Set at least one alert (a token-spend threshold, or a 429-frequency threshold) and confirm it fires correctly with a simulated breach.
8. Assign explicit ownership: document who checks this dashboard and how often, and where alerts route.
9. Run the dashboard for a defined period (even a few days) and identify one actionable finding — a cost driver, a latency pattern, a quota trend — purely from what the dashboard shows.
10. Reflect on the six subtopics as one coherent operational loop: token tracking fed cost optimization, logging supported debugging, performance and rate-limit metrics fed the same dashboard, and the dashboard is what makes the whole loop actionable day to day.

13 Limitations and Caveats

• Cost-optimization techniques trade quality or freshness risk for savings; every optimization should be validated against tutorial 22's evaluation practice, not adopted on cost grounds alone.
• Model tiering requires a reliable way to classify task difficulty in the first place; a poor tiering decision can route a genuinely hard task to a model that underperforms on it.
• Prompt caching is only valid for requests that are truly identical or near-identical in the ways that matter; caching a response for a subtly different question can serve a wrong or stale answer.
• Debug logging, even scoped and redacted, still carries residual risk; redaction patterns can miss unusual PII formats, so scope, access control, and short retention remain essential even with redaction in place.
• p95 latency and time to first token are better than averages but still summary statistics; a dashboard should retain the ability to drill into individual slow requests, not stop at the percentile.
• Quota and rate-limit behavior can change with provider updates; monitoring must be revisited periodically rather than configured once against assumptions that may not hold indefinitely.
• Dashboards and alerts require genuine ownership and response capacity; a well-built dashboard with nobody assigned to act on it provides limited real operational value.
• This tutorial's KQL/query examples are illustrative of intent; exact query syntax and available fields depend on the specific telemetry schema and platform version in use.

14 Best Practices

• Track token usage per feature and, where feasible, per user from day one — retrofitting this breakdown after cost becomes a problem is much harder than building it in from the start.
• Validate every cost optimization against an evaluation suite (tutorial 22) before and after — cost savings that regress quality are a false economy.
• Keep prompt/response body logging off by default, scoped and redacted when enabled, with automatic expiry rather than a standing always-on debug mode.
• Track p95 latency and time to first token separately from averages and from each other, especially for any streaming feature.
• Monitor quota utilization and 429 frequency as leading indicators, requesting more quota or optimizing usage before limits cause user-visible incidents.
• Build dashboards around the questions a team actually needs answered daily — cost, performance, quota headroom — not every metric that's technically available.
• Assign explicit ownership for dashboards and alerts; an unowned alert is functionally the same as no alert.
• Revisit cost, performance, and rate-limit assumptions periodically as usage patterns, models, and provider limits evolve over time.

20 Summary

• Token-usage tracking, broken down by feature and user, is the foundation cost-optimization, dashboards, and budget planning all build on.
• Cost-optimization techniques — model tiering, prompt caching, prompt size reduction, token budgets — trade quality or freshness risk for savings and must be validated against tutorial 22's evaluation practice, not adopted on cost grounds alone.
• Logging prompts and responses safely means metadata-only by default, with a scoped, redacted, short-retention debug mechanism as the deliberate exception, extending tutorial 24's privacy rules operationally.
• Performance monitoring for AI calls needs p95 latency and time to first token specifically, not just averages, since both reveal what averages hide or miss entirely for streaming responses.
• Rate limits and throttling need ongoing trend monitoring (429 frequency, quota headroom) as a leading indicator, distinct from and complementary to tutorial 13's per-request retry logic.
• Operational dashboards tie every subtopic together into one daily operating picture — but only provide real value when someone owns them and acts on what they show.

You now have the day-to-day operational discipline that keeps a deployed AI system economical, safe to debug, genuinely fast (not just fast-on-average), and ahead of its own rate limits — the practice that turns tutorial 23's deployed system into one a team can actually run sustainably over time. With cost, logging, performance, and capacity all under active management, the course turns next to the deeper question these dashboards can only partially answer: how do you actually evaluate whether an AI system's responses are good, not just fast and affordable?

21 Next Steps

Next tutorial: AI Evaluation and Response Quality (ai-evaluation-response-quality). Having covered the operational metrics — cost, latency, reliability — that dashboards can measure automatically, the next tutorial goes deeper into evaluating something dashboards can't measure by themselves: whether an AI system's actual responses are good, building further on tutorial 22's evaluation foundation.

• Practice: add feature-tagged token telemetry to an existing project and identify one genuine model-tiering opportunity using the resulting breakdown.
• Practice: implement the scoped, redacted debug-logging mechanism and verify it captures nothing when disabled and only redacted content when enabled.
• Practice: instrument time to first token separately from total completion time on a streaming endpoint and compare the two numbers on real traffic.
• Practice: build the four-tile operational dashboard from the walkthrough and identify one real, actionable finding purely from what it shows.
• Read: the official documentation for 'Application Insights custom metrics', 'Azure Monitor alerts and anomaly detection', and any current guidance on Azure OpenAI quota management and rate limits.