Chapter 6: AgentOps: Evaluation, Autoraters, Deployment, Tracing & Cost Optimization

Subsections Covered: 4.1, 4.2 (7 Considerations · 14 Custom Visual Diagrams)

Chapter Focus: Testing agents with golden datasets, trajectory metrics, and LLM-as-a-Judge autoraters; choosing between Agent Runtime, Cloud Run, and GKE; and troubleshooting latency, loops, and costs with Cloud Trace and caching.

Section 4 covers AgentOps—how you test, deploy, monitor, troubleshoot, and optimize agents in production. Subsection 4.1 focuses on Evaluating and Testing Agents (golden datasets, trajectory metrics, ADK evalset, and Autoraters), while Subsection 4.2 focuses on Deploying, Monitoring, and Troubleshooting (choosing between Agent Runtime, Cloud Run, and GKE, tracing latency bottlenecks, breaking infinite loops, and cutting token costs).

Subsection 4.1: Evaluating and Testing Agents in Development and Production

Section 4: Testing, Deploying, Monitoring, and Scaling Production Agents

The Big Picture

You cannot test an AI agent with simple assert response == 'Hello' unit tests. An agent might phrase its final answer nicely while calling the wrong tools, skipping a mandatory security check, or hallucinating facts not found in the retrieved documents. Subsection 4.1 teaches how to build Golden Datasets, measure Tool Trajectory and the RAG Triad, and automate grading with ADK evalset and Autoraters (LLM-as-a-Judge).

Consideration 4.1a: Creating Test Sets for Agent Evaluation (Golden Data, Prompts, and Edge Cases)

Anatomy of an Agent Evaluation Golden Dataset — A complete agent Golden Dataset includes happy-path prompts, adversarial edge cases, expected tool trajectories, and reference answers.

A Golden Dataset is your trusted benchmark for measuring whether a prompt change, model upgrade, or new tool makes your agent better or worse. Unlike traditional ML datasets that only have (input, label), an Agent Golden Dataset must capture the full interaction journey.

What Belongs in an Agent Golden Dataset

Diverse Scenario Coverage:
Happy-Path Prompts: Typical single-turn and multi-turn user requests.
Multi-Turn Context Drift Cases: Conversations where the user changes their mind mid-chat ("Actually, change the flight from Tuesday to Thursday") to verify the agent updates state properly.
Tool-Calling Edge Cases: Missing parameters, ambiguous dates, or API error responses (503 Service Unavailable) to verify the agent asks clarifying questions or retries gracefully.
Adversarial & Red-Team Prompts: Prompt injections, out-of-scope questions, and harmful requests where the expected behavior is a safe refusal.
Expected Tool Trajectory + Reference Answer: Each golden test case records both the expected sequence of tool calls and arguments (expected_trajectory) and the ground-truth reference facts.

Production Feedback Loop into the Golden Dataset — Continuously sampling failed or thumbs-down production traces, de-identifying PII, and adding human-verified cases to the Golden Dataset.

Building and Updating the Dataset (`Synthetic + HITL + Production Drift`)

Bootstrapping with Synthetic Data + Human Review: When launching a new agent with zero historical logs, use an LLM to generate synthetic test scenarios across your personas and edge cases, then have domain experts review and approve (Human-in-the-Loop) the golden pairs before locking them into the benchmark.
Continuous Production Drift Flywheel: In production, automatically sample conversations that triggered user thumbs-down ratings, fallback escalations, or low confidence scores. Run them through Sensitive Data Protection (SDP) to redact PII, have a human annotator verify the correct expected outcome, and append them to your Golden Dataset so the agent never repeats the same mistake twice.

Key Takeaways (4.1a)
Cold-start a golden dataset before launch → Generate synthetic scenarios with an LLM + validate with Human-in-the-Loop (HITL) domain experts.
Keep the golden dataset relevant over time → Continuously feed redacted production failures/drift back into the test set.

Consideration 4.1b: Creating Continuous Evaluation Pipelines to Assess Tool Execution

Syntactic Schema Errors vs. Semantic Tool Argument Errors — Testing both syntactic validity (valid JSON types) and semantic accuracy (picking the right airport code or date from context).

Even if an agent selects the right tool name, its tool call can fail in two very different ways: syntactic errors versus semantic errors. Your continuous evaluation pipeline must test for both.

Syntactic vs. Semantic Tool Evaluation

Syntactic / Schema Validation Errors: The model outputs malformed JSON, omits a required field, or passes a string "twenty" where an integer 20 is required by the OpenAPI/MCP schema.
Semantic Argument Errors: The JSON is 100% valid according to the schema, but the values are wrong for the conversation! For example, the user says "Book a flight from New York to London next Friday," and the agent passes valid strings {"origin": "LON", "destination": "NYC"} (swapping origin and destination) or calculates the wrong Friday date.
Trajectory Ordering Checks: Sometimes an agent must call tools in a strict order—for example, it must call verify_kyc_status() before calling execute_wire_transfer(). Trajectory evaluation checks that required prerequisite tools are never skipped or called out of order.

Automated CI/CD Deployment Gate Based on Tool Evaluation Metrics — Gating Cloud Build deployments automatically when tool selection precision, argument accuracy, or latency regresses.

Automating CI/CD Deployment Gates

Block Regressions Before Production: Wire your evaluation suite into Cloud Build (or GitHub Actions) so every pull request that modifies a prompt, tool docstring, or model version runs against your Golden Dataset against mocked/sandboxed tools.
Hard Pass/Fail Thresholds: Configure the pipeline to automatically fail the build and block deployment if tool selection accuracy, trajectory ordering, or P95 latency drops below your target SLO threshold.

Key Takeaways (4.1b)
JSON passes schema validation but has swapped or wrong parameter values → Test for Semantic Argument Accuracy against golden tool inputs.
Ensure verify_kyc always runs before transfer_funds → Use Trajectory Evaluation to verify directed tool execution order.
Prevent bad prompt edits from reaching users → Gate CI/CD deployments in Cloud Build on automated evaluation scores.

Consideration 4.1c: Determining the Appropriate Evaluation Framework and Tooling (ADK Evalset, Gen AI Evaluation Service, and Custom Autoraters)

ADK Evalset vs. Vertex AI Gen AI Evaluation Service — Using ADK evalset for fast developer/CI unit tests on trajectories, and Gen AI Evaluation Service for managed rubric and pairwise grading.

Google Cloud provides three complementary tools for agent evaluation, each designed for a distinct stage of the development lifecycle:

Choosing Between `ADK evalset`, `Gen AI Evaluation Service`, and `Custom Autoraters`

ADK Evaluation Tooling (evalset — Local Dev & CI/CD Regression):
Built directly into the Agent Development Kit (adk eval / Web UI).
Ideal for developers and CI/CD pipelines to rapidly replay saved multi-turn test files (*.evalset.json) and check tool trajectory match scores (EXACT, IN_ORDER, ANY_ORDER) and response similarity.
Vertex AI Gen AI Evaluation Service (Managed Fleet & Model Benchmarking):
Fully managed cloud service for running large-scale evaluations across hundreds or thousands of prompts, comparing candidate models, and monitoring production quality metrics (such as Groundedness, Safety, Instruction Following, and Verbosity).
Custom Autoraters (LLM-as-a-Judge — Domain-Specific Rubrics):
Standard string metrics like ROUGE or BLEU only count overlapping words—they cannot tell if a medical or financial disclaimer is legally compliant or empathetic.
When you need to grade open-ended answers against custom business or regulatory rules, configure a Custom Autorater (LLM-as-a-Judge) with an explicit step-by-step scoring rubric (e.g., a 1-to-5 scale with clear criteria) and calibrate it against human expert grades.

Custom Autoraters: Pointwise Scoring vs. Pairwise Comparison — Configuring LLM-as-a-Judge Autoraters with explicit scoring rubrics for domain-specific compliance and pairwise A/B model selection.

Pointwise vs. Pairwise Evaluation in Gen AI Evaluation Service

Pointwise Evaluation: Grades a single model's response on an absolute scale (e.g., "Score Groundedness from 0 to 1" or "Pass/Fail on HIPAA compliance rubric"). Use this for continuous production monitoring and CI/CD pass/fail gates.
Pairwise Evaluation: Shows an Autorater two responses side-by-side (Model A vs. Model B) for the same prompt and asks which one is better (win/loss/tie rate). Use Pairwise Evaluation when deciding whether to migrate from Model A to Model B or comparing two major prompt rewrites.

Key Takeaways (4.1c)
Fast local/CI regression testing of ADK agent tool trajectories → ADK evalset.
Compare two candidate models (Model A vs. Model B) head-to-head → Pairwise Evaluation in Gen AI Evaluation Service.
Grade specialized legal/clinical tone that BLEU/ROUGE cannot measure → Build a Custom Autorater (LLM-as-a-Judge) with a clear rubric.

Consideration 4.1d: Evaluating an Agentic System Against a Golden Dataset for Response and Retrieval Quality

The RAG Triad: Context Relevance, Groundedness, and Answer Relevance — Isolating whether an agent failure happened in the Retriever (Context Precision/Recall) or the Generator (Groundedness and Answer Relevance).

When a RAG-enabled agent gives a bad answer, how do you know whether the Retriever fetched the wrong documents or the LLM Generator hallucinated? Subsection 4.1d covers how to diagnose agent quality using Retrieval Metrics, the RAG Triad, and Trajectory Matching Modes.

Diagnosing RAG Quality: Retriever vs. Generator Metrics

Retriever Metrics (Did we fetch the right chunks?):
Context Recall: Did the retriever find all the essential ground-truth facts needed to answer the question? (Low recall = needed chunks were missed; fix by improving embeddings, chunking, or enabling Hybrid Search).
Context Precision: Are the most relevant chunks ranked right at the top of the retrieved list, without lots of noisy filler chunks? (Low precision = too much noise; fix by adding a Cross-Encoder Reranker).
Generator Metrics (Did the LLM use the chunks properly?):
Groundedness (Faithfulness): Is every claim in the LLM's answer supported by the retrieved context? (Low Groundedness = Hallucination!)
Answer Relevance: Did the LLM actually answer the user's specific question?
Diagnostic Example: What if Groundedness is 98% (no hallucination), but Answer Relevance is 65%? That means the agent faithfully summarized the retrieved documents without making anything up, but failed to directly answer what the user asked!

ADK Trajectory Evaluation Modes: Exact, In-Order, and Any-Order — Choosing between Exact Match (strict sequence), In-Order Match (required sequence allowing extra lookups), and Any-Order Match.

Choosing the Right ADK Trajectory Match Mode

When evaluating tool calls in ADK against a golden sequence [tool_A, tool_B]:

EXACT Match: The agent must call only [tool_A, tool_B] in that exact order with zero extra tool calls.
IN_ORDER Match: The agent must call tool_A followed by tool_B, but is allowed to make extra harmless helper calls in between (for example, [list_schemas, tool_A, get_time, tool_B]).
ANY_ORDER Match: The agent must call both tool_A and tool_B, but the order does not matter (great for independent parallel lookups).

Key Takeaways (4.1d)
High Groundedness (98%) + Low Answer Relevance (70%) → The answer is factually faithful to the docs (not hallucinating), but didn't answer the user's actual question.
Required tools must run in sequence A → B, even if the agent calls extra helper tools in between → Use IN_ORDER trajectory match.
Measure whether an agent refuses unsafe/harmful prompts → Evaluate Safety Compliance & Refusal Accuracy on an adversarial red-team evalset.

Quick Decision Cheat Sheet (Subsection 4.1)

If the Scenario Requires...	Choose This Google Cloud Pattern	Avoid This Common Mistake
Create a golden evaluation dataset for a brand-new agent before production logs exist	Generate synthetic test scenarios with an LLM and validate them with Human-in-the-Loop (HITL) domain experts	Waiting 6 months after launch before writing any evaluation tests
Ensure an agent always calls `verify_identity` before `issue_refund`, even if it makes extra read-only helper calls	Use `IN_ORDER` Trajectory Evaluation in ADK `evalset`	Using `ANY_ORDER` match (which allows refund before verification) or `BLEU` text scores
Compare two foundation models head-to-head to see which produces better customer support replies	Run Pairwise Evaluation using an Autorater in Vertex AI Gen AI Evaluation Service	Manually reading 3 prompts in the console
Grade whether an agent's responses follow strict company legal disclaimers and tone	Create a Custom Autorater (`LLM-as-a-Judge`) with an explicit scoring rubric	Using `ROUGE` or `BLEU` exact word-overlap metrics
Interpret evaluation results showing 99% Groundedness but 68% Answer Relevance	The agent stayed strictly faithful to the retrieved documents (no hallucinations), but did not directly answer the user's question	Assuming the model is hallucinating facts outside the knowledge base
Safely test a new system prompt on 5% of live users and measure real conversion impact	Use Cloud Run traffic splitting (5% canary) and track user satisfaction/conversion metrics	Deploying the unverified prompt to 100% of users at once