Book TOC · Previous · Chapter 4 of 6 · Next

Chapter 4: Improving Gen AI Output: Overcoming Hallucinations, Prompt Engineering & Grounding (RAG)

Subsections Covered: 3.1, 3.2, 3.3 (10 Considerations · 20 Custom Visual Diagrams)

Chapter Focus: Mitigating hallucinations, bias & knowledge cutoffs, Zero-Shot/Few-Shot/Role/Chain-of-Thought/ReAct prompting, Enterprise RAG grounding, HITL, and temperature/token sampling.

Subsection 3.1: Describe how to proactively overcome foundation model limitations

Section 3: Techniques to improve gen AI model output

The Big Picture

Section 3 focuses on how to systematically improve and control generative AI model outputs. A core responsibility of a Generative AI Leader is understanding that out-of-the-box foundation models are probabilistic next-token predictors, not infallible truth engines.

In Subsection 3.1, production architecture focuses on a three-part operational discipline:

1. Diagnosing the 6 Core Foundation Model Limitations (3.1a):
2. Hallucinations: Generating plausible-sounding, grammatically fluent, yet factually fabricated statements (e.g., inventing a fake court case citation or non-existent product feature).
3. Knowledge Cutoff: Complete blindness to any world event, price change, or company policy created after the model's static pre-training dataset was frozen.
4. Data Dependency: Output quality is strictly bounded by the coverage and accuracy of the training/context data.
5. Bias & Fairness Risks: Amplifying historical, demographic, or cultural prejudices embedded in web-scale training corpora.
6. Edge Cases: Failing on rare, ambiguous, adversarial, or out-of-distribution inputs.
7. Matching Each Limitation to the Right Mitigation Practice (3.1b): Follow the Cost-Complexity Escalation Ladder: Prompt Engineering (clarity, constraints, examples) → Grounding & RAG (connecting to authoritative live data to cure knowledge cutoffs and hallucinations) → Fine-Tuning (teaching domain-specific jargon, style, or complex formatting) → Human-in-the-Loop (HITL) (mandatory human review gates for high-stakes medical, legal, or financial decisions).
8. Continuous Post-Deployment Monitoring & Evaluation (3.1c): AI is never "deploy and forget." Leaders must establish KPIs, model versioning, drift monitoring (detecting when real-world input distributions shift away from training baselines), security patching, and centralized feature governance via Vertex AI Feature Store.

Consideration 3.1a: Common limitations: dependency, cutoff, bias, fairness, hallucination, edge cases

• Hallucinations vs. Knowledge Cutoff (Know the Exact Difference):
• Knowledge Cutoff: A model trained through January 2025 cannot answer "What was our company's Q3 2026 revenue?" or "What is today's stock price?" because its weights are frozen in time.
• Hallucination: Because the LLM is trained to predict the most statistically probable next word rather than say "I don't know", when asked about an unfamiliar court case or post-cutoff event, it may confidently fabricate a fake citation ("Smith v. Jones, 2023") or invent a false policy rule.
• Data Dependency, Bias & Edge Cases: Models inherit every blind spot and demographic stereotype present in their training corpus (data dependency & bias), and degrade sharply when encountering rare, atypical edge cases outside their normal distribution.

Key Implementation Patterns

• Mitigating Hallucinations in Financial Reporting: Use Integrate the model with Vertex AI Search to ground the responses in the PDF documents. Grounding the model with Vertex AI Search allows it to retrieve and use facts from your private documents. (Avoid alternative: Fine-tuning is better for adapting style or behavior, not for reliably retrieving specific facts from documents.)
• Reducing Demographic Bias in Recruitment: Use Update the system instructions to explicitly define a neutral, professional persona and include examples of unbiased questions. System instructions guide the model's behavior and persona, effectively reducing bias through explicit directives.
• Addressing Knowledge Cutoffs in News Analysis: Use Configure the model to use Grounding with Google Search. Grounding connects the model to live web search results to overcome knowledge cutoffs. (Avoid alternative: Daily fine-tuning is too slow and expensive for real-time news updates.)

Key Takeaways (3.1a)

• Hallucinations occur when models generate incorrect information confidently.
• Grounding (often via RAG) connects models to real-world data sources to improve accuracy.
• Vertex AI Search is a managed service that can be used to index and retrieve private documents for grounding.
• Fine-tuning is generally for style and behavior, while Grounding is for factual accuracy and knowledge retrieval.

Consideration 3.1b: Mitigation practices: grounding, RAG, prompting, tuning, HITL

• The Google Cloud Mitigation Escalation Ladder (Memorize This Order):
• Prompt Engineering (First Step — Lowest Cost & Zero Infra): Add explicit instructions, output constraints, few-shot examples, and "If the answer is not in the provided text, state that you do not know" rules.
• Grounding & Retrieval-Augmented Generation (RAG — Best for Facts & Freshness): Dynamically fetch current, authoritative snippets from enterprise data stores or Google Search at query time and inject them into the prompt context. RAG is always the correct answer for fixing knowledge cutoffs and factual hallucinations without retraining.
• Fine-Tuning (Best for Style, Tone & Specialized Vocabulary): Update model weights (or PEFT adapters) on a curated labeled dataset when the model must master specialized domain shorthand (e.g., clinical oncology notes), strict brand voice, or complex proprietary schemas.
• Human-in-the-Loop (HITL — Mandatory for High-Stakes Decisions): Place a human expert review/approval gate in the workflow whenever an error carries legal, financial, medical, or safety consequences—or when model confidence falls below a threshold.

Key Implementation Patterns

• Mitigating Hallucinations in Legal Document Summarization: Use Configure the model to use grounding with the firm's internal documents. Grounding anchors the model's response to your specific data, preventing it from making up facts. (Avoid alternative: Training a model from scratch is unnecessary and costly; grounding a pre-trained model is the standard solution.)
• Addressing Knowledge Cutoffs for News Aggregation: Use Implement Retrieval-Augmented Generation (RAG) to retrieve current news and include it in the prompt. RAG connects the model to external, real-time data sources to provide fresh context for generating answers. (Avoid alternative: Fine-tuning is too slow and expensive for daily updates and does not guarantee factual accuracy for real-time data.)

Key Takeaways (3.1b)

• Hallucinations are when models invent incorrect facts.
• Grounding connects a model to enterprise data to ensure accuracy.
• Temperature controls creativity; lower values are better for facts, but grounding is superior for retrieval.
• Avoid training from scratch for standard summarization tasks; use RAG/Grounding instead.

Consideration 3.1c: Continuous monitoring: upgrades, KPIs, patches, versioning, performance, drift, Feature Store

• Model & Data Drift Monitoring: Over time, real-world customer behavior, vocabulary, and economic conditions shift (data/concept drift), causing model accuracy to decay silently. Vertex AI Model Monitoring continuously compares live production request distributions against training baselines and alerts teams when drift thresholds are breached.
• Model Versioning, Upgrades & Security Patches: Pin production applications to specific model versions in Vertex AI Model Registry so upstream model updates do not unexpectedly alter prompt behavior, while testing new model versions and security patches in staging against golden evaluation metrics (e.g., ROUGE for summarization recall, BLEU for translation, or LLM-as-a-judge autoraters).
• Agent Platform Feature Store (Vertex AI Feature Store): Centralizes, versions, and serves curated ML features across both training and real-time online serving—eliminating training-serving skew.

Key Implementation Patterns

• Foundation Model Version Pinning vs. Auto-Updates: Use Keep your production code pinned to the specific model version. Deploy the new version to a staging environment and evaluate it against a golden dataset. Pinning the version prevents unexpected changes, and testing against a golden dataset in staging validates compliance before production traffic is affected.
• Detecting Semantic Drift in RAG Embeddings: Use Enable monitoring for feature drift on the query embeddings. Monitoring feature drift on embeddings detects changes in the semantic patterns of user queries. (Avoid alternative: Latency alerts measure response speed, not the accuracy or relevance of the content.)
• Automated Side-by-Side (AutoSxS) Evaluation: Use Use Vertex AI AutoSxS (Automatic Side-by-Side) to compare model versions. AutoSxS uses an autorater model to simulate human review, offering a scalable and accurate way to evaluate generative model quality continuously. (Avoid alternative: BLEU and ROUGE measure word overlap, which does not effectively capture the semantic quality or alignment with human judgment needed for complex summarization.)

Key Takeaways (3.1c)

• Avoid using @latest model tags in production for regulated or critical workloads.
• Always pin foundation models to specific versions (e.g., @001) to ensure consistency.
• Perform regression testing using a Golden Dataset in a staging environment before upgrading models.
• Continuous monitoring is crucial but serves as a safety net, not a substitute for pre-deployment testing.

Quick Decision Cheat Sheet (Subsection 3.1)

Further Reading (Subsection 3.1)

• Grounding overview — Explains how grounding connects model output to verifiable sources like Google Search or your own data.
• Vertex AI Search — Product page for Vertex AI Search, detailing how it enables retrieval for generative AI apps.
• System instructions in Vertex AI — Explains how system instructions can be used to set the persona and behavior of the model to ensure specific output qualities.
• Responsible AI best practices — Discusses safety settings, bias mitigation, and best practices for deploying generative AI responsibly.
• Model versions and lifecycle — Explains the difference between stable versions and the latest tag, and how to govern model updates.
• Introduction to Vertex AI Model Monitoring — Provides an overview of how Vertex AI monitors models for skew and drift to ensure prediction quality remains high.

Subsection 3.2: Describe prompt engineering techniques and how they drive better results

Section 3: Techniques to improve gen AI model output

The Big Picture

When a business user types "Summarize this article" into an LLM and gets back a rambling 600-word essay instead of a 3-bullet executive brief, the model didn't fail—the prompt architecture failed.

Prompt engineering is the highest-ROI lever in generative AI because it requires zero model weight modifications, zero training GPU hours, and zero infrastructure deployment. In Subsection 3.2, production architecture focuses on your ability to select the exact prompting technique that matches a scenario's structural complexity:

1. Prompt Design Fundamentals (3.2a): A production-grade prompt template explicitly defines four components: Persona/Role (who the model is), Task/Instruction (what action to take), Context/Constraints (length limits, tone, what to exclude), and Output Format (e.g., strict JSON schema or Markdown table).
2. Core In-Context Techniques (3.2b — Zero-Shot, One-Shot, Few-Shot, Role Prompting, Prompt Chaining):
3. Zero-Shot: Direct instruction with 0 examples—ideal for well-known, straightforward tasks.
4. One-Shot: Instruction + 1 complete input-output demonstration example to anchor structure.
5. Few-Shot: Instruction + 2 to 5+ input-output demonstration examples—the gold standard for teaching custom output formatting, edge-case classification, or nuanced sentiment rules without fine-tuning.
6. Role Prompting: Assigning a specific identity/expertise ("Act as an empathetic retail bank mortgage specialist" or "Act as a senior cybersecurity auditor"), which shapes vocabulary, depth, and tone.
7. Prompt Chaining: Breaking a complex macro workflow into a sequential pipeline of smaller prompts where the output of Prompt 1 becomes the input to Prompt 2 (e.g., Step 1: Extract key facts → Step 2: Translate to Spanish → Step 3: Format as HTML email).
8. Advanced Reasoning & Agentic Prompting (3.2c — Chain-of-Thought [CoT] & ReAct):
9. Chain-of-Thought (CoT): Instructing the model to "think step by step" and emit intermediate logical reasoning steps before stating its final answer—dramatically improving accuracy on multi-step math, logic, and policy calculations.
10. ReAct (Reason + Act): Interleaving internal Thought traces with external Action tool calls and Observation results in a loop until the goal is verified.

Consideration 3.2a: Definition and significance

• What Prompt Engineering Is & Why It Matters: Prompt engineering is the iterative discipline of designing, structuring, and refining natural-language inputs to guide an LLM toward accurate, relevant, and deterministic outputs without altering the model's underlying weights.
• The Anatomy of a High-Signal Enterprise Prompt:
• Clear Directive & Specificity: Replace vague verbs ("Summarize this") with concrete bounds ("Summarize this article in exactly 3 bullet points, maximum 50 words total, focusing only on financial risks").
• Explicit Output Schema Constraints: Specify exact machine-parsable formats ("Return ONLY valid JSON matching keys date, title, participants with no conversational preamble").

Key Implementation Patterns

• Addressing Logical Errors with Chain-of-Thought: Use Update the prompt to include the instruction "Think step-by-step" before generating the final answer. Chain-of-Thought prompting encourages the model to decompose complex reasoning tasks into steps, improving arithmetic accuracy.
• Standardizing Output Formats via Few-Shot Prompting: Use Include a few examples of input reviews and their corresponding output codes in the prompt. Few-shot prompting provides examples within the prompt to teach the model the specific output pattern required. (Avoid alternative: Fine-tuning is resource-intensive and unnecessary for simple formatting tasks solvable by prompting.)

Key Takeaways (3.2a)

• LLMs function as token predictors, not calculators; they struggle with direct complex math.
• Chain-of-Thought (CoT) prompting forces the model to decompose problems into intermediate steps.
• "Let's think step-by-step" is a key phrase to trigger CoT reasoning.
• Temperature controls randomness, while Personas control tone; neither directly fixes logic failures like CoT does.

Consideration 3.2b: Techniques: zero-shot, one-shot, few-shot, role prompting, chaining

• Zero-Shot vs. One-Shot vs. Few-Shot (Count the Examples!):
• Zero-Shot: Zero demonstration examples in the prompt. Best for simple, general-knowledge tasks where minimizing input token cost matters.
• One-Shot: Exactly one Input → Output demonstration example. Useful when a single canonical template is enough to show the desired output layout.
• Few-Shot: Multiple (2–5+) Input → Output demonstration examples. Best when downstream code requires a strict, non-standard classification schema (e.g., [POS], [NEG], [NEU]) or nuanced domain pattern matching.
• Role Prompting: Assigns an explicit persona or professional lens ("Act as a compassionate pediatric nurse explaining a procedure to a 10-year-old" vs. "Act as a Wall Street equity analyst"). Directly governs tone, empathy, jargon level, and perspective.
• Prompt Chaining: Decomposes a brittle, multi-objective mega-prompt into a sequence of modular prompts where each step's output feeds the next step's input—improving reliability, debuggability, and step-by-step validation.

Key Implementation Patterns

• Addressing Formatting Inconsistencies: Use Update the prompt to include a single example of an input email and the corresponding desired JSON output. One-shot prompting provides a concrete template for the model to copy, ensuring consistent structure and key names. (Avoid alternative: Instructions alone (Zero-shot) may not strictly enforce specific key names or formatting nuances.)
• Classifying Niche Industry Terminology: Use Update the prompt to include several examples of specific legacy queries mapped to their correct classifications. Few-shot prompting provides examples that teach the model how to interpret specific niche terms. (Avoid alternative: Role prompting sets a persona but does not define specific niche vocabulary.)
• Enhancing Complex Arithmetic Reasoning: Use Append the instruction "Let's think step by step" to the end of the prompt. Adding "Let's think step by step" triggers Chain-of-Thought reasoning, significantly improving performance on arithmetic and logic tasks. (Avoid alternative: One-shot prompting primarily helps with formatting and style, not complex arithmetic reasoning.)

Key Takeaways (3.2b)

• Use One-shot prompting to fix formatting inconsistencies or enforce specific schemas (like JSON).
• LLMs are pattern matchers; an example is often worth a paragraph of instructions.
• Avoid Chain-of-thought if you need concise, machine-readable output without conversational filler.
• Start with prompting techniques (Zero-shot -> One-shot -> Few-shot) before considering model fine-tuning.

Consideration 3.2c: Advanced: chain-of-thought, ReAct

• Chain-of-Thought (CoT) Prompting: When an LLM is forced to jump straight from a complex word problem (e.g., tiered retail discounts + loyalty points + tax calculation) to a final number in a single token, it often guesses wrong. By instructing the model to generate intermediate reasoning steps first ("Let's think step by step", or providing few-shot examples that show the working calculation before the final answer), CoT unlocks the model's sequential logic using its own generated tokens as working scratchpad memory.
• ReAct (Reason + Act) Prompting: Combines Chain-of-Thought reasoning with external tool invocation. Instead of reasoning in a vacuum, the model alternates between:
• Thought: "I need to find the user's current order status before I can calculate their refund eligibility."
• Action: Invokes get_order_status(order_id="9941").
• Observation: Reads the API response ("Delivered 45 days ago").
• Thought / Final Answer: Applies the 30-day return policy to the observed fact and responds accurately.

Key Implementation Patterns

• Zero-Shot Chain-of-Thought Activation: Use Append the phrase "Let's think step by step" to the end of your prompt. Adding "Let's think step by step" triggers Zero-Shot Chain-of-Thought reasoning, forcing the model to explain its steps without needing examples. (Avoid alternative: This describes Few-Shot Chain-of-Thought, which contradicts the requirement to avoid providing specific examples.)
• Few-Shot CoT for Legal Classification: Use Update the examples to include a reasoning step that explains the risk before stating the label. Providing examples that include the reasoning process (Chain-of-Thought) teaches the model how to derive the correct answer. (Avoid alternative: Adding more standard examples provides more data points but does not demonstrate the reasoning logic needed for complex cases.)

Key Takeaways (3.2c)

• Chain-of-Thought (CoT) improves reasoning capabilities in LLMs by breaking problems into steps.
• Zero-Shot CoT is triggered by the phrase "Let's think step by step" and requires no examples.
• Few-Shot CoT requires providing examples of questions and their step-by-step answers.
• Prompt engineering can often solve logic issues without needing model fine-tuning.

Quick Decision Cheat Sheet (Subsection 3.2)

Further Reading (Subsection 3.2)

• Prompt engineering strategies — Discusses various prompt design strategies including giving the model time to think (Chain of Thought) to reduce errors in reasoning tasks.
• Overview of prompting strategies — Google Cloud documentation detailing strategies like One-shot and Few-shot prompting to improve model response quality.
• Prompt engineering — Google Cloud documentation on prompt design strategies, including zero-shot, few-shot, and role prompting.
• Introduction to prompting — Review the section on 'Chain-of-thought prompting' to understand how breaking down tasks helps the model reason.
• Large Language Models are Zero-Shot Reasoners — The original research paper that established 'Let's think step by step' as a robust zero-shot trigger for reasoning.
• Prompt Engineering Guide (Google Cloud) — See the section on 'Chain-of-thought prompting' to understand how to guide the model to break down complex tasks.

Subsection 3.3: Identify grounding techniques and their use cases

Section 3: Techniques to improve gen AI model output

The Big Picture

Even with a well-structured prompt (3.2), an enterprise generative AI system requires two additional runtime controls before it can be trusted in production:

1. Grounding & Retrieval-Augmented Generation (3.3a, 3.3b, 3.3c): Anchoring the model's output in verifiable, authoritative data sources rather than relying solely on its parametric memory. You must distinguish both what data source you are grounding against (First-Party Enterprise Data like internal HR PDFs and BigQuery tables, Third-Party Licensed Data, or public World Data via Google Search) and which Google Cloud grounding offering fits the team (Pre-built RAG with Agent Search, modular RAG APIs, or Grounding with Google Search).
2. Sampling Parameters & Generation Controls (3.3d — Temperature, Top-P, Top-K, Token Count, Safety Settings): Every time an LLM generates a token, it computes a probability distribution over its entire vocabulary. Sampling parameters govern how the next token is chosen from that distribution:
3. Temperature (0.0 to 2.0): Controls randomness/flatness of the distribution. Low (0.0) = deterministic, factual, greedy decoding (mandatory for SQL, code translation, financial extraction). High (0.8–1.0+) = creative, diverse brainstorming (marketing slogans, poetry).
4. Top-P (Nucleus Sampling, 0.0 to 1.0): Dynamically selects from the smallest set of top tokens whose cumulative probability exceeds $P$ (e.g., 0.90 = top 90% probability mass).
5. Top-K (1 to N): Hard-caps the candidate pool to the $K$ most probable tokens (e.g., K=40).
6. Max Output Tokens / Output Length: Caps the maximum number of generated tokens (controlling cost and latency).
7. Safety Settings: Configurable content moderation thresholds across harm categories (Hate Speech, Harassment, Sexually Explicit, Dangerous Content).

Consideration 3.3a: Grounding concepts and data sources

• What Grounding Means: Grounding is the process of tethering an LLM's generated responses to verifiable, authoritative information sources—Providing factual accuracy, up-to-date freshness beyond the knowledge cutoff, and auditability via source citations.
• The Three Grounding Data Spheres (3.3a):
• First-Party Enterprise Data: Proprietary internal data owned by your organization—employee handbooks, internal SharePoint/Drive PDFs, customer CRM records, and BigQuery warehouse tables.
• Third-Party Data: External specialized datasets licensed from commercial providers (e.g., Bloomberg financial feeds, LexisNexis legal databases, or third-party SaaS connectors).
• World Data: Publicly available, real-time global information on the open internet (e.g., breaking news, live weather, public company filings, current stock prices) accessed via Grounding with Google Search.

Key Implementation Patterns

• Mitigating Hallucinations with Internal Knowledge: Use Configure the model to use Vertex AI Search as a grounding source pointing to the PDFs. Grounding with Vertex AI Search forces the model to use the indexed documents as the factual basis for its answers. (Avoid alternative: Fine-tuning is resource-intensive and not ideal for dynamic fact retrieval.)
• Verifiability and Citations: Use Configure the model to use grounding with Vertex AI Search and display the returned citations. Grounding with Vertex AI Search provides citation metadata that links generated text back to specific source documents. (Avoid alternative: Prompt instructions cannot replace the technical capability of grounding services to retrieve and link to source data.)
• Grounding with Google Search for Real-Time Data: Use Enable grounding with Google Search in Vertex AI. Grounding with Google Search allows the model to access real-time information from the web to answer queries about current events and weather. (Avoid alternative: Fine-tuning uses static data and cannot provide real-time information like current weather forecasts.)

Key Takeaways (3.3a)

• Grounding (RAG) is the primary method to prevent hallucinations when dealing with private or specific data.
• Vertex AI Search can ingest unstructured data like PDFs and serve as a grounding source.
• Fine-tuning is for adapting behavior or style, not for serving as a primary knowledge retrieval mechanism for frequently changing documents.
• Temperature controls randomness but does not inject new knowledge.

Consideration 3.3b: Impact of RAG

• How the RAG Workflow Transforms Model Output (Retrieve → Augment → Generate):
• Retrieve: When a user asks a question, the system converts the query into an embedding and retrieves the most semantically relevant passages from an authoritative knowledge base.
• Augment: Those retrieved passages are dynamically injected into the LLM's prompt context alongside instructions to answer strictly using the provided context.
• Generate: The LLM synthesizes a response grounded in the retrieved facts and attaches verifiable citations/links to the source documents.
• Business Impact of RAG: Drastically reduces hallucinations, eliminates the knowledge cutoff problem (the moment a new PDF or database row is indexed, the model can use it without retraining), enforces user-level ACL permissions on retrieved docs, and builds user trust through clickable citations.

Key Implementation Patterns

• Mitigating Hallucinations in Financial Advisory: Use Implement Retrieval-Augmented Generation (RAG) to retrieve current rates from your internal knowledge base before generating an answer. RAG allows the model to look up the specific, current facts from an external source to answer accurately.
• Handling High-Velocity Data Updates: Use Implement a Retrieval-Augmented Generation (RAG) architecture to query the inventory database in real time. RAG enables the model to access real-time external data sources, ensuring accuracy for frequently changing information. (Avoid alternative: Daily fine-tuning cannot keep up with minute-by-minute updates, resulting in stale data.)

Key Takeaways (3.3b)

• Hallucinations occur when LLMs generate incorrect but plausible-sounding information.
• RAG (Retrieval-Augmented Generation) is the primary method to ground LLMs in factual, private, or up-to-date data.
• Fine-tuning is better for adapting style or domain vocabulary, not for injecting frequently changing facts.
• Temperature controls randomness; lower temperature reduces creativity but does not solve data absence.

Consideration 3.3c: Google offerings: Vertex RAG, RAG APIs, Google Search grounding

• Choosing Among Google Cloud's 3 Grounding Offerings (3.3c):
• a. Pre-built RAG with Agent Search (Vertex AI Search): Best when you want a fully managed, low-code/out-of-the-box RAG solution over first-party enterprise documents (Cloud Storage, BigQuery, Google Drive, Confluence, Jira) with automatic parsing, chunking, embedding, hybrid search, and citation generation.
• b. RAG APIs (Vertex AI RAG Engine, Embeddings API, Vector Search, Ranking API, Layout Parser): Best when engineering teams need custom, modular control over chunking boundaries, custom vector databases, or bespoke retrieval/reranking logic.
• c. Grounding with Google Search: Best when the model needs live public world knowledge (recent news, public facts, real-time web verification) rather than private internal files.

Key Implementation Patterns

• Real-time Public Information Retrieval: Use Configure the model to use Grounding with Google Search in Vertex AI. Grounding with Google Search enables the model to access real-time public data directly from the web. (Avoid alternative: Fine-tuning creates a static knowledge snapshot and cannot provide real-time updates.)
• Verifiability and Trust: Use Parse the 'groundingMetadata' field in the API response to extract the source URIs from the retrieved contexts. The 'groundingMetadata' field contains the verified source links and citations returned by the grounding service. (Avoid alternative: Prompting for links is unreliable and can lead to hallucinations; use structured metadata instead.)
• Internal Unstructured Knowledge Base: Use Create a data store in Vertex AI Search using the Google Cloud Storage bucket. Configure the generative model to use this data store for grounding. Vertex AI Search creates a searchable index of your private PDFs, which serves as the grounding source for the model to generate accurate, document-based answers.

Key Takeaways (3.3c)

• LLMs have a static knowledge base defined by their training data cutoff.
• Grounding (RAG) is required to inject fresh or private data into the model's context.
• Use 'Grounding with Google Search' when the requirement is to access real-time, public information from the web.
• Use 'Vertex AI Search' (Enterprise RAG) when the requirement is to access private, internal documents.

Consideration 3.3d: Sampling controls: token, temperature, top-p, safety, length

• Mastering Every Sampling & Control Parameter (3.3d — High-Frequency architectures Table):
• Temperature: Controls randomness of token selection. Set Temperature = 0 (or near 0.0) for deterministic, repeatable, factual tasks: Text-to-SQL generation, COBOL-to-Python code conversion, financial figure extraction, legal compliance Q&A. Set Temperature high (0.7–1.0) for creative tasks: brainstorming marketing slogans, story writing, diverse ad copy.
• Top-P (Nucleus Sampling): Dynamically filters the token pool to the smallest set of words whose cumulative probability sums to $P$ (e.g., Top-P = 0.9 keeps tokens making up the top 90% of probability mass, cutting off the long tail of bizarre words).
• Top-K: Filters the candidate pool to a fixed count of the $K$ most probable tokens at each step.
• Token Count & Max Output Length: A token is roughly 4 characters or \sim\frac{3}{4} of an English word. Setting max_output_tokens caps response length, preventing runaway verbosity, bounding latency, and controlling API billing. Note that cutting max_output_tokens abruptly truncates the text mid-sentence unless paired with a prompt length instruction!
• Safety Settings: Configurable per-request filters in Vertex AI Studio/API across four core harm categories (Hate Speech, Harassment, Sexually Explicit, Dangerous Content) with block thresholds (Block none, Block few, Block some, Block most).

Key Implementation Patterns

• Ensuring Deterministic Code Generation: Use Set the temperature parameter to 0. Setting temperature to 0 eliminates randomness, forcing the model to choose the most probable token every time, which is required for deterministic code generation. (Avoid alternative: Higher temperature increases randomness and creativity, which causes inconsistency and syntax errors in code generation.)
• Encouraging Creativity in Marketing Copy: Use Increase the temperature parameter to a higher value. High temperature increases randomness, leading to more creative and varied outputs. (Avoid alternative: Low temperature forces the model to pick the most likely words, resulting in generic and repetitive text.)

Key Takeaways (3.3d)

• Temperature = 0 is the standard setting for code generation and math tasks to ensure determinism.
• Higher temperature settings introduce randomness, which fosters creativity but risks hallucinations and syntax errors.
• Top-k and Top-p are alternative sampling methods that also influence diversity but are often used in conjunction with temperature.
• Consistent inputs should yield consistent outputs for functional tasks like SQL generation.

Quick Decision Cheat Sheet (Subsection 3.3)

Further Reading (Subsection 3.3)

• Grounding overview — Explains how grounding connects models to real-world data to reduce hallucinations.
• Grounding with Google Search — Documentation on how to use Vertex AI Search as a datastore for grounding generative models.
• Grounding overview — Details the process of using Vertex AI Search as a grounding source for private data.
• Grounding with Google Search — This page explains how to use Grounding with Google Search in Vertex AI to connect models to fresh world knowledge.
• Model parameters and configuration — Refer to the section on 'Temperature' to understand how it controls the degree of randomness in token selection and why lower values are better for deterministic responses.

Book TOC · Previous · Chapter 4 of 6 · Next