Book TOC · Previous · Chapter 7 of 7

Chapter 7: Master Look-Alike Tool Comparison & Decision Playbook

Chapter Focus: Side-by-side comparison of look-alike PMLE services, a 5-Step Architecture Decision Framework, cross-domain production patterns, and the Top 25 Golden Rules.

1. The PMLE Look-Alike Service Disambiguation Matrix

In complex Google Cloud Certified Professional Machine Learning Engineer architectures, multiple Google Cloud services often appear to solve similar problems. Use the matrix below to distinguish between closely related services based on latency, scale, security boundaries, and operational overhead.

2. The 5-Step Architecture Decision Framework

When evaluating a multi-service Google Cloud design, apply this structured 5-step architectural review:

1. Step 1 — Pinpoint Core Workload & Non-Functional Requirements: Identify the primary workload driver (e.g., sub-10ms OLTP vs. petabyte OLAP, synchronous REST vs. asynchronous event streaming, strict regional data residency vs. global active-active availability).
2. Step 2 — Prefer Managed, Cloud-Native Services First: Default to fully managed, serverless, or Google-curated services before introducing self-managed VMs, custom glue scripts, or unscalable manual operations.
3. Step 3 — Enforce Zero-Trust Identity & Keyless Authentication: Replace static credentials, long-lived service account JSON keys, and broad basic roles (Owner/Editor) with Workload Identity Federation, attached service accounts, and predefined IAM roles.
4. Step 4 — Verify Network Isolation & Data Perimeter Guardrails: Combine private connectivity (Private Service Connect, Private Google Access, IAP TCP forwarding) with VPC Service Controls and Organization Policy constraints.
5. Step 5 — Optimize for Cost, Observability, and Day-2 Reliability: Validate autoscaling boundaries, SLO burn-rate alerting, sink-level log filtering, and automated rollback pipelines before production rollout.

3. Review: Architecture Decision List

1. Use Cloud Vision API for generic, common object detection.
2. Use Vertex AI AutoML when you have custom data and need to detect specific, non-generic items or patterns.
3. AutoML provides a low-code environment suitable for users with limited ML coding experience.
4. TPUs are optimized for bfloat16 and float32 operations and lack native hardware support for float64.
5. GPUs (specifically data center models like A100) support float64 (double precision) natively.
6. Use TPUs for high-throughput deep learning training where lower precision is acceptable.
7. Use GPUs for workloads requiring high numerical precision, such as scientific computing or financial modeling.
8. Accuracy is misleading for imbalanced datasets (e.g., fraud, rare diseases).
9. AUC-ROC is good for balanced data but can be optimistic for imbalanced data.
10. AUC-PR is the preferred metric for highly imbalanced classification problems where the positive class is rare.
11. Vertex AI AutoML allows you to explicitly select the optimization objective based on your business goal.
12. Vertex AI Endpoints support multiple deployed models simultaneously.
13. Traffic splitting is configured at the Endpoint level using percentages that sum to 100.
14. This feature enables A/B testing and canary deployments without client-side changes.
15. Machine types determine capacity, not traffic distribution.
16. Real-time requirements (sub-second latency) demand streaming architectures, not batch.
17. Vertex AI Feature Store has distinct APIs for Batch Import (high throughput) and Streaming Ingestion (low latency).
18. Dataflow streaming jobs bridge the gap between event sources like Pub/Sub and the Feature Store Online serving node.
19. Training-Serving Skew occurs when the data or logic used in production differs from training.
20. Never calculate normalization statistics (mean/std) on the serving batch if the batch size is small or 1.
21. Use tf.Transform to compute global statistics over the full training set.
22. Embed preprocessing logic into the SavedModel graph to guarantee consistency.
23. Embed Preprocessing: Always try to include preprocessing logic inside the model graph to ensure consistency between training and serving.
24. Use Analyzers: Operations that require a full pass over data (like mean, vocabularies, or quantiles) should use tf.transform analyzers.
25. Avoid Client Logic: Minimizing preprocessing logic in the client application reduces the risk of skew.
26. Vertex AI Endpoints support hosting multiple models simultaneously to enable gradual rollouts.
27. Use the trafficSplit configuration to define how traffic is distributed between deployed models.
28. Avoid client-side routing for deployment strategies; use server-side controls to keep clients decoupled.
29. Traffic splitting is distinct from autoscaling (replica counts); replica counts ensure capacity, while traffic split ensures routing logic.
30. Vertex AI AutoML Forecasting requires data to have a constant time granularity (e.g., hourly, daily).
31. Irregular event logs must be aggregated or resampled into fixed intervals before training.
32. Timestamps should be preserved as time data types, not converted to strings or indices, to maintain seasonality signals.
33. Training-Serving Skew often results from dual implementations of feature logic.
34. Vertex AI Feature Store eliminates skew by serving the exact same feature values for both training and inference.
35. Use Batch Serving for fetching large historical datasets for model training.
36. Use Online Serving for low-latency retrieval of the latest feature values during prediction.
37. Use Vision API for general object detection (e.g., "this is a car").
38. Use AutoML Vision when you need to detect custom labels specific to your business domain (e.g., "this is a 2022 model car with a scratch").
39. BigQuery ML is typically used for structured data, not raw image analysis.
40. Use Vertex AI Online Prediction when you need immediate results (low latency) for individual requests.
41. Use Vertex AI Batch Prediction when you have a large dataset, results are not needed instantly (asynchronous), and you want to optimize for cost and throughput.
42. Batch Prediction natively supports reading from and writing to BigQuery or Cloud Storage.
43. Windowing: Essential for aggregating unbounded data streams in Dataflow.
44. Sliding Windows: Use these when you need overlapping windows to calculate moving averages or rolling counts.
45. Fixed Windows: Use these for standard periodic reporting where overlaps are not needed.
46. Dataflow & Pub/Sub: The standard pattern for real-time data ingestion and processing on GCP.
47. Small files (KB size) stored individually in object storage cause high I/O latency due to request overhead.
48. I/O starvation occurs when the training hardware (GPU/TPU) processes data faster than the storage system can deliver it.
49. TFRecord is the recommended format for TensorFlow to aggregate small data items into larger files (shards) to maximize read throughput.
50. Optimizing the input pipeline is often more effective than upgrading hardware when the bottleneck is data ingestion.
51. Time-series data must never be split randomly; doing so causes data leakage.
52. Look-ahead bias occurs when a model trains on future data to predict past events, inflating validation metrics.
53. Always use a temporal (chronological) split for time-series models: train on the past, validate on the future.
54. A massive gap between validation accuracy and production performance is a strong indicator of data leakage.
55. Lightweight Python Components are best for simple, self-contained code with minimal dependencies.
56. Containerized Components are required when you need custom dependencies, specific OS libraries, or conflicting package versions.
57. Always prefer containerization for legacy migrations to ensure the runtime environment matches the original requirements.
58. RMSE optimizes for the mean and assumes symmetric error costs (over-prediction cost equals under-prediction cost).
59. Quantile Loss is used for probabilistic forecasting and handling asymmetric costs.
60. Target a low quantile (e.g., p10) to minimize over-predictions (reduce spoilage).
61. Target a high quantile (e.g., p90) to minimize under-predictions (avoid stockouts).
62. MirroredStrategy is for single-host, multi-GPU training where the model fits in memory.
63. MultiWorkerMirroredStrategy is for multi-host training where the model fits in memory on each device.
64. ParameterServerStrategy is best for models with large variables (like embeddings in recommendation systems) that need to be sharded across servers.
65. AutoML Forecasting is the correct choice for time-series predictions involving future horizons (e.g., stock planning, weather, financial forecasting).
66. AutoML Regression is for predicting numbers where time is not the primary dimension or structural component.
67. AutoML Classification is for predicting categories, not continuous numbers.
68. Always look for constraints like 'low-code' or 'no data science expertise' to favor AutoML over custom training.
69. Vertex AI Data Labeling Service allows you to offload the labor-intensive task of annotation to human labelers.
70. A labeling task requires a defined Label Set and a PDF of Instructions to ensure quality.
71. Supervised learning models (like Object Detection) cannot be trained on completely unlabeled data.
72. Use model_type='ARIMA_PLUS' for time-series forecasting in BigQuery ML.
73. Use the time_series_id_col option to forecast multiple items (e.g., products, stores) in a single query.
74. Avoid training individual models for each item in a large dataset; let BigQuery handle the partitioning.
75. Schemas are contracts: They define what valid data looks like.
76. Update schemas for valid changes: If data changes intentionally (evolution), update the schema, do not suppress the error.
77. Distinguish errors from evolution: Real errors (drift/skew) require fixing the data; evolution requires fixing the schema.
78. Vertex AI Workbench instances are Compute Engine VMs that adhere to standard VPC networking rules.
79. To secure instances, disable external IP addresses.
80. Use Private Google Access to allow private instances to reach Google APIs (like GCS and BigQuery).
81. Avoid Cloud NAT if strict 'no public internet' is a requirement, as NAT allows broader outbound access.
82. Use Cloud DLP to detect and mask PII in unstructured data.
83. Integrate Cloud DLP with Dataflow for real-time streaming transformations.
84. Always sanitize sensitive data before writing it to permanent storage to minimize risk.
85. Pub/Sub schemas validate structure, not content; they cannot redact text.
86. Linear models treat features independently by default.
87. Feature Crossing combines two or more features to capture non-linear interactions.
88. In BigQuery ML, use ML.FEATURE_CROSS within the TRANSFORM clause to create these interactions.
89. Feature crosses are especially useful for categorical features in linear algorithms like Logistic Regression.
90. Vertex AI Endpoints support deploying multiple models to a single endpoint ID.
91. Traffic splitting allows you to route requests by percentage between deployed models without changing client code.
92. Canary deployments minimize risk by exposing new models to a small subset of traffic first.
93. Blue/Green deployments involve switching traffic fully but allow for quick rollbacks.
94. Vertex AI Feature Store preserves the history of feature values using timestamps.
95. Training data for historical events requires point-in-time correctness to prevent data leakage.
96. Use the batch serving API with a read_instances list containing timestamps to perform time-travel lookups.
97. Domain Expertise Matters: For specialized fields (medical, legal), standard crowdsourced labeling is insufficient. Use specialized pools.
98. Guidance is Key: Always provide comprehensive instruction guides to human labelers to ensure consistency and accuracy.
99. Garbage In, Garbage Out: High-quality models start with high-quality labels. Do not compromise on labeling accuracy for cost in critical domains.
100. Jupyter Notebooks are JSON files that store both code and output results.

Book TOC · Previous · Chapter 7 of 7