Professional Data Engineer — Architectural Takeaways

Book TOC · Chapter 0 of 6 · Next: Chapter 1

Chapter 0: The PDE Reference Stack & 10 Core Principles

Chapter Focus: A visual guide to the Google Cloud Certified Professional Data Engineer (PDE) architecture stack, how the core Google Cloud services fit together, and the 10 Core Architectural Principles that govern production design decisions.

1. The Master Architectural Map: All In-Scope Google Cloud Data Engineering Services

The official Google Cloud Professional Data Engineer Curriculum spans 5 Sections, 19 Subsections, and 70 Considerations. Rather than memorizing services in isolation, organize them in your mind across Six Architectural Planes:

1. Ingestion, Messaging & Migration Plane (Sections 1.4 & 2.1) Pub/Sub & Pub/Sub Lite (Global Event Ingestion, Ordering Keys, DLQs) Datastream & DMS (Serverless CDC & Homogeneous DB Migration) Storage Transfer Service · Transfer Appliance BigQuery Data Transfer Service (DTS) 2. Stream & Batch Processing Plane (Sections 1.2 & 2.2) Dataflow (Apache Beam) (Unified Batch/Stream, Windows, Watermarks, RunInference) Dataproc ( GCE · GKE · Serverless ) (Managed Apache Spark, Hadoop, Preemptible/Spot) Dataform · Cloud Data Fusion · Dataprep (SQL ELT Assertions, Visual CDAP, Self-Service Prep) 3. Analytical Warehouse & Lakehouse Plane (Sections 3.2, 3.3 & 3.4) BigQuery (Partitioning, Clustering, Editions, Storage Write API, Time Travel) BigLake & BigQuery Omni (Unified Fine-Grained Security over GCS, AWS S3 & Azure Blob) Cloud Storage (Standard, Nearline, Coldline, Archive, Bucket Lock, Turbo Replication) 4. Operational Databases, NoSQL & Caching Plane (Section 3.1) Cloud SQL · Cloud Spanner · AlloyDB (Regional RDBMS, Global External Consistency, HTAP Columnar Engine) Cloud Bigtable · Firestore · Memorystore (Wide-Column Time-Series, Document Sync, Sub-ms Redis/Memcached) 5. BI, AI/ML & Data Sharing Plane (Section 4) Looker · Looker Studio · BI Engine · Materialized Views (Semantic LookML, Sub-Second In-Memory Acceleration) BigQuery ML · Vertex AI · Vector Search (In-Warehouse SQL ML, Embeddings, RAG Pipelines) Analytics Hub & Data Clean Rooms (Zero-Copy Linked Datasets & Privacy-Safe Sharing) 6. Security, Governance, Orchestration & Ops Plane (Sections 1.1, 2.3 & 5) Cloud IAM · Cloud KMS (CMEK/EKM/HSM) Sensitive Data Protection (DLP) · VPC Service Controls Dataplex & Data Catalog (Lakes/Zones, Data Quality, Lineage, Policy Tags) Cloud Composer (Airflow) · Workflows · Cloud Build Cloud Monitoring · Cloud Logging · INFORMATION_SCHEMA

Complete Reference Table of In-Scope PDE Services & Capabilities

#
Google Cloud Service / Feature
Architectural Plane
Primary Architectural Responsibility & Sweet Spot
1
BigQuery
Warehouse & Analytics
Serverless petabyte-scale columnar data warehouse; supports time/integer/ingestion partitioning, up to 4 clustering columns, Storage Write API, Time Travel (7 days), Snapshots, Clones, and Editions slot autoscaling.
2
BigLake & BigQuery Omni
Lakehouse & Multi-Cloud
BigLake extends BigQuery row/column-level security and governance to open formats (Parquet/ORC/Iceberg/Delta) on GCS; BigQuery Omni executes SQL in-place on AWS S3 and Azure Blob Storage without egress.
3
Dataflow (Apache Beam)
Stream & Batch Compute
Fully managed serverless runner for Apache Beam (PCollection, ParDo, Fixed/Sliding/Session windows, Watermarks, Triggers, Side Inputs, RunInference, Streaming Engine, Dynamic Work Rebalancing).
4
Dataproc (GCE, GKE, Serverless)
Open-Source Big Data
Managed Apache Spark/Hadoop ecosystem; decouple storage using the GCS connector (gs://), use ephemeral job-scoped clusters or Dataproc Serverless, and scale cost-effectively with Spot/Preemptible secondary workers.
5
Pub/Sub & Pub/Sub Lite
Messaging & Ingestion
Global, serverless asynchronous event bus supporting fan-out subscriptions, ordering keys, Avro/Protobuf schemas, Dead-Letter Topics (DLQ), and direct BigQuery/GCS export subscriptions.
6
Dataform
Warehouse ELT
Serverless SQL-based transformation orchestrator inside BigQuery supporting Git version control, dependency DAGs (ref()), incremental tables, and built-in data quality assertions.
7
Cloud Data Fusion & Dataprep
Low-Code ETL / Cleaning
Cloud Data Fusion: Code-free visual CDAP integration platform deploying batch/streaming pipelines onto Dataproc/Dataflow. Dataprep (Trifacta): Visual self-service data wrangling and anomaly discovery.
8
Datastream & Database Migration Service
CDC & Migration
Datastream: Serverless Change Data Capture (CDC) replicating Oracle, MySQL, PostgreSQL, SQL Server, and Salesforce into BigQuery/GCS. DMS: Homogeneous/heterogeneous operational DB migration to Cloud SQL/AlloyDB.
9
Storage Transfer Service & Transfer Appliance
Bulk Data Migration
Storage Transfer Service: High-speed online transfers from AWS S3, Azure, or on-prem POSIX via agents. Transfer Appliance: Encrypted physical rackable hardware (40 TB / 300 TB) shipped when network bandwidth is too slow.
10
Cloud Storage
Data Lake Storage
Unified object storage across 4 classes (Standard, Nearline 30d, Coldline 90d, Archive 365d) with Object Lifecycle Management, Autoclass, Bucket Lock (WORM compliance), and Dual-Region Turbo Replication (15-min RPO).
11
Cloud Bigtable
NoSQL Wide-Column Store
Petabyte-scale, sub-10ms latency wide-column store (~10,000 QPS/node) for IoT time-series and financial telemetry; requires compound row keys (prefix#entity#reverse_timestamp) and Key Visualizer tuning.
12
Cloud Spanner
Global Relational DB
Horizontally scalable relational database offering External Consistency (TrueTime), multi-region 99.999% availability, interleaved parent-child tables, and non-sequential primary keys (UUIDv4/bit-reversed) to prevent hotspots.
13
AlloyDB for PostgreSQL
Enterprise HTAP DB
PostgreSQL-compatible database with decoupled storage, 4x faster transactional throughput, and an adaptive in-memory Columnar Engine for hybrid transactional/analytical processing (HTAP) and pgvector AI workloads.
14
Cloud SQL
Regional Relational DB
Managed MySQL, PostgreSQL, and SQL Server (<= 64 TB); supports regional High Availability via synchronous standby replication across zones, read replicas, and Cloud SQL Auth Proxy for IAM/TLS connections.
15
Firestore & Memorystore
Document & In-Memory
Firestore: Serverless NoSQL document DB with real-time client listeners and mobile offline sync. Memorystore (Redis / Memcached): Sub-millisecond managed in-memory cache and sorted-set leaderboard store.
16
Dataplex & Data Catalog
Data Fabric & Governance
Logical governance mesh organizing GCS buckets and BigQuery datasets into Lakes, Zones (Raw/Bronze, Curated/Silver, Gold), and Assets with automated Data Quality, Data Lineage, and Policy Tags.
17
Sensitive Data Protection (Cloud DLP)
Privacy & De-Identification
Automated PII/PHI discovery, classification (InfoTypes), redaction, masking, bucketing, and deterministic pseudonymization (Crypto-Hash / Format-Preserving Encryption [FPE]) for referential integrity.
18
Cloud KMS, EKM, HSM & VPC-SC
Security & Perimeter
CMEK / Cloud HSM / Cloud EKM for cryptographic key control and crypto-shredding; VPC Service Controls (VPC-SC) for service perimeters preventing unauthorized data exfiltration across project boundaries.
19
Looker, Looker Studio, BI Engine & Materialized Views
Business Intelligence
Looker: Enterprise BI platform with Git-versioned LookML semantic modeling. BI Engine: Sub-second in-memory BigQuery cache. Materialized Views: Precomputed incremental views with Smart Tuning query rewrite.
20
BigQuery ML & Vertex AI
AI, ML & RAG
BigQuery ML (BQML): Train and serve models (LOGISTIC_REG, ARIMA_PLUS, KMEANS, XGBOOST), generate embeddings (ML.GENERATE_EMBEDDING), run VECTOR_SEARCH, and invoke remote Vertex AI Gemini models in SQL.
21
Analytics Hub
Data Sharing
Zero-copy cross-organization dataset publishing via Linked Datasets (subscriber pays compute, publisher pays storage) and privacy-enforced Data Clean Rooms.
22
Cloud Composer & Cloud Workflows
Orchestration
Cloud Composer (Managed Apache Airflow): Multi-cloud/hybrid DAG orchestration (use deferrable operators, KubernetesPodOperator, and Composer 2 GKE autoscaling). Workflows: Lightweight serverless HTTP/API state machine.
23
Cloud Monitoring, Logging & INFORMATION_SCHEMA
Observability & FinOps
Track Dataflow System Lag, Bigtable Key Visualizer, BigQuery INFORMATION_SCHEMA.JOBS_BY_PROJECT / RESERVATIONS_TIMELINE, and Billing Export with labels for granular chargeback.

2. The 10 Core Architectural Principles of the Architecture Stack

Whenever you are torn between two plausible-sounding options In Google Cloud architectures, apply these 10 Core Architectural Principles—they eliminate over 80% of look-alike service comparisons immediately:

  1. Principle 1 — Decouple Compute from Storage on Every Pipeline: Never store persistent data on Dataproc local HDFS disks! Always replace HDFS with the Cloud Storage connector (gs://), run ephemeral job-scoped Dataproc clusters (or Dataproc Serverless), and share hive metadata via Dataproc Metastore. If you use Preemptible/Spot VMs on Dataproc, place them strictly in Secondary Worker Groups (compute-only, zero HDFS DataNodes).
  2. Principle 2 — Match the Storage Engine to Latency, Structure, and Scale:
  3. Structured SQL Analytics (TBs to PBs, append-heavy) → BigQuery.
  4. High-throughput NoSQL Writes & Sub-10ms Time-Series/IoT Reads → Cloud Bigtable.
  5. Global Relational OLTP with Strong External Consistency (99.999% SLA) → Cloud Spanner.
  6. Enterprise PostgreSQL OLTP + Real-Time HTAP Analytics → AlloyDB for PostgreSQL.
  7. Regional Relational OLTP (MySQL / PostgreSQL / SQL Server ≤ 64 TB) → Cloud SQL.
  8. Mobile/Web Document Sync with Offline Support → Firestore (Native Mode).
  9. Sub-millisecond Caching / Session State / Sorted Sets → Memorystore for Redis.
  10. Principle 3 — Prevent Hotspotting by Eliminating Monotonically Increasing Keys: Never start a Cloud Bigtable row key or Cloud Spanner primary key with a raw timestamp, sequential auto-increment integer, or static prefix! In Bigtable, use a high-cardinality prefix followed by a reverse timestamp (sensor_id#metric#(MAX_LONG - timestamp)). In Spanner, use UUID v4, hash prefixes, or bit-reversed sequences.
  11. Principle 4 — Optimize BigQuery with Partitioning + Clustering + Storage Write API:
  12. Partition tables by a date/timestamp column (or integer range) and enable require_partition_filter = true to prevent full-table scans. If daily partitions over 15+ years exceed the 4,000-partition limit, switch to Monthly or Yearly partitioning + Clustering.
  13. Order up to 4 clustering columns from most frequently filtered/aggregated high-cardinality column to lowest.
  14. For high-throughput streaming ingestion with exactly-once semantics and lower cost, always prefer the BigQuery Storage Write API (with COMMITTED streams) over the legacy tabledata.insertAll API.
  15. Principle 5 — Use BigLake for Fine-Grained Governance on Lake Files & Multi-Cloud: Standard BigQuery External Tables require granting users raw Cloud Storage bucket permissions and cannot enforce row- or column-level security. Whenever a scenario asks for Row-Level Security (RLS), Column-Level Security (Policy Tags), or Dynamic Data Masking over Parquet/CSV/Iceberg files in Cloud Storage, AWS S3, or Azure Blob Storage (via BigQuery Omni), convert them to BigLake Tables using a Cloud Resource Connection.
  16. Principle 6 — Master Apache Beam Windowing, Watermarks, Triggers, and Allowed Lateness:
  17. Non-overlapping fixed intervals → Fixed (Tumbling) Windows.
  18. Overlapping moving averages ("every 5 minutes over the last 1 hour") → Sliding (Hopping) Windows.
  19. User activity bursts separated by an inactivity gap → Session Windows.
  20. Straggler events arriving after the watermark passes window end → Configure Allowed Lateness + AfterWatermark trigger with early/late firings + Accumulating mode (if downstream sinks overwrite with updated totals) or Discarding mode (if downstream sinks sum deltas).
  21. Principle 7 — Preserve Referential Integrity During PII De-Identification with FPE or Crypto-Hash: When Sensitive Data Protection (Cloud DLP) must de-identify sensitive identifiers (like SSN or Customer_ID) across multiple tables while still allowing analysts to JOIN those tables or reverse the token in authorized workflows, never use irreversible redaction ([REDACTED]). Choose Deterministic Encryption (CryptoDeterministicConfig / FPE) or Cryptographic Hashing (CryptoHashConfig).
  22. Principle 8 — Enforce Least-Privilege Data Access Without Duplicating Tables:
  23. Restrict specific columns (e.g., Salary, SSN) → Dataplex / Data Catalog Policy Tags with Dynamic Data Masking (or Authorized Views).
  24. Restrict specific rows by user/region (SESSION_USER()) → BigQuery Row-Level Security (Row Access Policies).
  25. Share live datasets across organizations with zero storage duplication → Analytics Hub Linked Datasets (or Data Clean Rooms).
  26. Block data exfiltration even if credentials are stolen → VPC Service Controls (VPC-SC).
  27. Principle 9 — Keep Cloud Composer (Airflow) Lightweight; Never Process Data Inside Airflow Workers: Cloud Composer is an orchestrator, not a data processing engine! Never load multi-gigabyte DataFrames inside a PythonOperator. Push heavy compute to BigQuery, Dataflow, or Dataproc; use Deferrable Operators (or mode="reschedule") for long-running sensors so worker slots aren't blocked; and isolate conflicting Python dependencies with KubernetesPodOperator.
  28. Principle 10 — Isolate Multi-Tenant Compute with BigQuery Editions & Reservations: Under On-Demand pricing, a runaway ad-hoc query can consume massive byte-scan budgets. To guarantee predictable spend and isolate critical executive dashboards from batch ETL or ad-hoc data science queries, use BigQuery Editions (Standard, Enterprise, Enterprise Plus) with separate Reservations (BACKGROUND/BATCH vs INTERACTIVE query priorities) and query cost guards (maximum_bytes_billed).

Book TOC · Chapter 0 of 6 · Next: Chapter 1

AwesomeGCP.com Learning

The approach taken is a question-led learning approach: you explore realistic project scenarios where you need to make decisions or take actions to achieve business goals within specific constraints. Instead of passive learning, you are challenged to think and work through the options to master the Professional Data Engineer (PDE) certification!

Practice PDE Questions