Google Cloud Data Engineer

Data rarely starts in Google Cloud. You assess what to move, choose between online and offline transfer, and use migration services for databases, files and warehouses with limited disruption.

• Assessing sources, size and dependencies
• Storage Transfer Service and Transfer Appliance
• Database Migration Service and Datastream
• Migrating a data warehouse
• Validating migrated data
• Cutover and rollback planning

You build A migration plan for a database and a file archive, with validation checks.

Batch is still the backbone of most data work. You design pipelines in Apache Beam, run them on Dataflow, handle schema and bad records, and tune them for time and cost.

• Beam concepts: PCollections, transforms and I/O
• Running pipelines on Dataflow
• Schemas and handling bad records
• Joins, aggregations and side inputs
• Performance and cost tuning
• Templates and parameterized runs

You build A Beam batch pipeline that cleans files and loads BigQuery with a dead-letter path.

Streaming adds time, order and duplicates to the problem. You publish and consume with Pub/Sub, window and aggregate events in Dataflow, and decide how to treat late and repeated data.

• Pub/Sub topics, subscriptions and delivery
• Windows, triggers and watermarks
• Late data and allowed lateness
• Exactly-once processing and deduplication
• Streaming into BigQuery and Bigtable
• Scaling and backlog monitoring

You build A streaming pipeline that aggregates events in windows and handles late data.

Many organizations arrive with existing Spark and Hadoop jobs. You run them on Dataproc clusters and serverless Spark, separate storage from compute, and decide when to modernize rather than lift and shift.

• Dataproc cluster types and ephemeral clusters
• Dataproc Serverless for Spark
• Moving Hadoop jobs and HDFS data
• Cloud Storage as the data layer
• Sizing, autoscaling and preemptible workers
• When to rewrite for Dataflow or BigQuery

You build A migrated Spark job running on ephemeral clusters with data in Cloud Storage.

Pipelines must run on schedule, recover from failure and change safely. You build workflows in Cloud Composer, test and deploy pipelines through CI/CD, and make runs repeatable.

• Cloud Composer environments and DAG design
• Dependencies, retries and sensors
• Testing pipeline code
• CI/CD for data pipelines with Cloud Build
• Infrastructure as code for data resources
• Backfills and reruns

You build An orchestrated pipeline deployed through CI/CD with tests and a backfill procedure.

Each store makes some queries easy and others unaffordable. You compare BigQuery, Bigtable, Spanner, Cloud SQL and Cloud Storage by latency, scale, consistency and cost, and choose from the access pattern.

• Analytic storage with BigQuery
• Wide-column storage with Bigtable
• Global relational storage with Spanner
• Cloud SQL and Firestore for application data
• Cloud Storage classes for raw and archive data
• Matching a store to an access pattern

You build A storage plan for six datasets, with the reason and the cost driver for each.

A well-modeled warehouse is faster and cheaper to query. You design schemas, partition and cluster tables, use nested and repeated fields, and control query cost and capacity.

• Dimensional models and denormalization
• Partitioning and clustering
• Nested and repeated fields
• Materialized views and query optimization
• Slots, reservations and on-demand pricing
• Time travel and table snapshots

You build A modeled, partitioned warehouse with a measured query cost reduction.

A lake without governance becomes a swamp. You organize raw, refined and curated zones, apply cataloging, quality and access policy with Dataplex, and bridge lakes and warehouses.

• Lake zones and file formats
• BigLake and open table formats
• Dataplex for discovery and governance
• Data quality rules and lineage
• Data mesh ideas and ownership
• Access policies across lake and warehouse

You build A governed lake with zones, a catalog, quality rules and access policy.

Data earns its value when others use it. You prepare data for dashboards and models, speed up BI queries, mask sensitive columns, share datasets safely, and shape unstructured data for retrieval.

• Preparing data for BI and visualization
• BI Engine and query acceleration
• Training models with BigQuery ML
• Column masking and row-level security
• Sharing data with Analytics Hub
• Embeddings and vector search for unstructured data

You build Curated datasets with masking and sharing in place, plus a vector index over documents.

Data platforms need steady care. You tune resources and capacity, schedule workloads, monitor jobs and costs, and troubleshoot slow queries and stuck pipelines before they affect users.

• Reservations and capacity management
• Optimizing Dataflow and Dataproc resources
• Cloud Monitoring for data jobs
• Alerts for freshness, failures and cost
• Troubleshooting slow queries and stuck jobs
• Cost attribution by team and workload

You build A monitoring and cost report with alerts for a running data platform.

Data systems fail in particular ways: late files, bad deploys, deleted tables. You plan replication, backups and recovery, recover from corruption, and test the recovery path.

• Replication and multi-region storage
• Backups, snapshots and time travel
• Recovering from accidental deletion and corruption
• Replaying streaming data
• Disaster recovery targets for data
• Rehearsing recovery

You build A recovery plan with tested restores for a warehouse and a stream.

Assemble the whole discipline in one project. You design for security and trust, ingest batch and streaming data, store and model it, prepare it for analysis and AI, and show that it can be operated and recovered.

• Defining requirements and a design
• Building ingestion and processing
• Storing, modeling and governing the data
• Operating, monitoring and recovering
• Presenting the platform

You build A working data platform with ingestion, warehouse, governance and operating runbooks.