AWS Data Engineer

Batch ingestion is where most pipelines begin. You pull data from databases, files and software services, schedule loads, and design them so a failed run can be repeated without duplicating records.

• Database replication with DMS
• Moving files with DataSync and Transfer Family
• SaaS data with AppFlow
• S3 landing zones and partitioned layouts
• Scheduling and replayable loads
• Handling late and duplicate data

You build A batch ingestion pipeline that loads two sources and can be safely rerun.

Streams change the rules on ordering, loss and scale. You capture events with Kinesis and Kafka, deliver them to storage and consumers, and handle throttling, replay and ordering correctly.

• Kinesis Data Streams: shards, throughput and retention
• Firehose delivery to S3 and Redshift
• Managed Kafka with MSK
• Consumers with Lambda and enhanced fan-out
• Ordering, duplicates and replay
• Handling throttling and backpressure

You build A streaming pipeline that delivers events to storage and to a live consumer.

Raw data is rarely usable. You transform it with Glue, EMR, Lambda and SQL, convert formats, partition for speed, and choose the processing engine that fits the size and shape of the job.

• Glue ETL jobs and Spark basics
• EMR clusters and serverless options
• Lambda for light transformation
• SQL transformation in Athena and Redshift
• Partitioning, compaction and file sizes
• Choosing a processing engine

You build A transformation job that converts raw files into partitioned, query-ready tables.

Pipelines are dependencies, schedules and failure plans. You coordinate multi-step flows with Step Functions, Airflow and Glue workflows, and write steps that survive retries.

• Step Functions for data workflows
• Managed Airflow DAGs
• Glue workflows and triggers
• Event-driven pipelines with EventBridge
• Retries, idempotency and error paths
• Backfills and reruns

You build An orchestrated pipeline with dependencies, retries and a backfill procedure.

Engineering means code that others can run and change. You write SQL and Python for data work, manage it in version control, package it, and test and deploy it like any other software.

• SQL joins, windows and aggregation
• Python for data processing
• Git workflows for data code
• Packaging with containers and Lambda
• Infrastructure as code for data resources
• Testing and deploying pipelines with CI/CD

You build A tested pipeline in version control deployed through an automated process.

Each store makes some queries cheap and others impossible. You compare S3, Redshift, DynamoDB, relational and search stores by access pattern, scale and cost.

• S3 as the data lake foundation
• Redshift for warehouse analytics
• DynamoDB for key-value access
• Relational stores for transactional data
• Search and vector stores
• Matching store to access pattern and cost

You build A data store selection for five workloads with reasons and cost notes.

Without a catalog, data becomes unfindable. You register and discover datasets, model them for analytics, and manage how data ages and how schemas change without breaking consumers.

• Glue Data Catalog and crawlers
• Dimensional modeling and star schemas
• Redshift distribution and sort keys
• Lifecycle rules, tiering and expiry
• Schema evolution and compatibility
• Table formats and partition design

You build A cataloged, modeled dataset with lifecycle rules and a schema change plan.

Pipelines exist so that people can ask questions. You query with Athena and Redshift, prepare and visualize data, and automate recurring processing so analysis stays current.

• Interactive querying with Athena
• Redshift workload patterns
• Preparing data with DataBrew
• Dashboards with QuickSight
• Scheduled and event-driven processing
• Controlling query cost

You build A dashboard on top of an automated, scheduled dataset.

A pipeline that runs is not a pipeline that is right. You monitor jobs and latency, write quality rules, alert on bad data and troubleshoot failures from logs and metrics.

• Pipeline metrics, logs and alarms
• Data quality rules with Glue Data Quality
• Detecting freshness and volume anomalies
• Troubleshooting failed jobs
• Handling bad records without stopping the flow
• Reporting quality to data users

You build A monitored pipeline with quality rules and alerts that triggers on bad input.

Data platforms hold the most sensitive information a company has. You control who reads which table and column, encrypt storage and transport, and mask or tokenize sensitive values.

• IAM and Lake Formation permissions
• Column, row and cell-level controls
• Encryption with KMS across data services
• Masking and tokenization
• Finding sensitive data with Macie
• Cross-account data sharing

You build An access model for a data lake with column controls and masked fields.

Governance answers who used what, and whether they were allowed. You log data access, enforce retention and residency rules, track lineage, and meet privacy obligations as pipelines change.

• Audit logs with CloudTrail and service logs
• Retention and deletion rules
• Data residency and sovereignty
• Lineage and data ownership
• Privacy requirements in pipeline design
• Reporting for auditors

You build A governance plan with audit logging, retention rules and a lineage diagram.

Build one platform from source to dashboard. You ingest batch and streaming data, transform and catalog it, secure and monitor it, and explain every choice you made.

• Scoping sources, consumers and constraints
• Building ingestion and transformation
• Cataloging, securing and governing the data
• Monitoring quality and operations
• Presenting the platform

You build A working data platform with ingestion, catalog, access control, monitoring and a dashboard.