Databricks Data Engineer

The right compute is the cheapest that finishes on time. You compare all-purpose and job compute, SQL warehouses and serverless options, and learn what drives cost so you can match each workload to a sensible choice.

• All-purpose, job and serverless compute
• SQL warehouses and when to use them
• Autoscaling and auto-termination
• What drives cost on the platform
• Matching a workload to a compute option

You build A compute plan that assigns five workloads to a compute type with a short reason each.

Every pipeline starts with getting data in reliably. You choose between Auto Loader, COPY INTO and managed connectors for batch, streaming and incremental loads, and read from files, JDBC sources and REST APIs.

• Batch, incremental and streaming loads
• Auto Loader for files arriving over time
• COPY INTO for idempotent batch loads
• Lakeflow Connect managed connectors
• Reading JDBC sources and REST APIs
• Choosing a load method for a stated source

You build Three loads of the same kind of data using three methods, with a note on when to use each.

Raw data is never ready to use. You fix types, handle nulls, parse nested fields and deduplicate with PySpark and SQL, adding checks that stop bad records, and so turn bronze tables into silver tables other people can trust.

• Casting types and handling nulls
• Parsing JSON and nested fields
• Deduplicating records
• Standardizing text, dates and keys
• Data quality checks on a table

You build A bronze-to-silver notebook that cleans, deduplicates and checks a messy source.

Value comes from combining sources. You join, union and aggregate DataFrames, learn how join type changes results, and make basic tuning choices so that these steps run in reasonable time on larger data.

• Inner, outer, semi and anti joins
• Union and union by name
• Grouping, aggregates and window basics
• Broadcast joins and partition basics
• Checking row counts after a combine

You build A silver table built by joining three sources, with row counts verified at each step.

Analysts want tables shaped for questions, not for storage. You apply the medallion layers, model gold tables and views for BI, and decide what belongs in each layer so that the whole flow stays understandable.

• Bronze, silver and gold: what belongs where
• Aggregated and dimensional gold tables
• Views and materialized views for BI
• Naming, documentation and ownership
• Refreshing gold tables on a schedule

You build A gold layer of two tables and a view, ready for a dashboard to read.

A pipeline that runs only when you click is not a pipeline. You build task graphs with dependencies, retries, branching and loops, and pick time-based or data-driven triggers such as file arrival.

• Tasks, dependencies and task graphs
• Retries, timeouts and failure handling
• Conditional branches and loops over inputs
• Schedules and file arrival triggers
• Passing parameters between tasks
• Notifications when a run fails

You build A job with five tasks, a retry policy and a file-arrival trigger.

Moving from development to production should be a repeatable step, not a copy and paste. You work in Git folders, then package jobs and pipelines as bundles and promote the same code across environments using the Databricks CLI.

• Git folders: branches, commits and pull requests
• What a bundle contains
• Targets for development and production
• Validating and deploying with the CLI
• Keeping secrets and settings out of code

You build A bundle that deploys one job to a development and a production target.

When a run is slow or fails, guessing wastes hours. You read run history to see what changed, use Spark UI stage metrics to spot skew and spilling, and diagnose cluster failures, including how Liquid Clustering can help a slow read.

• Run history: durations, trends and failures
• Stages, tasks and the Spark UI timeline
• Spotting skew and spilling to disk
• Common cluster failures and their causes
• Liquid Clustering for slow table reads

You build A short diagnosis of a slow run that names the cause and the change that fixed it.

Access should be granted on purpose. You manage tables in Unity Catalog, assign privileges with GRANT and REVOKE, understand ownership and inheritance, and decide which group needs which level of access.

• Securable objects and their hierarchy
• GRANT and REVOKE on catalogs, schemas and tables
• Ownership and privilege inheritance
• Groups and service principals
• Reviewing who can see what

You build A privilege plan with the grants for three roles on one catalog.

Some people may see a table but not every value in it. You apply column masks and row filters, and write attribute-based access policies driven by tags so that protection follows the data as it moves.

• Column masks for sensitive fields
• Row filters for regional or team access
• Tags that mark sensitive data
• Attribute-based access control policies
• Testing protection as different users

You build A table with a mask and a row filter, verified from the point of view of two users.

You build one complete pipeline: load data, clean and model it, orchestrate it, protect it and deploy it from development to production. Then you check it with run history and fix what the checks reveal.

• Scoping a source and its consumers
• Loading, cleaning and modeling the data
• Scheduling and deploying the pipeline
• Securing the result for each audience
• Reviewing a run and fixing what it shows

You build A scheduled, governed, deployed pipeline with documentation of how to operate it.