Databricks Data Engineer

Streams fail slowly: state grows, schemas drift, late data arrives. You ingest files incrementally with Auto Loader, handle schema change and rescued data, and manage watermarks, checkpoints and state so that a stream can run for months.

• Auto Loader: discovery modes and incremental listing
• Schema inference, evolution and rescued data
• Triggers, checkpoints and exactly-once behavior
• Watermarks, late data and stateful aggregation
• Deduplicating a stream and controlling state size
• Writing to Delta with foreachBatch

You build A file-ingestion stream that survives a schema change and bounded late data.

Declarative pipelines move orchestration into the platform. You decide between a streaming table and a materialized view for each step, apply change data capture with AUTO CDC, and handle ordering, deletes and slowly changing history.

• Streaming tables or materialized views: the decision
• Defining a pipeline in SQL and in Python
• AUTO CDC: sequencing, deletes and out-of-order events
• Slowly changing dimensions, type 1 and type 2
• Triggered and continuous modes, and full refresh
• Reading the pipeline event log

You build A CDC pipeline that keeps a type 2 dimension correct when events arrive out of order.

Data arrives from Kafka topics, Kinesis streams, operational databases and cloud storage, each with its own failure modes. You choose managed connectors where they exist, write your own readers where they do not, and handle many file formats.

• Reading from Kafka and Kinesis
• Offsets, replay and consumer behavior
• Lakeflow Connect for databases and SaaS sources
• JDBC reads: partitioning and pushdown
• Parsing JSON, Avro, Parquet and CSV at scale
• Choosing between managed and hand-built ingestion

You build An ingestion layer reading a message bus and a database into governed tables.

Copying data creates a second copy to secure, pay for and keep fresh. You expose data to other organizations with Delta Sharing, query external systems in place through federation, and run joint analysis in Clean Rooms without exposing raw rows.

• Delta Sharing: providers, recipients and shares
• Open sharing and Databricks-to-Databricks sharing
• Lakehouse Federation: connections and foreign catalogs
• When to federate and when to ingest
• Clean Rooms for joint analysis
• Governance that travels with shared data

You build A share and a federated catalog, each with the governance a consumer inherits.

Correct logic is not enough when the data is huge and untidy. You use window functions and joins that scale, handle semi-structured records with VARIANT, call models from SQL with ai_query, and stop bad records using expectations.

• Window functions for ranking, sessions and running totals
• Join strategies, broadcast and skewed keys
• Semi-structured data with VARIANT
• Enriching rows with ai_query
• Expectations: warn, drop and fail
• Quarantining bad records instead of losing them

You build A transformation with expectations that routes invalid records to a quarantine table.

A good table layout makes every later query cheaper. You compare Delta and Iceberg layouts, shape dimensional models, serve them through materialized views, and define metrics once as governed metric views.

• Delta and Iceberg table layout compared
• Facts, dimensions and grain
• Materialized views as the serving layer
• Metric views: defining measures once
• Evolving a model without breaking consumers

You build A dimensional model with a materialized view and a governed metric view on top.

Tuning without a diagnosis is guessing. You choose clustering for how a table is actually queried, let Predictive Optimization handle maintenance, use Change Data Feed for incremental work, and read the query profile to find skew, shuffles and weak skipping.

• Liquid Clustering against partitioning and Z-order
• Predictive Optimization and routine maintenance
• File sizing and data skipping
• Change Data Feed for incremental processing
• Reading a query profile: skew, spill and shuffle
• Right-sizing compute for a workload

You build A tuning report that turns a slow query profile into a clustering change with measured gain.

Protecting data is design work, not a final step. You apply least-privilege access in Unity Catalog, write attribute-based policies and column masks, pseudonymize sensitive fields, and purge records so that retention and erasure rules can be met.

• Least-privilege grants on Unity Catalog objects
• Attribute-based access control with tags and policies
• Column masks and row filters
• Hashing and pseudonymizing personal data
• Erasing a person’s records, including history
• Retention rules and the limits of VACUUM

You build A policy set that masks personal data by role and a verified erasure routine.

People use only the data they can find and trust. You tag and describe assets so they are discoverable, rely on permission inheritance instead of per-table grants, and keep ownership clear so that access stays manageable as the estate grows.

• Tags and governed tags for discovery
• Comments and documentation as metadata
• Permission inheritance from catalog to table
• Ownership, stewardship and handover
• Lineage for impact analysis

You build A catalog layout with tags, descriptions and inherited grants for three teams.

A pipeline nobody watches fails silently. You query system tables for cost, usage and job health, read pipeline event logs, raise alerts on conditions that matter and use the Jobs API to act on what you find.

• System tables for billing, usage and lineage
• Job run history and failure patterns
• Pipeline event logs and data quality metrics
• SQL alerts and notification routing
• Driving the Jobs API from monitoring code

You build A monitoring dashboard with alerts on cost spikes and failing pipelines.

When a run fails you need the cause quickly. You read Spark UI stages, logs and metrics, repair a job from the failed task, and ship changes through Git folders and bundles with a CI/CD flow that promotes tested code.

• Reading Spark UI stages, tasks and storage
• Driver and executor logs for out-of-memory failures
• Repairing and rerunning part of a job
• Git folders and branching for data teams
• Bundles and CI/CD from commit to production

You build A CI/CD pipeline that tests, deploys and can repair a failing production job.

You assemble one pipeline end to end: ingestion, CDC, quality rules, optimized layout, access policy, monitoring and deployment. Then you defend each choice on cost, security and reliability, as you would in a design review.

• Scoping sources, consumers and constraints
• Choosing ingestion, streaming and modeling approaches
• Applying security, governance and quality together
• Monitoring cost and health from the first run
• Defending the design in review

You build A production-grade pipeline with security, monitoring, tests and a written design record.