Microsoft Azure Databricks Data Engineer

Unity Catalog is the spine of governance. You create catalogs, schemas, volumes and tables, decide how to lay them out for teams and environments, and choose between managed and external storage.

• The Unity Catalog object model
• Metastores, catalogs and schemas
• Managed and external tables
• Volumes for files
• Delta Lake tables: transactions, time travel and history
• Naming and layout conventions across environments

You build A catalog layout for development, test and production, with tables and a volume created in each.

Access is granted, never assumed. You grant privileges on Unity Catalog objects, manage groups and service principals, and set up the authentication that people and jobs use.

• Securable objects and privilege inheritance
• GRANT, REVOKE and ownership
• Groups and service principals
• Authentication for users, jobs and tools
• Storage credentials and external locations
• Secrets and secret scopes

You build A privilege model for three teams, built on groups and service principals.

Table-level grants are too coarse for sensitive data. You add row filters and column masks, trace how data flows through lineage, and use audit logs to answer who touched what.

• Row filters and how they are evaluated
• Column masks for sensitive fields
• Tags and attribute-based controls
• Lineage at table and column level
• System tables and audit logs
• Investigating an access question

You build A table with row filters and column masks, and a lineage and audit report on its use.

Sharing data safely is a design decision. You decide what to share and with whom, set up Delta Sharing for internal and external recipients, and manage what happens when a share must be withdrawn.

• Delta Sharing concepts: providers, recipients and shares
• Databricks-to-Databricks and open sharing
• Choosing what to share and at what grain
• Securing and auditing shares
• Revoking and changing a share
• Cost and egress considerations

You build A share with a documented sharing strategy, a recipient and an access audit.

A model shapes every query that follows. You design layered medallion tables, choose between wide and dimensional models, and decide how to partition, cluster and evolve schemas.

• Bronze, silver and gold layers
• Dimensional modeling in a lakehouse
• Partitioning and liquid clustering
• Schema enforcement and evolution
• Slowly changing dimensions
• Normalized or denormalized: choosing per table

You build A layered data model for a business domain, with its clustering choice explained.

Ingestion is where data quality starts. You bring data in from files, databases and SaaS sources with Lakeflow Connect and batch loads, and choose between full and incremental loads.

• Sources and ingestion patterns
• Managed connectors in Lakeflow Connect
• COPY INTO and batch loads from files
• Incremental compared with full loads
• Handling schema drift at the source
• Landing raw data in bronze

You build A batch ingestion from a database and a file source into bronze tables.

Data arrives continuously, and files arrive unannounced. You use Auto Loader and Structured Streaming to pick up new files, handle schema changes, and manage checkpoints and triggers.

• Structured Streaming fundamentals
• Auto Loader and file discovery
• Schema inference and evolution
• Triggers, checkpoints and exactly-once processing
• Streaming from message sources
• Late data and watermarks

You build An Auto Loader stream that ingests arriving files with schema evolution and checkpoints.

Source systems change, and your tables have to follow. You apply inserts, updates and deletes from change feeds with MERGE and declarative CDC, and keep history where the business needs it.

• Change feeds and what they contain
• MERGE for upserts and deletes
• Declarative CDC in pipelines
• SCD type 1 and type 2 handling
• Ordering and late-arriving changes
• Verifying that a target matches its source

You build A pipeline that applies a change feed to a target table and keeps history for one dimension.

Raw data is rarely fit to use. You cleanse and transform it with SQL and Python, join and aggregate it correctly, and add constraints and expectations that stop bad rows from spreading.

• Profiling data to find problems
• Cleansing: nulls, duplicates and types
• Transformations in SQL and PySpark
• Joins, windows and aggregations
• Table constraints and expectations
• Quarantining bad records

You build A silver transformation with quality rules and a quarantine table for rejected rows.

Transformations become a product when they run on their own. You build declarative pipelines for table dependencies, orchestrate tasks with Lakeflow Jobs, and add retries, schedules and alerts.

• Declarative pipelines: tables, views and flows
• Triggered and continuous pipelines
• Lakeflow Jobs: tasks, dependencies and parameters
• Scheduling and event triggers
• Retries, timeouts and failure alerts
• Choosing a job design for a workload

You build A scheduled job that runs a declarative pipeline, with retries and alerts.

Pipelines you cannot reproduce are pipelines you cannot trust. You put code in Git, define jobs and pipelines as Databricks Asset Bundles, and deploy them across environments through an automated flow.

• Git folders and branching for data teams
• Asset Bundles: structure and configuration
• Targets for development, test and production
• Deploying bundles from CI/CD
• Testing notebooks and pipeline code
• Rolling back a deployment

You build A bundle that deploys one pipeline and job to two environments from a Git workflow.

Slow and failing jobs cost money and trust. You monitor runs, read the Spark UI, find skew, spills and small files, and apply the fixes that change run time.

• Monitoring jobs and pipelines
• Reading the Spark UI and the query profile
• Data skew, shuffles and spills
• The small files problem and table maintenance
• Caching, broadcast joins and file layout
• Diagnosing a failed run

You build A tuning report on a slow job, with measured run times before and after.

Build one pipeline end to end. You set up compute and catalogs, secure the data, ingest and transform it with quality rules, orchestrate and deploy it, and share a result safely.

• Designing the environment and the model
• Securing and governing the data
• Ingesting and transforming with quality rules
• Orchestrating and deploying
• Monitoring and sharing the result

You build A deployed lakehouse pipeline with governance, quality rules, monitoring and a data share.