Microsoft Fabric Data Engineer

A workspace is where settings, compute and access meet. You configure Spark pools and runtimes, OneLake and domain settings, and the workflow options that decide how pipelines and notebooks behave for a whole team.

• Workspace settings and license modes
• Spark pools, runtimes and environments
• Shared libraries and resources for notebooks
• Domains and OneLake workspace settings
• Job and data workflow settings

You build A workspace set up for a data engineering team, with a documented Spark environment.

Analytics code needs the same discipline as software. You connect workspaces to Git, move items through stages with deployment pipelines, and use rules and database projects so a release never overwrites what belongs to one stage.

• Connecting a workspace to a Git repository
• Branching strategies for Fabric items
• Deployment pipelines across dev, test and production
• Deployment rules for connections and parameters
• Database projects for warehouse schemas

You build A release path from development to production for a lakehouse project, using Git and a deployment pipeline.

Data engineers decide who can see what, down to a single row or column. You apply workspace roles and item permissions, add masking and sensitivity labels, and use endorsement so governance travels with the data.

• Workspace roles and item-level permissions
• Row-level, column-level and object-level security
• Dynamic data masking
• Sensitivity labels and endorsement
• Folder-level access to lakehouse files
• Auditing who touched which data

You build A security plan with row, column and masking rules, tested for three different user personas.

Moving data is rarely one step. You decide when a pipeline, a notebook or a dataflow is the right tool, then chain them with parameters, schedules and event-based triggers so runs start on time and in the right order.

• Choosing between pipelines, notebooks and dataflows
• Pipeline activities and control flow
• Parameters and dynamic expressions
• Schedules and event-based triggers
• Running notebooks from pipelines and passing values back
• Retries and dependencies between steps

You build A parameterized pipeline that runs notebooks on a schedule and again when a new file arrives.

How you load data matters as much as where it lands. You compare full and incremental loads, design loads for a dimensional model, and pick between copy, dataflows, shortcuts and mirroring for each source.

• Full loads and incremental loads with watermarks
• Designing loads for a dimensional model
• Copy activity and staging areas
• Dataflows Gen2 for light transformation
• Mirroring an operational database
• Choosing a store and a method for each source

You build An incremental load from a transactional source into the silver layer of a lakehouse.

Raw data is dirty, duplicated and badly shaped. You clean it, join it and aggregate it in both PySpark and T-SQL, write upserts into Delta tables, and learn which engine suits which transformation.

• Cleaning data: duplicates, nulls and wrong types
• Joins and aggregations in PySpark
• Writing T-SQL transformations in a warehouse
• Upserts into Delta tables with merge
• Reshaping and denormalizing for reporting
• Choosing between Spark and SQL for a transformation

You build A bronze-to-gold transformation written in PySpark and in T-SQL, with a note on which fits better.

Some data cannot wait for a nightly batch. You bring events in with Eventstreams, shape them in flight, handle windows and late arrivals, and land them in an eventhouse or in Delta tables with Spark Structured Streaming.

• Streaming sources and destinations in Eventstreams
• Transforming events in flight
• Windowing: tumbling, hopping and sliding
• Spark Structured Streaming into Delta tables
• Late and out-of-order events
• Choosing between streaming and micro-batch

You build An eventstream that lands sensor events in an eventhouse, with windowed aggregates on top.

Kusto Query Language is how you read fast-moving data in an eventhouse. You write queries that filter, summarize and join, then add update policies and materialized views so results are ready before anyone asks.

• KQL basics: filtering, summarizing and joining
• Time series and window functions
• Update policies for transform-on-ingest
• Materialized views for fast aggregates
• Retention and caching policies in an eventhouse

You build A KQL queryset with an update policy and a materialized view behind a live dashboard.

A pipeline that fails silently is worse than one that never ran. You watch loads, refreshes and Spark jobs from the monitoring hub, track capacity use, and set alerts with Activator so the right person hears first.

• The monitoring hub and run histories
• Watching refresh, ingestion and Spark runs
• Alerts and actions with Activator
• Capacity metrics and throttling
• Logging from notebooks and pipelines

You build A monitoring plan with alerts that notify the right person when a load fails or runs late.

Each Fabric item fails in its own way. You learn to read the error, trace it to the cause and fix it, whether it comes from a pipeline activity, a dataflow refresh, a Spark job, an eventstream or a T-SQL query.

• Pipeline and activity errors
• Dataflow refresh failures
• Notebook and Spark job errors
• Eventstream and eventhouse errors
• T-SQL and warehouse query failures
• Shortcut and mirroring problems

You build A runbook that maps common failures across Fabric items to their causes and fixes.

Slow analytics cost money and patience. You maintain Delta tables, choose file layouts, tune Spark jobs and warehouse queries, and measure each change so a speed-up is shown rather than assumed.

• Delta table maintenance: compaction and vacuum
• V-Order, partitioning and file sizing
• Spark tuning: shuffles, caching and skew
• Warehouse query tuning
• Pipeline and dataflow speed
• Eventstream and eventhouse throughput

You build A before-and-after tuning report for a slow lakehouse load.

Everything meets in a single working solution. You ingest batch and streaming data, transform it, secure it, release it through a pipeline and monitor it, then defend each design choice as you would to a team.

• Scoping the solution and its sources
• Designing storage layers and loading patterns
• Building ingestion, transformation and orchestration
• Securing, releasing and monitoring it
• Reviewing the design and its trade-offs

You build A governed Fabric solution from ingestion to monitoring, released through a pipeline and fully documented.