Databricks Developer for Apache Spark

Nothing runs until an action asks for a result. You separate transformations from actions, follow how a query becomes jobs, stages and tasks, and read an execution plan to predict what Spark will do.

• Transformations and actions
• Narrow and wide transformations
• From a query to jobs, stages and tasks
• Reading explain output and the Spark UI
• Why a program can look fast and still be slow

You build A short program with its plan explained: which steps shuffle and why.

Most Spark work begins and ends with files and tables. You read and write formats through JDBC and files, choose save modes, store persistent tables, and query files directly or through temporary views.

• Reading and writing Parquet, JSON and CSV
• Schemas: inferred and declared
• JDBC reads and writes
• Save modes and their effects
• Persistent tables and temporary views
• Querying data files directly with SQL

You build A job that reads three formats, saves a table in each save mode and exposes it as a view.

The DataFrame API is a vocabulary you use every day. You select, rename, cast and compute columns, filter rows and sort them, and learn the habits that keep expressions readable and correct.

• Selecting, renaming and dropping columns
• Column expressions and casting
• Conditional logic with when and otherwise
• Filtering and sorting rows
• Handling nulls and missing values

You build A clean-up job that reshapes a raw DataFrame into an analysis-ready one.

Combining and summarizing data is where results go wrong quietly. You join and union DataFrames, aggregate, deduplicate, and use window functions, checking row counts so mistakes show up early.

• Join types and their results
• Union and combining schemas
• Grouping and aggregate functions
• Deduplication: dropDuplicates and its limits
• Window functions for ranking and running totals

You build A report built from three DataFrames with each step’s row count checked.

Real data holds timestamps, messy text and nested structures. You parse and format dates, work with string functions, and handle arrays, maps and structs, including exploding and flattening them.

• Parsing and formatting dates and timestamps
• Date arithmetic and time zones
• String functions and regular expressions
• Arrays, maps and structs
• Exploding and flattening nested data

You build A parser that turns nested JSON events into flat, typed columns.

Built-in functions cover most needs, and the rest require your own. You write user-defined functions and weigh their cost, share read-only data with broadcast variables, and count events across tasks with accumulators.

• When a UDF is needed and when it is not
• Writing and registering a UDF
• The performance cost of Python UDFs
• Broadcast variables for shared lookups
• Accumulators and their caveats

You build A transformation using a UDF, then rewritten with built-ins, with timings compared.

Performance in Spark is mostly about data movement. You control partition counts with repartition and coalesce, decide what to cache and for how long, and learn what causes a shuffle so you can avoid it.

• Repartition against coalesce
• What triggers a shuffle
• Caching and persistence levels
• Broadcast joins to avoid shuffles
• Partitioning on write

You build A job tuned from many shuffles to few, with before and after stage timings.

Slow jobs leave evidence. You explain how adaptive query execution rewrites plans at run time, spot and treat skewed keys, and read driver and executor logs to diagnose memory and utilization problems.

• What adaptive query execution changes
• Spotting data skew in stages
• Mitigating skew
• Driver and executor logs
• Memory errors: causes and remedies

You build A diagnosis of a skewed job, with the change that evened out its tasks.

Streaming applies the same DataFrame ideas to data that never stops. You explain micro-batch processing and exactly-once behavior, write streaming DataFrames, choose output modes, and deduplicate with and without watermarks.

• Micro-batches and the streaming model
• Sources, sinks and checkpoints
• Output modes: append, update and complete
• Watermarks and late data
• Deduplicating a stream

You build A streaming job that deduplicates events and aggregates them over time windows.

How an application reaches a cluster has changed. You describe Spark Connect and its client-server design, and compare client, cluster and local deployment modes to choose where an application should run.

• The Spark Connect client and server
• What Spark Connect changes for applications
• Local, client and cluster modes
• Where the driver runs in each mode
• Choosing a mode for an application

You build The same application run under two deployment modes, with a note on the differences.

Many teams think in pandas. You see why and when the pandas API on Spark is a good fit, and create and call pandas UDFs that process data in vectorized batches instead of row by row.

• Why use the pandas API on Spark
• Differences from single-machine pandas
• Converting between Spark and pandas
• Writing a pandas UDF
• Series and grouped pandas UDFs

You build A pandas-style analysis scaled across a cluster, plus a pandas UDF feature.

You build one application that reads several sources, transforms them with DataFrames, runs a streaming leg, and is tuned from a first slow run. Then you explain what Spark does at each step.

• Reading and modeling the input
• Transforming and joining the data
• Adding a streaming leg
• Profiling and tuning the whole job
• Explaining the final plan

You build A batch and streaming application with a written tuning log.