Databricks Machine Learning

Models inherit every flaw in their data. You compute summary statistics, plot distributions and relationships, and learn which patterns, such as skew or a heavy tail, should change how you prepare the features that follow.

• Summary statistics on Spark and pandas DataFrames
• Distributions, skew and when a log scale helps
• Spotting outliers visually and numerically
• Correlations and relationships between features
• Checking class balance in a target

You build An exploration notebook that lists the data problems a model would inherit.

Algorithms need numbers without gaps. You remove or cap outliers, impute missing values in a way that suits each column, and encode categories so that the model sees them correctly, avoiding the leaks that careless preparation causes.

• Removing and capping outliers
• Imputing missing values: mean, median and mode
• One-hot encoding and other category encodings
• Scaling numeric features
• Fitting on training data only, to avoid leakage

You build A preparation pipeline that cleans, imputes and encodes a messy data set reproducibly.

Features that are computed once and reused save effort and prevent drift between training and serving. You create feature tables with keys, write and update them, and look them up when you build a training set.

• Why teams keep features in tables
• Creating a feature table with primary keys
• Writing and updating feature values
• Joining features to labels into a training set
• Finding and reusing existing features

You build A feature table used to assemble a training set for two different models.

You match an algorithm to the problem: classification, regression or clustering. Then you chain preprocessing and a model into a pipeline, so that the same steps run in training and in prediction, using SparkML or scikit-learn as the job requires.

• Classification, regression and clustering: matching the problem
• Linear models, trees and ensembles compared
• Chaining steps into a pipeline
• Train, validation and test splits
• SparkML or scikit-learn: which and why

You build A pipeline that prepares data and trains a model, tested on a held-out split.

Default settings rarely give the best model. You define a search space, run Hyperopt or grid search, validate each candidate with cross-validation, and read the results without fooling yourself about how much improvement is real.

• Hyperparameters and what they control
• Grid, random and Bayesian search
• Defining a search space in Hyperopt
• Cross-validation and what it protects against
• Running trials in parallel on a cluster

You build A tuned model whose search is logged and compared with the untuned baseline.

A metric is a promise about what matters. You compare classification metrics such as precision, recall and AUC, and regression metrics such as RMSE and R squared, and choose the one that matches the cost of being wrong.

• Confusion matrices and what they reveal
• Precision, recall, F1 and ROC AUC
• Regression error metrics: MAE, RMSE and R squared
• Imbalanced classes and misleading accuracy
• Matching the metric to a business cost

You build An evaluation report that justifies the chosen metric for a stated business cost.

Without a record you cannot reproduce a result. You log parameters, metrics and models with MLflow, register the best one in Unity Catalog, and promote it with an alias such as champion rather than copying files.

• MLflow tracking: experiments, runs and autologging
• Comparing runs in the UI and in code
• Registering a model in Unity Catalog
• Versions and aliases: promoting a champion
• Loading a registered model for prediction

You build A tracked experiment with a registered model and a champion alias.

How a model is used decides how it is deployed. You compare batch scoring, real-time endpoints and streaming inference on latency, cost and freshness, and choose the one that suits a use case.

• Batch inference on a table
• Real-time inference behind an endpoint
• Streaming inference on arriving data
• Latency, throughput and cost compared
• Choosing a mode for a stated use case

You build A decision table matching three use cases to inference modes, with one batch scoring job.

A model starts to matter when something calls it. You create a Model Serving endpoint, query it from SQL and code, and split traffic between two versions to compare them safely.

• Creating a serving endpoint
• Querying an endpoint from code and SQL
• Serving two versions behind one endpoint
• Splitting traffic by percentage
• Checking that predictions look right

You build A serving endpoint that splits traffic between two model versions and answers queries.

You carry one problem through the whole path: explore and prepare data, train and tune a model, track and register it, and serve it. Then you explain each choice in plain terms to someone who has to rely on it.

• Framing the problem and the target
• Preparing features and splitting data
• Training, tuning and evaluating
• Registering and serving the model
• Explaining the result and its limits

You build A served model with a documented pipeline, metrics and known limitations.