Microsoft Machine Learning Operations Engineer

Machine learning workspaces touch data, compute and registries, so their access needs design. You assign roles, use managed identities, keep keys out of code, and place the workspace on private networking.

• Roles for data scientists, engineers and pipelines
• Managed identities for compute and endpoints
• Private endpoints and managed virtual networks
• Storage and registry access without shared keys
• Key Vault for the secrets that remain
• Auditing workspace activity

You build A locked-down workspace on private networking, with a role matrix for each team.

Reproducible training depends on versioned inputs. You register data assets, define environments as code, and choose compute that matches the job, so that any experiment can be rerun later by someone else.

• Datastores and data assets
• Versioning the data used in training
• Environments from images and conda files
• Compute instances, clusters and serverless compute
• Cost control: scaling to zero and quotas
• Reusing assets across workspaces with registries

You build Versioned data and environment assets, with a compute cluster that scales to zero.

Experiments are only useful if they can be compared. You log parameters, metrics and artifacts with MLflow, compare runs, and keep a record that explains why one model was chosen over another.

• MLflow runs, experiments and tracking
• Autologging and custom metrics
• Logging artifacts, plots and models
• Comparing runs and choosing a model
• Hyperparameter sweeps
• Tagging runs for traceability

You build A tracked series of experiments, with a comparison that justifies the chosen model.

A training script becomes a process when it runs as a pipeline. You split training into components, run them as pipelines, schedule and trigger runs, and cache steps so only changed work reruns.

• Components, inputs and outputs
• Pipeline jobs with the Python SDK and CLI
• Scheduling and event triggers
• Caching and reuse of steps
• Distributed training basics
• Debugging a failed pipeline step

You build A multi-step training pipeline that prepares data, trains and evaluates, and runs on a schedule.

A model in a notebook is not a deliverable. You register models with their lineage, version them, record the metrics they were chosen on, and control which versions may move toward production.

• The model registry and model versions
• MLflow models and signatures
• Linking a model to its data, code and run
• Tags and approval before release
• Sharing models across workspaces
• Model cards and documentation

You build A registry entry with lineage and metrics, and a documented approval path.

Serving a model is a release like any other. You deploy to managed online endpoints for real-time use and to batch endpoints for bulk scoring, split traffic between versions, and roll back fast.

• Managed online and batch endpoints
• Scoring scripts and no-code deployment
• Blue/green deployment with traffic splitting
• Testing a deployment before it takes traffic
• Autoscaling and instance sizing
• Rolling back to the previous version

You build An online endpoint with two deployments, a traffic split and a rehearsed rollback.

Models decay as the world changes. You monitor data drift, prediction drift and model quality in production, set alerts, and trigger retraining through the same pipeline that built the model.

• Data drift, prediction drift and concept drift
• Configuring model monitoring
• Collecting production data
• Alert thresholds and response
• Retraining triggers and automated pipelines
• Deciding when not to retrain

You build A monitor on a deployed model that raises an alert and starts a retraining pipeline.

Generative AI work has its own platform setup. You configure Microsoft Foundry projects, deploy foundation models, size them for throughput and cost, and keep environments consistent through code.

• Foundry resources and projects
• Choosing and deploying a foundation model
• Deployment types, capacity and quotas
• Sizing for throughput and token cost
• Provisioning Foundry with Bicep and the CLI
• Identity and private networking for Foundry

You build A Foundry project and model deployment defined as code, with capacity sized for a stated load.

Prompts change behavior as much as code does. You store them in Git, version and review them, test each change in a pipeline, and promote prompt versions through environments together with the app that uses them.

• Prompts as files: structure and templates
• Versioning and review in pull requests
• Testing prompt changes automatically
• Promoting prompts across environments
• Separating prompt, model and parameter changes
• Rolling back a prompt

You build A repository of versioned prompts, with a GitHub Actions check that evaluates every change.

Generative output cannot be checked by equality. You evaluate groundedness, relevance and safety with evaluators and datasets, then trace live traffic and watch latency, throughput and token cost.

• Evaluation datasets and reference answers
• Groundedness, relevance and coherence evaluators
• Safety evaluations and red teaming
• Tracing requests through a generative app
• Monitoring latency, throughput and token cost
• Setting quality gates for release
• Investigating a regression in production

You build An evaluation pipeline with quality gates, and a dashboard of latency, throughput and token cost.

Most RAG problems are retrieval problems. You change chunking, embedding models and search configuration, measure the effect on retrieval and on answers, and choose the settings the data justifies.

• Chunking strategies and chunk size
• Choosing and comparing embedding models
• Hybrid search and reranking
• Metadata filters and query rewriting
• Measuring retrieval separately from generation
• Reindexing without downtime

You build A tuning experiment that compares chunking and search settings, with retrieval measurements.

Fine-tuning pays off when prompting and retrieval fall short. You prepare training data, including synthetic data, run fine-tuning jobs, evaluate the result against the base model, and manage the tuned model through its lifecycle.

• When fine-tuning is the right answer
• Preparing and validating training data
• Generating synthetic training data
• Running and tracking a fine-tuning job
• Comparing a tuned model with its base
• Deploying, versioning and retiring tuned models

You build A fine-tuned model with a comparison against its base model and a deployment plan.

Combine classic and generative operations in one system. You provision the platform as code, train and deploy a model, ship a generative app beside it, and show that both are tested, monitored and reversible.

• Designing the platform and its environments
• Automating infrastructure and releases
• Running a model through its full lifecycle
• Running a generative app through its full lifecycle
• Reviewing monitoring, evaluation and rollback

You build A repository that provisions, deploys, evaluates and monitors both a model and a generative app.