Databricks Generative AI Engineer

Prompts are code that nobody compiles. You write them with clear instructions, format and examples, tune them for a given model task, and weigh models by quality, latency, context size and price rather than by reputation.

• Instructions, context and output format
• Few-shot examples and when they help
• Prompting for extraction and classification
• Comparing candidate models on your own task
• Latency, context size and cost as selection criteria

You build A prompt set tested across two models on a labeled sample, with a recorded choice.

Retrieval is only as good as the chunks behind it. You parse varied source files, clean and filter them, choose a chunking approach that keeps meaning together, and write the results into governed Delta tables.

• Choosing a document parsing package
• Cleaning and filtering extracted text
• Chunking: size, overlap and structure-aware splits
• Attaching metadata to chunks
• Writing chunks to Delta tables
• Refreshing chunks when sources change

You build A document-preparation pipeline that turns raw files into a governed table of chunks.

Searching by meaning needs the right embedding model and a well-built index. You create a vector index over your chunk table, combine retrieval with re-ranking, and judge results with metrics rather than a few good-looking examples.

• Choosing an embedding model
• Creating and syncing a vector index
• Similarity search with metadata filters
• Re-ranking retrieved results
• Retrieval metrics: recall, precision and ranking
• Diagnosing poor retrieval

You build A vector index with re-ranking and a retrieval report on a set of test questions.

You write the application itself: chains in LangChain, agents with tool calling, and the order in which tools run for multi-step reasoning. MLflow and the Agent Framework make the pieces loggable, testable and shippable.

• Composing chains with LangChain
• Tool calling and ordering tools for multi-step tasks
• Agent Framework and Agent Bricks compared
• Connecting external tools through MCP
• Logging a chain or agent as an MLflow model
• Keeping state and context across turns

You build A retrieval agent with two tools, logged as an MLflow model and tested end to end.

Users and documents can both carry attacks. You add input and output guardrails, defend against prompt injection and jailbreak attempts, and test the defenses with adversarial examples instead of trusting them.

• Threats: prompt injection and jailbreaks
• Input filters and output checks
• Separating instructions from untrusted content
• Refusals and safe fallback answers
• Red-teaming your own application

You build A guardrail layer tested against a set of adversarial prompts.

What goes in comes out. You mask sensitive fields before they reach a model or an index, check that source data may legally be used, and screen documents for toxic or misleading text before it shapes answers.

• Finding and masking sensitive data
• Unity Catalog access for source tables and indexes
• Licensing and permitted use of source data
• Screening for problematic text
• Keeping an audit trail of what was used

You build A governed source table with masked fields and a documented licensing review.

A working notebook is not yet an application. You package a chain as a model, register it in Unity Catalog, serve it with its vector index, manage prompts like code, and expose it through a user interface and a CI/CD flow.

• Packaging a chain as a model with its dependencies
• Registering and versioning in Unity Catalog
• Deploying to Model Serving with a vector index
• Prompt management and versioning
• CI/CD for generative AI applications
• Putting a user interface in front of an endpoint

You build A served application with versioned prompts and an automated deployment flow.

A demo proves little. You choose metrics that fit the task, build judges and scorers for quality that cannot be counted, trace every call with MLflow, and use the results to compare versions before release.

• Choosing metrics for generation and retrieval
• LLM judges and custom scorers
• Building an evaluation dataset
• MLflow tracing across a chain
• Comparing two versions of an application
• Evaluating agents, not only single answers

You build An evaluation suite with judges and traces that scores each application version.

After launch the real questions arrive. You log inference to tables, track cost and token use, apply limits and routing through AI Gateway, and watch live endpoints for drift in quality, latency and spend.

• Inference tables for requests and responses
• Tracking token use and cost
• Rate limits and routing in AI Gateway
• Watching latency and error rates on endpoints
• Feeding production data back into evaluation

You build A monitoring setup with inference logging, cost limits and quality alerts.

You build one application across the whole lifecycle: design, data preparation, retrieval, an agent, guardrails, deployment and monitoring. Then you explain why each choice was made and what it cost.

• Scoping the use case and its risks
• Preparing data and building retrieval
• Assembling the agent and its guardrails
• Deploying, evaluating and monitoring it
• Presenting the design and its trade-offs

You build A deployed retrieval agent with governance, evaluation and monitoring documented.