AWS Generative AI Developer

Prompts are production code that nobody reviews unless you make them. You write system prompts and templates that hold across inputs, then put them under version control, approval and rollback so a wording change cannot quietly break an application.

• System prompts, roles and constraints
• Few-shot examples and step-by-step reasoning prompts
• Prompt templates with typed variables
• Versioning, reviewing and approving prompts
• Managing prompts centrally across teams
• Rolling back a prompt that regressed

You build A versioned prompt library with an approval flow and a rollback path.

Retrieval is only as good as the documents behind it. You build pipelines that check source data, strip noise, split text sensibly and attach the metadata that later filtering depends on, including images and scanned files.

• Data quality checks before anything is indexed
• Extracting text from PDFs, scans and images
• Chunking strategies and chunk size trade-offs
• Metadata that supports filtering and access control
• Batch and event-driven ingestion pipelines
• Handling updates and deletions in source data

You build An ingestion pipeline that validates, cleans and chunks a mixed document set.

Embeddings turn text into numbers you can search by meaning. You choose an embedding model, then design a vector store around it: index structure, similarity metric, filters and the sync jobs that keep results current.

• How embeddings capture meaning
• Choosing an embedding model and dimension size
• Vector stores on AWS: OpenSearch and PostgreSQL with pgvector
• Similarity metrics and approximate search
• Metadata filtering and tenant isolation
• Keeping an index in sync with its source

You build A vector store design with an index schema, a sync job and a tenant filter.

RAG decides what a model gets to read before it answers. You compare managed knowledge bases with custom retrieval, then improve results with query rewriting, hybrid search and reranking, and make every answer cite what it used.

• Managed knowledge bases versus custom retrieval
• Query rewriting and decomposition
• Hybrid keyword and vector search
• Reranking retrieved passages
• Grounding answers with citations
• Handling questions the sources cannot answer

You build A RAG application with hybrid retrieval, reranking and cited answers.

Agents let a model call your systems, which is where most production risk sits. You design tools a model can use correctly, orchestrate multi-step work, and control how much an agent may do before a person checks.

• When an agent is the right design, and when it is not
• Defining tools and action groups with clear schemas
• Model Context Protocol for sharing tools between agents
• Orchestrating multi-step work with Step Functions
• Session state, memory and context limits
• Approval steps and timeouts for risky actions

You build An agent that calls three tools, with retries and a human approval step.

A model becomes useful when other systems can call it safely. You decide between managed and self-hosted deployment, expose models through APIs with streaming and quotas, and connect them to enterprise data and event flows.

• Managed models versus SageMaker endpoints
• Importing and deploying customized models
• Exposing a model through API Gateway and Lambda
• Streaming responses to clients
• Asynchronous patterns with queues and events
• Throttling, quotas and retries against model APIs
• Connecting to enterprise data sources and identity

You build A GenAI API with streaming, per-client quotas and an asynchronous fallback.

Anything a user types can attack the model, and anything it says can mislead the user. You configure content filters, topic limits, sensitive-data handling and grounding checks, then test them against real attack patterns.

• Content filters and denied topics
• Detecting and masking personal data in prompts and answers
• Prompt injection and jailbreak patterns
• Contextual grounding checks against hallucination
• Validating structured output before using it
• Red-teaming a guardrail configuration

You build A guardrail configuration tested against injection, jailbreak and personal-data cases.

GenAI moves sensitive data in new ways and makes decisions people will question. You lock down model access, keep traffic and logs private, record which model and prompt produced which result, and test for bias and explain behavior to users.

• Least-privilege access to models and knowledge bases
• Private connectivity and encryption for prompts and logs
• Finding and redacting sensitive data in source corpora
• Invocation logging and audit trails
• Bias, fairness and transparency in outputs
• Documenting models, prompts and approvals
• Mapping an application to regulatory duties

You build A privacy, audit and responsible AI plan for a workload that handles personal data.

Tokens are the bill and the delay. You measure where both go, then cut them with smaller prompts, caching, batching, model choice and capacity planning, without letting answer quality fall below what the task needs.

• Counting tokens: where cost and delay come from
• Shorter prompts and context compression
• Prompt caching and semantic caching
• Batch inference for work that can wait
• Right-sizing models and routing by difficulty
• Streaming and parallel calls for perceived speed
• Provisioned capacity versus pay per token

You build A cost and latency profile of one application, with three measured savings.

Generative systems fail quietly: answers get worse and nobody sees an error. You instrument every call, track tokens, latency and quality signals, and build alarms and dashboards that show trouble before users report it.

• Metrics worth tracking: tokens, latency, errors and throttles
• Invocation logs and what belongs in them
• Tracing a request through retrieval, model and tools
• Quality signals: user feedback, refusals and groundedness
• Alarms and dashboards in CloudWatch
• Spotting drift after a model or data change

You build A dashboard and alarm set covering cost, latency, errors and answer quality.

Without a test set you cannot tell whether a change helped. You build golden datasets, apply automated metrics, model-based judges and human review, and wire the results into a gate that blocks regressions before release.

• Building a golden dataset from real questions
• Automated metrics and their blind spots
• Using a model as a judge, and checking the judge
• Human review workflows and rating guides
• Evaluating retrieval separately from generation
• Comparing models and prompts side by side
• Release gates and regression tests in CI

You build An evaluation suite that scores a RAG application and blocks regressions.

Most GenAI faults have recognizable causes. You learn to trace a bad answer back to retrieval, prompt, model or tool, and to resolve throttling, truncation, timeouts and runaway agent loops.

• Tracing a bad answer to its cause
• Retrieval misses and stale indexes
• Hallucination despite good context
• Throttling, timeouts and quota errors
• Truncated or malformed output
• Agent loops and failed tool calls
• Latency spikes under load

You build A troubleshooting runbook that maps failure symptoms to causes and fixes.

Everything meets in one system. You scope a real use case, choose models, retrieval and agent design, build in guardrails, evaluate and deploy with monitoring, then defend your choices on quality, cost, safety and governance.

• Scoping the use case and its constraints
• Choosing models, retrieval and agent design
• Building with guardrails and access controls
• Evaluating and tuning against a test set
• Deploying with monitoring and cost controls
• Defending the design in review

You build A deployed, evaluated and documented GenAI application with its design record.