NVIDIA Generative AI Multimodal

Each data type has its own form and its own faults. You load and preprocess text, images, audio, time series and geospatial data, and handle missing or inconsistent samples across modalities.

• Representing text, images and audio
• Time series and geospatial data
• Preprocessing for each modality
• Aligning samples across modalities
• Missing and inconsistent data
• Curating a multimodal dataset

You build A curated paired dataset of images and captions, with a data-quality report.

To compare a sentence with a photo, both must live in the same space. You learn how encoders embed each modality, how contrastive training such as CLIP aligns them, and how to test the result.

• Encoders for text and images
• Joint embedding spaces
• Contrastive learning and CLIP
• Zero-shot image classification
• Image–text retrieval
• Limits of joint embeddings

You build A text-to-image search over a small collection, using joint embeddings.

Attention is how a model relates a word to a region of an image. You learn self-attention and cross-attention, read attention maps to see what a model focused on, and use them to analyze data.

• Self-attention and cross-attention
• Vision transformers and patches
• Combining modalities in one model
• Plotting attention maps
• What attention maps can and cannot show
• Using maps to check data and model behavior

You build Attention maps for a set of image–text pairs, with notes on what they reveal.

Multimodal training breaks in ways single-modality training does not. You choose loss functions that combine modalities, keep training stable, and reuse pretrained models so you start from knowledge instead of from scratch.

• Multimodal and combined loss functions
• Exploding and vanishing gradients
• Learning-rate schedules and warm-up
• Balancing modalities during training
• Transfer learning and freezing layers
• Fine-tuning a pretrained multimodal model

You build A fine-tuned image–text model with a log of stability problems and their fixes.

Image generators learn to remove noise step by step. You build a U-Net, understand the forward and reverse diffusion process, and see how text conditioning steers what the model draws.

• The U-Net architecture
• Adding noise: the forward process
• Learning to denoise: the reverse process
• Time-step and text conditioning
• Sampling and the number of steps
• Guidance and its effect on output

You build A small U-Net that denoises images, with samples from each stage of training.

Text steers image generation through a shared embedding. You see how CLIP guides generation, write prompts that control subject, style and composition, and compare results across settings.

• How CLIP conditions an image model
• Prompt structure for images
• Style, composition and negative prompts
• Seeds, guidance scale and repeatability
• Evaluating generated images
• Common failure modes

You build A prompt set and results grid showing how each setting changes a generated image.

A multimodal model needs checks a text model does not. You run experiments, test quality and consistency across modalities, and use multimodal models themselves to explain what another model did.

• Designing experiments for multimodal models
• Checking data quality across modalities
• Metrics for captioning, retrieval and generation
• Consistency between text and image outputs
• Explainability for multimodal models
• Using a multimodal model to explain results

You build A test plan and results report for one multimodal model.

Efficiency matters once a model works. You tune hyperparameters, apply transfer learning and training improvements, and use NeMo to build and Triton to serve a multimodal pipeline.

• Hyperparameter search for multimodal training
• Reducing training and inference cost
• Mixed precision
• Building with NeMo
• Serving a pipeline with Triton
• Measuring accuracy against speed

You build A tuned multimodal pipeline served through Triton, with an accuracy and speed comparison.

Generated images and mixed data raise their own questions of fairness and privacy. You review bias in training data, consent and privacy, energy use, and the choices that make a system more ethical and less wasteful.

• Bias in image and text data
• Consent and privacy for images and voices
• Misuse, provenance and labeling of generated content
• Measuring and reducing energy use
• Fairness checks for generated output
• Documenting a system’s limits

You build A responsibility review of a generative image system, with bias, privacy and energy notes.

Build and explain one multimodal system. You prepare mixed data, train or tune a model, generate from prompts, test it across modalities, and review it for fairness and cost.

• Choosing a task and preparing data
• Training or tuning the model
• Generating and prompting
• Testing and explaining the results
• Reviewing fairness and cost

You build A small text-to-image system with data notes, tests and a responsibility review.