NVIDIA AI Operations

Base Command Manager is the starting point for many NVIDIA clusters. You install it, define node categories and software images, provision nodes over the network, and bring them in as one managed system.

• Head node installation and network setup
• Node categories and software images
• Provisioning and re-imaging nodes
• Cluster roles and services
• High availability for the head node
• Updating images and rolling them out

You build A provisioned cluster with two node categories and a rollback path.

Containers are how AI software ships. You pull and verify images from NGC, run them with GPU access, manage storage and networking for them, and learn the faults that appear when a container and a driver disagree.

• NGC catalog and authentication
• Running GPU containers with Docker
• Image storage, caching and registries
• Volumes, networks and shared memory settings
• Driver and CUDA compatibility with images
• Container security basics

You build A set of verified NGC containers running on GPU nodes with shared storage.

Kubernetes schedules a growing share of AI work. You set up a cluster for GPUs, expose GPUs to pods, label and taint nodes, and administer namespaces, quotas and storage for several teams.

• Cluster components and node roles
• GPU Operator and device plugin
• Labels, taints and scheduling
• Namespaces, quotas and access control
• Persistent storage for datasets and checkpoints
• Running jobs and long-lived services

You build A Kubernetes cluster that schedules GPU pods across namespaces with quotas.

BlueField cards can run services beside the host. You deploy DOCA services onto them, manage their lifecycle, and learn what to monitor so the network side of a cluster stays healthy.

• DOCA software and platform overview
• Deploying services to a BlueField card
• Managing service lifecycle and updates
• Networking and security offloads
• Monitoring DPU services
• Finding out when a DPU is the cause

You build A DOCA service deployed to a BlueField card and monitored.

Slurm still shares most large clusters. You configure partitions, accounts and limits, tune scheduling for GPU jobs, and read accounting data to keep every user fairly served.

• Controller, nodes and configuration files
• Partitions, QOS and limits
• GPU resources and gres configuration
• Accounts, fair share and priorities
• Prolog, epilog and node health
• Reading accounting and queue data
• Draining and returning nodes

You build A Slurm cluster with partitions, accounts and fair-share rules for two teams.

Run:ai lets teams share GPUs by policy rather than by hand. You set up projects and quotas, tune over-quota and fractional use, and read utilization to show where capacity goes.

• Run:ai architecture on Kubernetes
• Departments, projects and quotas
• Scheduling policies and priorities
• Fractional GPUs and over-quota use
• Workloads: training, inference and interactive
• Utilization reports

You build A Run:ai setup with projects, quotas and a utilization report.

MIG lets one GPU serve several users with hard isolation. You decide when partitions beat whole GPUs, create instances under Slurm and Kubernetes, and keep layouts consistent across nodes.

• When partitions beat whole GPUs
• Profiles for training, inference and HPC
• MIG under Slurm and Kubernetes
• Changing layouts across many nodes
• Monitoring and accounting for instances
• Limits and known trade-offs

You build A MIG layout used by both a Slurm job and a Kubernetes pod.

You cannot manage what you cannot see. You read GPU health and utilization with nvidia-smi, DCGM and NVSM, export metrics to dashboards, and set alerts that point at the failing component.

• nvidia-smi queries and fields
• DCGM health checks and diagnostics
• NVSM on DGX systems
• Exporting metrics and building dashboards
• Alerting on thermal, ECC and link faults
• Correlating GPU and workload data

You build A dashboard with alerts for GPU health across a cluster.

Clusters grow and age. You add and retire nodes, update software images and firmware in waves, manage remote edge sites with Fleet Command, and keep configuration identical across hundreds of machines.

• Adding and removing nodes
• Rolling updates for images and drivers
• Firmware management at scale
• Configuration drift and how to detect it
• Managing edge sites with Fleet Command
• Backups and recovery of cluster state

You build A written lifecycle plan with a tested rolling update.

Faults in the lower layers show up as slow or failing jobs. You diagnose Docker and runtime problems, the NVLink and NVSwitch fabric manager, and the communication libraries in the Magnum IO family.

• Docker and container runtime failures
• Fabric manager states and NVLink errors
• NCCL hangs and slow collectives
• GPUDirect RDMA and GPUDirect Storage issues
• Reading logs across layers
• A method for isolating the layer at fault

You build A troubleshooting guide with one reproduced fault for each layer.

Storage and the cluster manager are shared dependencies, so their faults hit everyone. You trace slow file systems, mount problems and provisioning failures to their cause and recover without losing running work.

• Slow or stalled shared file systems
• Mount, permission and quota problems
• Provisioning and imaging failures
• Services and health checks in the cluster manager
• Recovering a node or a head node
• Writing incident notes

You build A recovery playbook for storage and provisioning incidents.

Operate a cluster you built end to end. You deploy the stack, onboard teams under Slurm and Kubernetes, then work through injected faults in containers, fabric and storage, fixing each one while jobs keep running.

• Deploying the stack from scratch
• Onboarding two teams with fair sharing
• Finding faults from symptoms alone
• Fixing without stopping other work
• Writing the post-incident review

You build A working multi-team AI cluster, with a post-incident review of the faults you fixed.