NVIDIA AI Infrastructure

A cluster fails first at the facility. You check power budgets, circuit redundancy and airflow or liquid cooling before the hardware arrives, and learn the limits that stop a dense server from running at full speed.

• Rack power budgets and PDU capacity
• Redundant feeds and failure behavior
• Air cooling, containment and airflow direction
• Liquid cooling basics and leak response
• Environmental monitoring and thresholds
• Racking, rails and weight limits

You build A readiness checklist for one rack, with power and cooling calculations.

Every server starts with out-of-band management. You set up the BMC, secure it, bring firmware to a consistent level, and prepare each machine so it can be controlled remotely and reproduced exactly.

• First power-on and BMC network setup
• Redfish and IPMI for remote management
• Securing management interfaces and accounts
• TPM and secure boot settings
• Firmware inventory and update order
• BIOS settings for GPU servers
• Recording a golden configuration

You build A scripted bring-up that leaves a server at a known firmware and BIOS baseline.

Many early cluster faults are cable faults. You choose and seat copper, fiber and transceivers correctly, map every port against the design, and verify link state, speed and error counters before moving on.

• Copper, active optical and fiber links
• OSFP and QSFP transceivers and compatibility
• Port maps and labeling conventions
• Checking link speed, state and negotiation
• Reading error counters and signal quality
• Cable handling and cleaning

You build A verified port map and link report for a small rack-to-switch layout.

BlueField cards do more than move packets. You set their operating mode, install DOCA, configure networking and storage offloads, and manage each card like the small computer it is.

• DPU mode and NIC mode
• Installing DOCA and card firmware
• Configuring interfaces and virtual functions
• Offloading network and storage tasks
• Managing the DPU’s own operating system
• Monitoring a card’s health

You build A BlueField card configured, updated and verified in both operating modes.

Multi-Instance GPU turns one GPU into several isolated ones. You plan partitions around workloads, create and destroy instances, and check that they behave correctly once the cluster software sees them.

• What MIG isolates and what it shares
• Supported profiles and how they combine
• Enabling MIG and creating instances
• Exposing instances to containers and schedulers
• Reconfiguring without disturbing running jobs
• Monitoring MIG instances

You build A MIG layout for a mixed workload, built and verified on a GPU.

Hardware becomes a cluster once the software agrees with itself. You install the operating system, drivers, CUDA libraries, networking stack and container toolkit, and learn how mismatched versions show up as faults.

• Operating system and kernel choices
• GPU driver and CUDA installation
• DOCA and the network driver stack
• NVIDIA Container Toolkit
• Fabric manager for NVLink systems
• NGC CLI and pulling containers
• Version compatibility across the stack

You build A repeatable node image with drivers and container tools that passes a smoke test.

A cluster manager provisions nodes and a scheduler shares them. You install Base Command Manager, define node categories and images, then configure Slurm partitions and GPU resources so that users can submit jobs.

• Base Command Manager installation
• Node categories, images and provisioning
• Head nodes and high availability
• Slurm configuration and partitions
• GPU resources and job accounting
• Submitting a first multi-node job
• Health checks run on every node

You build A small cluster provisioned by Base Command Manager that runs a multi-node Slurm job.

Before nodes join a cluster, each one must hold up alone. You stress GPUs, memory and CPUs, run NVIDIA’s diagnostics, and watch temperatures and clocks to catch weak parts early.

• Running DCGM diagnostics
• GPU and memory stress tests
• NVLink and PCIe bandwidth checks
• Watching thermals, clocks and power draw
• Reading Xid errors and logs
• Writing an acceptance checklist per node

You build A per-node acceptance test script with a written report.

A cluster is only as fast as its slowest link. You run HPL and NCCL tests, measure east–west bandwidth with ClusterKit, and check signal quality and firmware consistency across every node.

• Running and tuning HPL
• NCCL tests and bus bandwidth
• ClusterKit and pairwise node tests
• Measuring east–west bandwidth
• Signal, link and firmware consistency checks
• Interpreting results against expected figures

You build A cluster verification report with NCCL and HPL results compared with expected values.

Early failures hide in the first days of heavy use. You design a burn-in that loads compute, memory and network together for long periods, set the criteria that decide acceptance, and record what you found.

• Designing a burn-in plan
• Running combined compute and network load
• Capturing logs and counters during the run
• Acceptance criteria and sign-off
• Reading failure patterns
• Handing the cluster over with records

You build A burn-in plan with acceptance criteria and a results log.

Faults rarely announce themselves clearly. You trace failing GPUs, network cards, power supplies and cables from symptoms to parts, replace them safely, and confirm that the node is healthy again.

• From symptom to suspect: GPU errors and Xid codes
• Diagnosing network cards and cables
• Power supply and fan failures
• Safe replacement and reflash procedure
• Validating a node after repair
• Keeping a fault and repair history

You build A fault-finding runbook with worked cases for GPUs, cards and power supplies.

Healthy hardware can still be slow. You tune BIOS settings, CPU and memory affinity, GPU-to-NIC paths and storage access so data reaches the GPUs without waiting, and you measure the gain.

• NUMA layout and CPU affinity
• GPU-to-NIC paths and GPUDirect RDMA
• Storage protocols and parallel file systems
• Local cache and dataset staging
• Measuring before and after a change
• Spotting the bottleneck against a baseline

You build A before-and-after tuning report for one node and its storage path.

Work through a whole build. You plan the rack, bring up servers, install the control plane and verify the cluster, then break something on purpose, fix it, and hand the cluster over with complete records.

• Planning from a specification
• Bring-up and cabling checks
• Installing the control plane
• Verification and burn-in
• Fault finding under a time limit
• Writing the handover pack

You build A verified small AI cluster with its test results and handover documentation.