Revision history for Rex-GPU

0.004     2026-10-01 16:14:46Z
  - Not run on GPU hardware yet; checked against the commands sent to the
    host.

  [CDI]
  - generate_cdi_specs checks with `nvidia-ctk cdi list` that
    nvidia.com/gpu=all and management.nvidia.com/gpu=all resolve, and
    writes only what is missing under /etc/cdi: nvidia.yaml, and
    management.nvidia.yaml via --mode=management
    --vendor=management.nvidia.com --class=gpu. Fixes hosts without
    nvidia-cdi-refresh (DGX OS / GB10, toolkit 1.19), where /run/cdi is
    empty after a reboot and the GPU Operator validator failed on
    "unresolvable CDI devices management.nvidia.com/gpu=all". Also in the
    "toolkit already present" case. Specs that resolve are left alone
    (also nvidia-cdi-refresh's under /run/cdi); next to nvidia-cdi-refresh
    no static nvidia.yaml is written. A CLI without the management mode,
    or without `cdi list`, is a warning, not a die. An existing
    /etc/cdi/nvidia.yaml is no longer rewritten on every run (k76).

0.003     2026-10-01 04:00:20Z
  - None of the changes below has run on GPU hardware yet; they are
    checked against the commands they send to the host.

  [Driver]
  - Grace Hopper GH200 (10de:2342, 10de:2348) gets the open kernel module
    only, as Blackwell: Ubuntu -server-open, Debian 12/13 NVIDIA's CUDA
    repository, RHEL cuda-open-dkms, openSUSE its open driver.
  - Ubuntu: only the running kernel's linux-headers-$(uname -r) are
    installed, no longer linux-headers-generic, which brought headers for
    another kernel on HWE and vendor kernels (DGX Spark). A kernel
    installed later needs its own headers for DKMS.
  - Ubuntu: new opt-in setup Rex::GPU::NVIDIA::Setup::UbuntuDrivers
    (experimental): `ubuntu-drivers list --gpgpu` names the driver
    package, installed by apt-get and verified with dpkg; nothing named
    dies before any driver install. The default stays apt-cache search.

  [Detection]
  - An AMD GPU's name is read from lspci as for NVIDIA, e.g.
    "Navi 31 [Radeon RX 7900 XTX]", instead of "Unknown AMD GPU" for
    every card. Detect-only as before: AMD stays compute => 0.
  - New opt-in Rex::GPU::Detect::Sysfs (experimental): detection from
    /sys/bus/pci/devices, without lspci and without installing pciutils;
    same result shape including vgpu and nvswitch. No product names, so a
    GPU the generation table does not cover is compute => undef, not 0.
    Dies where sysfs cannot be read. Chosen with detect => or set
    gpu_detect_class; lspci stays the default.

  [API]
  - Every function of Rex::GPU, Rex::GPU::Detect and Rex::GPU::NVIDIA
    also works as a class method, and a subclass of Rex::GPU::NVIDIA or
    Rex::GPU::Detect can replace any step (experimental): chosen with
    gpu_setup(nvidia => ..., detect => ...) or set gpu_nvidia_class /
    set gpu_detect_class; one that cannot be loaded dies before the host
    is touched. Function calls, their arguments and the defaults are
    unchanged.

  [Container toolkit]
  - install_container_toolkit(binaries_suffice => 1) counts the toolkit as
    present when nvidia-container-runtime and nvidia-ctk are on the PATH and
    nvidia-ctk runs, without asking the package manager (vendor images
    such as DGX Spark); gpu_setup and the default are unchanged

0.002     2026-09-24 21:34:32Z
  - None of the changes below has run on GPU hardware yet; they are
    checked against the commands they send to the host.

  [Detection]
  - A GPU counts as compute by its PCI device ID generation, not its name:
    every Maxwell or newer NVIDIA GPU (GeForce included) now gets a driver.
    Kepler and older are compute => 0 with a warning.
  - A virtual display (virtio, QXL, VMware, VirtualBox) no longer hides a
    real GPU on passthrough hosts and cloud GPU VMs.
  - New keys: nvswitch (HGX NVSwitch bridges) and per GPU vgpu, vgpu_type,
    subsystem_vendor_id, subsystem_id (NVIDIA vGPU guests).
  - pciutils is installed only when lspci is missing, also on Rocky, Alma
    and CentOS Stream; detection dies if lspci is still missing.

  [Driver]
  - gpu_setup installs one driver that fits all compute GPUs (new option
    gpus => [...]). It dies before any driver package is installed when
    they cannot share one, for a Kepler GPU, or for a vGPU guest without a
    working driver.
  - The driver fits the GPU generation: Blackwell (B200/B300, RTX 50xx,
    RTX PRO, GB10) gets the open kernel module, on Debian 12/13 from
    NVIDIA's CUDA repository; Maxwell/Pascal/Volta stay on the proprietary
    580 branch. GB10 (DGX Spark) needs 580 or newer.
  - A host with a working driver (nvidia-smi lists a GPU, libcuda is
    present) is left alone: no reinstall, no reboot.
  - HGX hosts with NVSwitch get NVIDIA Fabric Manager; HGX B200/B300 also
    get the NVLink fabric (nvlsm, ib_umad). An existing driver gets a
    missing Fabric Manager from the host's own package sources.
  - Ubuntu: the newest nvidia-driver-NNN-server package is picked after
    apt-get update; no silent fallback to 570.
  - Debian: contrib/non-free are enabled on every Debian archive entry,
    deb822 .sources included. An own mirror counts when it is signed-by the
    Debian archive keyring.
  - RHEL family: Rocky, Alma and CentOS Stream work under their lsb_release
    names; RHEL gets EPEL and CodeReady Builder; aarch64 uses NVIDIA's sbsa
    repository; a CUDA repository that cannot be added dies early.
  - reboot => 1 waits until the host answers a command again.
  - Own driver setup: setup => 'My::Setup' or set gpu_nvidia_setup, plus
    requirement => { kernel_module, min_branch, max_branch }. Setup classes
    and Rex::GPU::NVIDIA::Requirement are experimental API; examples in
    eg/custom-setup/ and eg/ubuntu-drivers/.

  [Container toolkit, CDI, containerd]
  - install_container_toolkit skips an installed toolkit, survives the
    dpkg lock after a reboot, and dies on a failed key or repository
    download instead of installing from a broken source.
  - generate_cdi_specs leaves CDI to nvidia-cdi-refresh when the toolkit
    ships it, so nvidia.com/gpu is no longer defined twice.
  - configure_containerd adds the nvidia runtime to RKE2/K3s's own config
    instead of replacing it (SystemdCgroup, sandbox image and registry
    config were lost). A template written by 0.001 is removed with a
    warning; restart rke2/k3s to regenerate the config.
  - An unknown containerd_config dies before anything runs on the host.

  [openSUSE (unverified, unsupported)]
  - Leap 15.x gets the Leap 15 repository and G06 driver again (it got
    Leap 16's). zypper waits for the zypp lock, repository failures die,
    and every install is verified.

  [Examples]
  - eg/ keeps SSH host-key verification on and scans the new host into
    known_hosts first.

0.001     2026-03-29 04:20:30Z
  - Initial release
  - GPU hardware detection via PCI class codes (NVIDIA, AMD)
  - NVIDIA driver installation (Debian 12/13, Ubuntu 22.04/24.04, RHEL/Rocky 8/9/10, openSUSE Leap 15.6/16.0)
  - NVIDIA Container Toolkit installation with automatic repo setup
  - CDI spec generation via nvidia-ctk cdi generate
  - Containerd runtime configuration (RKE2, K3s, standalone)
  - Support for Ubuntu 24.04: auto-detect nvidia-driver-NNN-server, handle virtual nvidia-smi package
  - DPkg::Lock::Timeout=120 on all apt-get calls for resilience on fresh-boot systems
  - All package installs use apt-get/dnf directly with dpkg/rpm verification
    (Rex::Pkg dies on non-zero from DKMS/grub/initramfs post-install scripts)
  - Reboot support with reconnect polling via Rex::LibSSH
  - Tested on Hetzner dedicated servers with NVIDIA RTX 4000 SFF Ada Generation
    (Debian 13, Rocky Linux 10.1, Ubuntu 24.04 LTS)
  - eg/Rexfile example for GPU detection and cluster deployment
  - Requires Rex::LibSSH for deployment to SFTP-less hosts