GPU Passthrough with VFIO: Giving a Virtual Machine Real Graphics

GPU Passthrough with VFIO: Giving a Virtual Machine Real Graphics

GPU passthrough gives a virtual machine direct control of a physical graphics card. The guest gets close to native performance because it is driving real hardware, not an emulation.

It also has more prerequisites than almost anything else in Linux virtualization, and most of the failures happen before you get anywhere near a virtual machine.

Check IOMMU groups first

This step decides whether passthrough is possible on your machine, so do it before anything else.

The IOMMU divides PCI devices into groups based on how the hardware can isolate them. Passthrough works on whole groups: everything in a group goes to the guest, or nothing does.

Enable IOMMU first as a kernel parameter:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"
# or amd_iommu=on for AMD
sudo update-grub && sudo reboot

Then look at the groups:

for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU group %s: ' "$n"
  lspci -nns "${d##*/}"
done | sort -V

You want your GPU, and usually its audio function, alone in a group. If it shares a group with a USB controller or a network card the host needs, you cannot pass it through cleanly.

Some motherboards isolate poorly. The ACS override patch exists to split groups artificially and it weakens the isolation guarantee, which is a real security consideration rather than a formality.

Bind the GPU to VFIO

The host must not claim the card. Find its IDs:

lspci -nn | grep -E 'VGA|Audio'
# 01:00.0 VGA compatible controller [0300]: NVIDIA ... [10de:2484]
# 01:00.1 Audio device [0403]: NVIDIA ... [10de:228b]

Bind both functions to vfio-pci:

echo 'options vfio-pci ids=10de:2484,10de:228b' | sudo tee /etc/modprobe.d/vfio.conf
echo -e 'vfio\nvfio_iommu_type1\nvfio_pci' | sudo tee /etc/modules-load.d/vfio.conf
sudo update-initramfs -u          # or dracut -f
sudo reboot

Rebuilding the initramfs matters, because the binding has to happen before the real driver loads.

Confirm:

lspci -nnk -d 10de:2484
# Kernel driver in use: vfio-pci

If it still says nvidia or amdgpu, the host driver won the race. Blacklisting it is the usual next step.

Attach it to the VM

In virt-manager, add both PCI host devices. From the command line, edit the domain XML with virsh edit.

For NVIDIA cards with older drivers, hide the hypervisor to avoid the notorious error 43:

<features>
  <hyperv>
    <vendor_id state='on' value='randomid'/>
  </hyperv>
  <kvm>
    <hidden state='on'/>
  </kvm>
</features>

Recent NVIDIA driver releases removed the restriction, so this matters less than it used to.

Two things worth doing afterwards

CPU pinning. Assign specific host cores to the guest so the scheduler stops moving threads around. This is where most of the remaining latency comes from.

Hugepages. Reduce memory translation overhead for the guest.

Both are more noticeable for gaming and real-time work than for transcoding.

The honest assessment

Passthrough is genuinely excellent when it works: near-native graphics in a guest, with the host untouched.

Getting there depends on hardware you may not be able to change. Bad IOMMU grouping, a motherboard that will not isolate the slot, or a laptop with a muxless hybrid GPU can all make it impossible regardless of effort.

Check the IOMMU groups before buying anything or committing an evening. That one command answers most of the question.

Frequently Asked Questions

What is an IOMMU group and why does it matter?

The IOMMU divides PCI devices into groups based on how the hardware isolates them, and passthrough operates on whole groups. If your GPU shares a group with a device the host needs, you cannot pass it through without also passing that device, which is the most common reason passthrough is impossible on a given machine.

Do I need two GPUs?

Usually yes for a desktop, because the host needs something to display on once the passed-through card is bound to VFIO. Integrated graphics count as the second GPU. On a headless server you can pass through the only discrete card if the host uses serial or network access.

Will games detect that they are in a virtual machine?

Some anti-cheat systems do, and treat it as suspicious. Hiding the hypervisor is possible with CPU feature flags and helps in some cases, and there is no guarantee any particular title will work.

What is the performance cost?

Graphics performance is close to bare metal because the guest drives the card directly. The overhead lives in CPU and I/O, and CPU pinning plus hugepages recovers most of it. Expect a small single-digit percentage difference for most workloads.

Why does my VM show error 43 with an NVIDIA card?

Older NVIDIA drivers refused to run in a detected virtual machine. Hiding the hypervisor with kvm hidden state resolves it. Recent driver releases removed the restriction, so this affects older setups more than new ones.

Can I use the GPU for the host and the guest at the same time?

Not with plain passthrough, which assigns the whole device to one owner. Vendor technologies like SR-IOV or vGPU allow sharing on supported hardware, and consumer cards generally do not support them.