docs: add kbase entry showing KVM real time guest config

There are many different settings that required to config a KVM guest
for real time, low latency workoads. The documentation included here is
based on guidance developed & tested by the Red Hat KVM real time team.
Reviewed-by: NJiri Denemark <jdenemar@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

docs/kbase.html.in

@@ -36,6 +36,9 @@
 
         <dt><a href="kbase/virtiofs.html">Virtio-FS</a></dt>
         <dd>Share a filesystem between the guest and the host</dd>
+
+        <dt><a href="kbase/kvm-realtime.html">KVM real time</a></dt>
+        <dd>Run real time workloads in guests on a KVM hypervisor</dd>
       </dl>
     </div>
 

docs/kbase/kvm-realtime.rst

+==========================
+KVM Real Time Guest Config
+==========================
+
+.. contents::
+
+The KVM hypervisor is capable of running real time guest workloads. This page
+describes the key pieces of configuration required in the domain XML to achieve
+the low latency needs of real time workloads.
+
+For the most part, configuration of the host OS is out of scope of this
+documentation. Refer to the operating system vendor's guidance on configuring
+the host OS and hardware for real time. Note in particular that the default
+kernel used by most Linux distros is not suitable for low latency real time and
+must be replaced by a special kernel build.
+
+
+Host partitioning plan
+======================
+
+Running real time workloads requires carefully partitioning up the host OS
+resources, such that the KVM / QEMU processes are strictly separated from any
+other workload running on the host, both userspace processes and kernel threads.
+
+As such, some subset of host CPUs need to be reserved exclusively for running
+KVM guests. This requires that the host kernel be booted using the ``isolcpus``
+kernel command line parameter. This parameter removes a set of CPUs from the
+scheduler, such that that no kernel threads or userspace processes will ever get
+placed on those CPUs automatically. KVM guests are then manually placed onto
+these CPUs.
+
+Deciding which host CPUs to reserve for real time requires understanding of the
+guest workload needs and balancing with the host OS needs. The trade off will
+also vary based on the physical hardware available.
+
+For the sake of illustration, this guide will assume a physical machine with two
+NUMA nodes, each with 2 sockets and 4 cores per socket, giving a total of 16
+CPUs on the host. Furthermore, it is assumed that hyperthreading is either not
+supported or has been disabled in the BIOS, since it is incompatible with real
+time. Each NUMA node is assumed to have 32 GB of RAM, giving 64 GB total for
+the host.
+
+It is assumed that 2 CPUs in each NUMA node are reserved for the host OS, with
+the remaining 6 CPUs available for KVM real time. With this in mind, the host
+kernel should have booted with ``isolcpus=2-7,10-15`` to reserve CPUs.
+
+To maximise efficiency of page table lookups for the guest, the host needs to be
+configured with most RAM exposed as huge pages, ideally 1 GB sized. 6 GB of RAM
+in each NUMA node will be reserved for general host OS usage as normal sized
+pages, leaving 26 GB for KVM usage as huge pages.
+
+Once huge pages are reserved on the hypothetical machine, the ``virsh
+capabilities`` command output is expected to look approximately like:
+
+::
+
+   <topology>
+     <cells num='2'>
+       <cell id='0'>
+         <memory unit='KiB'>33554432</memory>
+         <pages unit='KiB' size='4'>1572864</pages>
+         <pages unit='KiB' size='2048'>0</pages>
+         <pages unit='KiB' size='1048576'>26</pages>
+         <distances>
+           <sibling id='0' value='10'/>
+           <sibling id='1' value='21'/>
+         </distances>
+         <cpus num='8'>
+           <cpu id='0' socket_id='0' core_id='0' siblings='0'/>
+           <cpu id='1' socket_id='0' core_id='1' siblings='1'/>
+           <cpu id='2' socket_id='0' core_id='2' siblings='2'/>
+           <cpu id='3' socket_id='0' core_id='3' siblings='3'/>
+           <cpu id='4' socket_id='1' core_id='0' siblings='4'/>
+           <cpu id='5' socket_id='1' core_id='1' siblings='5'/>
+           <cpu id='6' socket_id='1' core_id='2' siblings='6'/>
+           <cpu id='7' socket_id='1' core_id='3' siblings='7'/>
+         </cpus>
+       </cell>
+       <cell id='1'>
+         <memory unit='KiB'>33554432</memory>
+         <pages unit='KiB' size='4'>1572864</pages>
+         <pages unit='KiB' size='2048'>0</pages>
+         <pages unit='KiB' size='1048576'>26</pages>
+         <distances>
+           <sibling id='0' value='21'/>
+           <sibling id='1' value='10'/>
+         </distances>
+         <cpus num='8'>
+           <cpu id='8' socket_id='0' core_id='0' siblings='8'/>
+           <cpu id='9' socket_id='0' core_id='1' siblings='9'/>
+           <cpu id='10' socket_id='0' core_id='2' siblings='10'/>
+           <cpu id='11' socket_id='0' core_id='3' siblings='11'/>
+           <cpu id='12' socket_id='1' core_id='0' siblings='12'/>
+           <cpu id='13' socket_id='1' core_id='1' siblings='13'/>
+           <cpu id='14' socket_id='1' core_id='2' siblings='14'/>
+           <cpu id='15' socket_id='1' core_id='3' siblings='15'/>
+          </cpus>
+       </cell>
+     </cells>
+   </topology>
+
+Be aware that CPU ID numbers are not always allocated sequentially as shown
+here. It is not unusual to see IDs interleaved between sockets on the two NUMA
+nodes, such that ``0-3,8-11`` are on the first node and ``4-7,12-15`` are on
+the second node.  Carefully check the ``virsh capabilities`` output to determine
+the CPU ID numbers when configiring both ``isolcpus`` and the guest ``cpuset``
+values.
+
+Guest configuration
+===================
+
+What follows is an overview of the key parts of the domain XML that need to be
+configured to achieve low latency for real time workflows. The following example
+will assume a 4 CPU guest, requiring 16 GB of RAM. It is intended to be placed
+on the second host NUMA node.
+
+CPU configuration
+-----------------
+
+Real time KVM guests intended to run Linux should have a minimum of 2 CPUs.
+One vCPU is for running non-real time processes and performing I/O. The other
+vCPUs will run real time applications. Some non-Linux OS may not require a
+special non-real time CPU to be available, in which case the 2 CPU minimum would
+not apply.
+
+Each guest CPU, even the non-real time one, needs to be pinned to a dedicated
+host core that is in the `isolcpus` reserved set. The QEMU emulator threads
+need to be pinned to host CPUs that are not in the `isolcpus` reserved set.
+The vCPUs need to be given a real time CPU scheduler policy.
+
+When configuring the `guest CPU count <../formatdomain.html#elementsCPUAllocation>`_,
+do not include any CPU affinity at this stage:
+
+::
+
+   <vcpu placement='static'>4</vcpu>
+
+The guest CPUs now need to be placed individually. In this case, they will all
+be put within the same host socket, such that they can be exposed as core
+siblings. This is achieved using the `CPU tunning config <../formatdomain.html#elementsCPUTuning>`_:
+
+::
+
+   <cputune>
+     <emulatorpin cpuset="8-9"/>
+     <vcpupin vcpu="0" cpuset="12"/>
+     <vcpupin vcpu="1" cpuset="13"/>
+     <vcpupin vcpu="2" cpuset="14"/>
+     <vcpupin vcpu="3" cpuset="15"/>
+     <vcpusched vcpus='0-4' scheduler='fifo' priority='1'/>
+   </cputune>
+
+The `guest CPU model <formatdomain.html#elementsCPU>`_ now needs to be
+configured to pass through the host model unchanged, with topology matching the
+placement:
+
+::
+
+   <cpu mode='host-passthrough'>
+     <topology sockets='1' dies='1' cores='4' threads='1'/>
+     <feature policy='require' name='tsc-deadline'/>
+   </cpu>
+
+The performance monitoring unit virtualization needs to be disabled
+via the `hypervisor features <../formatdomain.html#elementsFeatures>`_:
+
+::
+
+   <features>
+     ...
+     <pmu state='off'/>
+   </features>
+
+
+Memory configuration
+--------------------
+
+The host memory used for guest RAM needs to be allocated from huge pages on the
+second NUMA node, and all other memory allocation needs to be locked into RAM
+with memory page sharing disabled.
+This is achieved by using the `memory backing config <formatdomain.html#elementsMemoryBacking>`_:
+
+::
+
+   <memoryBacking>
+     <hugepages>
+       <page size="1" unit="G" nodeset="1"/>
+     </hugepages>
+     <locked/>
+     <nosharepages/>
+   </memoryBacking>
+
+
+Device configuration
+--------------------
+
+Libvirt adds a few devices by default to maintain historical QEMU configuration
+behaviour. It is unlikely these devices are required by real time guests, so it
+is wise to disable them. Remove all USB controllers that may exist in the XML
+config and replace them with:
+
+::
+
+   <controller type="usb" model="none"/>
+
+Similarly the memory balloon config should be changed to
+
+::
+
+   <memballoon model="none"/>
+
+If the guest had a graphical console at installation time this can also be
+disabled, with remote access being over SSH, with a minimal serial console
+for emergencies.