introduce a bidirectionnal "comm page", a page of memory shared between
the kernel and userland, and used to transfer data in and out in a more
performant manner than ioctls.
The comm page contains the VCPU state, plus three flags:
- "wanted": the states the kernel must get/set when requested via ioctls
- "cached": the states that are in the comm page
- "commit": the states the kernel must set in vcpu_run
The idea is to avoid performing expensive syscalls, by using the VCPU
state cached, either explicitly or speculatively, in the comm page. For
example, if the state is cached we do a direct 1->5 with no syscall:
+---------------------------------------------+
| Qemu |
+---------------------------------------------+
| ^
| (0) nvmm_vcpu_getstate | (6) Done
| |
V |
+---------------------------------------+
| libnvmm |
+---------------------------------------+
| ^ | ^
(1) State | | (2) No | (3) Ioctl: | (5) Ok, state
cached? | | | "please cache | fetched
| | | the state" |
V | | |
+-----------+ | |
| Comm Page |------+---------------+
+-----------+ |
^ |
(4) "Alright | V
babe" | +--------+
+-----| Kernel |
+--------+
The main changes in behavior are:
- nvmm_vcpu_getstate(): won't emit a syscall if the state is already
cached in the comm page, will just fetch from the comm page directly
- nvmm_vcpu_setstate(): won't emit a syscall at all, will just cache
the wanted state in the comm page
- nvmm_vcpu_run(): will commit the to-be-set state in the comm page,
as previously requested by nvmm_vcpu_setstate()
In addition to this, the kernel NVMM driver is changed to speculatively
cache certain states known to be of interest, so that the future
nvmm_vcpu_getstate() calls libnvmm or the emulator will perform will use
the comm page rather than expensive syscalls. For example, if an I/O
VMEXIT occurs, the I/O Assist in libnvmm will want GPRS+SEGS+CRS+MSRS,
and now the kernel caches all of that in the comm page before returning
to userland.
Overall, in a normal run of Windows 10, this saves several millions of
syscalls. Eg on a 4CPU Intel with 4VCPUs, booting the Win10 install ISO
goes from taking 1min35 to taking 1min16.
The libnvmm API is not changed, but the ABI is. If we changed the API it
would be possible to save expensive memcpys on libnvmm's side. This will
be avoided in a future version. The comm page can also be extended to
implement future services.
software to effortlessly create and manage virtual machines via NVMM.
It is mostly complete, only nvmm_assist_mem needs to be filled -- I have
a draft for that, but it needs some more care. This Mem Assist should
not be needed when emulating a system in x2apic mode, so theoretically
the current form of libnvmm is sufficient to emulate a whole class of
systems.
Generally speaking, there are so many modes in x86 that it is difficult
to handle each corner case without introducing a ton of checks that just
slow down the common-case execution. Currently we check a limited number
of things; we may add more checks in the future if they turn out to be
needed, but that's rather low priority.
Libnvmm is compiled and installed only on amd64. A man page (reviewed by
wiz@) is provided.
