vfio/migration: Add VFIO migration pre-copy support

Pre-copy support allows the VFIO device data to be transferred while the
VM is running. This helps to accommodate VFIO devices that have a large
amount of data that needs to be transferred, and it can reduce migration
downtime.

Pre-copy support is optional in VFIO migration protocol v2.
Implement pre-copy of VFIO migration protocol v2 and use it for devices
that support it. Full description of it can be found in the following
Linux commit: 4db52602a607 ("vfio: Extend the device migration protocol
with PRE_COPY").

Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Tested-by: YangHang Liu <yanghliu@redhat.com>
Acked-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Cédric Le Goater <clg@redhat.com>
This commit is contained in:
Avihai Horon 2023-06-21 14:12:00 +03:00 committed by Cédric Le Goater
parent 6cd1fe1159
commit eda7362af9
5 changed files with 190 additions and 22 deletions

View File

@ -7,12 +7,14 @@ the guest is running on source host and restoring this saved state on the
destination host. This document details how saving and restoring of VFIO destination host. This document details how saving and restoring of VFIO
devices is done in QEMU. devices is done in QEMU.
Migration of VFIO devices currently consists of a single stop-and-copy phase. Migration of VFIO devices consists of two phases: the optional pre-copy phase,
During the stop-and-copy phase the guest is stopped and the entire VFIO device and the stop-and-copy phase. The pre-copy phase is iterative and allows to
data is transferred to the destination. accommodate VFIO devices that have a large amount of data that needs to be
transferred. The iterative pre-copy phase of migration allows for the guest to
The pre-copy phase of migration is currently not supported for VFIO devices. continue whilst the VFIO device state is transferred to the destination, this
Support for VFIO pre-copy will be added later on. helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy
support by reporting the VFIO_MIGRATION_PRE_COPY flag in the
VFIO_DEVICE_FEATURE_MIGRATION ioctl.
Note that currently VFIO migration is supported only for a single device. This Note that currently VFIO migration is supported only for a single device. This
is due to VFIO migration's lack of P2P support. However, P2P support is planned is due to VFIO migration's lack of P2P support. However, P2P support is planned
@ -29,10 +31,20 @@ VFIO implements the device hooks for the iterative approach as follows:
* A ``load_setup`` function that sets the VFIO device on the destination in * A ``load_setup`` function that sets the VFIO device on the destination in
_RESUMING state. _RESUMING state.
* A ``state_pending_estimate`` function that reports an estimate of the
remaining pre-copy data that the vendor driver has yet to save for the VFIO
device.
* A ``state_pending_exact`` function that reads pending_bytes from the vendor * A ``state_pending_exact`` function that reads pending_bytes from the vendor
driver, which indicates the amount of data that the vendor driver has yet to driver, which indicates the amount of data that the vendor driver has yet to
save for the VFIO device. save for the VFIO device.
* An ``is_active_iterate`` function that indicates ``save_live_iterate`` is
active only when the VFIO device is in pre-copy states.
* A ``save_live_iterate`` function that reads the VFIO device's data from the
vendor driver during iterative pre-copy phase.
* A ``save_state`` function to save the device config space if it is present. * A ``save_state`` function to save the device config space if it is present.
* A ``save_live_complete_precopy`` function that sets the VFIO device in * A ``save_live_complete_precopy`` function that sets the VFIO device in
@ -111,8 +123,10 @@ Flow of state changes during Live migration
=========================================== ===========================================
Below is the flow of state change during live migration. Below is the flow of state change during live migration.
The values in the brackets represent the VM state, the migration state, and The values in the parentheses represent the VM state, the migration state, and
the VFIO device state, respectively. the VFIO device state, respectively.
The text in the square brackets represents the flow if the VFIO device supports
pre-copy.
Live migration save path Live migration save path
------------------------ ------------------------
@ -124,11 +138,12 @@ Live migration save path
| |
migrate_init spawns migration_thread migrate_init spawns migration_thread
Migration thread then calls each device's .save_setup() Migration thread then calls each device's .save_setup()
(RUNNING, _SETUP, _RUNNING) (RUNNING, _SETUP, _RUNNING [_PRE_COPY])
| |
(RUNNING, _ACTIVE, _RUNNING) (RUNNING, _ACTIVE, _RUNNING [_PRE_COPY])
If device is active, get pending_bytes by .state_pending_exact() If device is active, get pending_bytes by .state_pending_{estimate,exact}()
If total pending_bytes >= threshold_size, call .save_live_iterate() If total pending_bytes >= threshold_size, call .save_live_iterate()
[Data of VFIO device for pre-copy phase is copied]
Iterate till total pending bytes converge and are less than threshold Iterate till total pending bytes converge and are less than threshold
| |
On migration completion, vCPU stops and calls .save_live_complete_precopy for On migration completion, vCPU stops and calls .save_live_complete_precopy for

View File

@ -492,7 +492,8 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainer *container)
} }
if (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF && if (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF &&
migration->device_state == VFIO_DEVICE_STATE_RUNNING) { (migration->device_state == VFIO_DEVICE_STATE_RUNNING ||
migration->device_state == VFIO_DEVICE_STATE_PRE_COPY)) {
return false; return false;
} }
} }
@ -537,7 +538,8 @@ static bool vfio_devices_all_running_and_mig_active(VFIOContainer *container)
return false; return false;
} }
if (migration->device_state == VFIO_DEVICE_STATE_RUNNING) { if (migration->device_state == VFIO_DEVICE_STATE_RUNNING ||
migration->device_state == VFIO_DEVICE_STATE_PRE_COPY) {
continue; continue;
} else { } else {
return false; return false;

View File

@ -68,6 +68,8 @@ static const char *mig_state_to_str(enum vfio_device_mig_state state)
return "STOP_COPY"; return "STOP_COPY";
case VFIO_DEVICE_STATE_RESUMING: case VFIO_DEVICE_STATE_RESUMING:
return "RESUMING"; return "RESUMING";
case VFIO_DEVICE_STATE_PRE_COPY:
return "PRE_COPY";
default: default:
return "UNKNOWN STATE"; return "UNKNOWN STATE";
} }
@ -241,6 +243,25 @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev,
return 0; return 0;
} }
static int vfio_query_precopy_size(VFIOMigration *migration)
{
struct vfio_precopy_info precopy = {
.argsz = sizeof(precopy),
};
migration->precopy_init_size = 0;
migration->precopy_dirty_size = 0;
if (ioctl(migration->data_fd, VFIO_MIG_GET_PRECOPY_INFO, &precopy)) {
return -errno;
}
migration->precopy_init_size = precopy.initial_bytes;
migration->precopy_dirty_size = precopy.dirty_bytes;
return 0;
}
/* Returns the size of saved data on success and -errno on error */ /* Returns the size of saved data on success and -errno on error */
static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration) static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration)
{ {
@ -249,6 +270,14 @@ static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration)
data_size = read(migration->data_fd, migration->data_buffer, data_size = read(migration->data_fd, migration->data_buffer,
migration->data_buffer_size); migration->data_buffer_size);
if (data_size < 0) { if (data_size < 0) {
/*
* Pre-copy emptied all the device state for now. For more information,
* please refer to the Linux kernel VFIO uAPI.
*/
if (errno == ENOMSG) {
return 0;
}
return -errno; return -errno;
} }
if (data_size == 0) { if (data_size == 0) {
@ -265,6 +294,38 @@ static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration)
return qemu_file_get_error(f) ?: data_size; return qemu_file_get_error(f) ?: data_size;
} }
static void vfio_update_estimated_pending_data(VFIOMigration *migration,
uint64_t data_size)
{
if (!data_size) {
/*
* Pre-copy emptied all the device state for now, update estimated sizes
* accordingly.
*/
migration->precopy_init_size = 0;
migration->precopy_dirty_size = 0;
return;
}
if (migration->precopy_init_size) {
uint64_t init_size = MIN(migration->precopy_init_size, data_size);
migration->precopy_init_size -= init_size;
data_size -= init_size;
}
migration->precopy_dirty_size -= MIN(migration->precopy_dirty_size,
data_size);
}
static bool vfio_precopy_supported(VFIODevice *vbasedev)
{
VFIOMigration *migration = vbasedev->migration;
return migration->mig_flags & VFIO_MIGRATION_PRE_COPY;
}
/* ---------------------------------------------------------------------- */ /* ---------------------------------------------------------------------- */
static int vfio_save_setup(QEMUFile *f, void *opaque) static int vfio_save_setup(QEMUFile *f, void *opaque)
@ -285,6 +346,28 @@ static int vfio_save_setup(QEMUFile *f, void *opaque)
return -ENOMEM; return -ENOMEM;
} }
if (vfio_precopy_supported(vbasedev)) {
int ret;
switch (migration->device_state) {
case VFIO_DEVICE_STATE_RUNNING:
ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_PRE_COPY,
VFIO_DEVICE_STATE_RUNNING);
if (ret) {
return ret;
}
vfio_query_precopy_size(migration);
break;
case VFIO_DEVICE_STATE_STOP:
/* vfio_save_complete_precopy() will go to STOP_COPY */
break;
default:
return -EINVAL;
}
}
trace_vfio_save_setup(vbasedev->name, migration->data_buffer_size); trace_vfio_save_setup(vbasedev->name, migration->data_buffer_size);
qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE);
@ -299,26 +382,42 @@ static void vfio_save_cleanup(void *opaque)
g_free(migration->data_buffer); g_free(migration->data_buffer);
migration->data_buffer = NULL; migration->data_buffer = NULL;
migration->precopy_init_size = 0;
migration->precopy_dirty_size = 0;
vfio_migration_cleanup(vbasedev); vfio_migration_cleanup(vbasedev);
trace_vfio_save_cleanup(vbasedev->name); trace_vfio_save_cleanup(vbasedev->name);
} }
static void vfio_state_pending_estimate(void *opaque, uint64_t *must_precopy,
uint64_t *can_postcopy)
{
VFIODevice *vbasedev = opaque;
VFIOMigration *migration = vbasedev->migration;
if (migration->device_state != VFIO_DEVICE_STATE_PRE_COPY) {
return;
}
*must_precopy +=
migration->precopy_init_size + migration->precopy_dirty_size;
trace_vfio_state_pending_estimate(vbasedev->name, *must_precopy,
*can_postcopy,
migration->precopy_init_size,
migration->precopy_dirty_size);
}
/* /*
* Migration size of VFIO devices can be as little as a few KBs or as big as * Migration size of VFIO devices can be as little as a few KBs or as big as
* many GBs. This value should be big enough to cover the worst case. * many GBs. This value should be big enough to cover the worst case.
*/ */
#define VFIO_MIG_STOP_COPY_SIZE (100 * GiB) #define VFIO_MIG_STOP_COPY_SIZE (100 * GiB)
/*
* Only exact function is implemented and not estimate function. The reason is
* that during pre-copy phase of migration the estimate function is called
* repeatedly while pending RAM size is over the threshold, thus migration
* can't converge and querying the VFIO device pending data size is useless.
*/
static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy,
uint64_t *can_postcopy) uint64_t *can_postcopy)
{ {
VFIODevice *vbasedev = opaque; VFIODevice *vbasedev = opaque;
VFIOMigration *migration = vbasedev->migration;
uint64_t stop_copy_size = VFIO_MIG_STOP_COPY_SIZE; uint64_t stop_copy_size = VFIO_MIG_STOP_COPY_SIZE;
/* /*
@ -328,8 +427,48 @@ static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy,
vfio_query_stop_copy_size(vbasedev, &stop_copy_size); vfio_query_stop_copy_size(vbasedev, &stop_copy_size);
*must_precopy += stop_copy_size; *must_precopy += stop_copy_size;
if (migration->device_state == VFIO_DEVICE_STATE_PRE_COPY) {
vfio_query_precopy_size(migration);
*must_precopy +=
migration->precopy_init_size + migration->precopy_dirty_size;
}
trace_vfio_state_pending_exact(vbasedev->name, *must_precopy, *can_postcopy, trace_vfio_state_pending_exact(vbasedev->name, *must_precopy, *can_postcopy,
stop_copy_size); stop_copy_size, migration->precopy_init_size,
migration->precopy_dirty_size);
}
static bool vfio_is_active_iterate(void *opaque)
{
VFIODevice *vbasedev = opaque;
VFIOMigration *migration = vbasedev->migration;
return migration->device_state == VFIO_DEVICE_STATE_PRE_COPY;
}
static int vfio_save_iterate(QEMUFile *f, void *opaque)
{
VFIODevice *vbasedev = opaque;
VFIOMigration *migration = vbasedev->migration;
ssize_t data_size;
data_size = vfio_save_block(f, migration);
if (data_size < 0) {
return data_size;
}
qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE);
vfio_update_estimated_pending_data(migration, data_size);
trace_vfio_save_iterate(vbasedev->name, migration->precopy_init_size,
migration->precopy_dirty_size);
/*
* A VFIO device's pre-copy dirty_bytes is not guaranteed to reach zero.
* Return 1 so following handlers will not be potentially blocked.
*/
return 1;
} }
static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
@ -338,7 +477,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
ssize_t data_size; ssize_t data_size;
int ret; int ret;
/* We reach here with device state STOP only */ /* We reach here with device state STOP or STOP_COPY only */
ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP_COPY, ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP_COPY,
VFIO_DEVICE_STATE_STOP); VFIO_DEVICE_STATE_STOP);
if (ret) { if (ret) {
@ -457,7 +596,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
static const SaveVMHandlers savevm_vfio_handlers = { static const SaveVMHandlers savevm_vfio_handlers = {
.save_setup = vfio_save_setup, .save_setup = vfio_save_setup,
.save_cleanup = vfio_save_cleanup, .save_cleanup = vfio_save_cleanup,
.state_pending_estimate = vfio_state_pending_estimate,
.state_pending_exact = vfio_state_pending_exact, .state_pending_exact = vfio_state_pending_exact,
.is_active_iterate = vfio_is_active_iterate,
.save_live_iterate = vfio_save_iterate,
.save_live_complete_precopy = vfio_save_complete_precopy, .save_live_complete_precopy = vfio_save_complete_precopy,
.save_state = vfio_save_state, .save_state = vfio_save_state,
.load_setup = vfio_load_setup, .load_setup = vfio_load_setup,
@ -470,13 +612,18 @@ static const SaveVMHandlers savevm_vfio_handlers = {
static void vfio_vmstate_change(void *opaque, bool running, RunState state) static void vfio_vmstate_change(void *opaque, bool running, RunState state)
{ {
VFIODevice *vbasedev = opaque; VFIODevice *vbasedev = opaque;
VFIOMigration *migration = vbasedev->migration;
enum vfio_device_mig_state new_state; enum vfio_device_mig_state new_state;
int ret; int ret;
if (running) { if (running) {
new_state = VFIO_DEVICE_STATE_RUNNING; new_state = VFIO_DEVICE_STATE_RUNNING;
} else { } else {
new_state = VFIO_DEVICE_STATE_STOP; new_state =
(migration->device_state == VFIO_DEVICE_STATE_PRE_COPY &&
(state == RUN_STATE_FINISH_MIGRATE || state == RUN_STATE_PAUSED)) ?
VFIO_DEVICE_STATE_STOP_COPY :
VFIO_DEVICE_STATE_STOP;
} }
/* /*

View File

@ -162,6 +162,8 @@ vfio_save_block(const char *name, int data_size) " (%s) data_size %d"
vfio_save_cleanup(const char *name) " (%s)" vfio_save_cleanup(const char *name) " (%s)"
vfio_save_complete_precopy(const char *name, int ret) " (%s) ret %d" vfio_save_complete_precopy(const char *name, int ret) " (%s) ret %d"
vfio_save_device_config_state(const char *name) " (%s)" vfio_save_device_config_state(const char *name) " (%s)"
vfio_save_iterate(const char *name, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy initial size 0x%"PRIx64" precopy dirty size 0x%"PRIx64
vfio_save_setup(const char *name, uint64_t data_buffer_size) " (%s) data buffer size 0x%"PRIx64 vfio_save_setup(const char *name, uint64_t data_buffer_size) " (%s) data buffer size 0x%"PRIx64
vfio_state_pending_exact(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t stopcopy_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" stopcopy size 0x%"PRIx64 vfio_state_pending_estimate(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" precopy initial size 0x%"PRIx64" precopy dirty size 0x%"PRIx64
vfio_state_pending_exact(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t stopcopy_size, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" stopcopy size 0x%"PRIx64" precopy initial size 0x%"PRIx64" precopy dirty size 0x%"PRIx64
vfio_vmstate_change(const char *name, int running, const char *reason, const char *dev_state) " (%s) running %d reason %s device state %s" vfio_vmstate_change(const char *name, int running, const char *reason, const char *dev_state) " (%s) running %d reason %s device state %s"

View File

@ -67,6 +67,8 @@ typedef struct VFIOMigration {
void *data_buffer; void *data_buffer;
size_t data_buffer_size; size_t data_buffer_size;
uint64_t mig_flags; uint64_t mig_flags;
uint64_t precopy_init_size;
uint64_t precopy_dirty_size;
} VFIOMigration; } VFIOMigration;
typedef struct VFIOAddressSpace { typedef struct VFIOAddressSpace {