2014-06-10 14:02:57 +04:00
|
|
|
Vhost-user Protocol
|
|
|
|
===================
|
|
|
|
|
|
|
|
Copyright (c) 2014 Virtual Open Systems Sarl.
|
|
|
|
|
|
|
|
This work is licensed under the terms of the GNU GPL, version 2 or later.
|
|
|
|
See the COPYING file in the top-level directory.
|
|
|
|
===================
|
|
|
|
|
|
|
|
This protocol is aiming to complement the ioctl interface used to control the
|
|
|
|
vhost implementation in the Linux kernel. It implements the control plane needed
|
|
|
|
to establish virtqueue sharing with a user space process on the same host. It
|
|
|
|
uses communication over a Unix domain socket to share file descriptors in the
|
|
|
|
ancillary data of the message.
|
|
|
|
|
|
|
|
The protocol defines 2 sides of the communication, master and slave. Master is
|
|
|
|
the application that shares its virtqueues, in our case QEMU. Slave is the
|
|
|
|
consumer of the virtqueues.
|
|
|
|
|
|
|
|
In the current implementation QEMU is the Master, and the Slave is intended to
|
|
|
|
be a software Ethernet switch running in user space, such as Snabbswitch.
|
|
|
|
|
|
|
|
Master and slave can be either a client (i.e. connecting) or server (listening)
|
|
|
|
in the socket communication.
|
|
|
|
|
|
|
|
Message Specification
|
|
|
|
---------------------
|
|
|
|
|
|
|
|
Note that all numbers are in the machine native byte order. A vhost-user message
|
|
|
|
consists of 3 header fields and a payload:
|
|
|
|
|
|
|
|
------------------------------------
|
|
|
|
| request | flags | size | payload |
|
|
|
|
------------------------------------
|
|
|
|
|
|
|
|
* Request: 32-bit type of the request
|
|
|
|
* Flags: 32-bit bit field:
|
|
|
|
- Lower 2 bits are the version (currently 0x01)
|
|
|
|
- Bit 2 is the reply flag - needs to be sent on each reply from the slave
|
2016-08-05 13:53:50 +03:00
|
|
|
- Bit 3 is the need_reply flag - see VHOST_USER_PROTOCOL_F_REPLY_ACK for
|
|
|
|
details.
|
2014-06-10 14:02:57 +04:00
|
|
|
* Size - 32-bit size of the payload
|
|
|
|
|
|
|
|
|
|
|
|
Depending on the request type, payload can be:
|
|
|
|
|
|
|
|
* A single 64-bit integer
|
|
|
|
-------
|
|
|
|
| u64 |
|
|
|
|
-------
|
|
|
|
|
|
|
|
u64: a 64-bit unsigned integer
|
|
|
|
|
|
|
|
* A vring state description
|
|
|
|
---------------
|
2017-12-14 23:22:19 +03:00
|
|
|
| index | num |
|
|
|
|
---------------
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
Index: a 32-bit index
|
|
|
|
Num: a 32-bit number
|
|
|
|
|
|
|
|
* A vring address description
|
|
|
|
--------------------------------------------------------------
|
|
|
|
| index | flags | size | descriptor | used | available | log |
|
|
|
|
--------------------------------------------------------------
|
|
|
|
|
|
|
|
Index: a 32-bit vring index
|
|
|
|
Flags: a 32-bit vring flags
|
2017-12-14 23:22:20 +03:00
|
|
|
Descriptor: a 64-bit ring address of the vring descriptor table
|
|
|
|
Used: a 64-bit ring address of the vring used ring
|
|
|
|
Available: a 64-bit ring address of the vring available ring
|
2014-06-10 14:02:57 +04:00
|
|
|
Log: a 64-bit guest address for logging
|
|
|
|
|
2017-12-14 23:22:20 +03:00
|
|
|
Note that a ring address is an IOVA if VIRTIO_F_IOMMU_PLATFORM has been
|
|
|
|
negotiated. Otherwise it is a user address.
|
|
|
|
|
2014-06-10 14:02:57 +04:00
|
|
|
* Memory regions description
|
|
|
|
---------------------------------------------------
|
|
|
|
| num regions | padding | region0 | ... | region7 |
|
|
|
|
---------------------------------------------------
|
|
|
|
|
|
|
|
Num regions: a 32-bit number of regions
|
|
|
|
Padding: 32-bit
|
|
|
|
|
|
|
|
A region is:
|
2014-06-27 01:01:32 +04:00
|
|
|
-----------------------------------------------------
|
|
|
|
| guest address | size | user address | mmap offset |
|
|
|
|
-----------------------------------------------------
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
Guest address: a 64-bit guest address of the region
|
|
|
|
Size: a 64-bit size
|
|
|
|
User address: a 64-bit user address
|
2014-06-29 18:58:48 +04:00
|
|
|
mmap offset: 64-bit offset where region starts in the mapped memory
|
2014-06-10 14:02:57 +04:00
|
|
|
|
2015-11-15 22:25:11 +03:00
|
|
|
* Log description
|
|
|
|
---------------------------
|
|
|
|
| log size | log offset |
|
|
|
|
---------------------------
|
|
|
|
log size: size of area used for logging
|
|
|
|
log offset: offset from start of supplied file descriptor
|
|
|
|
where logging starts (i.e. where guest address 0 would be logged)
|
|
|
|
|
2017-06-02 13:18:31 +03:00
|
|
|
* An IOTLB message
|
|
|
|
---------------------------------------------------------
|
|
|
|
| iova | size | user address | permissions flags | type |
|
|
|
|
---------------------------------------------------------
|
|
|
|
|
|
|
|
IOVA: a 64-bit I/O virtual address programmed by the guest
|
|
|
|
Size: a 64-bit size
|
|
|
|
User address: a 64-bit user address
|
|
|
|
Permissions: a 8-bit value:
|
|
|
|
- 0: No access
|
|
|
|
- 1: Read access
|
|
|
|
- 2: Write access
|
|
|
|
- 3: Read/Write access
|
|
|
|
Type: a 8-bit IOTLB message type:
|
|
|
|
- 1: IOTLB miss
|
|
|
|
- 2: IOTLB update
|
|
|
|
- 3: IOTLB invalidate
|
|
|
|
- 4: IOTLB access fail
|
|
|
|
|
2018-01-04 04:53:31 +03:00
|
|
|
* Virtio device config space
|
|
|
|
-----------------------------------
|
|
|
|
| offset | size | flags | payload |
|
|
|
|
-----------------------------------
|
|
|
|
|
|
|
|
Offset: a 32-bit offset of virtio device's configuration space
|
|
|
|
Size: a 32-bit configuration space access size in bytes
|
|
|
|
Flags: a 32-bit value:
|
|
|
|
- 0: Vhost master messages used for writeable fields
|
|
|
|
- 1: Vhost master messages used for live migration
|
|
|
|
Payload: Size bytes array holding the contents of the virtio
|
|
|
|
device's configuration space
|
|
|
|
|
2014-06-10 14:02:57 +04:00
|
|
|
In QEMU the vhost-user message is implemented with the following struct:
|
|
|
|
|
|
|
|
typedef struct VhostUserMsg {
|
|
|
|
VhostUserRequest request;
|
|
|
|
uint32_t flags;
|
|
|
|
uint32_t size;
|
|
|
|
union {
|
|
|
|
uint64_t u64;
|
|
|
|
struct vhost_vring_state state;
|
|
|
|
struct vhost_vring_addr addr;
|
|
|
|
VhostUserMemory memory;
|
2015-11-11 17:26:02 +03:00
|
|
|
VhostUserLog log;
|
2017-06-02 13:18:31 +03:00
|
|
|
struct vhost_iotlb_msg iotlb;
|
2018-01-04 04:53:31 +03:00
|
|
|
VhostUserConfig config;
|
2014-06-10 14:02:57 +04:00
|
|
|
};
|
|
|
|
} QEMU_PACKED VhostUserMsg;
|
|
|
|
|
|
|
|
Communication
|
|
|
|
-------------
|
|
|
|
|
|
|
|
The protocol for vhost-user is based on the existing implementation of vhost
|
|
|
|
for the Linux Kernel. Most messages that can be sent via the Unix domain socket
|
|
|
|
implementing vhost-user have an equivalent ioctl to the kernel implementation.
|
|
|
|
|
|
|
|
The communication consists of master sending message requests and slave sending
|
|
|
|
message replies. Most of the requests don't require replies. Here is a list of
|
|
|
|
the ones that do:
|
|
|
|
|
2016-11-24 06:20:56 +03:00
|
|
|
* VHOST_USER_GET_FEATURES
|
|
|
|
* VHOST_USER_GET_PROTOCOL_FEATURES
|
|
|
|
* VHOST_USER_GET_VRING_BASE
|
|
|
|
* VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
|
2014-06-10 14:02:57 +04:00
|
|
|
|
2016-08-05 13:53:50 +03:00
|
|
|
[ Also see the section on REPLY_ACK protocol extension. ]
|
|
|
|
|
2014-06-10 14:02:57 +04:00
|
|
|
There are several messages that the master sends with file descriptors passed
|
|
|
|
in the ancillary data:
|
|
|
|
|
2016-11-24 06:20:56 +03:00
|
|
|
* VHOST_USER_SET_MEM_TABLE
|
|
|
|
* VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
|
|
|
|
* VHOST_USER_SET_LOG_FD
|
|
|
|
* VHOST_USER_SET_VRING_KICK
|
|
|
|
* VHOST_USER_SET_VRING_CALL
|
|
|
|
* VHOST_USER_SET_VRING_ERR
|
2017-06-02 13:18:30 +03:00
|
|
|
* VHOST_USER_SET_SLAVE_REQ_FD
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
If Master is unable to send the full message or receives a wrong reply it will
|
|
|
|
close the connection. An optional reconnection mechanism can be implemented.
|
|
|
|
|
2015-09-23 07:19:56 +03:00
|
|
|
Any protocol extensions are gated by protocol feature bits,
|
|
|
|
which allows full backwards compatibility on both master
|
|
|
|
and slave.
|
|
|
|
As older slaves don't support negotiating protocol features,
|
|
|
|
a feature bit was dedicated for this purpose:
|
|
|
|
#define VHOST_USER_F_PROTOCOL_FEATURES 30
|
|
|
|
|
2015-11-15 22:25:11 +03:00
|
|
|
Starting and stopping rings
|
|
|
|
----------------------
|
2015-11-23 13:48:52 +03:00
|
|
|
Client must only process each ring when it is started.
|
|
|
|
|
|
|
|
Client must only pass data between the ring and the
|
|
|
|
backend, when the ring is enabled.
|
|
|
|
|
|
|
|
If ring is started but disabled, client must process the
|
|
|
|
ring without talking to the backend.
|
|
|
|
|
|
|
|
For example, for a networking device, in the disabled state
|
|
|
|
client must not supply any new RX packets, but must process
|
|
|
|
and discard any TX packets.
|
2015-11-17 14:55:48 +03:00
|
|
|
|
|
|
|
If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, the ring is initialized
|
|
|
|
in an enabled state.
|
2015-11-15 22:25:11 +03:00
|
|
|
|
2015-11-17 14:55:48 +03:00
|
|
|
If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, the ring is initialized
|
2015-11-23 13:48:52 +03:00
|
|
|
in a disabled state. Client must not pass data to/from the backend until ring is enabled by
|
2015-11-17 14:55:48 +03:00
|
|
|
VHOST_USER_SET_VRING_ENABLE with parameter 1, or after it has been disabled by
|
|
|
|
VHOST_USER_SET_VRING_ENABLE with parameter 0.
|
|
|
|
|
|
|
|
Each ring is initialized in a stopped state, client must not process it until
|
|
|
|
ring is started, or after it has been stopped.
|
2015-11-15 22:25:11 +03:00
|
|
|
|
2015-11-17 14:55:48 +03:00
|
|
|
Client must start ring upon receiving a kick (that is, detecting that file
|
|
|
|
descriptor is readable) on the descriptor specified by
|
|
|
|
VHOST_USER_SET_VRING_KICK, and stop ring upon receiving
|
|
|
|
VHOST_USER_GET_VRING_BASE.
|
2015-11-15 22:25:11 +03:00
|
|
|
|
2015-11-23 13:48:52 +03:00
|
|
|
While processing the rings (whether they are enabled or not), client must
|
2015-11-17 14:55:48 +03:00
|
|
|
support changing some configuration aspects on the fly.
|
2015-11-15 22:25:11 +03:00
|
|
|
|
vhost-user: add multiple queue support
This patch is initially based a patch from Nikolay Nikolaev.
This patch adds vhost-user multiple queue support, by creating a nc
and vhost_net pair for each queue.
Qemu exits if find that the backend can't support the number of requested
queues (by providing queues=# option). The max number is queried by a
new message, VHOST_USER_GET_QUEUE_NUM, and is sent only when protocol
feature VHOST_USER_PROTOCOL_F_MQ is present first.
The max queue check is done at vhost-user initiation stage. We initiate
one queue first, which, in the meantime, also gets the max_queues the
backend supports.
In older version, it was reported that some messages are sent more times
than necessary. Here we came an agreement with Michael that we could
categorize vhost user messages to 2 types: non-vring specific messages,
which should be sent only once, and vring specific messages, which should
be sent per queue.
Here I introduced a helper function vhost_user_one_time_request(), which
lists following messages as non-vring specific messages:
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUM
For above messages, we simply ignore them when they are not sent the first
time.
Signed-off-by: Nikolay Nikolaev <n.nikolaev@virtualopensystems.com>
Signed-off-by: Changchun Ouyang <changchun.ouyang@intel.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Jason Wang <jasowang@redhat.com>
Tested-by: Marcel Apfelbaum <marcel@redhat.com>
2015-09-23 07:20:00 +03:00
|
|
|
Multiple queue support
|
|
|
|
----------------------
|
|
|
|
|
|
|
|
Multiple queue is treated as a protocol extension, hence the slave has to
|
|
|
|
implement protocol features first. The multiple queues feature is supported
|
2015-10-09 18:17:29 +03:00
|
|
|
only when the protocol feature VHOST_USER_PROTOCOL_F_MQ (bit 0) is set.
|
vhost-user: add multiple queue support
This patch is initially based a patch from Nikolay Nikolaev.
This patch adds vhost-user multiple queue support, by creating a nc
and vhost_net pair for each queue.
Qemu exits if find that the backend can't support the number of requested
queues (by providing queues=# option). The max number is queried by a
new message, VHOST_USER_GET_QUEUE_NUM, and is sent only when protocol
feature VHOST_USER_PROTOCOL_F_MQ is present first.
The max queue check is done at vhost-user initiation stage. We initiate
one queue first, which, in the meantime, also gets the max_queues the
backend supports.
In older version, it was reported that some messages are sent more times
than necessary. Here we came an agreement with Michael that we could
categorize vhost user messages to 2 types: non-vring specific messages,
which should be sent only once, and vring specific messages, which should
be sent per queue.
Here I introduced a helper function vhost_user_one_time_request(), which
lists following messages as non-vring specific messages:
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUM
For above messages, we simply ignore them when they are not sent the first
time.
Signed-off-by: Nikolay Nikolaev <n.nikolaev@virtualopensystems.com>
Signed-off-by: Changchun Ouyang <changchun.ouyang@intel.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Jason Wang <jasowang@redhat.com>
Tested-by: Marcel Apfelbaum <marcel@redhat.com>
2015-09-23 07:20:00 +03:00
|
|
|
|
|
|
|
The max number of queues the slave supports can be queried with message
|
|
|
|
VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of
|
|
|
|
requested queues is bigger than that.
|
|
|
|
|
|
|
|
As all queues share one connection, the master uses a unique index for each
|
2015-09-23 07:20:01 +03:00
|
|
|
queue in the sent message to identify a specified queue. One queue pair
|
|
|
|
is enabled initially. More queues are enabled dynamically, by sending
|
|
|
|
message VHOST_USER_SET_VRING_ENABLE.
|
vhost-user: add multiple queue support
This patch is initially based a patch from Nikolay Nikolaev.
This patch adds vhost-user multiple queue support, by creating a nc
and vhost_net pair for each queue.
Qemu exits if find that the backend can't support the number of requested
queues (by providing queues=# option). The max number is queried by a
new message, VHOST_USER_GET_QUEUE_NUM, and is sent only when protocol
feature VHOST_USER_PROTOCOL_F_MQ is present first.
The max queue check is done at vhost-user initiation stage. We initiate
one queue first, which, in the meantime, also gets the max_queues the
backend supports.
In older version, it was reported that some messages are sent more times
than necessary. Here we came an agreement with Michael that we could
categorize vhost user messages to 2 types: non-vring specific messages,
which should be sent only once, and vring specific messages, which should
be sent per queue.
Here I introduced a helper function vhost_user_one_time_request(), which
lists following messages as non-vring specific messages:
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUM
For above messages, we simply ignore them when they are not sent the first
time.
Signed-off-by: Nikolay Nikolaev <n.nikolaev@virtualopensystems.com>
Signed-off-by: Changchun Ouyang <changchun.ouyang@intel.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Jason Wang <jasowang@redhat.com>
Tested-by: Marcel Apfelbaum <marcel@redhat.com>
2015-09-23 07:20:00 +03:00
|
|
|
|
2015-10-09 18:17:29 +03:00
|
|
|
Migration
|
|
|
|
---------
|
|
|
|
|
|
|
|
During live migration, the master may need to track the modifications
|
|
|
|
the slave makes to the memory mapped regions. The client should mark
|
|
|
|
the dirty pages in a log. Once it complies to this logging, it may
|
|
|
|
declare the VHOST_F_LOG_ALL vhost feature.
|
|
|
|
|
2015-11-15 22:25:11 +03:00
|
|
|
To start/stop logging of data/used ring writes, server may send messages
|
|
|
|
VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with
|
|
|
|
VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively.
|
|
|
|
|
2015-10-09 18:17:29 +03:00
|
|
|
All the modifications to memory pointed by vring "descriptor" should
|
|
|
|
be marked. Modifications to "used" vring should be marked if
|
2015-11-15 22:25:11 +03:00
|
|
|
VHOST_VRING_F_LOG is part of ring's flags.
|
2015-10-09 18:17:29 +03:00
|
|
|
|
|
|
|
Dirty pages are of size:
|
|
|
|
#define VHOST_LOG_PAGE 0x1000
|
|
|
|
|
|
|
|
The log memory fd is provided in the ancillary data of
|
|
|
|
VHOST_USER_SET_LOG_BASE message when the slave has
|
|
|
|
VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature.
|
|
|
|
|
2015-11-15 22:25:11 +03:00
|
|
|
The size of the log is supplied as part of VhostUserMsg
|
|
|
|
which should be large enough to cover all known guest
|
|
|
|
addresses. Log starts at the supplied offset in the
|
|
|
|
supplied file descriptor.
|
|
|
|
The log covers from address 0 to the maximum of guest
|
2015-10-09 18:17:29 +03:00
|
|
|
regions. In pseudo-code, to mark page at "addr" as dirty:
|
|
|
|
|
|
|
|
page = addr / VHOST_LOG_PAGE
|
|
|
|
log[page / 8] |= 1 << page % 8
|
|
|
|
|
2015-11-15 22:25:11 +03:00
|
|
|
Where addr is the guest physical address.
|
|
|
|
|
2015-10-09 18:17:29 +03:00
|
|
|
Use atomic operations, as the log may be concurrently manipulated.
|
|
|
|
|
2015-11-15 22:25:11 +03:00
|
|
|
Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG
|
|
|
|
is set for this ring), log_guest_addr should be used to calculate the log
|
|
|
|
offset: the write to first byte of the used ring is logged at this offset from
|
|
|
|
log start. Also note that this value might be outside the legal guest physical
|
|
|
|
address range (i.e. does not have to be covered by the VhostUserMemory table),
|
|
|
|
but the bit offset of the last byte of the ring must fall within
|
|
|
|
the size supplied by VhostUserLog.
|
|
|
|
|
2015-10-09 18:17:29 +03:00
|
|
|
VHOST_USER_SET_LOG_FD is an optional message with an eventfd in
|
|
|
|
ancillary data, it may be used to inform the master that the log has
|
|
|
|
been modified.
|
|
|
|
|
2015-11-15 22:25:11 +03:00
|
|
|
Once the source has finished migration, rings will be stopped by
|
|
|
|
the source. No further update must be done before rings are
|
|
|
|
restarted.
|
2015-10-09 18:17:29 +03:00
|
|
|
|
2017-12-14 23:22:20 +03:00
|
|
|
Memory access
|
|
|
|
-------------
|
|
|
|
|
|
|
|
The master sends a list of vhost memory regions to the slave using the
|
|
|
|
VHOST_USER_SET_MEM_TABLE message. Each region has two base addresses: a guest
|
|
|
|
address and a user address.
|
|
|
|
|
|
|
|
Messages contain guest addresses and/or user addresses to reference locations
|
|
|
|
within the shared memory. The mapping of these addresses works as follows.
|
|
|
|
|
|
|
|
User addresses map to the vhost memory region containing that user address.
|
|
|
|
|
|
|
|
When the VIRTIO_F_IOMMU_PLATFORM feature has not been negotiated:
|
|
|
|
|
|
|
|
* Guest addresses map to the vhost memory region containing that guest
|
|
|
|
address.
|
|
|
|
|
|
|
|
When the VIRTIO_F_IOMMU_PLATFORM feature has been negotiated:
|
|
|
|
|
|
|
|
* Guest addresses are also called I/O virtual addresses (IOVAs). They are
|
|
|
|
translated to user addresses via the IOTLB.
|
|
|
|
|
|
|
|
* The vhost memory region guest address is not used.
|
|
|
|
|
2017-06-02 13:18:31 +03:00
|
|
|
IOMMU support
|
|
|
|
-------------
|
|
|
|
|
|
|
|
When the VIRTIO_F_IOMMU_PLATFORM feature has been negotiated, the master
|
|
|
|
sends IOTLB entries update & invalidation by sending VHOST_USER_IOTLB_MSG
|
|
|
|
requests to the slave with a struct vhost_iotlb_msg as payload. For update
|
|
|
|
events, the iotlb payload has to be filled with the update message type (2),
|
|
|
|
the I/O virtual address, the size, the user virtual address, and the
|
|
|
|
permissions flags. Addresses and size must be within vhost memory regions set
|
|
|
|
via the VHOST_USER_SET_MEM_TABLE request. For invalidation events, the iotlb
|
|
|
|
payload has to be filled with the invalidation message type (3), the I/O virtual
|
|
|
|
address and the size. On success, the slave is expected to reply with a zero
|
|
|
|
payload, non-zero otherwise.
|
|
|
|
|
|
|
|
The slave relies on the slave communcation channel (see "Slave communication"
|
|
|
|
section below) to send IOTLB miss and access failure events, by sending
|
|
|
|
VHOST_USER_SLAVE_IOTLB_MSG requests to the master with a struct vhost_iotlb_msg
|
|
|
|
as payload. For miss events, the iotlb payload has to be filled with the miss
|
|
|
|
message type (1), the I/O virtual address and the permissions flags. For access
|
|
|
|
failure event, the iotlb payload has to be filled with the access failure
|
|
|
|
message type (4), the I/O virtual address and the permissions flags.
|
|
|
|
For synchronization purpose, the slave may rely on the reply-ack feature,
|
|
|
|
so the master may send a reply when operation is completed if the reply-ack
|
|
|
|
feature is negotiated and slaves requests a reply. For miss events, completed
|
|
|
|
operation means either master sent an update message containing the IOTLB entry
|
|
|
|
containing requested address and permission, or master sent nothing if the IOTLB
|
|
|
|
miss message is invalid (invalid IOVA or permission).
|
|
|
|
|
|
|
|
The master isn't expected to take the initiative to send IOTLB update messages,
|
|
|
|
as the slave sends IOTLB miss messages for the guest virtual memory areas it
|
|
|
|
needs to access.
|
|
|
|
|
2017-06-02 13:18:30 +03:00
|
|
|
Slave communication
|
|
|
|
-------------------
|
|
|
|
|
|
|
|
An optional communication channel is provided if the slave declares
|
|
|
|
VHOST_USER_PROTOCOL_F_SLAVE_REQ protocol feature, to allow the slave to make
|
|
|
|
requests to the master.
|
|
|
|
|
|
|
|
The fd is provided via VHOST_USER_SET_SLAVE_REQ_FD ancillary data.
|
|
|
|
|
|
|
|
A slave may then send VHOST_USER_SLAVE_* messages to the master
|
|
|
|
using this fd communication channel.
|
|
|
|
|
2015-10-09 18:17:29 +03:00
|
|
|
Protocol features
|
|
|
|
-----------------
|
|
|
|
|
|
|
|
#define VHOST_USER_PROTOCOL_F_MQ 0
|
|
|
|
#define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
|
2015-10-09 18:17:32 +03:00
|
|
|
#define VHOST_USER_PROTOCOL_F_RARP 2
|
2016-08-05 13:53:50 +03:00
|
|
|
#define VHOST_USER_PROTOCOL_F_REPLY_ACK 3
|
2016-12-10 18:30:36 +03:00
|
|
|
#define VHOST_USER_PROTOCOL_F_MTU 4
|
2017-06-02 13:18:30 +03:00
|
|
|
#define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5
|
2017-06-14 20:44:38 +03:00
|
|
|
#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
|
2015-10-09 18:17:29 +03:00
|
|
|
|
2017-06-02 13:18:30 +03:00
|
|
|
Master message types
|
|
|
|
--------------------
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
* VHOST_USER_GET_FEATURES
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 1
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_GET_FEATURES
|
|
|
|
Master payload: N/A
|
|
|
|
Slave payload: u64
|
|
|
|
|
|
|
|
Get from the underlying vhost implementation the features bitmask.
|
2015-09-23 07:19:56 +03:00
|
|
|
Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for
|
|
|
|
VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES.
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
* VHOST_USER_SET_FEATURES
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 2
|
2014-06-10 14:02:57 +04:00
|
|
|
Ioctl: VHOST_SET_FEATURES
|
|
|
|
Master payload: u64
|
|
|
|
|
|
|
|
Enable features in the underlying vhost implementation using a bitmask.
|
2015-09-23 07:19:56 +03:00
|
|
|
Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for
|
|
|
|
VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES.
|
|
|
|
|
|
|
|
* VHOST_USER_GET_PROTOCOL_FEATURES
|
|
|
|
|
|
|
|
Id: 15
|
|
|
|
Equivalent ioctl: VHOST_GET_FEATURES
|
|
|
|
Master payload: N/A
|
|
|
|
Slave payload: u64
|
|
|
|
|
|
|
|
Get the protocol feature bitmask from the underlying vhost implementation.
|
|
|
|
Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
|
|
|
|
VHOST_USER_GET_FEATURES.
|
|
|
|
Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support
|
|
|
|
this message even before VHOST_USER_SET_FEATURES was called.
|
|
|
|
|
|
|
|
* VHOST_USER_SET_PROTOCOL_FEATURES
|
|
|
|
|
|
|
|
Id: 16
|
|
|
|
Ioctl: VHOST_SET_FEATURES
|
|
|
|
Master payload: u64
|
|
|
|
|
|
|
|
Enable protocol features in the underlying vhost implementation.
|
|
|
|
Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
|
|
|
|
VHOST_USER_GET_FEATURES.
|
|
|
|
Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support
|
|
|
|
this message even before VHOST_USER_SET_FEATURES was called.
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
* VHOST_USER_SET_OWNER
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 3
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_OWNER
|
|
|
|
Master payload: N/A
|
|
|
|
|
|
|
|
Issued when a new connection is established. It sets the current Master
|
|
|
|
as an owner of the session. This can be used on the Slave as a
|
|
|
|
"session start" flag.
|
|
|
|
|
2015-11-11 16:24:37 +03:00
|
|
|
* VHOST_USER_RESET_OWNER
|
2014-06-10 14:02:57 +04:00
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 4
|
2014-06-10 14:02:57 +04:00
|
|
|
Master payload: N/A
|
|
|
|
|
2015-11-23 13:48:52 +03:00
|
|
|
This is no longer used. Used to be sent to request disabling
|
2015-11-15 22:25:11 +03:00
|
|
|
all rings, but some clients interpreted it to also discard
|
|
|
|
connection state (this interpretation would lead to bugs).
|
|
|
|
It is recommended that clients either ignore this message,
|
2015-11-23 13:48:52 +03:00
|
|
|
or use it to disable all rings.
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
* VHOST_USER_SET_MEM_TABLE
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 5
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_MEM_TABLE
|
|
|
|
Master payload: memory regions description
|
|
|
|
|
|
|
|
Sets the memory map regions on the slave so it can translate the vring
|
|
|
|
addresses. In the ancillary data there is an array of file descriptors
|
|
|
|
for each memory mapped region. The size and ordering of the fds matches
|
|
|
|
the number and ordering of memory regions.
|
|
|
|
|
|
|
|
* VHOST_USER_SET_LOG_BASE
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 6
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_LOG_BASE
|
|
|
|
Master payload: u64
|
2015-10-09 18:17:29 +03:00
|
|
|
Slave payload: N/A
|
2014-06-10 14:02:57 +04:00
|
|
|
|
2015-11-11 17:26:02 +03:00
|
|
|
Sets logging shared memory space.
|
|
|
|
When slave has VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol
|
|
|
|
feature, the log memory fd is provided in the ancillary data of
|
|
|
|
VHOST_USER_SET_LOG_BASE message, the size and offset of shared
|
|
|
|
memory area provided in the message.
|
|
|
|
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
* VHOST_USER_SET_LOG_FD
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 7
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_LOG_FD
|
|
|
|
Master payload: N/A
|
|
|
|
|
|
|
|
Sets the logging file descriptor, which is passed as ancillary data.
|
|
|
|
|
|
|
|
* VHOST_USER_SET_VRING_NUM
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 8
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_VRING_NUM
|
|
|
|
Master payload: vring state description
|
|
|
|
|
2016-04-09 16:50:35 +03:00
|
|
|
Set the size of the queue.
|
2014-06-10 14:02:57 +04:00
|
|
|
|
|
|
|
* VHOST_USER_SET_VRING_ADDR
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 9
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_VRING_ADDR
|
|
|
|
Master payload: vring address description
|
|
|
|
Slave payload: N/A
|
|
|
|
|
|
|
|
Sets the addresses of the different aspects of the vring.
|
|
|
|
|
|
|
|
* VHOST_USER_SET_VRING_BASE
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 10
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_VRING_BASE
|
|
|
|
Master payload: vring state description
|
|
|
|
|
|
|
|
Sets the base offset in the available vring.
|
|
|
|
|
|
|
|
* VHOST_USER_GET_VRING_BASE
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 11
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_USER_GET_VRING_BASE
|
|
|
|
Master payload: vring state description
|
|
|
|
Slave payload: vring state description
|
|
|
|
|
|
|
|
Get the available vring base offset.
|
|
|
|
|
|
|
|
* VHOST_USER_SET_VRING_KICK
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 12
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_VRING_KICK
|
|
|
|
Master payload: u64
|
|
|
|
|
|
|
|
Set the event file descriptor for adding buffers to the vring. It
|
|
|
|
is passed in the ancillary data.
|
|
|
|
Bits (0-7) of the payload contain the vring index. Bit 8 is the
|
|
|
|
invalid FD flag. This flag is set when there is no file descriptor
|
|
|
|
in the ancillary data. This signals that polling should be used
|
|
|
|
instead of waiting for a kick.
|
|
|
|
|
|
|
|
* VHOST_USER_SET_VRING_CALL
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 13
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_VRING_CALL
|
|
|
|
Master payload: u64
|
|
|
|
|
|
|
|
Set the event file descriptor to signal when buffers are used. It
|
|
|
|
is passed in the ancillary data.
|
|
|
|
Bits (0-7) of the payload contain the vring index. Bit 8 is the
|
|
|
|
invalid FD flag. This flag is set when there is no file descriptor
|
|
|
|
in the ancillary data. This signals that polling will be used
|
|
|
|
instead of waiting for the call.
|
|
|
|
|
|
|
|
* VHOST_USER_SET_VRING_ERR
|
|
|
|
|
2014-06-25 16:49:36 +04:00
|
|
|
Id: 14
|
2014-06-10 14:02:57 +04:00
|
|
|
Equivalent ioctl: VHOST_SET_VRING_ERR
|
|
|
|
Master payload: u64
|
|
|
|
|
|
|
|
Set the event file descriptor to signal when error occurs. It
|
|
|
|
is passed in the ancillary data.
|
|
|
|
Bits (0-7) of the payload contain the vring index. Bit 8 is the
|
|
|
|
invalid FD flag. This flag is set when there is no file descriptor
|
|
|
|
in the ancillary data.
|
2015-09-23 07:19:58 +03:00
|
|
|
|
|
|
|
* VHOST_USER_GET_QUEUE_NUM
|
|
|
|
|
|
|
|
Id: 17
|
|
|
|
Equivalent ioctl: N/A
|
|
|
|
Master payload: N/A
|
|
|
|
Slave payload: u64
|
|
|
|
|
|
|
|
Query how many queues the backend supports. This request should be
|
2016-04-09 16:50:36 +03:00
|
|
|
sent only when VHOST_USER_PROTOCOL_F_MQ is set in queried protocol
|
2015-09-23 07:19:58 +03:00
|
|
|
features by VHOST_USER_GET_PROTOCOL_FEATURES.
|
2015-09-23 07:20:01 +03:00
|
|
|
|
|
|
|
* VHOST_USER_SET_VRING_ENABLE
|
|
|
|
|
|
|
|
Id: 18
|
|
|
|
Equivalent ioctl: N/A
|
|
|
|
Master payload: vring state description
|
|
|
|
|
|
|
|
Signal slave to enable or disable corresponding vring.
|
2015-11-15 22:25:11 +03:00
|
|
|
This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
|
|
|
|
has been negotiated.
|
2015-10-09 18:17:32 +03:00
|
|
|
|
|
|
|
* VHOST_USER_SEND_RARP
|
|
|
|
|
|
|
|
Id: 19
|
|
|
|
Equivalent ioctl: N/A
|
|
|
|
Master payload: u64
|
|
|
|
|
|
|
|
Ask vhost user backend to broadcast a fake RARP to notify the migration
|
|
|
|
is terminated for guest that does not support GUEST_ANNOUNCE.
|
|
|
|
Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
|
|
|
|
VHOST_USER_GET_FEATURES and protocol feature bit VHOST_USER_PROTOCOL_F_RARP
|
|
|
|
is present in VHOST_USER_GET_PROTOCOL_FEATURES.
|
|
|
|
The first 6 bytes of the payload contain the mac address of the guest to
|
|
|
|
allow the vhost user backend to construct and broadcast the fake RARP.
|
2016-08-05 13:53:50 +03:00
|
|
|
|
2016-12-10 18:30:36 +03:00
|
|
|
* VHOST_USER_NET_SET_MTU
|
|
|
|
|
|
|
|
Id: 20
|
|
|
|
Equivalent ioctl: N/A
|
|
|
|
Master payload: u64
|
|
|
|
|
|
|
|
Set host MTU value exposed to the guest.
|
|
|
|
This request should be sent only when VIRTIO_NET_F_MTU feature has been
|
|
|
|
successfully negotiated, VHOST_USER_F_PROTOCOL_FEATURES is present in
|
|
|
|
VHOST_USER_GET_FEATURES and protocol feature bit
|
|
|
|
VHOST_USER_PROTOCOL_F_NET_MTU is present in
|
|
|
|
VHOST_USER_GET_PROTOCOL_FEATURES.
|
|
|
|
If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated, slave must respond
|
|
|
|
with zero in case the specified MTU is valid, or non-zero otherwise.
|
|
|
|
|
2017-06-02 13:18:30 +03:00
|
|
|
* VHOST_USER_SET_SLAVE_REQ_FD
|
|
|
|
|
|
|
|
Id: 21
|
|
|
|
Equivalent ioctl: N/A
|
|
|
|
Master payload: N/A
|
|
|
|
|
|
|
|
Set the socket file descriptor for slave initiated requests. It is passed
|
|
|
|
in the ancillary data.
|
|
|
|
This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
|
|
|
|
has been negotiated, and protocol feature bit VHOST_USER_PROTOCOL_F_SLAVE_REQ
|
|
|
|
bit is present in VHOST_USER_GET_PROTOCOL_FEATURES.
|
|
|
|
If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated, slave must respond
|
|
|
|
with zero for success, non-zero otherwise.
|
|
|
|
|
2017-06-02 13:18:31 +03:00
|
|
|
* VHOST_USER_IOTLB_MSG
|
|
|
|
|
|
|
|
Id: 22
|
|
|
|
Equivalent ioctl: N/A (equivalent to VHOST_IOTLB_MSG message type)
|
|
|
|
Master payload: struct vhost_iotlb_msg
|
|
|
|
Slave payload: u64
|
|
|
|
|
|
|
|
Send IOTLB messages with struct vhost_iotlb_msg as payload.
|
|
|
|
Master sends such requests to update and invalidate entries in the device
|
|
|
|
IOTLB. The slave has to acknowledge the request with sending zero as u64
|
|
|
|
payload for success, non-zero otherwise.
|
|
|
|
This request should be send only when VIRTIO_F_IOMMU_PLATFORM feature
|
|
|
|
has been successfully negotiated.
|
|
|
|
|
2017-06-14 20:44:38 +03:00
|
|
|
* VHOST_USER_SET_VRING_ENDIAN
|
|
|
|
|
|
|
|
Id: 23
|
|
|
|
Equivalent ioctl: VHOST_SET_VRING_ENDIAN
|
|
|
|
Master payload: vring state description
|
|
|
|
|
|
|
|
Set the endianess of a VQ for legacy devices. Little-endian is indicated
|
|
|
|
with state.num set to 0 and big-endian is indicated with state.num set
|
|
|
|
to 1. Other values are invalid.
|
|
|
|
This request should be sent only when VHOST_USER_PROTOCOL_F_CROSS_ENDIAN
|
|
|
|
has been negotiated.
|
|
|
|
Backends that negotiated this feature should handle both endianesses
|
|
|
|
and expect this message once (per VQ) during device configuration
|
|
|
|
(ie. before the master starts the VQ).
|
|
|
|
|
2018-01-04 04:53:31 +03:00
|
|
|
* VHOST_USER_GET_CONFIG
|
|
|
|
|
|
|
|
Id: 24
|
|
|
|
Equivalent ioctl: N/A
|
|
|
|
Master payload: virtio device config space
|
|
|
|
Slave payload: virtio device config space
|
|
|
|
|
|
|
|
Submitted by the vhost-user master to fetch the contents of the virtio
|
|
|
|
device configuration space, vhost-user slave's payload size MUST match
|
|
|
|
master's request, vhost-user slave uses zero length of payload to
|
|
|
|
indicate an error to vhost-user master. The vhost-user master may
|
|
|
|
cache the contents to avoid repeated VHOST_USER_GET_CONFIG calls.
|
|
|
|
|
|
|
|
* VHOST_USER_SET_CONFIG
|
|
|
|
|
|
|
|
Id: 25
|
|
|
|
Equivalent ioctl: N/A
|
|
|
|
Master payload: virtio device config space
|
|
|
|
Slave payload: N/A
|
|
|
|
|
|
|
|
Submitted by the vhost-user master when the Guest changes the virtio
|
|
|
|
device configuration space and also can be used for live migration
|
|
|
|
on the destination host. The vhost-user slave must check the flags
|
|
|
|
field, and slaves MUST NOT accept SET_CONFIG for read-only
|
|
|
|
configuration space fields unless the live migration bit is set.
|
|
|
|
|
2017-06-02 13:18:31 +03:00
|
|
|
Slave message types
|
|
|
|
-------------------
|
|
|
|
|
|
|
|
* VHOST_USER_SLAVE_IOTLB_MSG
|
|
|
|
|
|
|
|
Id: 1
|
|
|
|
Equivalent ioctl: N/A (equivalent to VHOST_IOTLB_MSG message type)
|
|
|
|
Slave payload: struct vhost_iotlb_msg
|
|
|
|
Master payload: N/A
|
|
|
|
|
|
|
|
Send IOTLB messages with struct vhost_iotlb_msg as payload.
|
|
|
|
Slave sends such requests to notify of an IOTLB miss, or an IOTLB
|
|
|
|
access failure. If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated,
|
|
|
|
and slave set the VHOST_USER_NEED_REPLY flag, master must respond with
|
|
|
|
zero when operation is successfully completed, or non-zero otherwise.
|
|
|
|
This request should be send only when VIRTIO_F_IOMMU_PLATFORM feature
|
|
|
|
has been successfully negotiated.
|
|
|
|
|
2018-01-04 04:53:31 +03:00
|
|
|
* VHOST_USER_SLAVE_CONFIG_CHANGE_MSG
|
|
|
|
|
|
|
|
Id: 2
|
|
|
|
Equivalent ioctl: N/A
|
|
|
|
Slave payload: N/A
|
|
|
|
Master payload: N/A
|
|
|
|
|
|
|
|
Vhost-user slave sends such messages to notify that the virtio device's
|
|
|
|
configuration space has changed, for those host devices which can support
|
|
|
|
such feature, host driver can send VHOST_USER_GET_CONFIG message to slave
|
|
|
|
to get the latest content. If VHOST_USER_PROTOCOL_F_REPLY_ACK is
|
|
|
|
negotiated, and slave set the VHOST_USER_NEED_REPLY flag, master must
|
|
|
|
respond with zero when operation is successfully completed, or non-zero
|
|
|
|
otherwise.
|
|
|
|
|
2016-08-05 13:53:50 +03:00
|
|
|
VHOST_USER_PROTOCOL_F_REPLY_ACK:
|
|
|
|
-------------------------------
|
|
|
|
The original vhost-user specification only demands replies for certain
|
|
|
|
commands. This differs from the vhost protocol implementation where commands
|
|
|
|
are sent over an ioctl() call and block until the client has completed.
|
|
|
|
|
|
|
|
With this protocol extension negotiated, the sender (QEMU) can set the
|
|
|
|
"need_reply" [Bit 3] flag to any command. This indicates that
|
|
|
|
the client MUST respond with a Payload VhostUserMsg indicating success or
|
|
|
|
failure. The payload should be set to zero on success or non-zero on failure,
|
|
|
|
unless the message already has an explicit reply body.
|
|
|
|
|
|
|
|
The response payload gives QEMU a deterministic indication of the result
|
|
|
|
of the command. Today, QEMU is expected to terminate the main vhost-user
|
|
|
|
loop upon receiving such errors. In future, qemu could be taught to be more
|
|
|
|
resilient for selective requests.
|
|
|
|
|
|
|
|
For the message types that already solicit a reply from the client, the
|
|
|
|
presence of VHOST_USER_PROTOCOL_F_REPLY_ACK or need_reply bit being set brings
|
|
|
|
no behavioural change. (See the 'Communication' section for details.)
|