spapr_pci_vfio: Add spapr-pci-vfio-host-bridge to support vfio
The patch adds a spapr-pci-vfio-host-bridge device type
which is a PCI Host Bridge with VFIO support. The new device
inherits from the spapr-pci-host-bridge device and adds an "iommu"
property which is an IOMMU id. This ID represents a minimal entity
for which IOMMU isolation can be guaranteed. In SPAPR architecture IOMMU
group is called a Partitionable Endpoint (PE).
Current implementation supports one IOMMU id per QEMU VFIO PHB. Since
SPAPR allows multiple PHB for no extra cost, this does not seem to
be a problem. This limitation may change in the future though.
Example of use:
Configure and Add 3 functions of a multifunctional device to QEMU:
(the NEC PCI USB card is used as an example here):
-device spapr-pci-vfio-host-bridge,id=USB,iommu=4,index=7 \
-device vfio-pci,host=4:0:1.0,addr=1.0,bus=USB,multifunction=true
-device vfio-pci,host=4:0:1.1,addr=1.1,bus=USB
-device vfio-pci,host=4:0:1.2,addr=1.2,bus=USB
where:
* index=7 is a QEMU PHB index (used as source for MMIO/MSI/IO windows
offset);
* iommu=4 is an IOMMU id which can be found in sysfs:
[aik@vpl2 ~]$ cd /sys/bus/pci/devices/0004:00:00.0/
[aik@vpl2 0004:00:00.0]$ ls -l iommu_group
lrwxrwxrwx 1 root root 0 Jun 5 12:49 iommu_group -> ../../../kernel/iommu_groups/4
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Alexander Graf <agraf@suse.de>
2014-06-10 09:39:23 +04:00
|
|
|
/*
|
|
|
|
* QEMU sPAPR PCI host for VFIO
|
|
|
|
*
|
|
|
|
* Copyright (c) 2011-2014 Alexey Kardashevskiy, IBM Corporation.
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or modify
|
|
|
|
* it under the terms of the GNU General Public License as published by
|
|
|
|
* the Free Software Foundation; either version 2 of the License,
|
|
|
|
* or (at your option) any later version.
|
|
|
|
*
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
* along with this program; if not, see <http://www.gnu.org/licenses/>.
|
|
|
|
*/
|
|
|
|
|
2016-01-26 21:16:58 +03:00
|
|
|
#include "qemu/osdep.h"
|
2016-06-22 20:11:19 +03:00
|
|
|
#include <linux/vfio.h>
|
spapr_pci_vfio: Add spapr-pci-vfio-host-bridge to support vfio
The patch adds a spapr-pci-vfio-host-bridge device type
which is a PCI Host Bridge with VFIO support. The new device
inherits from the spapr-pci-host-bridge device and adds an "iommu"
property which is an IOMMU id. This ID represents a minimal entity
for which IOMMU isolation can be guaranteed. In SPAPR architecture IOMMU
group is called a Partitionable Endpoint (PE).
Current implementation supports one IOMMU id per QEMU VFIO PHB. Since
SPAPR allows multiple PHB for no extra cost, this does not seem to
be a problem. This limitation may change in the future though.
Example of use:
Configure and Add 3 functions of a multifunctional device to QEMU:
(the NEC PCI USB card is used as an example here):
-device spapr-pci-vfio-host-bridge,id=USB,iommu=4,index=7 \
-device vfio-pci,host=4:0:1.0,addr=1.0,bus=USB,multifunction=true
-device vfio-pci,host=4:0:1.1,addr=1.1,bus=USB
-device vfio-pci,host=4:0:1.2,addr=1.2,bus=USB
where:
* index=7 is a QEMU PHB index (used as source for MMIO/MSI/IO windows
offset);
* iommu=4 is an IOMMU id which can be found in sysfs:
[aik@vpl2 ~]$ cd /sys/bus/pci/devices/0004:00:00.0/
[aik@vpl2 0004:00:00.0]$ ls -l iommu_group
lrwxrwxrwx 1 root root 0 Jun 5 12:49 iommu_group -> ../../../kernel/iommu_groups/4
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Alexander Graf <agraf@suse.de>
2014-06-10 09:39:23 +04:00
|
|
|
#include "hw/ppc/spapr.h"
|
|
|
|
#include "hw/pci-host/spapr.h"
|
2015-07-02 09:23:28 +03:00
|
|
|
#include "hw/pci/msix.h"
|
2022-12-22 13:03:28 +03:00
|
|
|
#include "hw/pci/pci_device.h"
|
2014-12-20 01:24:06 +03:00
|
|
|
#include "hw/vfio/vfio.h"
|
2016-02-29 09:19:50 +03:00
|
|
|
#include "qemu/error-report.h"
|
spapr_pci_vfio: Add spapr-pci-vfio-host-bridge to support vfio
The patch adds a spapr-pci-vfio-host-bridge device type
which is a PCI Host Bridge with VFIO support. The new device
inherits from the spapr-pci-host-bridge device and adds an "iommu"
property which is an IOMMU id. This ID represents a minimal entity
for which IOMMU isolation can be guaranteed. In SPAPR architecture IOMMU
group is called a Partitionable Endpoint (PE).
Current implementation supports one IOMMU id per QEMU VFIO PHB. Since
SPAPR allows multiple PHB for no extra cost, this does not seem to
be a problem. This limitation may change in the future though.
Example of use:
Configure and Add 3 functions of a multifunctional device to QEMU:
(the NEC PCI USB card is used as an example here):
-device spapr-pci-vfio-host-bridge,id=USB,iommu=4,index=7 \
-device vfio-pci,host=4:0:1.0,addr=1.0,bus=USB,multifunction=true
-device vfio-pci,host=4:0:1.1,addr=1.1,bus=USB
-device vfio-pci,host=4:0:1.2,addr=1.2,bus=USB
where:
* index=7 is a QEMU PHB index (used as source for MMIO/MSI/IO windows
offset);
* iommu=4 is an IOMMU id which can be found in sysfs:
[aik@vpl2 ~]$ cd /sys/bus/pci/devices/0004:00:00.0/
[aik@vpl2 0004:00:00.0]$ ls -l iommu_group
lrwxrwxrwx 1 root root 0 Jun 5 12:49 iommu_group -> ../../../kernel/iommu_groups/4
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Alexander Graf <agraf@suse.de>
2014-06-10 09:39:23 +04:00
|
|
|
|
spapr: Use CamelCase properly
The qemu coding standard is to use CamelCase for type and structure names,
and the pseries code follows that... sort of. There are quite a lot of
places where we bend the rules in order to preserve the capitalization of
internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR".
That was a bad idea - it frequently leads to names ending up with hard to
read clusters of capital letters, and means they don't catch the eye as
type identifiers, which is kind of the point of the CamelCase convention in
the first place.
In short, keeping type identifiers look like CamelCase is more important
than preserving standard capitalization of internal "words". So, this
patch renames a heap of spapr internal type names to a more standard
CamelCase.
In addition to case changes, we also make some other identifier renames:
VIOsPAPR* -> SpaprVio*
The reverse word ordering was only ever used to mitigate the capital
cluster, so revert to the natural ordering.
VIOsPAPRVTYDevice -> SpaprVioVty
VIOsPAPRVLANDevice -> SpaprVioVlan
Brevity, since the "Device" didn't add useful information
sPAPRDRConnector -> SpaprDrc
sPAPRDRConnectorClass -> SpaprDrcClass
Brevity, and makes it clearer this is the same thing as a "DRC"
mentioned in many other places in the code
This is 100% a mechanical search-and-replace patch. It will, however,
conflict with essentially any and all outstanding patches touching the
spapr code.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
|
|
|
bool spapr_phb_eeh_available(SpaprPhbState *sphb)
|
2016-02-29 09:19:42 +03:00
|
|
|
{
|
|
|
|
return vfio_eeh_as_ok(&sphb->iommu_as);
|
|
|
|
}
|
|
|
|
|
spapr: Use CamelCase properly
The qemu coding standard is to use CamelCase for type and structure names,
and the pseries code follows that... sort of. There are quite a lot of
places where we bend the rules in order to preserve the capitalization of
internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR".
That was a bad idea - it frequently leads to names ending up with hard to
read clusters of capital letters, and means they don't catch the eye as
type identifiers, which is kind of the point of the CamelCase convention in
the first place.
In short, keeping type identifiers look like CamelCase is more important
than preserving standard capitalization of internal "words". So, this
patch renames a heap of spapr internal type names to a more standard
CamelCase.
In addition to case changes, we also make some other identifier renames:
VIOsPAPR* -> SpaprVio*
The reverse word ordering was only ever used to mitigate the capital
cluster, so revert to the natural ordering.
VIOsPAPRVTYDevice -> SpaprVioVty
VIOsPAPRVLANDevice -> SpaprVioVlan
Brevity, since the "Device" didn't add useful information
sPAPRDRConnector -> SpaprDrc
sPAPRDRConnectorClass -> SpaprDrcClass
Brevity, and makes it clearer this is the same thing as a "DRC"
mentioned in many other places in the code
This is 100% a mechanical search-and-replace patch. It will, however,
conflict with essentially any and all outstanding patches touching the
spapr code.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
|
|
|
static void spapr_phb_vfio_eeh_reenable(SpaprPhbState *sphb)
|
2015-07-02 09:23:27 +03:00
|
|
|
{
|
2016-02-29 06:00:34 +03:00
|
|
|
vfio_eeh_as_op(&sphb->iommu_as, VFIO_EEH_PE_ENABLE);
|
2015-07-02 09:23:27 +03:00
|
|
|
}
|
|
|
|
|
2016-02-29 09:45:05 +03:00
|
|
|
void spapr_phb_vfio_reset(DeviceState *qdev)
|
spapr_pci_vfio: Add spapr-pci-vfio-host-bridge to support vfio
The patch adds a spapr-pci-vfio-host-bridge device type
which is a PCI Host Bridge with VFIO support. The new device
inherits from the spapr-pci-host-bridge device and adds an "iommu"
property which is an IOMMU id. This ID represents a minimal entity
for which IOMMU isolation can be guaranteed. In SPAPR architecture IOMMU
group is called a Partitionable Endpoint (PE).
Current implementation supports one IOMMU id per QEMU VFIO PHB. Since
SPAPR allows multiple PHB for no extra cost, this does not seem to
be a problem. This limitation may change in the future though.
Example of use:
Configure and Add 3 functions of a multifunctional device to QEMU:
(the NEC PCI USB card is used as an example here):
-device spapr-pci-vfio-host-bridge,id=USB,iommu=4,index=7 \
-device vfio-pci,host=4:0:1.0,addr=1.0,bus=USB,multifunction=true
-device vfio-pci,host=4:0:1.1,addr=1.1,bus=USB
-device vfio-pci,host=4:0:1.2,addr=1.2,bus=USB
where:
* index=7 is a QEMU PHB index (used as source for MMIO/MSI/IO windows
offset);
* iommu=4 is an IOMMU id which can be found in sysfs:
[aik@vpl2 ~]$ cd /sys/bus/pci/devices/0004:00:00.0/
[aik@vpl2 0004:00:00.0]$ ls -l iommu_group
lrwxrwxrwx 1 root root 0 Jun 5 12:49 iommu_group -> ../../../kernel/iommu_groups/4
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Alexander Graf <agraf@suse.de>
2014-06-10 09:39:23 +04:00
|
|
|
{
|
2015-07-02 09:23:27 +03:00
|
|
|
/*
|
|
|
|
* The PE might be in frozen state. To reenable the EEH
|
|
|
|
* functionality on it will clean the frozen state, which
|
|
|
|
* ensures that the contained PCI devices will work properly
|
|
|
|
* after reboot.
|
|
|
|
*/
|
2016-02-29 06:00:34 +03:00
|
|
|
spapr_phb_vfio_eeh_reenable(SPAPR_PCI_HOST_BRIDGE(qdev));
|
spapr_pci_vfio: Add spapr-pci-vfio-host-bridge to support vfio
The patch adds a spapr-pci-vfio-host-bridge device type
which is a PCI Host Bridge with VFIO support. The new device
inherits from the spapr-pci-host-bridge device and adds an "iommu"
property which is an IOMMU id. This ID represents a minimal entity
for which IOMMU isolation can be guaranteed. In SPAPR architecture IOMMU
group is called a Partitionable Endpoint (PE).
Current implementation supports one IOMMU id per QEMU VFIO PHB. Since
SPAPR allows multiple PHB for no extra cost, this does not seem to
be a problem. This limitation may change in the future though.
Example of use:
Configure and Add 3 functions of a multifunctional device to QEMU:
(the NEC PCI USB card is used as an example here):
-device spapr-pci-vfio-host-bridge,id=USB,iommu=4,index=7 \
-device vfio-pci,host=4:0:1.0,addr=1.0,bus=USB,multifunction=true
-device vfio-pci,host=4:0:1.1,addr=1.1,bus=USB
-device vfio-pci,host=4:0:1.2,addr=1.2,bus=USB
where:
* index=7 is a QEMU PHB index (used as source for MMIO/MSI/IO windows
offset);
* iommu=4 is an IOMMU id which can be found in sysfs:
[aik@vpl2 ~]$ cd /sys/bus/pci/devices/0004:00:00.0/
[aik@vpl2 0004:00:00.0]$ ls -l iommu_group
lrwxrwxrwx 1 root root 0 Jun 5 12:49 iommu_group -> ../../../kernel/iommu_groups/4
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Alexander Graf <agraf@suse.de>
2014-06-10 09:39:23 +04:00
|
|
|
}
|
|
|
|
|
spapr: Fix EEH capability issue on KVM guest for PCI passthru
With upstream kernel, especially after commit 98ba956f6a389
("powerpc/pseries/eeh: Rework device EEH PE determination") we see that KVM
guest isn't able to enable EEH option for PCI pass-through devices anymore.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.032337] EEH: pSeries platform initialized
[ 0.298207] EEH: No capable adapters found: recovery disabled.
[root@atest-guest ~]#
So far the linux kernel was assuming pe_config_addr equal to device's
config_addr and using it to enable EEH on the PE through ibm,set-eeh-option
RTAS call. Which wasn't the correct way as per PAPR. The linux kernel
commit 98ba956f6a389 fixed this flow. With that fixed, linux now uses PE
config address returned by ibm,get-config-addr-info2 RTAS call to enable
EEH option per-PE basis instead of per-device basis. However this has
uncovered a bug in qemu where ibm,set-eeh-option is treating PE config
address as per-device config address.
Hence in qemu guest with recent kernel the ibm,set-eeh-option RTAS call
fails with -3 return value indicating that there is no PCI device exist for
the specified PE config address. The rtas_ibm_set_eeh_option call uses
pci_find_device() to get the PC device that matches specific bus and devfn
extracted from PE config address passed as argument. Thus it tries to map
the PE config address to a single specific PCI device 'bus->devices[devfn]'
which always results into checking device on slot 0 'bus->devices[0]'.
This succeeds when there is a pass-through device (vfio-pci) present on
slot 0. But in cases where there is no pass-through device present in slot
0, but present in non-zero slots, ibm,set-eeh-option call fails to enable
the EEH capability.
hw/ppc/spapr_pci_vfio.c: spapr_phb_vfio_eeh_set_option()
case RTAS_EEH_ENABLE: {
PCIHostState *phb;
PCIDevice *pdev;
/*
* The EEH functionality is enabled on basis of PCI device,
* instead of PE. We need check the validity of the PCI
* device address.
*/
phb = PCI_HOST_BRIDGE(sphb);
pdev = pci_find_device(phb->bus,
(addr >> 16) & 0xFF, (addr >> 8) & 0xFF);
if (!pdev || !object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
return RTAS_OUT_PARAM_ERROR;
}
hw/pci/pci.c:pci_find_device()
PCIDevice *pci_find_device(PCIBus *bus, int bus_num, uint8_t devfn)
{
bus = pci_find_bus_nr(bus, bus_num);
if (!bus)
return NULL;
return bus->devices[devfn];
}
This patch fixes ibm,set-eeh-option to check for presence of any PCI device
(vfio-pci) under specified bus and enable the EEH if found. The current
code already makes sure that all the devices on that bus are from same
iommu group (within same PE) and fail very early if it does not.
After this fix guest is able to find EEH capable devices and enable EEH
recovery on it.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.048139] EEH: pSeries platform initialized
[ 0.405115] EEH: Capable adapter found: recovery enabled.
[root@atest-guest ~]#
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Message-Id: <162158429107.145117.5843504911924013125.stgit@jupiter>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-05-21 11:05:51 +03:00
|
|
|
static void spapr_eeh_pci_find_device(PCIBus *bus, PCIDevice *pdev,
|
|
|
|
void *opaque)
|
|
|
|
{
|
|
|
|
bool *found = opaque;
|
|
|
|
|
|
|
|
if (object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
|
|
|
|
*found = true;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
spapr: Use CamelCase properly
The qemu coding standard is to use CamelCase for type and structure names,
and the pseries code follows that... sort of. There are quite a lot of
places where we bend the rules in order to preserve the capitalization of
internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR".
That was a bad idea - it frequently leads to names ending up with hard to
read clusters of capital letters, and means they don't catch the eye as
type identifiers, which is kind of the point of the CamelCase convention in
the first place.
In short, keeping type identifiers look like CamelCase is more important
than preserving standard capitalization of internal "words". So, this
patch renames a heap of spapr internal type names to a more standard
CamelCase.
In addition to case changes, we also make some other identifier renames:
VIOsPAPR* -> SpaprVio*
The reverse word ordering was only ever used to mitigate the capital
cluster, so revert to the natural ordering.
VIOsPAPRVTYDevice -> SpaprVioVty
VIOsPAPRVLANDevice -> SpaprVioVlan
Brevity, since the "Device" didn't add useful information
sPAPRDRConnector -> SpaprDrc
sPAPRDRConnectorClass -> SpaprDrcClass
Brevity, and makes it clearer this is the same thing as a "DRC"
mentioned in many other places in the code
This is 100% a mechanical search-and-replace patch. It will, however,
conflict with essentially any and all outstanding patches touching the
spapr code.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
|
|
|
int spapr_phb_vfio_eeh_set_option(SpaprPhbState *sphb,
|
2016-02-29 09:45:05 +03:00
|
|
|
unsigned int addr, int option)
|
2015-02-20 07:58:53 +03:00
|
|
|
{
|
2016-02-29 06:00:34 +03:00
|
|
|
uint32_t op;
|
2015-02-20 07:58:53 +03:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
switch (option) {
|
|
|
|
case RTAS_EEH_DISABLE:
|
2016-02-29 06:00:34 +03:00
|
|
|
op = VFIO_EEH_PE_DISABLE;
|
2015-02-20 07:58:53 +03:00
|
|
|
break;
|
|
|
|
case RTAS_EEH_ENABLE: {
|
|
|
|
PCIHostState *phb;
|
spapr: Fix EEH capability issue on KVM guest for PCI passthru
With upstream kernel, especially after commit 98ba956f6a389
("powerpc/pseries/eeh: Rework device EEH PE determination") we see that KVM
guest isn't able to enable EEH option for PCI pass-through devices anymore.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.032337] EEH: pSeries platform initialized
[ 0.298207] EEH: No capable adapters found: recovery disabled.
[root@atest-guest ~]#
So far the linux kernel was assuming pe_config_addr equal to device's
config_addr and using it to enable EEH on the PE through ibm,set-eeh-option
RTAS call. Which wasn't the correct way as per PAPR. The linux kernel
commit 98ba956f6a389 fixed this flow. With that fixed, linux now uses PE
config address returned by ibm,get-config-addr-info2 RTAS call to enable
EEH option per-PE basis instead of per-device basis. However this has
uncovered a bug in qemu where ibm,set-eeh-option is treating PE config
address as per-device config address.
Hence in qemu guest with recent kernel the ibm,set-eeh-option RTAS call
fails with -3 return value indicating that there is no PCI device exist for
the specified PE config address. The rtas_ibm_set_eeh_option call uses
pci_find_device() to get the PC device that matches specific bus and devfn
extracted from PE config address passed as argument. Thus it tries to map
the PE config address to a single specific PCI device 'bus->devices[devfn]'
which always results into checking device on slot 0 'bus->devices[0]'.
This succeeds when there is a pass-through device (vfio-pci) present on
slot 0. But in cases where there is no pass-through device present in slot
0, but present in non-zero slots, ibm,set-eeh-option call fails to enable
the EEH capability.
hw/ppc/spapr_pci_vfio.c: spapr_phb_vfio_eeh_set_option()
case RTAS_EEH_ENABLE: {
PCIHostState *phb;
PCIDevice *pdev;
/*
* The EEH functionality is enabled on basis of PCI device,
* instead of PE. We need check the validity of the PCI
* device address.
*/
phb = PCI_HOST_BRIDGE(sphb);
pdev = pci_find_device(phb->bus,
(addr >> 16) & 0xFF, (addr >> 8) & 0xFF);
if (!pdev || !object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
return RTAS_OUT_PARAM_ERROR;
}
hw/pci/pci.c:pci_find_device()
PCIDevice *pci_find_device(PCIBus *bus, int bus_num, uint8_t devfn)
{
bus = pci_find_bus_nr(bus, bus_num);
if (!bus)
return NULL;
return bus->devices[devfn];
}
This patch fixes ibm,set-eeh-option to check for presence of any PCI device
(vfio-pci) under specified bus and enable the EEH if found. The current
code already makes sure that all the devices on that bus are from same
iommu group (within same PE) and fail very early if it does not.
After this fix guest is able to find EEH capable devices and enable EEH
recovery on it.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.048139] EEH: pSeries platform initialized
[ 0.405115] EEH: Capable adapter found: recovery enabled.
[root@atest-guest ~]#
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Message-Id: <162158429107.145117.5843504911924013125.stgit@jupiter>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-05-21 11:05:51 +03:00
|
|
|
bool found = false;
|
2015-02-20 07:58:53 +03:00
|
|
|
|
|
|
|
/*
|
spapr: Fix EEH capability issue on KVM guest for PCI passthru
With upstream kernel, especially after commit 98ba956f6a389
("powerpc/pseries/eeh: Rework device EEH PE determination") we see that KVM
guest isn't able to enable EEH option for PCI pass-through devices anymore.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.032337] EEH: pSeries platform initialized
[ 0.298207] EEH: No capable adapters found: recovery disabled.
[root@atest-guest ~]#
So far the linux kernel was assuming pe_config_addr equal to device's
config_addr and using it to enable EEH on the PE through ibm,set-eeh-option
RTAS call. Which wasn't the correct way as per PAPR. The linux kernel
commit 98ba956f6a389 fixed this flow. With that fixed, linux now uses PE
config address returned by ibm,get-config-addr-info2 RTAS call to enable
EEH option per-PE basis instead of per-device basis. However this has
uncovered a bug in qemu where ibm,set-eeh-option is treating PE config
address as per-device config address.
Hence in qemu guest with recent kernel the ibm,set-eeh-option RTAS call
fails with -3 return value indicating that there is no PCI device exist for
the specified PE config address. The rtas_ibm_set_eeh_option call uses
pci_find_device() to get the PC device that matches specific bus and devfn
extracted from PE config address passed as argument. Thus it tries to map
the PE config address to a single specific PCI device 'bus->devices[devfn]'
which always results into checking device on slot 0 'bus->devices[0]'.
This succeeds when there is a pass-through device (vfio-pci) present on
slot 0. But in cases where there is no pass-through device present in slot
0, but present in non-zero slots, ibm,set-eeh-option call fails to enable
the EEH capability.
hw/ppc/spapr_pci_vfio.c: spapr_phb_vfio_eeh_set_option()
case RTAS_EEH_ENABLE: {
PCIHostState *phb;
PCIDevice *pdev;
/*
* The EEH functionality is enabled on basis of PCI device,
* instead of PE. We need check the validity of the PCI
* device address.
*/
phb = PCI_HOST_BRIDGE(sphb);
pdev = pci_find_device(phb->bus,
(addr >> 16) & 0xFF, (addr >> 8) & 0xFF);
if (!pdev || !object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
return RTAS_OUT_PARAM_ERROR;
}
hw/pci/pci.c:pci_find_device()
PCIDevice *pci_find_device(PCIBus *bus, int bus_num, uint8_t devfn)
{
bus = pci_find_bus_nr(bus, bus_num);
if (!bus)
return NULL;
return bus->devices[devfn];
}
This patch fixes ibm,set-eeh-option to check for presence of any PCI device
(vfio-pci) under specified bus and enable the EEH if found. The current
code already makes sure that all the devices on that bus are from same
iommu group (within same PE) and fail very early if it does not.
After this fix guest is able to find EEH capable devices and enable EEH
recovery on it.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.048139] EEH: pSeries platform initialized
[ 0.405115] EEH: Capable adapter found: recovery enabled.
[root@atest-guest ~]#
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Message-Id: <162158429107.145117.5843504911924013125.stgit@jupiter>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-05-21 11:05:51 +03:00
|
|
|
* The EEH functionality is enabled per sphb level instead of
|
|
|
|
* per PCI device. We have already identified this specific sphb
|
|
|
|
* based on buid passed as argument to ibm,set-eeh-option rtas
|
|
|
|
* call. Now we just need to check the validity of the PCI
|
|
|
|
* pass-through devices (vfio-pci) under this sphb bus.
|
|
|
|
* We have already validated that all the devices under this sphb
|
2023-07-14 14:18:16 +03:00
|
|
|
* are from same iommu group (within same PE) before coming here.
|
spapr: Fix EEH capability issue on KVM guest for PCI passthru
With upstream kernel, especially after commit 98ba956f6a389
("powerpc/pseries/eeh: Rework device EEH PE determination") we see that KVM
guest isn't able to enable EEH option for PCI pass-through devices anymore.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.032337] EEH: pSeries platform initialized
[ 0.298207] EEH: No capable adapters found: recovery disabled.
[root@atest-guest ~]#
So far the linux kernel was assuming pe_config_addr equal to device's
config_addr and using it to enable EEH on the PE through ibm,set-eeh-option
RTAS call. Which wasn't the correct way as per PAPR. The linux kernel
commit 98ba956f6a389 fixed this flow. With that fixed, linux now uses PE
config address returned by ibm,get-config-addr-info2 RTAS call to enable
EEH option per-PE basis instead of per-device basis. However this has
uncovered a bug in qemu where ibm,set-eeh-option is treating PE config
address as per-device config address.
Hence in qemu guest with recent kernel the ibm,set-eeh-option RTAS call
fails with -3 return value indicating that there is no PCI device exist for
the specified PE config address. The rtas_ibm_set_eeh_option call uses
pci_find_device() to get the PC device that matches specific bus and devfn
extracted from PE config address passed as argument. Thus it tries to map
the PE config address to a single specific PCI device 'bus->devices[devfn]'
which always results into checking device on slot 0 'bus->devices[0]'.
This succeeds when there is a pass-through device (vfio-pci) present on
slot 0. But in cases where there is no pass-through device present in slot
0, but present in non-zero slots, ibm,set-eeh-option call fails to enable
the EEH capability.
hw/ppc/spapr_pci_vfio.c: spapr_phb_vfio_eeh_set_option()
case RTAS_EEH_ENABLE: {
PCIHostState *phb;
PCIDevice *pdev;
/*
* The EEH functionality is enabled on basis of PCI device,
* instead of PE. We need check the validity of the PCI
* device address.
*/
phb = PCI_HOST_BRIDGE(sphb);
pdev = pci_find_device(phb->bus,
(addr >> 16) & 0xFF, (addr >> 8) & 0xFF);
if (!pdev || !object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
return RTAS_OUT_PARAM_ERROR;
}
hw/pci/pci.c:pci_find_device()
PCIDevice *pci_find_device(PCIBus *bus, int bus_num, uint8_t devfn)
{
bus = pci_find_bus_nr(bus, bus_num);
if (!bus)
return NULL;
return bus->devices[devfn];
}
This patch fixes ibm,set-eeh-option to check for presence of any PCI device
(vfio-pci) under specified bus and enable the EEH if found. The current
code already makes sure that all the devices on that bus are from same
iommu group (within same PE) and fail very early if it does not.
After this fix guest is able to find EEH capable devices and enable EEH
recovery on it.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.048139] EEH: pSeries platform initialized
[ 0.405115] EEH: Capable adapter found: recovery enabled.
[root@atest-guest ~]#
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Message-Id: <162158429107.145117.5843504911924013125.stgit@jupiter>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-05-21 11:05:51 +03:00
|
|
|
*
|
|
|
|
* Prior to linux commit 98ba956f6a389 ("powerpc/pseries/eeh:
|
|
|
|
* Rework device EEH PE determination") kernel would call
|
|
|
|
* eeh-set-option for each device in the PE using the device's
|
|
|
|
* config_address as the argument rather than the PE address.
|
|
|
|
* Hence if we check validity of supplied config_addr whether
|
|
|
|
* it matches to this PHB will cause issues with older kernel
|
|
|
|
* versions v5.9 and older. If we return an error from
|
|
|
|
* eeh-set-option when the argument isn't a valid PE address
|
|
|
|
* then older kernels (v5.9 and older) will interpret that as
|
|
|
|
* EEH not being supported.
|
2015-02-20 07:58:53 +03:00
|
|
|
*/
|
|
|
|
phb = PCI_HOST_BRIDGE(sphb);
|
spapr: Fix EEH capability issue on KVM guest for PCI passthru
With upstream kernel, especially after commit 98ba956f6a389
("powerpc/pseries/eeh: Rework device EEH PE determination") we see that KVM
guest isn't able to enable EEH option for PCI pass-through devices anymore.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.032337] EEH: pSeries platform initialized
[ 0.298207] EEH: No capable adapters found: recovery disabled.
[root@atest-guest ~]#
So far the linux kernel was assuming pe_config_addr equal to device's
config_addr and using it to enable EEH on the PE through ibm,set-eeh-option
RTAS call. Which wasn't the correct way as per PAPR. The linux kernel
commit 98ba956f6a389 fixed this flow. With that fixed, linux now uses PE
config address returned by ibm,get-config-addr-info2 RTAS call to enable
EEH option per-PE basis instead of per-device basis. However this has
uncovered a bug in qemu where ibm,set-eeh-option is treating PE config
address as per-device config address.
Hence in qemu guest with recent kernel the ibm,set-eeh-option RTAS call
fails with -3 return value indicating that there is no PCI device exist for
the specified PE config address. The rtas_ibm_set_eeh_option call uses
pci_find_device() to get the PC device that matches specific bus and devfn
extracted from PE config address passed as argument. Thus it tries to map
the PE config address to a single specific PCI device 'bus->devices[devfn]'
which always results into checking device on slot 0 'bus->devices[0]'.
This succeeds when there is a pass-through device (vfio-pci) present on
slot 0. But in cases where there is no pass-through device present in slot
0, but present in non-zero slots, ibm,set-eeh-option call fails to enable
the EEH capability.
hw/ppc/spapr_pci_vfio.c: spapr_phb_vfio_eeh_set_option()
case RTAS_EEH_ENABLE: {
PCIHostState *phb;
PCIDevice *pdev;
/*
* The EEH functionality is enabled on basis of PCI device,
* instead of PE. We need check the validity of the PCI
* device address.
*/
phb = PCI_HOST_BRIDGE(sphb);
pdev = pci_find_device(phb->bus,
(addr >> 16) & 0xFF, (addr >> 8) & 0xFF);
if (!pdev || !object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
return RTAS_OUT_PARAM_ERROR;
}
hw/pci/pci.c:pci_find_device()
PCIDevice *pci_find_device(PCIBus *bus, int bus_num, uint8_t devfn)
{
bus = pci_find_bus_nr(bus, bus_num);
if (!bus)
return NULL;
return bus->devices[devfn];
}
This patch fixes ibm,set-eeh-option to check for presence of any PCI device
(vfio-pci) under specified bus and enable the EEH if found. The current
code already makes sure that all the devices on that bus are from same
iommu group (within same PE) and fail very early if it does not.
After this fix guest is able to find EEH capable devices and enable EEH
recovery on it.
[root@atest-guest ~]# dmesg | grep EEH
[ 0.048139] EEH: pSeries platform initialized
[ 0.405115] EEH: Capable adapter found: recovery enabled.
[root@atest-guest ~]#
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Message-Id: <162158429107.145117.5843504911924013125.stgit@jupiter>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-05-21 11:05:51 +03:00
|
|
|
pci_for_each_device(phb->bus, (addr >> 16) & 0xFF,
|
|
|
|
spapr_eeh_pci_find_device, &found);
|
|
|
|
|
|
|
|
if (!found) {
|
2015-02-20 07:58:53 +03:00
|
|
|
return RTAS_OUT_PARAM_ERROR;
|
|
|
|
}
|
|
|
|
|
2016-02-29 06:00:34 +03:00
|
|
|
op = VFIO_EEH_PE_ENABLE;
|
2015-02-20 07:58:53 +03:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
case RTAS_EEH_THAW_IO:
|
2016-02-29 06:00:34 +03:00
|
|
|
op = VFIO_EEH_PE_UNFREEZE_IO;
|
2015-02-20 07:58:53 +03:00
|
|
|
break;
|
|
|
|
case RTAS_EEH_THAW_DMA:
|
2016-02-29 06:00:34 +03:00
|
|
|
op = VFIO_EEH_PE_UNFREEZE_DMA;
|
2015-02-20 07:58:53 +03:00
|
|
|
break;
|
|
|
|
default:
|
|
|
|
return RTAS_OUT_PARAM_ERROR;
|
|
|
|
}
|
|
|
|
|
2016-02-29 06:00:34 +03:00
|
|
|
ret = vfio_eeh_as_op(&sphb->iommu_as, op);
|
2015-02-20 07:58:53 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return RTAS_OUT_HW_ERROR;
|
|
|
|
}
|
|
|
|
|
|
|
|
return RTAS_OUT_SUCCESS;
|
|
|
|
}
|
|
|
|
|
spapr: Use CamelCase properly
The qemu coding standard is to use CamelCase for type and structure names,
and the pseries code follows that... sort of. There are quite a lot of
places where we bend the rules in order to preserve the capitalization of
internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR".
That was a bad idea - it frequently leads to names ending up with hard to
read clusters of capital letters, and means they don't catch the eye as
type identifiers, which is kind of the point of the CamelCase convention in
the first place.
In short, keeping type identifiers look like CamelCase is more important
than preserving standard capitalization of internal "words". So, this
patch renames a heap of spapr internal type names to a more standard
CamelCase.
In addition to case changes, we also make some other identifier renames:
VIOsPAPR* -> SpaprVio*
The reverse word ordering was only ever used to mitigate the capital
cluster, so revert to the natural ordering.
VIOsPAPRVTYDevice -> SpaprVioVty
VIOsPAPRVLANDevice -> SpaprVioVlan
Brevity, since the "Device" didn't add useful information
sPAPRDRConnector -> SpaprDrc
sPAPRDRConnectorClass -> SpaprDrcClass
Brevity, and makes it clearer this is the same thing as a "DRC"
mentioned in many other places in the code
This is 100% a mechanical search-and-replace patch. It will, however,
conflict with essentially any and all outstanding patches touching the
spapr code.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
|
|
|
int spapr_phb_vfio_eeh_get_state(SpaprPhbState *sphb, int *state)
|
2015-02-20 07:58:53 +03:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2016-02-29 06:00:34 +03:00
|
|
|
ret = vfio_eeh_as_op(&sphb->iommu_as, VFIO_EEH_PE_GET_STATE);
|
2015-02-20 07:58:53 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return RTAS_OUT_PARAM_ERROR;
|
|
|
|
}
|
|
|
|
|
|
|
|
*state = ret;
|
|
|
|
return RTAS_OUT_SUCCESS;
|
|
|
|
}
|
|
|
|
|
2015-07-02 09:23:28 +03:00
|
|
|
static void spapr_phb_vfio_eeh_clear_dev_msix(PCIBus *bus,
|
|
|
|
PCIDevice *pdev,
|
|
|
|
void *opaque)
|
|
|
|
{
|
|
|
|
/* Check if the device is VFIO PCI device */
|
|
|
|
if (!object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The MSIx table will be cleaned out by reset. We need
|
|
|
|
* disable it so that it can be reenabled properly. Also,
|
|
|
|
* the cached MSIx table should be cleared as it's not
|
|
|
|
* reflecting the contents in hardware.
|
|
|
|
*/
|
|
|
|
if (msix_enabled(pdev)) {
|
|
|
|
uint16_t flags;
|
|
|
|
|
|
|
|
flags = pci_host_config_read_common(pdev,
|
|
|
|
pdev->msix_cap + PCI_MSIX_FLAGS,
|
|
|
|
pci_config_size(pdev), 2);
|
|
|
|
flags &= ~PCI_MSIX_FLAGS_ENABLE;
|
|
|
|
pci_host_config_write_common(pdev,
|
|
|
|
pdev->msix_cap + PCI_MSIX_FLAGS,
|
|
|
|
pci_config_size(pdev), flags, 2);
|
|
|
|
}
|
|
|
|
|
|
|
|
msix_reset(pdev);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void spapr_phb_vfio_eeh_clear_bus_msix(PCIBus *bus, void *opaque)
|
|
|
|
{
|
2021-10-28 07:31:26 +03:00
|
|
|
pci_for_each_device_under_bus(bus, spapr_phb_vfio_eeh_clear_dev_msix,
|
|
|
|
NULL);
|
2015-07-02 09:23:28 +03:00
|
|
|
}
|
|
|
|
|
spapr: Use CamelCase properly
The qemu coding standard is to use CamelCase for type and structure names,
and the pseries code follows that... sort of. There are quite a lot of
places where we bend the rules in order to preserve the capitalization of
internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR".
That was a bad idea - it frequently leads to names ending up with hard to
read clusters of capital letters, and means they don't catch the eye as
type identifiers, which is kind of the point of the CamelCase convention in
the first place.
In short, keeping type identifiers look like CamelCase is more important
than preserving standard capitalization of internal "words". So, this
patch renames a heap of spapr internal type names to a more standard
CamelCase.
In addition to case changes, we also make some other identifier renames:
VIOsPAPR* -> SpaprVio*
The reverse word ordering was only ever used to mitigate the capital
cluster, so revert to the natural ordering.
VIOsPAPRVTYDevice -> SpaprVioVty
VIOsPAPRVLANDevice -> SpaprVioVlan
Brevity, since the "Device" didn't add useful information
sPAPRDRConnector -> SpaprDrc
sPAPRDRConnectorClass -> SpaprDrcClass
Brevity, and makes it clearer this is the same thing as a "DRC"
mentioned in many other places in the code
This is 100% a mechanical search-and-replace patch. It will, however,
conflict with essentially any and all outstanding patches touching the
spapr code.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
|
|
|
static void spapr_phb_vfio_eeh_pre_reset(SpaprPhbState *sphb)
|
2015-07-02 09:23:28 +03:00
|
|
|
{
|
|
|
|
PCIHostState *phb = PCI_HOST_BRIDGE(sphb);
|
|
|
|
|
|
|
|
pci_for_each_bus(phb->bus, spapr_phb_vfio_eeh_clear_bus_msix, NULL);
|
|
|
|
}
|
|
|
|
|
spapr: Use CamelCase properly
The qemu coding standard is to use CamelCase for type and structure names,
and the pseries code follows that... sort of. There are quite a lot of
places where we bend the rules in order to preserve the capitalization of
internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR".
That was a bad idea - it frequently leads to names ending up with hard to
read clusters of capital letters, and means they don't catch the eye as
type identifiers, which is kind of the point of the CamelCase convention in
the first place.
In short, keeping type identifiers look like CamelCase is more important
than preserving standard capitalization of internal "words". So, this
patch renames a heap of spapr internal type names to a more standard
CamelCase.
In addition to case changes, we also make some other identifier renames:
VIOsPAPR* -> SpaprVio*
The reverse word ordering was only ever used to mitigate the capital
cluster, so revert to the natural ordering.
VIOsPAPRVTYDevice -> SpaprVioVty
VIOsPAPRVLANDevice -> SpaprVioVlan
Brevity, since the "Device" didn't add useful information
sPAPRDRConnector -> SpaprDrc
sPAPRDRConnectorClass -> SpaprDrcClass
Brevity, and makes it clearer this is the same thing as a "DRC"
mentioned in many other places in the code
This is 100% a mechanical search-and-replace patch. It will, however,
conflict with essentially any and all outstanding patches touching the
spapr code.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
|
|
|
int spapr_phb_vfio_eeh_reset(SpaprPhbState *sphb, int option)
|
2015-02-20 07:58:53 +03:00
|
|
|
{
|
2016-02-29 06:00:34 +03:00
|
|
|
uint32_t op;
|
2015-02-20 07:58:53 +03:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
switch (option) {
|
|
|
|
case RTAS_SLOT_RESET_DEACTIVATE:
|
2016-02-29 06:00:34 +03:00
|
|
|
op = VFIO_EEH_PE_RESET_DEACTIVATE;
|
2015-02-20 07:58:53 +03:00
|
|
|
break;
|
|
|
|
case RTAS_SLOT_RESET_HOT:
|
2015-07-02 09:23:28 +03:00
|
|
|
spapr_phb_vfio_eeh_pre_reset(sphb);
|
2016-02-29 06:00:34 +03:00
|
|
|
op = VFIO_EEH_PE_RESET_HOT;
|
2015-02-20 07:58:53 +03:00
|
|
|
break;
|
|
|
|
case RTAS_SLOT_RESET_FUNDAMENTAL:
|
2015-07-02 09:23:28 +03:00
|
|
|
spapr_phb_vfio_eeh_pre_reset(sphb);
|
2016-02-29 06:00:34 +03:00
|
|
|
op = VFIO_EEH_PE_RESET_FUNDAMENTAL;
|
2015-02-20 07:58:53 +03:00
|
|
|
break;
|
|
|
|
default:
|
|
|
|
return RTAS_OUT_PARAM_ERROR;
|
|
|
|
}
|
|
|
|
|
2016-02-29 06:00:34 +03:00
|
|
|
ret = vfio_eeh_as_op(&sphb->iommu_as, op);
|
2015-02-20 07:58:53 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return RTAS_OUT_HW_ERROR;
|
|
|
|
}
|
|
|
|
|
|
|
|
return RTAS_OUT_SUCCESS;
|
|
|
|
}
|
|
|
|
|
spapr: Use CamelCase properly
The qemu coding standard is to use CamelCase for type and structure names,
and the pseries code follows that... sort of. There are quite a lot of
places where we bend the rules in order to preserve the capitalization of
internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR".
That was a bad idea - it frequently leads to names ending up with hard to
read clusters of capital letters, and means they don't catch the eye as
type identifiers, which is kind of the point of the CamelCase convention in
the first place.
In short, keeping type identifiers look like CamelCase is more important
than preserving standard capitalization of internal "words". So, this
patch renames a heap of spapr internal type names to a more standard
CamelCase.
In addition to case changes, we also make some other identifier renames:
VIOsPAPR* -> SpaprVio*
The reverse word ordering was only ever used to mitigate the capital
cluster, so revert to the natural ordering.
VIOsPAPRVTYDevice -> SpaprVioVty
VIOsPAPRVLANDevice -> SpaprVioVlan
Brevity, since the "Device" didn't add useful information
sPAPRDRConnector -> SpaprDrc
sPAPRDRConnectorClass -> SpaprDrcClass
Brevity, and makes it clearer this is the same thing as a "DRC"
mentioned in many other places in the code
This is 100% a mechanical search-and-replace patch. It will, however,
conflict with essentially any and all outstanding patches touching the
spapr code.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
|
|
|
int spapr_phb_vfio_eeh_configure(SpaprPhbState *sphb)
|
2015-02-20 07:58:53 +03:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2016-02-29 06:00:34 +03:00
|
|
|
ret = vfio_eeh_as_op(&sphb->iommu_as, VFIO_EEH_PE_CONFIGURE);
|
2015-02-20 07:58:53 +03:00
|
|
|
if (ret < 0) {
|
|
|
|
return RTAS_OUT_PARAM_ERROR;
|
|
|
|
}
|
|
|
|
|
|
|
|
return RTAS_OUT_SUCCESS;
|
|
|
|
}
|