mirrors/qemu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ad08e67a96
qemu/include/hw/pci-host/spapr.h

219 lines
7.0 KiB
C
Raw Normal View History

pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
/*
* QEMU SPAPR PCI BUS definitions
*
* Copyright (c) 2011 Alexey Kardashevskiy <aik@au1.ibm.com>
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, see <http://www.gnu.org/licenses/>.
*/
Clean up header guards that don't match their file name Header guard symbols should match their file name to make guard collisions less likely. Offenders found with scripts/clean-header-guards.pl -vn. Cleaned up with scripts/clean-header-guards.pl, followed by some renaming of new guard symbols picked by the script to better ones. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Richard Henderson <rth@twiddle.net>
2016-06-29 11:12:57 +03:00
#ifndef PCI_HOST_SPAPR_H
#define PCI_HOST_SPAPR_H
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
spapr_pci: Include spapr.h instead of playing games with #error include/hw/pci-host/spapr.h needs hw/ppc/spapr.h. It checks whether its header guard is defined, and errors out if it isn't. Playing games with some other header's guard symbol is not a good idea. Just include the frackin' header already. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Richard Henderson <rth@twiddle.net>
2016-06-29 12:31:09 +03:00
#include "hw/ppc/spapr.h"
pci: update all users to look in pci/ update all users so we can remove the makefile hack. Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2012-12-12 16:24:50 +04:00
#include "hw/pci/pci.h"
#include "hw/pci/pci_host.h"
hw: move headers to include/ Many of these should be cleaned up with proper qdev-/QOM-ification. Right now there are many catch-all headers in include/hw/ARCH depending on cpu.h, and this makes it necessary to compile these files per-target. However, fixing this does not belong in these patches. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2013-02-05 20:06:20 +04:00
#include "hw/ppc/xics.h"
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
spapr_pci: QOM'ify sPAPR PCI host bridge Introduce type constant. Introduce cast macro to drop bogus busdev field that would've broken SYS_BUS_DEVICE(). Avoid accessing parent fields directly. Free the identifier phb as acronym for PCI_HOST_BRIDGE. Updated against conflicting merge from branch 'agraf/ppc-for-upstream' (0d16fdd732d1b211842fa96b7c90ddf9e6bde0e4), which removed busdev field differently, moved some code around and added new occurrences of 'phb'. Signed-off-by: Andreas Färber <afaerber@suse.de> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2012-08-20 21:08:05 +04:00
#define TYPE_SPAPR_PCI_HOST_BRIDGE "spapr-pci-host-bridge"
#define SPAPR_PCI_HOST_BRIDGE(obj) \
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
OBJECT_CHECK(SpaprPhbState, (obj), TYPE_SPAPR_PCI_HOST_BRIDGE)
spapr_pci: QOM'ify sPAPR PCI host bridge Introduce type constant. Introduce cast macro to drop bogus busdev field that would've broken SYS_BUS_DEVICE(). Avoid accessing parent fields directly. Free the identifier phb as acronym for PCI_HOST_BRIDGE. Updated against conflicting merge from branch 'agraf/ppc-for-upstream' (0d16fdd732d1b211842fa96b7c90ddf9e6bde0e4), which removed busdev field differently, moved some code around and added new occurrences of 'phb'. Signed-off-by: Andreas Färber <afaerber@suse.de> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2012-08-20 21:08:05 +04:00
spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW) This adds support for Dynamic DMA Windows (DDW) option defined by the SPAPR specification which allows to have additional DMA window(s) The "ddw" property is enabled by default on a PHB but for compatibility the pseries-2.6 machine and older disable it. This also creates a single DMA window for the older machines to maintain backward migration. This implements DDW for PHB with emulated and VFIO devices. The host kernel support is required. The advertised IOMMU page sizes are 4K and 64K; 16M pages are supported but not advertised by default, in order to enable them, the user has to specify "pgsz" property for PHB and enable huge pages for RAM. The existing linux guests try creating one additional huge DMA window with 64K or 16MB pages and map the entire guest RAM to. If succeeded, the guest switches to dma_direct_ops and never calls TCE hypercalls (H_PUT_TCE,...) again. This enables VFIO devices to use the entire RAM and not waste time on map/unmap later. This adds a "dma64_win_addr" property which is a bus address for the 64bit window and by default set to 0x800.0000.0000.0000 as this is what the modern POWER8 hardware uses and this allows having emulated and VFIO devices on the same bus. This adds 4 RTAS handlers: * ibm,query-pe-dma-window * ibm,create-pe-dma-window * ibm,remove-pe-dma-window * ibm,reset-pe-dma-window These are registered from type_init() callback. These RTAS handlers are implemented in a separate file to avoid polluting spapr_iommu.c with PCI. This changes sPAPRPHBState::dma_liobn to an array to allow 2 LIOBNs and updates all references to dma_liobn. However this does not add 64bit LIOBN to the migration stream as in fact even 32bit LIOBN is rather pointless there (as it is a PHB property and the management software can/should pass LIOBNs via CLI) but we keep it for the backward migration support. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2016-07-04 06:33:07 +03:00
#define SPAPR_PCI_DMA_MAX_WINDOWS 2
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
typedef struct SpaprPhbState SpaprPhbState;
spapr_pci: Introduce a finish_realize() callback The spapr-pci PHB initializes IOMMU for emulated devices only. The upcoming VFIO support will do it different. However both emulated and VFIO PHB types share most of the initialization code. For the type specific things a new finish_realize() callback is introduced. This introduces sPAPRPHBClass derived from PCIHostBridgeClass and adds the callback pointer. This implements finish_realize() for emulated devices. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [agraf: Fix compilation] Signed-off-by: Alexander Graf <agraf@suse.de>
2014-05-27 09:36:31 +04:00
spapr_pci: Use XICS interrupt allocator and do not cache interrupts in PHB Currently SPAPR PHB keeps track of all allocated MSI (here and below MSI stands for both MSI and MSIX) interrupt because XICS used to be unable to reuse interrupts. This is a problem for dynamic MSI reconfiguration which happens when guest reloads a driver or performs PCI hotplug. Another problem is that the existing implementation can enable MSI on 32 devices maximum (SPAPR_MSIX_MAX_DEVS=32) and there is no good reason for that. This makes use of new XICS ability to reuse interrupts. This reorganizes MSI information storage in sPAPRPHBState. Instead of static array of 32 descriptors (one per a PCI function), this patch adds a GHashTable when @config_addr is a key and (first_irq, num) pair is a value. GHashTable can dynamically grow and shrink so the initial limit of 32 devices is gone. This changes migration stream as @msi_table was a static array while new @msi_devs is a dynamic hash table. This adds temporary array which is used for migration, it is populated in "spapr_pci"::pre_save() callback and expanded into the hash table in post_load() callback. Since the destination side does not know the number of MSI-enabled devices in advance and cannot pre-allocate the temporary array to receive migration state, this makes use of new VMSTATE_STRUCT_VARRAY_ALLOC macro which allocates the array automatically. This resets the MSI configuration space when interrupts are released by the ibm,change-msi RTAS call. This fixed traces to be more informative. This changes vmstate_spapr_pci_msi name from "...lsi" to "...msi" which was incorrect by accident. As the internal representation changed, thus bumps migration version number. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [agraf: drop g_malloc_n usage] Signed-off-by: Alexander Graf <agraf@suse.de>
2014-05-30 13:34:20 +04:00
typedef struct spapr_pci_msi {
uint32_t first_irq;
uint32_t num;
} spapr_pci_msi;
typedef struct spapr_pci_msi_mig {
uint32_t key;
spapr_pci_msi value;
} spapr_pci_msi_mig;
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
struct SpaprPhbState {
pci: Tidy up PCI host bridges Adopt the QOM parent field name and enforce QOM-style access via casts. Don't just typedef PCIHostState, either use it directly or embed it. Signed-off-by: Andreas Färber <afaerber@suse.de> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2012-08-20 21:08:09 +04:00
PCIHostState parent_obj;
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
pseries: Limit PCI host bridge "index" value pseries guests can have large numbers of PCI host bridges. To avoid the user having to specify a number of different configuration values for every one, the device supports an "index" property which is a shorthand setting the various window and configuration addresses from a predefined sensible set. There are some problems with the details at present: * The "index" propery is signed, but negative values will create PCI windows below where we expect, potentially colliding with other devices * No limit is imposed on the "index" property and large values can translate to extremely large window addresses. With PCI passthrough in particular this can mean we exceed various mapping and physical address limits causing the guest host bridge to not work in strange ways. This patch addresses this, by making "index" unsigned, and imposing a limit. Currently the limit allows indices from 0..255 which is probably enough host bridges for the time being. It's fairly easy to extend if we discover we need more. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com> Signed-off-by: Alexander Graf <agraf@suse.de>
2015-01-14 05:33:39 +03:00
uint32_t index;
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
uint64_t buid;
pseries: Configure PCI bridge using properties Currently, the function spapr_create_phb() uses its parameters to initialize the correct memory windows for the new PCI Host Bridge (PHB). This is not the way things are supposed to be done with qdevs, and means you can't create extra PHBs easily using -device. Since pSeries machines can and do have many PHBs with various configurations, this is a real limitation, not just a theoretical. This patch, therefore, alters the PHB initialization code to use qdev properties to set these parameters of the new bridge, moving most of the code from spapr_create_phb() to spapr_phb_init(). While we're at it, we change the naming of each PCI bus and its associated memory regions to be less arbitrary and make it easier to relate the guest and qemu views of memory to each other. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Andreas Färber <afaerber@suse.de> Signed-off-by: Alexander Graf <agraf@suse.de>
2012-03-12 21:50:24 +04:00
char *dtbusname;
spapr_pci: add dynamic-reconfiguration option for spapr-pci-host-bridge This option enables/disables PCI hotplug for a particular PHB. Also add machine compatibility code to disable it by default for machine types prior to pseries-2.4. Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> [agraf: move commas for compat fields] Signed-off-by: Alexander Graf <agraf@suse.de>
2015-05-07 08:33:52 +03:00
bool dr_enabled;
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
MemoryRegion memspace, iospace;
spapr_pci: Add a 64-bit MMIO window On real hardware, and under pHyp, the PCI host bridges on Power machines typically advertise two outbound MMIO windows from the guest's physical memory space to PCI memory space: - A 32-bit window which maps onto 2GiB..4GiB in the PCI address space - A 64-bit window which maps onto a large region somewhere high in PCI address space (traditionally this used an identity mapping from guest physical address to PCI address, but that's not always the case) The qemu implementation in spapr-pci-host-bridge, however, only supports a single outbound MMIO window, however. At least some Linux versions expect the two windows however, so we arranged this window to map onto the PCI memory space from 2 GiB..~64 GiB, then advertised it as two contiguous windows, the "32-bit" window from 2G..4G and the "64-bit" window from 4G..~64G. This approach means, however, that the 64G window is not naturally aligned. In turn this limits the size of the largest BAR we can map (which does have to be naturally aligned) to roughly half of the total window. With some large nVidia GPGPU cards which have huge memory BARs, this is starting to be a problem. This patch adds true support for separate 32-bit and 64-bit outbound MMIO windows to the spapr-pci-host-bridge implementation, each of which can be independently configured. The 32-bit window always maps to 2G.. in PCI space, but the PCI address of the 64-bit window can be configured (it defaults to the same as the guest physical address). So as not to break possible existing configurations, as long as a 64-bit window is not specified, a large single window can be specified. This will appear the same way to the guest as the old approach, although it's now implemented by two contiguous memory regions rather than a single one. For now, this only adds the possibility of 64-bit windows. The default configuration still uses the legacy mode. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Laurent Vivier <lvivier@redhat.com>
2016-10-11 06:23:33 +03:00
hwaddr mem_win_addr, mem_win_size, mem64_win_addr, mem64_win_size;
uint64_t mem64_win_pciaddr;
hwaddr io_win_addr, io_win_size;
MemoryRegion mem32window, mem64window, iowindow, msiwindow;
pseries: Add PCI MSI/MSI-X support This patch implements MSI and MSI-X support for the pseries PCI host bridge. To do this it adds: * A "config_space_address to msi_table" map, since the MSI RTAS calls take a PCI config space address as an identifier. * A MSIX memory region to catch msi_notify()/msix_notiry() from virtio-pci and pass them to the guest via qemu_irq_pulse(). * RTAS call "ibm,change-msi" which sets up MSI vectors for a device. Note that this call may configure and return lesser number of vectors than requested. * RTAS call "ibm,query-interrupt-source-number" which translates MSI vector to interrupt controller (XICS) IRQ number. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> [agraf: fix error case ndev < 0] Signed-off-by: Alexander Graf <agraf@suse.de>
2012-08-07 20:10:37 +04:00
spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW) This adds support for Dynamic DMA Windows (DDW) option defined by the SPAPR specification which allows to have additional DMA window(s) The "ddw" property is enabled by default on a PHB but for compatibility the pseries-2.6 machine and older disable it. This also creates a single DMA window for the older machines to maintain backward migration. This implements DDW for PHB with emulated and VFIO devices. The host kernel support is required. The advertised IOMMU page sizes are 4K and 64K; 16M pages are supported but not advertised by default, in order to enable them, the user has to specify "pgsz" property for PHB and enable huge pages for RAM. The existing linux guests try creating one additional huge DMA window with 64K or 16MB pages and map the entire guest RAM to. If succeeded, the guest switches to dma_direct_ops and never calls TCE hypercalls (H_PUT_TCE,...) again. This enables VFIO devices to use the entire RAM and not waste time on map/unmap later. This adds a "dma64_win_addr" property which is a bus address for the 64bit window and by default set to 0x800.0000.0000.0000 as this is what the modern POWER8 hardware uses and this allows having emulated and VFIO devices on the same bus. This adds 4 RTAS handlers: * ibm,query-pe-dma-window * ibm,create-pe-dma-window * ibm,remove-pe-dma-window * ibm,reset-pe-dma-window These are registered from type_init() callback. These RTAS handlers are implemented in a separate file to avoid polluting spapr_iommu.c with PCI. This changes sPAPRPHBState::dma_liobn to an array to allow 2 LIOBNs and updates all references to dma_liobn. However this does not add 64bit LIOBN to the migration stream as in fact even 32bit LIOBN is rather pointless there (as it is a PHB property and the management software can/should pass LIOBNs via CLI) but we keep it for the backward migration support. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2016-07-04 06:33:07 +03:00
uint32_t dma_liobn[SPAPR_PCI_DMA_MAX_WINDOWS];
spapr_pci: Allow PCI host bridge DMA window to be configured At present the PCI host bridge (PHB) for the pseries machine type has a fixed DMA window from 0..1GB (in PCI address space) which is mapped to real memory via the PAPR paravirtualized IOMMU. For better support of VFIO devices, we're going to want to allow for different configurations of the DMA window. Eventually we'll want to allow the guest itself to reconfigure the window via the PAPR dynamic DMA window interface, but as a preliminary this patch allows the user to reconfigure the window with new properties on the PHB device. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Laurent Vivier <lvivier@redhat.com>
2015-09-24 02:56:44 +03:00
hwaddr dma_win_addr, dma_win_size;
pci: use memory core for iommu support Use the new iommu support in the memory core for iommu support. The only user, spapr, is also converted, but it still provides a DMAContext interface until the non-PCI bits switch to AddressSpace. Reviewed-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Avi Kivity <avi.kivity@gmail.com> [ Do not calls memory_region_del_subregion() on the device's bus_master_enable_region, it is an alias; return an AddressSpace from the IOMMU hook and remove the destructor hook. - David Gibson ] Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2012-10-30 15:47:48 +04:00
AddressSpace iommu_as;
spapr_pci: spapr_iommu: Make DMA window a subregion Currently the default DMA window is represented by a single MemoryRegion. However there can be more than just one window so we need a "root" memory region to be separated from the actual DMA window(s). This introduces a "root" IOMMU memory region and adds a subregion for the default DMA 32bit window. Following patches will add other subregion(s). This initializes a default DMA window subregion size to the guest RAM size as this window can be switched into "bypass" mode which implements direct DMA mapping. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: Alexander Graf <agraf@suse.de>
2014-05-27 09:36:32 +04:00
MemoryRegion iommu_root;
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
pseries: savevm support for PCI host bridge This adds the necessary support for saving the state of the PAPR virtual PCI host bridge (or host bridges). Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Anthony Liguori <aliguori@us.ibm.com> Message-id: 1374175984-8930-10-git-send-email-aliguori@us.ibm.com Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2013-07-18 23:33:02 +04:00
struct spapr_pci_lsi {
pseries: Rework irq assignment to avoid carrying qemu_irqs around Currently, the interfaces in the pseries machine code for assignment and setup of interrupts pass around qemu_irq objects. That was done in an attempt not to be too closely linked to the specific XICS interrupt controller. However interactions with the device tree setup made that attempt rather futile, and XICS is part of the PAPR spec anyway, so this really just meant we had to carry both the qemu_irq pointers and the XICS irq numbers around. This mess will just get worse when we add upcoming PCI MSI support, since that will require tracking a bunch more interrupt. Therefore, this patch reworks the spapr code to just use XICS irq numbers (roughly equivalent to GSIs on x86) and only retrieve the qemu_irq pointers from the XICS code when we need them (a trivial lookup). This is a reworked and generalized version of an earlier spapr_pci specific patch from Alexey Kardashevskiy. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> [agraf: fix checkpath warning] Signed-off-by: Alexander Graf <agraf@suse.de>
2012-08-07 20:10:32 +04:00
uint32_t irq;
pseries: Use the same interrupt swizzling for host bridges as p2p bridges Currently the pseries PCI code uses a somewhat strange scheme of PCI irq allocation - one per slot up to a maximum that's greater than the usual 4. This scheme more or less worked, because we were able to tell the guest the irq mapping in the device tree, however it's a bit odd and may break assumptions in the future. Worse, the array used to construct the dev tree interrupt map was mis-sized, we got away with it only because it happened that our SPAPR_PCI_NUM_LSI value was greater than 7. This patch changes the pseries PCI code to use the same interrupt swizzling scheme as is standardized for PCI to PCI bridges. This makes for better consistency, deals better with any devices which use multiple interrupt pins and will make life easier in the future when we add passthrough of what may be either a host bridge or a PCI to PCI bridge. This won't break existing guests, because they don't assume a particular mapping scheme for host bridges, but just follow what we tell them in the device tree (also updated to match, of course). This patch also fixes the allocation of the irq map. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2012-04-25 21:55:42 +04:00
} lsi_table[PCI_NUM_PINS];
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
spapr_pci: Use XICS interrupt allocator and do not cache interrupts in PHB Currently SPAPR PHB keeps track of all allocated MSI (here and below MSI stands for both MSI and MSIX) interrupt because XICS used to be unable to reuse interrupts. This is a problem for dynamic MSI reconfiguration which happens when guest reloads a driver or performs PCI hotplug. Another problem is that the existing implementation can enable MSI on 32 devices maximum (SPAPR_MSIX_MAX_DEVS=32) and there is no good reason for that. This makes use of new XICS ability to reuse interrupts. This reorganizes MSI information storage in sPAPRPHBState. Instead of static array of 32 descriptors (one per a PCI function), this patch adds a GHashTable when @config_addr is a key and (first_irq, num) pair is a value. GHashTable can dynamically grow and shrink so the initial limit of 32 devices is gone. This changes migration stream as @msi_table was a static array while new @msi_devs is a dynamic hash table. This adds temporary array which is used for migration, it is populated in "spapr_pci"::pre_save() callback and expanded into the hash table in post_load() callback. Since the destination side does not know the number of MSI-enabled devices in advance and cannot pre-allocate the temporary array to receive migration state, this makes use of new VMSTATE_STRUCT_VARRAY_ALLOC macro which allocates the array automatically. This resets the MSI configuration space when interrupts are released by the ibm,change-msi RTAS call. This fixed traces to be more informative. This changes vmstate_spapr_pci_msi name from "...lsi" to "...msi" which was incorrect by accident. As the internal representation changed, thus bumps migration version number. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [agraf: drop g_malloc_n usage] Signed-off-by: Alexander Graf <agraf@suse.de>
2014-05-30 13:34:20 +04:00
GHashTable *msi;
/* Temporary cache for migration purposes */
int32_t msi_devs_num;
spapr_pci_msi_mig *msi_devs;
pseries: Add PCI MSI/MSI-X support This patch implements MSI and MSI-X support for the pseries PCI host bridge. To do this it adds: * A "config_space_address to msi_table" map, since the MSI RTAS calls take a PCI config space address as an identifier. * A MSIX memory region to catch msi_notify()/msix_notiry() from virtio-pci and pass them to the guest via qemu_irq_pulse(). * RTAS call "ibm,change-msi" which sets up MSI vectors for a device. Note that this call may configure and return lesser number of vectors than requested. * RTAS call "ibm,query-interrupt-source-number" which translates MSI vector to interrupt controller (XICS) IRQ number. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> [agraf: fix error case ndev < 0] Signed-off-by: Alexander Graf <agraf@suse.de>
2012-08-07 20:10:37 +04:00
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
QLIST_ENTRY(SpaprPhbState) list;
spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW) This adds support for Dynamic DMA Windows (DDW) option defined by the SPAPR specification which allows to have additional DMA window(s) The "ddw" property is enabled by default on a PHB but for compatibility the pseries-2.6 machine and older disable it. This also creates a single DMA window for the older machines to maintain backward migration. This implements DDW for PHB with emulated and VFIO devices. The host kernel support is required. The advertised IOMMU page sizes are 4K and 64K; 16M pages are supported but not advertised by default, in order to enable them, the user has to specify "pgsz" property for PHB and enable huge pages for RAM. The existing linux guests try creating one additional huge DMA window with 64K or 16MB pages and map the entire guest RAM to. If succeeded, the guest switches to dma_direct_ops and never calls TCE hypercalls (H_PUT_TCE,...) again. This enables VFIO devices to use the entire RAM and not waste time on map/unmap later. This adds a "dma64_win_addr" property which is a bus address for the 64bit window and by default set to 0x800.0000.0000.0000 as this is what the modern POWER8 hardware uses and this allows having emulated and VFIO devices on the same bus. This adds 4 RTAS handlers: * ibm,query-pe-dma-window * ibm,create-pe-dma-window * ibm,remove-pe-dma-window * ibm,reset-pe-dma-window These are registered from type_init() callback. These RTAS handlers are implemented in a separate file to avoid polluting spapr_iommu.c with PCI. This changes sPAPRPHBState::dma_liobn to an array to allow 2 LIOBNs and updates all references to dma_liobn. However this does not add 64bit LIOBN to the migration stream as in fact even 32bit LIOBN is rather pointless there (as it is a PHB property and the management software can/should pass LIOBNs via CLI) but we keep it for the backward migration support. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2016-07-04 06:33:07 +03:00
bool ddw_enabled;
uint64_t page_size_mask;
uint64_t dma64_win_addr;
spapr_pci: Add numa node id This adds a numa id property to a PHB to allow linking passed PCI device to CPU/memory. It is up to the management stack to do CPU/memory pinning to the node with the actual PCI device. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [dwg: Renamed property from "node" to "numa_node" to match the similar one in the pxb device] Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2016-07-27 11:03:38 +03:00
uint32_t numa_node;
spapr: Fix 2.7<->2.8 migration of PCI host bridge daa2369 "spapr_pci: Add a 64-bit MMIO window" subtly broke migration from qemu-2.7 to the current version. It split the device's MMIO window into two pieces for 32-bit and 64-bit MMIO. The patch included backwards compatibility code to convert the old property into the new format. However, the property value was also transferred in the migration stream and compared with a (probably unwise) VMSTATE_EQUAL. So, the "raw" value from 2.7 is compared to the new style converted value from (pre-)2.8 giving a mismatch and migration failure. Along with the actual field that caused the breakage, there are several other ill-advised VMSTATE_EQUAL()s. To fix forwards migration, we read the values in the stream into scratch variables and ignore them, instead of comparing for equality. To fix backwards migration, we populate those scratch variables in pre_save() with adjusted values to match the old behaviour. To permit the eventual possibility of removing this cruft from the stream, we only include these compatibility fields if a new 'pre-2.8-migration' property is set. We clear it on the pseries-2.8 machine type, which obviously can't be migrated backwards, but set it on earlier machine type versions. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Greg Kurz <groug@kaod.org> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-11-23 02:26:38 +03:00
pseries: Don't expose PCIe extended config space on older machine types bb9986452 "spapr_pci: Advertise access to PCIe extended config space" allowed guests to access the extended config space of PCI Express devices via the PAPR interfaces, even though the paravirtualized bus mostly acts like plain PCI. However, that patch enabled access unconditionally, including for existing machine types, which is an unwise change in behaviour. This patch limits the change to pseries-2.9 (and later) machine types. Suggested-by: Andrea Bolognani <abologna@redhat.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2017-03-14 03:54:17 +03:00
bool pcie_ecs; /* Allow access to PCIe extended config space? */
spapr: Fix 2.7<->2.8 migration of PCI host bridge daa2369 "spapr_pci: Add a 64-bit MMIO window" subtly broke migration from qemu-2.7 to the current version. It split the device's MMIO window into two pieces for 32-bit and 64-bit MMIO. The patch included backwards compatibility code to convert the old property into the new format. However, the property value was also transferred in the migration stream and compared with a (probably unwise) VMSTATE_EQUAL. So, the "raw" value from 2.7 is compared to the new style converted value from (pre-)2.8 giving a mismatch and migration failure. Along with the actual field that caused the breakage, there are several other ill-advised VMSTATE_EQUAL()s. To fix forwards migration, we read the values in the stream into scratch variables and ignore them, instead of comparing for equality. To fix backwards migration, we populate those scratch variables in pre_save() with adjusted values to match the old behaviour. To permit the eventual possibility of removing this cruft from the stream, we only include these compatibility fields if a new 'pre-2.8-migration' property is set. We clear it on the pseries-2.8 machine type, which obviously can't be migrated backwards, but set it on earlier machine type versions. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Greg Kurz <groug@kaod.org> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-11-23 02:26:38 +03:00
/* Fields for migration compatibility hacks */
bool pre_2_8_migration;
uint32_t mig_liobn;
hwaddr mig_mem_win_addr, mig_mem_win_size;
hwaddr mig_io_win_addr, mig_io_win_size;
spapr: Support NVIDIA V100 GPU with NVLink2 NVIDIA V100 GPUs have on-board RAM which is mapped into the host memory space and accessible as normal RAM via an NVLink bus. The VFIO-PCI driver implements special regions for such GPUs and emulates an NVLink bridge. NVLink2-enabled POWER9 CPUs also provide address translation services which includes an ATS shootdown (ATSD) register exported via the NVLink bridge device. This adds a quirk to VFIO to map the GPU memory and create an MR; the new MR is stored in a PCI device as a QOM link. The sPAPR PCI uses this to get the MR and map it to the system address space. Another quirk does the same for ATSD. This adds additional steps to sPAPR PHB setup: 1. Search for specific GPUs and NPUs, collect findings in sPAPRPHBState::nvgpus, manage system address space mappings; 2. Add device-specific properties such as "ibm,npu", "ibm,gpu", "memory-block", "link-speed" to advertise the NVLink2 function to the guest; 3. Add "mmio-atsd" to vPHB to advertise the ATSD capability; 4. Add new memory blocks (with extra "linux,memory-usable" to prevent the guest OS from accessing the new memory until it is onlined) and npuphb# nodes representing an NPU unit for every vPHB as the GPU driver uses it for link discovery. This allocates space for GPU RAM and ATSD like we do for MMIOs by adding 2 new parameters to the phb_placement() hook. Older machine types set these to zero. This puts new memory nodes in a separate NUMA node to as the GPU RAM needs to be configured equally distant from any other node in the system. Unlike the host setup which assigns numa ids from 255 downwards, this adds new NUMA nodes after the user configures nodes or from 1 if none were configured. This adds requirement similar to EEH - one IOMMU group per vPHB. The reason for this is that ATSD registers belong to a physical NPU so they cannot invalidate translations on GPUs attached to another NPU. It is guaranteed by the host platform as it does not mix NVLink bridges or GPUs from different NPU in the same IOMMU group. If more than one IOMMU group is detected on a vPHB, this disables ATSD support for that vPHB and prints a warning. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [aw: for vfio portions] Acked-by: Alex Williamson <alex.williamson@redhat.com> Message-Id: <20190312082103.130561-1-aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-12 11:21:03 +03:00
hwaddr nv2_gpa_win_addr;
hwaddr nv2_atsd_win_addr;
struct spapr_phb_pci_nvgpu_config *nvgpus;
spapr_pci: Introduce a finish_realize() callback The spapr-pci PHB initializes IOMMU for emulated devices only. The upcoming VFIO support will do it different. However both emulated and VFIO PHB types share most of the initialization code. For the type specific things a new finish_realize() callback is introduced. This introduces sPAPRPHBClass derived from PCIHostBridgeClass and adds the callback pointer. This implements finish_realize() for emulated devices. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [agraf: Fix compilation] Signed-off-by: Alexander Graf <agraf@suse.de>
2014-05-27 09:36:31 +04:00
};
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
spapr-pci: Enable huge BARs At the moment sPAPR only supports 512MB window for MMIO BARs. However modern devices might want bigger 64bit BARs. This extends MMIO window from 512MB to 62GB (aligned to SPAPR_PCI_WINDOW_SPACING) and advertises it in 2 records in the PHB "ranges" property. 32bit gets the space from SPAPR_PCI_MEM_WIN_BUS_OFFSET till the end of 4GB, 64bit gets the rest of the space. If no space is left, 64bit range is not advertised. The MMIO space size is set to old value of 0x20000000 by default for pseries machines older than 2.3. The approach changes the device tree which is a guest visible change, however it won't break migration as: 1. we do not support migration to older QEMU versions 2. migration to newer QEMU will migrate the device tree as well and since the new layout only extends the old one and does not change address mappigns, no breakage is expected here too. SLOF change is required to utilize this extension. Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2015-01-30 04:53:19 +03:00
#define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL
spapr_pci: Add a 64-bit MMIO window On real hardware, and under pHyp, the PCI host bridges on Power machines typically advertise two outbound MMIO windows from the guest's physical memory space to PCI memory space: - A 32-bit window which maps onto 2GiB..4GiB in the PCI address space - A 64-bit window which maps onto a large region somewhere high in PCI address space (traditionally this used an identity mapping from guest physical address to PCI address, but that's not always the case) The qemu implementation in spapr-pci-host-bridge, however, only supports a single outbound MMIO window, however. At least some Linux versions expect the two windows however, so we arranged this window to map onto the PCI memory space from 2 GiB..~64 GiB, then advertised it as two contiguous windows, the "32-bit" window from 2G..4G and the "64-bit" window from 4G..~64G. This approach means, however, that the 64G window is not naturally aligned. In turn this limits the size of the largest BAR we can map (which does have to be naturally aligned) to roughly half of the total window. With some large nVidia GPGPU cards which have huge memory BARs, this is starting to be a problem. This patch adds true support for separate 32-bit and 64-bit outbound MMIO windows to the spapr-pci-host-bridge implementation, each of which can be independently configured. The 32-bit window always maps to 2G.. in PCI space, but the PCI address of the 64-bit window can be configured (it defaults to the same as the guest physical address). So as not to break possible existing configurations, as long as a 64-bit window is not specified, a large single window can be specified. This will appear the same way to the guest as the old approach, although it's now implemented by two contiguous memory regions rather than a single one. For now, this only adds the possibility of 64-bit windows. The default configuration still uses the legacy mode. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Laurent Vivier <lvivier@redhat.com>
2016-10-11 06:23:33 +03:00
#define SPAPR_PCI_MEM32_WIN_SIZE \
((1ULL << 32) - SPAPR_PCI_MEM_WIN_BUS_OFFSET)
spapr: Improved placement of PCI host bridges in guest memory map Currently, the MMIO space for accessing PCI on pseries guests begins at 1 TiB in guest address space. Each PCI host bridge (PHB) has a 64 GiB chunk of address space in which it places its outbound PIO and 32-bit and 64-bit MMIO windows. This scheme as several problems: - It limits guest RAM to 1 TiB (though we have a limited fix for this now) - It limits the total MMIO window to 64 GiB. This is not always enough for some of the large nVidia GPGPU cards - Putting all the windows into a single 64 GiB area means that naturally aligning things within there will waste more address space. In addition there was a miscalculation in some of the defaults, which meant that the MMIO windows for each PHB actually slightly overran the 64 GiB region for that PHB. We got away without nasty consequences because the overrun fit within an unused area at the beginning of the next PHB's region, but it's not pretty. This patch implements a new scheme which addresses those problems, and is also closer to what bare metal hardware and pHyp guests generally use. Because some guest versions (including most current distro kernels) can't access PCI MMIO above 64 TiB, we put all the PCI windows between 32 TiB and 64 TiB. This is broken into 1 TiB chunks. The first 1 TiB contains the PIO (64 kiB) and 32-bit MMIO (2 GiB) windows for all of the PHBs. Each subsequent TiB chunk contains a naturally aligned 64-bit MMIO window for one PHB each. This reduces the number of allowed PHBs (without full manual configuration of all the windows) from 256 to 31, but this should still be plenty in practice. We also change some of the default window sizes for manually configured PHBs to saner values. Finally we adjust some tests and libqos so that it correctly uses the new default locations. Ideally it would parse the device tree given to the guest, but that's a more complex problem for another time. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Laurent Vivier <lvivier@redhat.com>
2016-10-16 04:04:15 +03:00
#define SPAPR_PCI_MEM64_WIN_SIZE 0x10000000000ULL /* 1 TiB */
spapr-pci: Enable huge BARs At the moment sPAPR only supports 512MB window for MMIO BARs. However modern devices might want bigger 64bit BARs. This extends MMIO window from 512MB to 62GB (aligned to SPAPR_PCI_WINDOW_SPACING) and advertises it in 2 records in the PHB "ranges" property. 32bit gets the space from SPAPR_PCI_MEM_WIN_BUS_OFFSET till the end of 4GB, 64bit gets the rest of the space. If no space is left, 64bit range is not advertised. The MMIO space size is set to old value of 0x20000000 by default for pseries machines older than 2.3. The approach changes the device tree which is a guest visible change, however it won't break migration as: 1. we do not support migration to older QEMU versions 2. migration to newer QEMU will migrate the device tree as well and since the new layout only extends the old one and does not change address mappigns, no breakage is expected here too. SLOF change is required to utilize this extension. Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2015-01-30 04:53:19 +03:00
spapr_pci: Define SPAPR_MAX_PHBS in hw/pci-host/spapr.h PHB hotplug will bring more users for it. Let's define it along with the PHB defines from which it is derived for simplicity. While here fix a misleading comment about manual placement, which was abandoned with 30b3bc5aa9f4. Signed-off-by: Greg Kurz <groug@kaod.org> Reviewed-by: Cédric Le Goater <clg@kaod.org> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2018-12-21 03:36:53 +03:00
/* All PCI outbound windows will be within this range */
spapr: Improved placement of PCI host bridges in guest memory map Currently, the MMIO space for accessing PCI on pseries guests begins at 1 TiB in guest address space. Each PCI host bridge (PHB) has a 64 GiB chunk of address space in which it places its outbound PIO and 32-bit and 64-bit MMIO windows. This scheme as several problems: - It limits guest RAM to 1 TiB (though we have a limited fix for this now) - It limits the total MMIO window to 64 GiB. This is not always enough for some of the large nVidia GPGPU cards - Putting all the windows into a single 64 GiB area means that naturally aligning things within there will waste more address space. In addition there was a miscalculation in some of the defaults, which meant that the MMIO windows for each PHB actually slightly overran the 64 GiB region for that PHB. We got away without nasty consequences because the overrun fit within an unused area at the beginning of the next PHB's region, but it's not pretty. This patch implements a new scheme which addresses those problems, and is also closer to what bare metal hardware and pHyp guests generally use. Because some guest versions (including most current distro kernels) can't access PCI MMIO above 64 TiB, we put all the PCI windows between 32 TiB and 64 TiB. This is broken into 1 TiB chunks. The first 1 TiB contains the PIO (64 kiB) and 32-bit MMIO (2 GiB) windows for all of the PHBs. Each subsequent TiB chunk contains a naturally aligned 64-bit MMIO window for one PHB each. This reduces the number of allowed PHBs (without full manual configuration of all the windows) from 256 to 31, but this should still be plenty in practice. We also change some of the default window sizes for manually configured PHBs to saner values. Finally we adjust some tests and libqos so that it correctly uses the new default locations. Ideally it would parse the device tree given to the guest, but that's a more complex problem for another time. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Laurent Vivier <lvivier@redhat.com>
2016-10-16 04:04:15 +03:00
#define SPAPR_PCI_BASE (1ULL << 45) /* 32 TiB */
#define SPAPR_PCI_LIMIT (1ULL << 46) /* 64 TiB */
spapr_pci: Define SPAPR_MAX_PHBS in hw/pci-host/spapr.h PHB hotplug will bring more users for it. Let's define it along with the PHB defines from which it is derived for simplicity. While here fix a misleading comment about manual placement, which was abandoned with 30b3bc5aa9f4. Signed-off-by: Greg Kurz <groug@kaod.org> Reviewed-by: Cédric Le Goater <clg@kaod.org> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2018-12-21 03:36:53 +03:00
#define SPAPR_MAX_PHBS ((SPAPR_PCI_LIMIT - SPAPR_PCI_BASE) / \
SPAPR_PCI_MEM64_WIN_SIZE - 1)
pseries: Improve handling of multiple PCI host bridges Multiple - even many - PCI host bridges (i.e. PCI domains) are very common on real PAPR compliant hardware. For reasons related to the PAPR specified IOMMU interfaces, PCI device assignment with VFIO will generally require at least two (virtual) PHBs and possibly more depending on which devices are assigned. At the moment the qemu PAPR PCI code will not deal with this well, leaving several crucial parameters of PHBs other than the default one uninitialized. This patch reworks the code to allow this. Every PHB needs a unique BUID (Bus Unit Identifier, the id used for the PAPR PCI related interfaces) and a unique LIOBN (Logical IO Bus Number, the id used for the PAPR IOMMU related interfaces). In addition they need windows in CPU real address space to access PCI memory space, PCI IO space and MSIs. Properties are added to the PCI host bridge qdevice to allow configuration of all these. To simplify configuration of multiple PHBs for common cases, a convenience "index" property is also added. This can be set instead of the low-level properties, and will generate suitable values for the other parameters, different for each index value. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2013-01-23 21:20:39 +04:00
#define SPAPR_PCI_IO_WIN_SIZE 0x10000
spapr-pci: rework MSI/MSIX On the sPAPR platform a guest allocates MSI/MSIX vectors via RTAS hypercalls which return global IRQ numbers to a guest so it only operates with those and never touches MSIMessage. Therefore MSIMessage handling is completely hidden in QEMU. Previously every sPAPR PCI host bridge implemented its own MSI window to catch msi_notify()/msix_notify() calls from QEMU devices (virtio-pci or vfio) and route them to the guest via qemu_pulse_irq(). MSIMessage used to be encoded as: .addr - address within the PHB MSI window; .data - the device index on PHB plus vector number. The MSI MR write function translated this MSIMessage to a global IRQ number and called qemu_pulse_irq(). However the total number of IRQs is not really big (at the moment it is 1024 IRQs starting from 4096) and even 16bit data field of MSIMessage seems to be enough to store an IRQ number there. This simplifies MSI handling in sPAPR PHB. Specifically, this does: 1. remove a MSI window from a PHB; 2. add a single memory region for all MSIs to sPAPREnvironment and spapr_pci_msi_init() to initialize it; 3. encode MSIMessage as: * .addr - a fixed address of SPAPR_PCI_MSI_WINDOW==0x40000000000ULL; * .data as an IRQ number. 4. change IRQ allocator to align first IRQ number in a block for MSI. MSI uses lower bits to specify the vector number so the first IRQ has to be aligned. MSIX does not need any special allocator though. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Reviewed-by: Anthony Liguori <aliguori@us.ibm.com> Acked-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Alexander Graf <agraf@suse.de>
2013-07-12 11:38:24 +04:00
#define SPAPR_PCI_MSI_WINDOW 0x40000000000ULL
pseries: Improve handling of multiple PCI host bridges Multiple - even many - PCI host bridges (i.e. PCI domains) are very common on real PAPR compliant hardware. For reasons related to the PAPR specified IOMMU interfaces, PCI device assignment with VFIO will generally require at least two (virtual) PHBs and possibly more depending on which devices are assigned. At the moment the qemu PAPR PCI code will not deal with this well, leaving several crucial parameters of PHBs other than the default one uninitialized. This patch reworks the code to allow this. Every PHB needs a unique BUID (Bus Unit Identifier, the id used for the PAPR PCI related interfaces) and a unique LIOBN (Logical IO Bus Number, the id used for the PAPR IOMMU related interfaces). In addition they need windows in CPU real address space to access PCI memory space, PCI IO space and MSIs. Properties are added to the PCI host bridge qdevice to allow configuration of all these. To simplify configuration of multiple PHBs for common cases, a convenience "index" property is also added. This can be set instead of the low-level properties, and will generate suitable values for the other parameters, different for each index value. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2013-01-23 21:20:39 +04:00
spapr: Support NVIDIA V100 GPU with NVLink2 NVIDIA V100 GPUs have on-board RAM which is mapped into the host memory space and accessible as normal RAM via an NVLink bus. The VFIO-PCI driver implements special regions for such GPUs and emulates an NVLink bridge. NVLink2-enabled POWER9 CPUs also provide address translation services which includes an ATS shootdown (ATSD) register exported via the NVLink bridge device. This adds a quirk to VFIO to map the GPU memory and create an MR; the new MR is stored in a PCI device as a QOM link. The sPAPR PCI uses this to get the MR and map it to the system address space. Another quirk does the same for ATSD. This adds additional steps to sPAPR PHB setup: 1. Search for specific GPUs and NPUs, collect findings in sPAPRPHBState::nvgpus, manage system address space mappings; 2. Add device-specific properties such as "ibm,npu", "ibm,gpu", "memory-block", "link-speed" to advertise the NVLink2 function to the guest; 3. Add "mmio-atsd" to vPHB to advertise the ATSD capability; 4. Add new memory blocks (with extra "linux,memory-usable" to prevent the guest OS from accessing the new memory until it is onlined) and npuphb# nodes representing an NPU unit for every vPHB as the GPU driver uses it for link discovery. This allocates space for GPU RAM and ATSD like we do for MMIOs by adding 2 new parameters to the phb_placement() hook. Older machine types set these to zero. This puts new memory nodes in a separate NUMA node to as the GPU RAM needs to be configured equally distant from any other node in the system. Unlike the host setup which assigns numa ids from 255 downwards, this adds new NUMA nodes after the user configures nodes or from 1 if none were configured. This adds requirement similar to EEH - one IOMMU group per vPHB. The reason for this is that ATSD registers belong to a physical NPU so they cannot invalidate translations on GPUs attached to another NPU. It is guaranteed by the host platform as it does not mix NVLink bridges or GPUs from different NPU in the same IOMMU group. If more than one IOMMU group is detected on a vPHB, this disables ATSD support for that vPHB and prints a warning. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [aw: for vfio portions] Acked-by: Alex Williamson <alex.williamson@redhat.com> Message-Id: <20190312082103.130561-1-aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-12 11:21:03 +03:00
#define SPAPR_PCI_NV2RAM64_WIN_BASE SPAPR_PCI_LIMIT
#define SPAPR_PCI_NV2RAM64_WIN_SIZE (2 * TiB) /* For up to 6 GPUs 256GB each */
/* Max number of these GPUsper a physical box */
#define NVGPU_MAX_NUM 6
/* Max number of NVLinks per GPU in any physical box */
#define NVGPU_MAX_LINKS 3
/*
* GPU RAM starts at 64TiB so huge DMA window to cover it all ends at 128TiB
* which is enough. We do not need DMA for ATSD so we put them at 128TiB.
*/
#define SPAPR_PCI_NV2ATSD_WIN_BASE (128 * TiB)
#define SPAPR_PCI_NV2ATSD_WIN_SIZE (NVGPU_MAX_NUM * NVGPU_MAX_LINKS * \
64 * KiB)
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
static inline qemu_irq spapr_phb_lsi_qirq(struct SpaprPhbState *phb, int pin)
pseries: Rework irq assignment to avoid carrying qemu_irqs around Currently, the interfaces in the pseries machine code for assignment and setup of interrupts pass around qemu_irq objects. That was done in an attempt not to be too closely linked to the specific XICS interrupt controller. However interactions with the device tree setup made that attempt rather futile, and XICS is part of the PAPR spec anyway, so this really just meant we had to carry both the qemu_irq pointers and the XICS irq numbers around. This mess will just get worse when we add upcoming PCI MSI support, since that will require tracking a bunch more interrupt. Therefore, this patch reworks the spapr code to just use XICS irq numbers (roughly equivalent to GSIs on x86) and only retrieve the qemu_irq pointers from the XICS code when we need them (a trivial lookup). This is a reworked and generalized version of an earlier spapr_pci specific patch from Alexey Kardashevskiy. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> [agraf: fix checkpath warning] Signed-off-by: Alexander Graf <agraf@suse.de>
2012-08-07 20:10:32 +04:00
{
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
SpaprMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
spapr: Merge sPAPREnvironment into sPAPRMachineState The code for -machine pseries maintains a global sPAPREnvironment structure which keeps track of general state information about the guest platform. This predates the existence of the MachineState structure, but performs basically the same function. Now that we have the generic MachineState, fold sPAPREnvironment into sPAPRMachineState, the pseries specific subclass of MachineState. This is mostly a matter of search and replace, although a few places which relied on the global spapr variable are changed to find the structure via qdev_get_machine(). Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2015-07-02 09:23:04 +03:00
spapr: introduce a spapr_qirq() helper xics_get_qirq() is only used by the sPAPR machine. Let's move it there and change its name to reflect its scope. It will be useful for XIVE support which will use its own set of qirqs. Signed-off-by: Cédric Le Goater <clg@kaod.org> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2017-12-01 19:06:04 +03:00
return spapr_qirq(spapr, phb->lsi_table[pin].irq);
pseries: Rework irq assignment to avoid carrying qemu_irqs around Currently, the interfaces in the pseries machine code for assignment and setup of interrupts pass around qemu_irq objects. That was done in an attempt not to be too closely linked to the specific XICS interrupt controller. However interactions with the device tree setup made that attempt rather futile, and XICS is part of the PAPR spec anyway, so this really just meant we had to carry both the qemu_irq pointers and the XICS irq numbers around. This mess will just get worse when we add upcoming PCI MSI support, since that will require tracking a bunch more interrupt. Therefore, this patch reworks the spapr code to just use XICS irq numbers (roughly equivalent to GSIs on x86) and only retrieve the qemu_irq pointers from the XICS code when we need them (a trivial lookup). This is a reworked and generalized version of an earlier spapr_pci specific patch from Alexey Kardashevskiy. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> [agraf: fix checkpath warning] Signed-off-by: Alexander Graf <agraf@suse.de>
2012-08-07 20:10:32 +04:00
}
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
int spapr_populate_pci_dt(SpaprPhbState *phb, uint32_t intc_phandle, void *fdt,
spapr_pci: provide node start offset via spapr_populate_pci_dt() PHB hotplug re-uses PHB device tree generation code and passes it to a guest via RTAS. Doing this requires knowledge of where exactly in the device tree the node describing the PHB begins. Provide this via a new optional pointer that can be used to store the PHB node's start offset. Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Greg Kurz <groug@kaod.org> Message-Id: <155059671912.1466090.10891589403973703473.stgit@bahia.lab.toulouse-stg.fr.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-02-19 20:18:39 +03:00
uint32_t nr_msis, int *node_offset);
pseries: Add partial support for PCI This patch adds a PCI bus to the pseries machine. This instantiates the qemu generic PCI bus code, advertises a PCI host bridge in the guest's device tree and implements the RTAS methods specified by PAPR to access PCI config space. It also sets up the memory regions we need to provide windows into the PCI memory and IO space, and advertises those to the guest. However, because qemu can't yet emulate an IOMMU, which is mandatory on pseries, PCI devices which use DMA (i.e. most of them) will not work with this code alone. Still, this is enough to support the virtio_pci device (which probably _should_ use emulated PCI DMA, but is specced to use direct hypervisor access to guest physical memory instead). [agraf] remove typedef which could cause compile errors Signed-off-by: Alexey Kardashevskiy <aik@au1.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2011-10-30 21:16:46 +04:00
pseries: Separate PCI RTAS setup from common from emulation specific PCI setup Currently the RTAS functions for handling PCI are registered from the class init code for the PCI host bridge. That sort of makes sense now, but will break in the future when vfio gives us multiple types of host bridge for pseries (emulated and pass-through, at least). The RTAS functions will be common across all host bridge types (and will call out to different places internally depending on the type). So, this patch moves the RTAS registration into its own function called direct from the machine setup code. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2012-08-07 20:10:33 +04:00
void spapr_pci_rtas_init(void);
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
SpaprPhbState *spapr_pci_find_phb(SpaprMachineState *spapr, uint64_t buid);
PCIDevice *spapr_pci_find_dev(SpaprMachineState *spapr, uint64_t buid,
spapr_pci: Make find_phb()/find_dev() public This makes find_phb()/find_dev() public and changed its names to spapr_pci_find_phb()/spapr_pci_find_dev() as they are going to be used from other parts of QEMU such as VFIO DDW (dynamic DMA window) or VFIO PCI error injection or VFIO EEH handling - in all these cases there are RTAS calls which are addressed to BUID+config_addr in IEEE1275 format. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Alexander Graf <agraf@suse.de>
2015-05-07 08:33:34 +03:00
uint32_t config_addr);
spapr/pci: Generate FDT fragment at configure connector time Signed-off-by: Greg Kurz <groug@kaod.org> Message-Id: <155059667346.1466090.326696113231137772.stgit@bahia.lab.toulouse-stg.fr.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-02-19 20:17:53 +03:00
/* DRC callbacks */
hw/ppc: removing drc->detach_cb and drc->detach_cb_opaque The pointer drc->detach_cb is being used as a way of informing the detach() function inside spapr_drc.c which cb to execute. This information can also be retrieved simply by checking drc->type and choosing the right callback based on it. In this context, detach_cb is redundant information that must be managed. After the previous spapr_lmb_release change, no detach_cb_opaques are being used by any of the three callbacks functions. This is yet another information that is now unused and, on top of that, can't be migrated either. This patch makes the following changes: - removal of detach_cb_opaque. the 'opaque' argument was removed from the callbacks and from the detach() function of sPAPRConnectorClass. The attribute detach_cb_opaque of sPAPRConnector was removed. - removal of detach_cb from the detach() call. The function pointer detach_cb of sPAPRConnector was removed. detach() now uses a switch(drc->type) to execute the apropriate callback. To achieve this, spapr_core_release, spapr_lmb_release and spapr_phb_remove_pci_device_cb callbacks were made public to be visible inside detach(). Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2017-05-22 22:35:48 +03:00
void spapr_phb_remove_pci_device_cb(DeviceState *dev);
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
int spapr_pci_dt_populate(SpaprDrc *drc, SpaprMachineState *spapr,
spapr/pci: Generate FDT fragment at configure connector time Signed-off-by: Greg Kurz <groug@kaod.org> Message-Id: <155059667346.1466090.326696113231137772.stgit@bahia.lab.toulouse-stg.fr.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-02-19 20:17:53 +03:00
void *fdt, int *fdt_start_offset, Error **errp);
hw/ppc: removing drc->detach_cb and drc->detach_cb_opaque The pointer drc->detach_cb is being used as a way of informing the detach() function inside spapr_drc.c which cb to execute. This information can also be retrieved simply by checking drc->type and choosing the right callback based on it. In this context, detach_cb is redundant information that must be managed. After the previous spapr_lmb_release change, no detach_cb_opaques are being used by any of the three callbacks functions. This is yet another information that is now unused and, on top of that, can't be migrated either. This patch makes the following changes: - removal of detach_cb_opaque. the 'opaque' argument was removed from the callbacks and from the detach() function of sPAPRConnectorClass. The attribute detach_cb_opaque of sPAPRConnector was removed. - removal of detach_cb from the detach() call. The function pointer detach_cb of sPAPRConnector was removed. detach() now uses a switch(drc->type) to execute the apropriate callback. To achieve this, spapr_core_release, spapr_lmb_release and spapr_phb_remove_pci_device_cb callbacks were made public to be visible inside detach(). Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2017-05-22 22:35:48 +03:00
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
/* VFIO EEH hooks */
#ifdef CONFIG_LINUX
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
bool spapr_phb_eeh_available(SpaprPhbState *sphb);
int spapr_phb_vfio_eeh_set_option(SpaprPhbState *sphb,
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
unsigned int addr, int option);
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
int spapr_phb_vfio_eeh_get_state(SpaprPhbState *sphb, int *state);
int spapr_phb_vfio_eeh_reset(SpaprPhbState *sphb, int option);
int spapr_phb_vfio_eeh_configure(SpaprPhbState *sphb);
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
void spapr_phb_vfio_reset(DeviceState *qdev);
spapr: Support NVIDIA V100 GPU with NVLink2 NVIDIA V100 GPUs have on-board RAM which is mapped into the host memory space and accessible as normal RAM via an NVLink bus. The VFIO-PCI driver implements special regions for such GPUs and emulates an NVLink bridge. NVLink2-enabled POWER9 CPUs also provide address translation services which includes an ATS shootdown (ATSD) register exported via the NVLink bridge device. This adds a quirk to VFIO to map the GPU memory and create an MR; the new MR is stored in a PCI device as a QOM link. The sPAPR PCI uses this to get the MR and map it to the system address space. Another quirk does the same for ATSD. This adds additional steps to sPAPR PHB setup: 1. Search for specific GPUs and NPUs, collect findings in sPAPRPHBState::nvgpus, manage system address space mappings; 2. Add device-specific properties such as "ibm,npu", "ibm,gpu", "memory-block", "link-speed" to advertise the NVLink2 function to the guest; 3. Add "mmio-atsd" to vPHB to advertise the ATSD capability; 4. Add new memory blocks (with extra "linux,memory-usable" to prevent the guest OS from accessing the new memory until it is onlined) and npuphb# nodes representing an NPU unit for every vPHB as the GPU driver uses it for link discovery. This allocates space for GPU RAM and ATSD like we do for MMIOs by adding 2 new parameters to the phb_placement() hook. Older machine types set these to zero. This puts new memory nodes in a separate NUMA node to as the GPU RAM needs to be configured equally distant from any other node in the system. Unlike the host setup which assigns numa ids from 255 downwards, this adds new NUMA nodes after the user configures nodes or from 1 if none were configured. This adds requirement similar to EEH - one IOMMU group per vPHB. The reason for this is that ATSD registers belong to a physical NPU so they cannot invalidate translations on GPUs attached to another NPU. It is guaranteed by the host platform as it does not mix NVLink bridges or GPUs from different NPU in the same IOMMU group. If more than one IOMMU group is detected on a vPHB, this disables ATSD support for that vPHB and prints a warning. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [aw: for vfio portions] Acked-by: Alex Williamson <alex.williamson@redhat.com> Message-Id: <20190312082103.130561-1-aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-12 11:21:03 +03:00
void spapr_phb_nvgpu_setup(SpaprPhbState *sphb, Error **errp);
void spapr_phb_nvgpu_free(SpaprPhbState *sphb);
void spapr_phb_nvgpu_populate_dt(SpaprPhbState *sphb, void *fdt, int bus_off,
Error **errp);
void spapr_phb_nvgpu_ram_populate_dt(SpaprPhbState *sphb, void *fdt);
void spapr_phb_nvgpu_populate_pcidev_dt(PCIDevice *dev, void *fdt, int offset,
SpaprPhbState *sphb);
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
#else
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
static inline bool spapr_phb_eeh_available(SpaprPhbState *sphb)
spapr_pci: Allow EEH on spapr-pci-host-bridge Now that the EEH code is independent of the special spapr-vfio-pci-host-bridge device, we can allow it on all spapr PCI host bridges instead. We do this by changing spapr_phb_eeh_available() to be based on the vfio_eeh_as_ok() call instead of the host bridge class. Because the value of vfio_eeh_as_ok() can change with devices being hotplugged or unplugged, this can potentially lead to some strange edge cases where the guest starts using EEH, then it starts failing because of a change in status. However, it's not really any worse than the current situation. Cases that would have worked previously will still work (i.e. VFIO devices from at most one VFIO IOMMU group per vPHB), it's just that it's no longer necessary to use spapr-vfio-pci-host-bridge with the groupid pre-specified. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:19:42 +03:00
{
return false;
}
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
static inline int spapr_phb_vfio_eeh_set_option(SpaprPhbState *sphb,
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
unsigned int addr, int option)
{
return RTAS_OUT_HW_ERROR;
}
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
static inline int spapr_phb_vfio_eeh_get_state(SpaprPhbState *sphb,
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
int *state)
{
return RTAS_OUT_HW_ERROR;
}
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
static inline int spapr_phb_vfio_eeh_reset(SpaprPhbState *sphb, int option)
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
{
return RTAS_OUT_HW_ERROR;
}
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
static inline int spapr_phb_vfio_eeh_configure(SpaprPhbState *sphb)
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
{
return RTAS_OUT_HW_ERROR;
}
static inline void spapr_phb_vfio_reset(DeviceState *qdev)
{
}
spapr: Support NVIDIA V100 GPU with NVLink2 NVIDIA V100 GPUs have on-board RAM which is mapped into the host memory space and accessible as normal RAM via an NVLink bus. The VFIO-PCI driver implements special regions for such GPUs and emulates an NVLink bridge. NVLink2-enabled POWER9 CPUs also provide address translation services which includes an ATS shootdown (ATSD) register exported via the NVLink bridge device. This adds a quirk to VFIO to map the GPU memory and create an MR; the new MR is stored in a PCI device as a QOM link. The sPAPR PCI uses this to get the MR and map it to the system address space. Another quirk does the same for ATSD. This adds additional steps to sPAPR PHB setup: 1. Search for specific GPUs and NPUs, collect findings in sPAPRPHBState::nvgpus, manage system address space mappings; 2. Add device-specific properties such as "ibm,npu", "ibm,gpu", "memory-block", "link-speed" to advertise the NVLink2 function to the guest; 3. Add "mmio-atsd" to vPHB to advertise the ATSD capability; 4. Add new memory blocks (with extra "linux,memory-usable" to prevent the guest OS from accessing the new memory until it is onlined) and npuphb# nodes representing an NPU unit for every vPHB as the GPU driver uses it for link discovery. This allocates space for GPU RAM and ATSD like we do for MMIOs by adding 2 new parameters to the phb_placement() hook. Older machine types set these to zero. This puts new memory nodes in a separate NUMA node to as the GPU RAM needs to be configured equally distant from any other node in the system. Unlike the host setup which assigns numa ids from 255 downwards, this adds new NUMA nodes after the user configures nodes or from 1 if none were configured. This adds requirement similar to EEH - one IOMMU group per vPHB. The reason for this is that ATSD registers belong to a physical NPU so they cannot invalidate translations on GPUs attached to another NPU. It is guaranteed by the host platform as it does not mix NVLink bridges or GPUs from different NPU in the same IOMMU group. If more than one IOMMU group is detected on a vPHB, this disables ATSD support for that vPHB and prints a warning. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> [aw: for vfio portions] Acked-by: Alex Williamson <alex.williamson@redhat.com> Message-Id: <20190312082103.130561-1-aik@ozlabs.ru> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-12 11:21:03 +03:00
static inline void spapr_phb_nvgpu_setup(SpaprPhbState *sphb, Error **errp)
{
}
static inline void spapr_phb_nvgpu_free(SpaprPhbState *sphb)
{
}
static inline void spapr_phb_nvgpu_populate_dt(SpaprPhbState *sphb, void *fdt,
int bus_off, Error **errp)
{
}
static inline void spapr_phb_nvgpu_ram_populate_dt(SpaprPhbState *sphb,
void *fdt)
{
}
static inline void spapr_phb_nvgpu_populate_pcidev_dt(PCIDevice *dev, void *fdt,
int offset,
SpaprPhbState *sphb)
{
}
spapr_pci: Eliminate class callbacks The EEH operations in the spapr-vfio-pci-host-bridge no longer rely on the special groupid field in sPAPRPHBVFIOState. So we can simplify, removing the class specific callbacks with direct calls based on a simple spapr_phb_eeh_enabled() helper. For now we implement that in terms of a boolean in the class, but we'll continue to clean that up later. On its own this is a rather strange way of doing things, but it's a useful intermediate step to further cleanups. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2016-02-29 09:45:05 +03:00
#endif
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
void spapr_phb_dma_reset(SpaprPhbState *sphb);
spapr_pci: Add and export DMA resetting helper This will be later used by the "ibm,reset-pe-dma-window" RTAS handler which resets the DMA configuration to the defaults. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2016-06-01 11:57:39 +03:00
spapr: Use CamelCase properly The qemu coding standard is to use CamelCase for type and structure names, and the pseries code follows that... sort of. There are quite a lot of places where we bend the rules in order to preserve the capitalization of internal acronyms like "PHB", "TCE", "DIMM" and most commonly "sPAPR". That was a bad idea - it frequently leads to names ending up with hard to read clusters of capital letters, and means they don't catch the eye as type identifiers, which is kind of the point of the CamelCase convention in the first place. In short, keeping type identifiers look like CamelCase is more important than preserving standard capitalization of internal "words". So, this patch renames a heap of spapr internal type names to a more standard CamelCase. In addition to case changes, we also make some other identifier renames: VIOsPAPR* -> SpaprVio* The reverse word ordering was only ever used to mitigate the capital cluster, so revert to the natural ordering. VIOsPAPRVTYDevice -> SpaprVioVty VIOsPAPRVLANDevice -> SpaprVioVlan Brevity, since the "Device" didn't add useful information sPAPRDRConnector -> SpaprDrc sPAPRDRConnectorClass -> SpaprDrcClass Brevity, and makes it clearer this is the same thing as a "DRC" mentioned in many other places in the code This is 100% a mechanical search-and-replace patch. It will, however, conflict with essentially any and all outstanding patches touching the spapr code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-03-06 07:35:37 +03:00
static inline unsigned spapr_phb_windows_supported(SpaprPhbState *sphb)
spapr_pci: add PHB unrealize To support PHB hotplug we need to clean up lingering references, memory, child properties, etc. prior to the PHB object being finalized. Generally this will be called as a result of calling object_unparent() on the PHB object, which in turn would normally be called as the result of an unplug() operation. When the PHB is finalized, child objects will be unparented in turn, and finalized if the PHB was the only reference holder. so we don't bother to explicitly unparent child objects of the PHB, with the notable exception of DRCs. This is needed to avoid a QEMU crash when unplugging a PHB and resetting the machine before the guest could handle the event. The DRCs are removed from the QOM tree by pci_unregister_root_bus() and we must make sure we're not leaving stale aliases under the global /dr-connector path. The formula that gives the number of DMA windows is moved to an inline function in the hw/pci-host/spapr.h header because it will have other users. The unrealize function is able to cope with partially realized PHBs. It is hence used to implement proper rollback on the realize error path. Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com> Signed-off-by: Greg Kurz <groug@kaod.org> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Message-Id: <155059669881.1466090.13515030705986041517.stgit@bahia.lab.toulouse-stg.fr.ibm.com> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2019-02-19 20:18:18 +03:00
{
return sphb->ddw_enabled ? SPAPR_PCI_DMA_MAX_WINDOWS : 1;
}
Clean up header guards that don't match their file name Header guard symbols should match their file name to make guard collisions less likely. Offenders found with scripts/clean-header-guards.pl -vn. Cleaned up with scripts/clean-header-guards.pl, followed by some renaming of new guard symbols picked by the script to better ones. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Richard Henderson <rth@twiddle.net>
2016-06-29 11:12:57 +03:00
#endif /* PCI_HOST_SPAPR_H */
Reference in New Issue Copy Permalink