Devices supporting Process Address Space Identifiers
(PASIDs) can use an IOMMU to access multiple IO address
spaces at the same time. A PCIe device indicates support for
this feature by implementing the PASID capability. This
patch adds support for the capability to the Linux kernel.
Reviewed-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Implement the necessary functions to handle PRI capabilities
on PCIe devices. With PRI devices behind an IOMMU can signal
page fault conditions to software and recover from such
faults.
Reviewed-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Latency tolerance reporting allows devices to send messages to the root
complex indicating their latency tolerance for snooped & unsnooped
memory transactions.  Add support for enabling & disabling this
feature, along with a routine to set the max latencies a device should
send upstream.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

OBFF (optimized buffer flush/fill), where supported, can help improve
energy efficiency by giving devices information about when interrupts
and other activity will have a reduced power impact.  It requires
support from both the device and system (i.e. not only does the device
need to respond to OBFF messages, but the platform must be capable of
generating and routing them to the end point).
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Add support to allow drivers to enable/disable ID-based ordering.  Where
supported, ID-based ordering can significantly improve the latency of
individual requests by preventing them from queueing up behind unrelated
traffic.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Fixes generated by 'codespell' and manually reviewed.
Signed-off-by: NLucas De Marchi <lucas.demarchi@profusion.mobi>

I noticed that PCI Express PMEs don't work on my Toshiba Portege R500
after the system has been woken up from a sleep state by a PME
(through Wake-on-LAN).  After some investigation it turned out that
the BIOS didn't clear the Root PME Status bit in the root port that
received the wakeup PME and since the Requester ID was also set in
the port's Root Status register, any subsequent PMEs didn't trigger
interrupts.

This problem can be avoided by clearing the Root PME Status bits in
all PCI Express root ports during early resume.  For this purpose,
add an early resume routine to the PCIe port driver and make this
driver be always registered, even if pci_ports_disable is set (in
which case the driver's only function is to provide the early
resume callback).
Signed-off-by: NRafael J. Wysocki <rjw@sisk.pl>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Then we can use it instead of magic number 1.
Reviewed-by: NHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
Cc: Matthew Wilcox <willy@linux.intel.com>
Signed-off-by: NSheng Yang <sheng@linux.intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Then it can be used by others.
Reviewed-by: NHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
Reviewed-by: NMatthew Wilcox <willy@linux.intel.com>
Signed-off-by: NSheng Yang <sheng@linux.intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

These are already defined in pcilib's pci/header.h but not in kernel's
linux/pci_regs.h.  Copy them to avoid using magic numbers.
Signed-off-by: NHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

The Error Source Identification Register (Offset 34h) is 4 byte
which contains a couple of 2 byte field, "[15:0] ERR_COR Source
Identification" and "[31:16] ERR_FATAL/NONFATAL Source Identification."

This patch defines PCI_ERR_ROOT_ERR_SRC to make dword access sensible.
Signed-off-by: NHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
Reviewed-by: NKenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

This change corrects the fact that we were not reporting Gen2 link speeds
when we were in fact connected at Gen2 rates.
Signed-off-by: NAlexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The Moorestown platform only has a few devices that actually support
PCI config cycles.  The rest of the devices use an in-RAM MCFG space
for the purposes of device enumeration and initialization.

There are a few uglies in the fake support, like BAR sizes that aren't
a power of two, sizing detection, and writes to the real devices, but
other than that it's pretty straightforward.

Another way to think of this is not really as PCI at all, but just a
table in RAM describing which devices are present, their capabilities
and their offsets in MMIO space.  This could have been done with a
special new firmware table on this platform, but given that we do have
some real PCI devices too, simply describing things in an MCFG type
space was pretty simple.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
LKML-Reference: <43F901BD926A4E43B106BF17856F07559FB80D08@orsmsx508.amr.corp.intel.com>
Signed-off-by: NJacob Pan <jacob.jun.pan@intel.com>
Signed-off-by: NH. Peter Anvin <hpa@zytor.com>

Change to populate the subsystem vendor and subsytem device IDs for
PCI-PCI bridges that implement the PCI Subsystem Vendor ID capability.
Previously bridges left subsystem vendor IDs unpopulated.
Signed-off-by: NGabe Black <gabe.black@ni.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Note: dom0 checking in v4 has been separated out into 2/2.

This patch enables P2P upstream forwarding in ACS capable PCIe switches.
It solves two potential problems in virtualization environment where a PCIe
device is assigned to a guest domain using a HW iommu such as VT-d:

1) Unintentional failure caused by guest physical address programmed
   into the device's DMA that happens to match the memory address range
   of other downstream ports in the same PCIe switch.  This causes the PCI
   transaction to go to the matching downstream port instead of go to the
   root complex to get translated by VT-d as it should be.

2) Malicious guest software intentionally attacks another downstream
   PCIe device by programming the DMA address into the assigned device
   that matches memory address range of the downstream PCIe port.

We are in process of implementing device filtering software in KVM/XEN
management software to allow device assignment of PCIe devices behind a PCIe
switch only if it has ACS capability and with the P2P upstream forwarding bits
enabled.  This patch is intended to work for both KVM and Xen environments.
Signed-off-by: NAllen Kay <allen.m.kay@intel.com>
Reviewed-by: NMathew Wilcox <willy@linux.intel.com>
Reviewed-by: NChris Wright <chris@sous-sol.org>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

This adds a generic uio driver that can bind to any PCI device.  First
user will be virtualization where a qemu userspace process needs to give
guest OS access to the device.

Interrupts are handled using the Interrupt Disable bit in the PCI
command register and Interrupt Status bit in the PCI status register.
All devices compliant to PCI 2.3 (circa 2002) and all compliant PCI
Express devices should support these bits.  Driver detects this support,
and won't bind to devices which do not support the Interrupt Disable Bit
in the command register.

It's expected that more features of interest to virtualization will be
added to this driver in the future. Possibilities are: mmap for device
resources, MSI/MSI-X, eventfd (to interface with kvm), iommu.
Signed-off-by: NMichael S. Tsirkin <mst@redhat.com>
Acked-by: NChris Wright <chrisw@redhat.com>
Signed-off-by: NHans J. Koch <hjk@linutronix.de>
Acked-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Impact: cleanup, improve readability

Define PCI_MSI_MASK_32/64 for 32/64bit devices, instead of using
implicit offset (-4), "PCI_MSI_MASK_BIT - 4" and "PCI_MSI_MASK_BIT".
Signed-off-by: NHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
Reviewed-by: NMatthew Wilcox <willy@linux.intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Impact: cleanup, spec compliance

This patch does:

 - Remove unused msi/msix_enable/disable macros.
   User should use msi/msix_set_enable() functions instead.

 - Remove unused msix_mask/unmask/pending macros.
   These macros are useless because they are not based on any of
   the PCI Local Bus Specifications properly.
   It seems that they were written based on a draft of PCI spec,
   and that the draft was the MSI-X ECN that underwent membership
   review in September 2002.
   (* In the draft, the size of a entry in MSI-X table was 64bit,
      containing 32bit message data and DWORD aligned lower address
      plus a pending bit and a mask bit.(30+1+1bit)  The higher
      address was placed in MSI-X capability structure and shared
      by all entries.)

 - Remove PCI_MSIX_FLAGS_BITMASK.
   This definition also come from the draft ECN.
Signed-off-by: NHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
Reviewed-by: NMatthew Wilcox <willy@linux.intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

The PCIe ATS capability makes the Endpoint be able to request the
DMA address translation from the IOMMU and cache the translation
in the device side, thus alleviate IOMMU pressure and improve the
hardware performance in the I/O virtualization environment.
Signed-off-by: NYu Zhao <yu.zhao@intel.com>
Acked-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

PCIe 1.1 base neither requires the endpoint to implement the entire
PCIe capability structure nor specifies default values of registers
that are not implemented by the device. So we only save and restore
registers that must be implemented by different device types if the
device PCIe capability version is 1.

PCIe 1.1 Capability Structure Expansion ECN and PCIe 2.0 requires
all registers in the PCIe capability to be either implemented or
hardwired to 0. Their PCIe capability version is 2.
Signed-off-by: NYu Zhao <yu.zhao@intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

PCIe 2.0 defines several new registers (Device Control 2, Link Control 2,
and Slot Control 2). Save and retore them in pci_save_pcie_state() and
pci_restore_pcie_state().
Signed-off-by: NYu Zhao <yu.zhao@intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

If a device has the SR-IOV capability, initialize it (set the ARI
Capable Hierarchy in the lowest numbered PF if necessary; calculate
the System Page Size for the VF MMIO, probe the VF Offset, Stride
and BARs). A lock for the VF bus allocation is also initialized if
a PF is the lowest numbered PF.
Reviewed-by: NMatthew Wilcox <willy@linux.intel.com>
Signed-off-by: NYu Zhao <yu.zhao@intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Reviewed-by: NMatthew Wilcox <matthew@wil.cx>
Signed-off-by: NYu Zhao <yu.zhao@intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Clean up register definitions related to PCI Express Hot plug.

  - Add register definitions into include/linux/pci_regs.h, and use
    them instead of pciehp's locally definied register definitions.
  - Remove pciehp's locally defined register definitions
  - Remove unused register definitions in pciehp.
  - Some minor cleanups.
Signed-off-by: NKenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

PCI Advanced Features Capability is introduced by "Conventional PCI
Advanced Caps ECN" (can be downloaded in pcisig.com).  Add defines for
the various AF capabilities, including function level reset (FLR).
Reviewed-by: NMatthew Wilcox <willy@linux.intel.com>
Signed-off-by: NSheng Yang <sheng@linux.intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Sometimes, it's necessary to enable software's ability to quiesce and
reset endpoint hardware with function-level granularity, so provide
support for it.

The patch implement Function Level Reset(FLR) feature following PCI-e
spec. And this is the first step. We would add more generic method, like
D0/D3, to allow more devices support this function.

The patch contains two functions. pcie_reset_function() is the new
driver API, and, contains some action to quiesce a device.  The other
function is a helper:  pcie_execute_reset_function() just executes the
reset for a particular device function.

Current the usage model is in KVM. Function reset is necessary for
assigning device to a guest, or moving it between partitions.

For Function Level Reset(FLR), please refer to PCI Express spec chapter
6.6.2.
Signed-off-by: NSheng Yang <sheng@linux.intel.com>
Signed-off-by: NMatthew Wilcox <willy@linux.intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

This patch adds support for PCI Express Alternative Routing-ID
Interpretation (ARI) capability.

The ARI capability extends the Function Number field of the PCI Express
Endpoint by reusing the Device Number which is otherwise hardwired to 0.
With ARI, an Endpoint can have up to 256 functions.
Signed-off-by: NYu Zhao <yu.zhao@intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Disable ASPM on pre-1.1 PCIe devices, as many of them don't implement it
correctly.
Tested-by: NJack Howarth <howarth@bromo.msbb.uc.edu>
Signed-off-by: NShaohua Li <shaohua.li@intel.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

If the offset of PCI device's PM capability in its configuration space,
the mask of states that the device supports PME# from and the D1 and D2
support bits are cached in the corresponding struct pci_dev, the PCI
device PM code can be simplified quite a bit.
Signed-off-by: NRafael J. Wysocki <rjw@sisk.pl>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

PCI Express ASPM defines a protocol for PCI Express components in the D0
state to reduce Link power by placing their Links into a low power state
and instructing the other end of the Link to do likewise. This
capability allows hardware-autonomous, dynamic Link power reduction
beyond what is achievable by software-only controlled power management.
However, The device should be configured by software appropriately.
Enabling ASPM will save power, but will introduce device latency.

This patch adds ASPM support in Linux. It introduces a global policy for
ASPM, a sysfs file /sys/module/pcie_aspm/parameters/policy can control
it. The interface can be used as a boot option too. Currently we have
below setting:
        -default, BIOS default setting
        -powersave, highest power saving mode, enable all available ASPM
state and clock power management
        -performance, highest performance, disable ASPM and clock power
management
By default, the 'default' policy is used currently.

In my test, power difference between powersave mode and performance mode
is about 1.3w in a system with 3 PCIE links.

Note: some devices might not work well with aspm, either because chipset
issue or device issue. The patch provide API (pci_disable_link_state),
driver can disable ASPM for specific device.
Signed-off-by: NShaohua Li <shaohua.li@intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

This reverts commit 6c723d5b.

It caused build errors on non-x86 platforms, config file confusion, and
even some boot errors on some x86-64 boxes.  All around, not quite ready
for prime-time :(

Cc: Shaohua Li <shaohua.li@intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

PCI Express ASPM defines a protocol for PCI Express components in the D0
state to reduce Link power by placing their Links into a low power state
and instructing the other end of the Link to do likewise. This
capability allows hardware-autonomous, dynamic Link power reduction
beyond what is achievable by software-only controlled power management.
However, The device should be configured by software appropriately.
Enabling ASPM will save power, but will introduce device latency.

This patch adds ASPM support in Linux. It introduces a global policy for
ASPM, a sysfs file /sys/module/pcie_aspm/parameters/policy can control
it. The interface can be used as a boot option too. Currently we have
below setting:
        -default, BIOS default setting
        -powersave, highest power saving mode, enable all available ASPM
state
and clock power management
        -performance, highest performance, disable ASPM and clock power
management
By default, the 'default' policy is used currently.

In my test, power difference between powersave mode and performance mode
is about 1.3w in a system with 3 PCIE links.
Signed-off-by: NShaohua Li <shaohua.li@intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Modify PCI Bridge Control ISA flag for clarity

This patch changes PCI_BRIDGE_CTL_NO_ISA to PCI_BRIDGE_CTL_ISA
and modifies it's clarifying comment and locations where used.
The change reduces the chance of future confusion since it makes
the set/unset meaning of the bit the same in both the bridge
control register and bridge_ctl field of the pci_bus struct.
Signed-off-by: NGary Hade <garyhade@us.ibm.com>
Acked-by: NLinas Vepstas <linas@austin.ibm.com>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

These IDs are in pciutils, but haven't been added to the kernel
yet.
Signed-off-by: NAlex Chiang <achiang@hp.com>
Signed-off-by: NMatthew Wilcox <matthew@wil.cx>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Newer tg3 devices shuffle around the registers in PCI configuration
space.  This patch changes the way the driver accesses the PCI
capabilities registers.  Hardcoded register locations are replaced with
offsets from pci_find_capability() return values.
Signed-off-by: NMatt Carlson <mcarlson@broadcom.com>
Signed-off-by: NMichael Chan <mchan@broadcom.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

There are two ways pci_save_state and pci_restore_state are used.  As
helper functions during suspend/resume, and as helper functions around
a hardware reset event.  When used as helper functions around a hardware
reset event there is no reason to believe the calls will be paired, nor
is there a good reason to believe that if we restore the msi state from
before the reset that it will match the current msi state.  Since arch
code may change the msi message without going through the driver, drivers
currently do not have enough information to even know when to call
pci_save_state to ensure they will have msi state in sync with the other
kernel irq reception data structures.

It turns out the solution is straight forward, cache the state in the
existing msi data structures (not the magic pci saved things) and
have the msi code update the cached state each time we write to the hardware.
This means we never need to read the hardware to figure out what the hardware
state should be.

By modifying the caching in this manner we get to remove our save_state
routines and only need to provide restore_state routines.

The only fields that were at all tricky to regenerate were the msi and msi-x
control registers and the way we regenerate them currently is a bit dependent
upon assumptions on how we use the allow msi registers to be configured and used
making the code a little bit brittle.  If we ever change what cases we allow
or how we configure the msi bits we can address the fragility then.
Signed-off-by: NEric W. Biederman <ebiederm@xmission.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>
Acked-by: NAuke Kok <auke-jan.h.kok@intel.com>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

In some cases when we are not using msi we need a way to ensure that the
hardware does not have an msi capability enabled.  Currently the code has been
calling disable_msi_mode to try and achieve that.  However disable_msi_mode
has several other side effects and is only available when msi support is
compiled in so it isn't really appropriate.

Instead this patch implements pci_msi_off which disables all msi and msix
capabilities unconditionally with no additional side effects.

pci_disable_device was redundantly clearing the bus master enable flag and
clearing the msi enable bit.  A device that is not allowed to perform bus
mastering operations cannot generate intx or msi interrupt messages as those
are essentially a special case of dma, and require bus mastering.  So the call
in pci_disable_device to disable msi capabilities was redundant.

quirk_pcie_pxh also called disable_msi_mode and is updated to use pci_msi_off.
Signed-off-by: NEric W. Biederman <ebiederm@xmission.com>
Cc: Michael Ellerman <michael@ellerman.id.au>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Greg KH <greg@kroah.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Add a few #defines for grabbing and working with the address fields
in a HT_CAPTYPE_MSI_MAPPING capability. All from the HT spec v3.00.
Signed-off-by: NMichael Ellerman <michael@ellerman.id.au>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

There are already several places in the kernel that want to search a PCI
device for a given Hypertransport capability. Although this is possible
using pci_find_capability() etc., it makes sense to encapsulate that
logic in a helper - pci_find_ht_capability().

To cater for searching exhaustively for a capability, we also provide
pci_find_next_ht_capability().

We also need to cater for the fact that the HT capability fields may be
either 3 or 5 bits wide. pci_find_ht_capability() deals with this for you,
but callers using the #defines directly must handle that themselves.
Signed-off-by: NMichael Ellerman <michael@ellerman.id.au>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Move some MSI-X #defines into pci_regs.h so they can be used
outside of drivers/pci.
Signed-off-by: NMichael Ellerman <michael@ellerman.id.au>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>