We don't need to be paranoid about the topology changing while handling an
error.  If the device has changed in a hotplug capable slot, we can rely on
the presence detection handling to react to a changing topology.

Restore the fatal error handling behavior that existed before merging DPC
with AER with 7e9084b3 ("PCI/AER: Handle ERR_FATAL with removal and
re-enumeration of devices").
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>

This patch provides DPC save and restore capabilities.  This is necessary
for the driver to observe DPC events in the event the configuration space
needs to be restored after a reset.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>

The PCI port driver saves the PCI state after initializing the device with
the applicable service devices.  This was, however, before the service
drivers were even registered because PCI probe happens before the
device_initcall initialized those service drivers.  The config space state
that the services set up were not being saved.  The end result would cause
PCI devices to not react to events that the drivers think they did if the
PCI state ever needed to be restored.

Fix this by changing the service drivers from using the init calls to
having the portdrv driver calling the services directly.  This will get the
state saved as desired, while making the relationship between the port
driver and the services under it more explicit in the code.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>

Simplify waiting for the contained link to become inactive, removing the
indirection to a unnecessary DPC-specific handler.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>
Reviewed-by: NOza Pawandeep <poza@codeaurora.org>

Remove the work struct that was being used to handle a DPC event and use a
threaded IRQ instead.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>
Reviewed-by: NOza Pawandeep <poza@codeaurora.org>

A DPC enabled device suppresses ERR_(NON)FATAL messages, preventing the AER
handler from reporting error details.  If the DPC trigger reason says the
downstream port detected the error, collect the AER uncorrectable status
for logging, then clear the status.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>
Reviewed-by: NOza Pawandeep <poza@codeaurora.org>

We don't need to save the rp pio status across multiple contexts as all
DPC event handling occurs in a single work queue context.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>
Reviewed-by: NOza Pawandeep <poza@codeaurora.org>

Move all event handling to the existing work queue, which will
make it simpler to pass event information to the handler.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>
Reviewed-by: NOza Pawandeep <poza@codeaurora.org>

Now that the DPC driver clears the interrupt status before exiting the
IRQ handler, we don't need to abuse the DPC control register to know if
a shared interrupt is for a new DPC event: a DPC port can not trigger
a second interrupt until the host clears the trigger status later in the
work queue handler.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>
Reviewed-by: NOza Pawandeep <poza@codeaurora.org>

Move pcie_aer_get_firmware_first() to portdrv.h, where it can be more
easily shared between AER and DPC.  Then DPC no longer needs to include
aer/aerdrv.h.  No functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>

Our goal is to handle ERR_FATAL errors similarly, whether they are reported
via AER or via DPC.  A previous commit changed AER so it handles ERR_FATAL
by calling driver .remove() methods and resetting the Link.  DPC already
does that (although the Link reset is done automatically by hardware and
happens before we call the driver .remove() methods).

Restructure the DPC code so it calls the same pcie_do_fatal_recovery()
interface used by AER.  This makes it clearer that we want to use the same
path.

Implement the .reset_link() method used by pcie_do_fatal_recovery().  For
DPC, the actual reset is done automatically by hardware, so we really only
have to wait for the Link to be inactive, then release the Port from DPC.
Signed-off-by: NOza Pawandeep <poza@codeaurora.org>
[bhelgaas: changelog, DPC_FATAL is not a bitfield, can be sequential]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

PCIe ERR_NONFATAL errors mean a particular transaction is unreliable but
the Link is otherwise fully functional (PCIe r4.0, sec 6.2.2).

The AER driver handles these by logging the error details and calling
driver-supplied pci_error_handlers callbacks.  It does not reset downstream
devices, does not remove them from the PCI subsystem, does not re-enumerate
them, and does not call their driver .remove() or .probe() methods.

But DPC driver previously enabled DPC on ERR_NONFATAL, so if the hardware
supports DPC, these errors caused a Link reset (performed automatically by
the hardware), followed by the DPC driver removing affected devices (which
calls their .remove() methods), bringing the Link back up, and
re-enumerating (which calls driver .probe() methods).

Disable ERR_NONFATAL DPC triggering so these errors will only be handled by
AER.  This means drivers won't have to deal with different usage of their
pci_error_handlers callbacks and .probe() and .remove() methods based on
whether the platform has DPC support.
Signed-off-by: NOza Pawandeep <poza@codeaurora.org>
[bhelgaas: changelog]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Clients such as hotplug and Downstream Port Containment (DPC) both need to
wait until a link becomes active or inactive.

Add a generic pcie_wait_link_active() interface and use it instead of
duplicating the code.
Signed-off-by: NOza Pawandeep <poza@codeaurora.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>

The generic IRQ handling code ensures that an interrupt handler runs with
its interrupt masked or disabled.  If the interrupt is level-triggered, the
interrupt handler must tell its device to stop asserting the interrupt
before returning.  If it doesn't, we will immediately take the interrupt
again when the handler returns and the generic code unmasks the interrupt.

The driver doesn't know whether its interrupt is edge- or level-triggered,
so it must clear its interrupt source directly in its interrupt handler.

Previously we cleared the DPC interrupt status in the bottom half, i.e., in
deferred work, which can cause an interrupt storm if the DPC interrupt
happens to be level-triggered, e.g., if we're using INTx instead of MSI.

Clear the DPC interrupt status bit in the interrupt handler, not in the
deferred work.
Signed-off-by: NOza Pawandeep <poza@codeaurora.org>
[bhelgaas: changelog]
Signed-off-by: NBjorn Helgaas <helgaas@kernel.org>
Reviewed-by: NKeith Busch <keith.busch@intel.com>

Rename pcie-dpc.c to dpc.c.  The path "drivers/pci/pcie/pcie-dpc.c" has
more occurrences of "pci" than necessary.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

pcieport_if.h contained the interfaces to register port service driver,
e.g., pcie_port_service_register().  portdrv.h contained internal data
structures of the port driver.

I don't think it's worth keeping those files separate, since both headers
and their users are all inside the PCI core.

Merge pcieport_if.h directly in drivers/pci/pcie/portdrv.h and update the
users to include that instead.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Move pcieport_if.h from include/linux to drivers/pci/pcie/pcieport_if.h
because the interfaces there are only used by the PCI core.

Replace all uses of #include<linux/pcieport_if.h> with relative paths to
the new file location, e.g., #include "../pcieport_if.h"
Signed-off-by: NFrederick Lawler <fred@fredlawl.com>
Signed-off-by: NBjorn Helgaas <helgaas@kernel.org>

Add definitions for DPC Status register fields and use them in the code.
No functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

dpc_process_rp_pio_error() only calls dpc_rp_pio_get_info(), so squash them
together.  No functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

We read and immediately print the RP PIO log registers.  We don't save
them, so there's no need to define structs for them.  Remove the structs
and read the registers into local variables instead.  No functional change
intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

Move the dpc->rp_pio_status assignment into dpc_rp_pio_get_info() since
that's where we read rp_pio->status anway.  No functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

Separating dpc_rp_pio_print_error() doesn't really provide any useful
abstraction, so squash it into its caller, dpc_rp_pio_get_info().  No
functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

In dpc_probe(), we set dpc->rp_log_size to zero if we think the hardware
reports an invalid size.  In this case, we could have dpc->rp_extensions
set but dpc->rp_log_size == 0, and we should print the basic RP PIO
registers but not the variable-size portion.  We already checked for
dpc->rp_log_size < 4 above, so this patch is just for consistency of style.
No functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

In dpc_rp_pio_get_info() rename the local "status" variable to
"dpc_status".  This is to make room for another variable named "status" in
a subsequent patch.  No functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

Separating dpc_rp_pio_print_tlp_header() doesn't really provide any useful
abstraction, so squash it into its caller, dpc_rp_pio_print_error().  No
functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

The RP PIO registers (status, mask, severity, etc) are only implemented if
the "RP Extensions for DPC" bit is set in the DPC Capabilities register.

Previously we called dpc_process_rp_pio_error(), which reads and decodes
those RP PIO registers, whenever the DPC Status register indicated an "RP
PIO error" (Trigger Reason == 3 and Trigger Reason Extension == 0).

It does seem reasonable to assume that DPC Status would only indicate an RP
PIO error if the RP extensions are supported, but PCIe r4.0, sec 7.9.15.4,
is actually not explicit about that: it does not say "Trigger Reason
Extension == 0 is valid only for Root Ports that support RP Extensions for
DPC."

Check whether the RP Extensions for DPC are supported before trying to read
the RP PIO registers.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

The RP PIO Log Size is a read-only field in the DPC Capability, so it is
constant and known at probe-time, but previously we read it every time we
processed an RP PIO error.

Read it once in dpc_probe() (if the RP Extensions for DPC are supported)
and remember the size in struct dpc_dev.  No functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

"rp" is ambiguous: it might mean "this DPC device is a Root Port."  But in
fact, it means "this DPC device is a Root Port *and* it supports a set of
DPC Extensions."

Rename "rp" to "rp_extensions" to make this more clear.  No functional
change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

Add a local variable for DPC capability offset and replace repeated use of
"dpc->cap_pos" with simply "cap".  No functional change intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

Add SPDX GPL-2.0 to all PCI files that referred to the kernel default
"COPYING" file, which specifies GPL version 2.

Remove the boilerplate language referring to the GPL and "COPYING", relying
on the assertion in b2441318 ("License cleanup: add SPDX GPL-2.0
license identifier to files with no license") that the SPDX identifier may
be used instead of the full boilerplate text.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Rename interrupt_event_handler() to dpc_work() so there's more useful
information in stack traces and similar situations.  No functional change
intended.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>
Reviewed-by: NSinan Kaya <okaya@codeaurora.org>

The interrupt message number is the first 5 bits, but the driver was
masking only the first 4 bits.  Fix that by using the existing
define.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
[bhelgaas: remove reformatting (done by another patch)]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The "Determination of DPC Control" implementation note in PCIe r4.0, sec
6.1.10, recommends the operating system always link DPC control to the
control of AER, as the two functionalities are strongly connected.

To avoid conflicts over whether platform firmware or the OS controls DPC,
enable DPC only if AER is enabled in the OS, and the device's error
handling does not have firmware-first AER handling.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <helgaas@kernel.org>

DPC supports shared interrupts, but it plays very loosely with testing
whether the interrupt is generated by DPC before generating spurious log
messages, such as:

  dpc 0000:10:01.2:pcie010: DPC containment event, status:0x1f00 source:0x0000

Testing the status register for zero or -1 is not sufficient when the
device supports the RP PIO First Error Pointer register.  Change this to
test whether the interrupt is enabled in the control register, retaining
the device present test, and that the status reports the interrupt as
signaled and DPC is triggered, clearing as a spurious interrupt otherwise.

Additionally, since the interrupt is actually serviced by a workqueue,
disable the interrupt in the control register until that completes or else
we may never see it execute due to further incoming interrupts.  A software
generated DPC floods the system otherwise.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NBjorn Helgaas <helgaas@kernel.org>
Reviewed-by: NKeith Busch <keith.busch@intel.com>

Use a local "struct device *dev" for brevity and consistency in DPC driver.
No functional change intended.
Signed-off-by: NDongdong Liu <liudongdong3@huawei.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>

Add eDPC support. Get and print the RP PIO error information when the
trigger condition is RP PIO error.

For more information on eDPC, please see PCI Express Base Specification
Revision 3.1, section 6.2.10.3, or view the PCI-SIG eDPC ECN here:
https://pcisig.com/sites/default/files/specification_documents/ECN_Enhanced_DPC_2012-11-19_final.pdfSigned-off-by: NDongdong Liu <liudongdong3@huawei.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>

This driver was OR'ing desired bits from the existing control setting.
That could create an invalid DPC Trigger Enabled configuration if the
platform previously set this to "ERR_FATAL", 01b.  The driver currently
wants to set this to ERR_NONFATAL/ERR_FATAL, 10b, and the logical OR of
this gets 11b, which is reserved.  Fix that by masking off the fields it is
setting.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The DPC interupt may be executed on a device that is being removed.  Skip
queuing event handling if the status is all 1's, which should be seen only
if the device is not present.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Add a new state to pci_dev to be set when it is unexpectedly disconnected.
The PCI driver tear down functions can observe this new device state so
they may skip operations that will fail.

The pciehp and pcie-dpc drivers are aware when the link is down, so these
set the flag when their handlers detect the device is disconnected.
Tested-by: NKrishna Dhulipala <krishnad@fb.com>
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NWei Zhang <wzhang@fb.com>

Per PCIe r3.1, sec 6.2.10 and sec 7.13.4, on Root Ports that support "RP
Extensions for DPC",

  When the DPC Trigger Status bit is Set and the DPC RP Busy bit is Set,
  software must leave the Root Port in DPC until the DPC RP Busy bit reads
  0b.

Wait up to 1 second for the Root Port to become non-busy.

[bhelgaas: changelog, spec references]
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>