Implement graceful handling of surprise or orderly removal of PCIe HBA:

 - Detect a hot removal of the controller at certain critical places in the
   driver. Early detection will help to reduce the time taken for cleaning
   up the hot-removed controller at the driver level.

 - Poll the status of the port enable issued after reset once every 5
   seconds to avoid a long delay in detecting unavailable controller.

Link: https://lore.kernel.org/r/20220912135742.11764-5-sreekanth.reddy@broadcom.comReported-by: Nkernel test robot <lkp@intel.com>
Signed-off-by: NSreekanth Reddy <sreekanth.reddy@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>