This patch removes a limitation on the maximum packet size that is read by
the device CPU as that limitation is not needed.

Therefore, the patch also removes an elaborate calculation that is based
on this limitation which is also not needed now. Instead, use a fixed
value for the memory pool size of the packets.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds support to the goya memset function to perform memset to
device memory with size larger then 4GB. In this case, we need to use
multiple LIN_DMA packets because a single packet supports up to 4GB.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch improves the error reporting in case of fatal and non-RAZWI
events such that the event name is printed in addition to the IRQ number.
Signed-off-by: NOmer Shpigelman <oshpigelman@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch increases the timeout for PCI ELBI configuration to support low
frequency Palladium images.
Signed-off-by: NOmer Shpigelman <oshpigelman@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds a new parameter that is passed to the
add_end_of_cb_packets() asic-specific function.

The parameter is the pointer to the driver's device structure. The
function needs this pointer for future ASICs.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch changes two polling functions to macros, in order to make their
API the same as the standard readl_poll_timeout so we would be able to
define the "condition for exit" when calling these macros.

This will simplify the code as it will eliminate the need to check both
for timeout and for the (cond) in the calling function.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The driver allocates memory for fence object with GFP_ZERO flag, so there
is no need to explicitly write 0 to the allocated object after the
allocation.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Driver-initiated DMA jobs are synchronized jobs, i.e. the driver polls on
fence object until the job is finished. There is no interrupt from the
device. Therefore, no need to add space for 2 * msg_prot packets to the
end of the CB. Only a single msg_prot is needed (to write the fence).
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch changes the order of checks when initializing the device CPU.
We want first to check if we need to load the F/W, and only if we need to,
then we want to check the status of the CPU boot program.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch removes some dead code that performs checks about variables
with hard-coded values.

The patch also moves the initialization of those variables to a separate
function, that will possibly have different values per ASIC.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds the implementation of the HL_DEBUG_OP_SET_MODE opcode in
the DEBUG IOCTL.

It forces the user who wants to debug the device to set the device into
debug mode before he can configure the debug engines. The patch also makes
sure to disable debug mode upon user releasing FD, in case the user forgot
to disable debug mode.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch fixes comments on various structure members and some spelling
errors in log messages.
Signed-off-by: NOmer Shpigelman <oshpigelman@habana.ai>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch removes redundant CPU availability checks in:
goya_test_queues() - will be done in goya_test_cpu_queue().
goya_ring_doorbell() - was done earlier in goya_send_cpu_message().
Signed-off-by: NOmer Shpigelman <oshpigelman@habana.ai>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch improves the error message that is shown when a new user tries
to open a new FD while there is already an existing user that is working
on the device.

It also improves the error message in case of missing firmware file.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

We cannot cast a 64-bit integer to a pointer on 32-bit architectures
without a warning:

drivers/misc/habanalabs/habanalabs_ioctl.c: In function 'debug_coresight':
drivers/misc/habanalabs/habanalabs_ioctl.c:143:23: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
   input = memdup_user((const void __user *) args->input_ptr,

Use the macro that was defined for this purpose.

Fixes: 315bc055 ("habanalabs: add new IOCTL for debug, tracing and profiling")
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The trace buffer address is 40 bits wide.
The end of the buffer is set in the RWP register (lower 32 bits), and in
the RWPHI register (upper 8 bits).
Currently only the lower 32 bits are read, and this patch fixes it and
concatenates the upper 8 bits to the output address.
Signed-off-by: NTomer Tayar <ttayar@habana.ai>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The debugfs interface for accessing DRAM virtual addresses currently
uses the 12 LSBs of a virtual address as an offset.
However, it should use the 20 LSBs in case the device MMU page size is
2MB instead of 4KB.
This patch fixes the offset calculation to be based on the page size.
Signed-off-by: NTomer Tayar <ttayar@habana.ai>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch fix a bug in the mmu code that checks whether we can use huge
page mappings for host pages.

The code is supposed to enable huge page mappings only if ALL DMA
addresses are aligned to 2MB AND the number of pages in each DMA chunk is
a modulo of the number of pages in 2MB. However, the code ignored the
first requirement for the first DMA chunk.

This patch fix that issue by making sure the requirement of address
alignment is validated against all DMA chunks.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The MMU cache mutex is used in the ASIC hw_init() functions, but it is
initialized only later in hl_mmu_init().
This patch prevents it by moving the initialization to the
device_early_init() function.
Signed-off-by: NTomer Tayar <ttayar@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This fixes multiple things in the habanalabs debugfs code, in particular:

 - mmu_write() was unnecessarily verbose, copying around between multiple
   buffers
 - mmu_write() could write a user-specified, unbounded amount of userspace
   memory into a kernel buffer (out-of-bounds write)
 - multiple debugfs read handlers ignored the user-supplied count,
   potentially corrupting out-of-bounds userspace data
 - hl_device_read() was unnecessarily verbose
 - hl_device_write() could read uninitialized stack memory
 - multiple debugfs read handlers copied terminating null characters to
   userspace
Signed-off-by: NJann Horn <jannh@google.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Cc: stable@vger.kernel.org

This patch fix a potential bug where a user's process has closed
unexpectedly without disabling the debug engines. In that case, the debug
engines might continue running but because the user's MMU mappings are
going away, we will get page fault errors.

This behavior is also opposed to the general rule where nothing runs on
the device after the user process closes.

The patch stops the debug H/W engines upon process termination and thus
makes sure nothing runs on the device after the process goes away.
Signed-off-by: NOmer Shpigelman <oshpigelman@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Add SPDX license identifiers to all Make/Kconfig files which:

 - Have no license information of any form

These files fall under the project license, GPL v2 only. The resulting SPDX
license identifier is:

  GPL-2.0-only
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

The CPU accessible DMA memory is general and not used only for PQ.
Accordingly, this patch renames the "free_cpu_pq_dma_mem" label with
"free_cpu_dma_mem".
Signed-off-by: NTomer Tayar <ttayar@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The CPU accessible DMA pool is general and not used only for PQ.
Accordingly, this patch rename the "free_cpu_pq_pool" label with
"free_cpu_accessible_dma_pool".
Signed-off-by: NTomer Tayar <ttayar@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Where there is a spike in the CPU consumption, it may cause
random failures in the C/I since the KMD timeout for CPU
and/or QMAN0 jobs expires and it stops communicating to the simulator.
This commit fixes it by increasing timeout on polling functions
if working with simulator.
Signed-off-by: NDalit Ben Zoor <dbenzoor@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

After removing the parsing of the command submission
when doing memset of the device memory, goya_validate_dma_pkt_host
is never called by the kernel, so there is no need to check
context id.
Signed-off-by: NDalit Ben Zoor <dbenzoor@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

use_virt_addr member was used for telling whether to treat the
addresses in the CB as virtual during parsing. We disabled it only
when calling the parser from the driver memset device function,
and since this call had been removed, it should always be enabled.
Signed-off-by: NDalit Ben Zoor <dbenzoor@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Routing device accesses to the host memory requires the usage of a base
offset, which is canceled by the iATU just before leaving the device.
The value of the base offset might be distinctive between different ASIC
types.
The manipulation of the addresses is currently used throughout the
driver code, and one should be aware to it whenever providing a host
memory address to the device.
This patch removes this manipulation from the driver common code, and
moves it to the ASIC specific functions that are responsible for
host memory allocation/mapping.
Signed-off-by: NTomer Tayar <ttayar@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch renames four functions in the ASIC-specific functions section,
so it will be easier to differentiate them from the generic kernel
functions with the same name.

This will help in future code reviews, to make sure we don't use the
kernel functions directly.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

There is no need to parse the command submission when doing memset
of the device memory using the DMA engine because only the driver calls
the memset function and therefore, the CS is trusted and doesn't require
validation and patching.
Signed-off-by: NDalit Ben Zoor <dbenzoor@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The device's CPU accessible memory on host is managed in a dedicated
pool, except for 2 regions - Primary Queue (PQ) and Event Queue (EQ) -
which are allocated from generic DMA pools.
Due to address length limitations of the CPU, the addresses of all these
memory regions must have the same MSBs starting at bit 40.
This patch modifies the allocation of the PQ and EQ to be also from the
dedicated pool, to ensure compliance with the limitation.
Signed-off-by: NTomer Tayar <ttayar@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch changes the ASIC interface function that changes the DRAM bar
window. The change is to return the old address that the DRAM bar pointed
to instead of an error code.

This simplifies the code that use this function (mainly in debugfs) to
restore the bar to the old setting.

This is also needed for easier support in future ASICs.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch only does renaming of certain variables and structure members,
and their accompanied comments.

This is done to better reflect the actions these variables and members
represent.

There is no functional change in this patch.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch slightly changes the macros of RREG32 and WREG32, which are
used when reading or writing from registers.

Instead of directly calling a function in the common code from these
macros, the new code calls a function from the ASIC functions interface.

This change allows us to share much more code between real ASICs and
simulators, which in turn reduces the maintenance burden and
the chances for forgetting to port code between the ASIC files.

The patch also implements the hl_poll_timeout macro, instead of calling
the generic readl_poll_timeout macro. This is required to allow use of
this macro in the simulator files.

As a result from this change, more functions in goya.c are shared with the
simulator and therefore, should not be defined as static.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds missing fields of start address 0 and 1 in the bmon
parameter structure that is received from the user in the debug IOCTL.

Without these fields, the functionality of the bmon trace is broken,
because there is no configuration of the base address of the filter of the
bus monitor.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch re-factors goya_parse_cb_no_ext_queue() to make it more
readable by inverting the check inside the first if statement so the bulk
of the function won't be inside an if statement.

The patch also fixes a spelling error in the name of the function.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

pr_fmt() should be defined before including linux/printk.h, either
directly or indirectly, in order to avoid redefinition of the macro.
Currently the macro definition is in habanalabs.h, which is included in
many files, and that makes the addition/reorder of includes to be prone
to compilation errors.
This patch cancels this dependency by defining the macro only in the few
source files that use it.
Signed-off-by: NTomer Tayar <ttayar@habana.ai>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

During hard-reset, contexts are closed as part of the tear-down process.
After a context is closed, the driver cleans up the page tables of that
context in the device's DRAM. This action is both dangerous and
unnecessary.

It is unnecessary, because the device is going through a hard-reset, which
means the device's DRAM contents are no longer valid and the device's MMU
is being reset.

It is dangerous, because if the hard-reset came as a result of a PCI
freeze, this action may cause the entire host machine to hang.

Therefore, prevent all device PTE updates when a hard-reset operation is
pending.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch makes some improvement in how IOCTLs behave when the device is
disabled or under reset.

The new code checks, at the start of every IOCTL, if the device is
disabled or in reset. If so, it prints an appropriate kernel message and
returns -EBUSY to user-space.

In addition, the code modifies the location of where the
hard_reset_pending flag is being set or cleared:

1. It is now cleared immediately after the reset *tear-down* flow is
   finished but before the re-initialization flow begins.

2. It is being set in the remove function of the device, to make the
   behavior the same with the hard-reset flow

There are two exceptions to the disable or in reset check:

1. The HL_INFO_DEVICE_STATUS opcode in the INFO IOCTL. This opcode allows
   the user to inquire about the status of the device, whether it is
   operational, in reset or malfunction (disabled). If the driver will
   block this IOCTL, the user won't be able to retrieve the status in
   case of malfunction or in reset.

2. The WAIT_FOR_CS IOCTL. This IOCTL allows the user to inquire about the
   status of a CS. We want to allow the user to continue to do so, even if
   we started a soft-reset process because it will allow the user to get
   the correct error code for each CS he submitted.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch fixes a bug in the implementation of the function that removes
the device.

The bug can happen when the device is removed but not the driver itself
(e.g. remove by the OS due to PCI freeze in Power architecture).

In that case, there maybe open users that are calling IOCTLs while the
device is removed. This is a possible race condition that the driver must
handle. Otherwise, a kernel panic may occur.

This race is prevented in the hard-reset flow, because the driver makes
sure the users are closed before continuing with the hard-reset. This
race can not occur when the driver itself is removed because the OS makes
sure all the file descriptors are closed.

The fix is to make sure the open users close their file descriptors and if
they don't (after a certain amount of time), the driver sends them a
SIGKILL, because the remove of the device can't be stopped.

The patch re-uses the same code that is called from the hard-reset flow.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>