This removes the ifdef clutter a bit and saves few lines.

It also makes it easier to detect the remaining places
where we have conditional building of code done based
on if defined for things like DMA.
Signed-off-by: NTony Lindgren <tony@atomide.com>
Signed-off-by: NFelipe Balbi <balbi@ti.com>