Store the source/destination slave address separately into the channel
structure.  This moves us towards being able to avoid a configuration
call each time we use the channel.
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>