In preparation for irq_domain support change the code to the not switch
based on the irq number. This actually makes things simpler, if slightly
repetitive.
Signed-off-by: NMark Brown <broonie@opensource.wolfsonmicro.com>