Otherwise we reference potentially non-existing members when
ipv6 is disabled.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>