pt2pt ndlp ref count prematurely goes to 0. There was reference removed
that should only be removed if connected to a switch, not if in
point-to-point mode.

Add a mode check before the reference remove.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NHannes Reinecke <hare@suse.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>