dm: clk: ignore default settings when node not valid

When the device not binded with a node, we need ignore
the parents and rate settings.

Cc: Simon Glass <sjg@chromium.org>
Cc: Jagan Teki <jagan@amarulasolutions.com>
Cc: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Cc: Neil Armstrong <narmstrong@baylibre.com>
Cc: Andreas Dannenberg <dannenberg@ti.com>
Signed-off-by: NPeng Fan <peng.fan@nxp.com>

drivers/clk/clk-uclass.c

@@ -285,6 +285,9 @@ int clk_set_defaults(struct udevice *dev)
 {
 	int ret;
 
+	if (!dev_of_valid(dev))
+		return 0;
+
 	/* If this not in SPL and pre-reloc state, don't take any action. */
 	if (!(IS_ENABLED(CONFIG_SPL_BUILD) || (gd->flags & GD_FLG_RELOC)))
 		return 0;