ASoC: SOF: sof-audio: unprepare when swidget->use_count > 0

We should unprepare the widget if its use_count = 1.

Fixes: 9862dcf7 ("ASoC: SOF: don't unprepare widget used other pipelines")
Cc: <stable@vger.kernel.org> # 6.1
Signed-off-by: NBard Liao <yung-chuan.liao@linux.intel.com>
Reviewed-by: NRanjani Sridharan <ranjani.sridharan@linux.intel.com>
Reviewed-by: NPierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: NRander Wang <rander.wang@intel.com>
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@linux.intel.com>
Link: https://lore.kernel.org/r/20230118101255.29139-2-peter.ujfalusi@linux.intel.comSigned-off-by: NMark Brown <broonie@kernel.org>

sound/soc/sof/sof-audio.c

@@ -272,7 +272,7 @@ sof_unprepare_widgets_in_path(struct snd_sof_dev *sdev, struct snd_soc_dapm_widg
 	struct snd_soc_dapm_path *p;
 
 	/* return if the widget is in use or if it is already unprepared */
-	if (!swidget->prepared || swidget->use_count > 1)
+	if (!swidget->prepared || swidget->use_count > 0)
 		return;
 
 	if (widget_ops[widget->id].ipc_unprepare)