[ Upstream commit d9e100829fca6cbd270d7e005b0c0bb2d14924b8 ]

We are not destroying the sysfs attribute groupe we registered during
the probe function which will make subsequent probe calls to that
driver fail. Correct that with adding a remove function which only
removes those attributes since the reference counting on clocks did its
job already.

Fixes: 415060b2 ("phy: usb: phy-brcm-usb: Add ability to force DRD mode to host or device")
Signed-off-by: NFlorian Fainelli <f.fainelli@gmail.com>
Signed-off-by: NKishon Vijay Abraham I <kishon@ti.com>
Signed-off-by: NSasha Levin <sashal@kernel.org>
Signed-off-by: NYang Yingliang <yangyingliang@huawei.com>