The USBNET_DEVADDR has nothing to do with the USB download gadget, but
rather with the USB Ethernet gadget. Move it out of the if statement.

Acked-by: Łukasz Majewski <lukma@denx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Maxime Ripard <maxime.ripard@free-electrons.com>