Instead of clearing 2 bits, all the other
bits were set because '|=' was used instead
of '&='.

Signed-off-by: Troy Kisky <troy.kisky@boundarydevices.com>
Acked-by: Marek Vasut <marex@denx.de>
Acked-by: Stefano Babic <sbabic@denx.de>