Zap an obscure unneeded cast and clean other minor nits in this function.

Signed-off-by: Marek Vasut <marex@denx.de>