serial: ns16550: Remove unwanted cast

This cast does not seem correct, since we should be writing to a pointer,
not a ulong.

This fixes the following warning on nds32:

-ns16550.c:49: warning: passing argument 2 of 'writeb' makes pointer from integer without a cast

Signed-off-by: Simon Glass <sjg@chromium.org>