Due to the peripheral clock driver improvement, remove the
unnecessary clock calling.

Signed-off-by: Wenyou Yang <wenyou.yang@atmel.com>
Acked-by: Stephen Warren <swarren@nvidia.com>