Let it swing further back behind the boat before reeling in.