A rule of thumb for pulling worms. Try to stay under 1.5mph. 2 oz weight , for every foot down in depth it takes 2 feet of line back. This works out for 1.2 to 1.5 mph. If you are going faster like 1.5 to 1.8. Its 3 ft back for same depth.
Worms are better with low speed. When I drift, which is most of the time, I try to keep my speed under 1.5. Same for trolling with worm. Feet back works if the fish are up high, then if they are very high, go with also less weight.