I travel back and forth all season long. I usually only carry my passport if I step foot on land. That being said, you are required to report to Canadian customs if you cross, when you cross. This topic is debated every year. What you saw was likely border patrol. Many of them are contracted agents that use a boat service for hire. At the end of the day, you need to do what you are comfortable with. IMO fishing near or on the line doesn't warrant carrying of your passport as Long as you have other ID.