Clearly you're in the right. Think about it this way, what if you'd been perching and had been anchored. Are you then responsible for pulling anchor and moving out of the way of every boat that came your way? Guy was just being a jerk.

18. Responsibilities between vesselsExcept in narrow channels, traffic separation schemes, and when overtaking (i.e., rules 9, 10, and 13) A power-driven vessel must give way to: a vessel not under command;
a vessel restricted in her ability to manoeuvre (this may include vessels towing one another[12]);
a vessel engaged in fishing;
a sailing vessel.

A vessel engaged in fishing when underway shall, so far as possible, keep out of the way of:
a vessel not under command;
a vessel restricted in her ability to manoeuvre.

Any vessel other than a vessel not under command or a vessel restricted in her ability to manoeuvre shall, if possible, not impede the safe passage of a vessel constrained by her draft, exhibiting the signals in Rule 28.
A vessel constrained by her draft shall navigate with particular caution having full regard to her special condition.[6]