Because watch invites can be sent by others while you’re playing, receiving such an invite can easily lead to getting thrown off.
After experiencing this first-hand, namely an invite distracting me and worsening my play, I’d be grateful if an option was added so I can prevent this from ever happening again.
As a side-note, delaying notifications about someone adding you to friend list until a match is over would be nice as well.