For northern Europe, I'm on the larger side. I primarily do Longsword fencing, with the occasional Shield and Axe or One-Hand-Spear mixed in.
I think it comes down to training - I started out fencing with friends who're about the same size as me. The first few shorter people I fought against abolutely obliterated me, because I could't defend against them properly. Nowadays I'd say it's balanced.
Another factor I rank higher than body size is physical fitness and motor control. It just doesn't matter much how taller you are if your opponent moves faster and more precise than you - especially as the sparring sessions drags out.
Though body mass is an advantage when it comes to ringing with your opponent. Then again that also goes for a lower center of gravity.
Don't judge oppontents by their gestalt, judge them by how they spar.
To answer your question: I don't agree with your opinion, as I argue that body size is not a key metric when it comes to determine an advantage over another fencer.