Neither of these - think of it as a triangle - there are a LOT less people above you than below you that's why most of the people who attack are from below. The higher you go up the triangle the less likely of seeing anyone above you.
And gems have nothing to do with matchmaking.



