The reason I don’t buy gems isn’t because I can’t afford them, or because I’m cheap. There’s just nothing in the game that makes me want to buy them.
Most things you can buy with gems are too overpriced. Buying resources directly is a total rip off. You would need to spend $20 to have enough gems for the gold needed to upgrade 1 wizard tower from level 6 to level 7.
Boosting collectors is more reasonable, but the process is too slow. It might be worthwhile if they produced 3 or 4 times the usual amount. As is, boosting maxed collectors will only get you an extra 500k per day. You’d need to boost them for an entire week just for one upgrade, and half of that loot would be stolen in raids anyway.
The only boost that really makes sense is to boost your barracks and farm non-stop. The problem with that is the boost lasts two hours, and people have real-life responsibilities. It’s hard to find a 2 hour chunk of time to use for non-stop raiding. A one hour boost that could be used during lunch break or on the bus ride home from work would be much more attractive.
In another thread, someone suggested an across-the-board timer boost. For example, perhaps you could spend 150 gems per day, or maybe 700 gems per week, to have all your timers move twice as fast. Troops would train at double speed, builders would build twice as fast, heroes would heal in half the time, etc. I would definitely want to buy something like that.
In sum, I would buy gems if there was something worth buying them for, and I'm sure many other free players would as well.

