And how many of the population has high enough strenght to make an effective fighter, that could take on the fantasy races with far superior scores? It's not just human vs human, when an owlbear could eat half of your village barony for breakfast.
Frankly, if you'd have realism in a fantasy setting, then humanity would become extinct, because apex predators in the fantasy settings are not only stronger, but also more wise and/or intelligent. Just look how much wisdom a worg has in this game. Does it actually act like it, though? Not really; that is the "no realism" part.
You don't have have to fend off a smilodon with a sword in real life, because intelligence does trump strenght given enough time. That is why humanity is now hunting all the stronger species into extinction, and not the other way around. (where are the heroic owlbears when you need them...)