Honestly, that sounds terrible.I'm taking actually being outright quasi immune to physical damage to the point that without the right tools, a ghost unit can practically solo an entire army if played right.
Imagine going up against that without ways to fight it - and no, you cannot always know what you'll be up against.
Getting wiped by a single unit of ghosts has to be some of the worst feelings

And I'm not sure if every faction even has access to magic before end game...
Just making them scarier and increasing the resistance sounds more than fine to me. No need to go over the top.