Good PvPers aren't phased by the amount of enemies there are. I've seen invaders slice through people in such a way that it almost looked like it was a rehearsed dance. I was recently in a 2v2v1 scenario where the lone Darkwraith came out as the winner. It was beautiful. Having a large army of phantoms at your side most of the time isn't going to be beneficial to you simply because they aren't usually nearly as good a player as the invaders tend to be. It's far better than being alone, sure, but if they are incompetent at PvP then they are just going to get backstabbed to death pretty fast. The invader can dance around the phantoms and then just kill the host if he becomes cocky and lets out a perfect opening. It's another good tactic, especially against clueless people.
Invaders often have the edge on stats and equipment as well. Usual people build their characters however they want and upgrade their gear as they go, but an invader is built purely with the sole purpose of ending your life. They take any ace they can to have even a minor advantage over the other player.
Of course most of this comes crumbling down when lag hits you and you get backstabbed from the other side of Darkroot. Then it's just gay.
Wow. Look, I see where you're coming from, but saying - or insinuating, rather - that "PvP is unbalanced in the hosts' disfavour because invaders are better players!" is the reason why tears are delicious. :D
It might seem that Darkwraiths have a categorical advantage in PvP, but honestly, it's only true in the early game. The reason for this gross imbalance is that you can only get a Red Eye Orb so late in the game that anyone who has one inevitably has access to vastly superior equipment than players in the early game, which makes Darkwraiths practically impossible to beat if they're using their best gear, no matter how many they're up against. I don't really like this, because it gives people the wrong idea about why invaders are scary. Honestly, I wish that Dark Souls II has some mechanic or other to scale invaders down when invading in the early areas, simply because the advantage in experience and skill should make Darkwraiths terrifying enough.
Be that as it may, have to say it's disingenuous to say that White Phantoms are somehow an insignificant advantage. They're only useless at the beginning, when hosts and allies of similar level have no weapons that can do adequate damage against invaders, but that playing field begins to level quickly as you progress and get better equipment. By Blighttown or Anor Londo at the latest, a host with two White Phantoms can give a decent fight to any typical Darkwraith just by ganging up on them. It's worth noting that matchmaking for Darkwraiths sets them up against foes that are on a higher level than they are, which is why people who invade normal players tend to stay on as low a level as possible to broaden the pool of people they can invade. Because of this, they don't get a major advantage from having a good build, since they simply don't have enough levels for it to make a huge difference - good builds are mainly used for dueling, which takes place at much higher levels. Meanwhile, hosts at all levels can use Estus to heal White Phantoms; if the invaders can't kill with one or two attacks, the host can simply heal his allies faster than the Darkwraith can do damage and win by attrition. For most of the game, both sides can have a fair shot.
So, it goes without saying that Darkwraiths are a terror for lone players unused to PvP, but aside of the ridiculous mechanical difference in the early game, that's exactly the way it's meant to be. But you should know that in the Dark Souls food chain, Darkwraiths are far below Darkmoon Blades and dedicated ganker teams, who can actually get mileage out of having high-level characters built for PvP. That's not to say that a decent invader couldn't defeat even a prepared ganker host with allies every now and then, with some combination of skill, luck and cheese. I myself (if I may say so) have beaten my share of ganker trios in full Havel's and danced the Channeler Dance upon their wretched corpses.
My point, though, is that for an invader that's a rare treat you get only rarely, after much toil and failure, but that's Dark Souls for you. No matter your playstyle (except ganking, I guess) you will eventually face odds grossly stacked against you, and often you lose. It's worth it, because wringing a victory out of a situation like that, from the greasy fingers of ganking, well-what-is-it-spamming scumbags, is one of the great pleasures of life.