I read complaints like this so often, it makes me wonder why people keep making small-scale tactics games that don't give a good reason for staying small-scale. (And full disclosure: I want you all to buy my game.)
If you're going to limit the number of units you can take to battle, you need to have a good reason to keep your B-squad around. That's what Battletech fails at. It would make sense if it took time for a mech to get repaired, so you would bring a weaker mech to fill out the lance until the better mech is repaired. But because engineering fixes only one mech at a time, you start cluttering your repair queue with your crappy mechs. And eventually you learn to use only your good mechs, and wait for repairs. There's not even a good trade-off for using lighter mechs.
I was annoyed at Battletech for not letting me deploy multiple lances. They could at least have let me buy another shuttle and send lances on multi-day missions, and we could still stick to the 4-mech lance battles.
Small battles and useless reserve units. It's the same reason I wished for a bigger Final Fantasy Tactics and Tactics Ogre. In Tactics Ogre, I would tame another dragon and feel great, and then I would remember I would never bring the dragons into battle because they were weaker than my mains. Twenty years later, HBS is still making the same weak design choices that make your B-squad useless. In Battletech, I build a new mech from salvage, and unless it's heavier than what I'm already using, I immediately sell it. There's really no use in keeping it around. In FFT, you could at least send your crappy guys on tavern missions and they'd get a few gil and XP for it.
It isn't a dumbing trend intended to cater to anyone. It's a dumbing trend in programmers, who are unable to optimize systems such that they don't choke and die with more units in play. You see this across all games, or even the SAME game, where the number of units has steadily shrunk in games, or even the SAME game, as the programmers pile more badly written crap onto an engine, and as a result, the number of units available in an instance of map falls.
I think most programmers would gladly optimize the game, given the chance, to allow for bigger battles. If it is indeed a technical hurdle, and not a design decision, then I would have to believe that anyone who can program would be able to put the player in charge of more than four units. I'm a lone developer with no budget, and I could do it. Surely HBS could too. I'm suspecting the 4-mech limit is a design decision to keep the battles difficult, to make you feel like a hero for defeating a larger enemy force, and to keep the battles short.