Yes, and this makes it so there's no strategic reason to expand. Why pick fights on the campaign map against an AI that can hire 10 armies while you can only hire 2 (which means that with your lord in the chaos realm you are hopelessly unable to defend a large empire), when instead you can just rush through the chaos realm to pick up the victory tokens?
There is a strategic reason to expand, it is called income. The game has simple economic function, you earn money by doing battles and especially attacking settlements, which you then use to build income buildings to increase your income. You will have more money thus more armies if you control more settlements since you'll earn more money by taking them and use that money to build quantitively more income buildings.
If you expand too much, you'll have more enemies and more internal issues to deal with however the chaos one in particular can be dealt with by using heroes to close rifts. Hell this is actually a good thing. Very few strategy games even properly deal with outcomes of overextension, in games like EU4 you can endlessly expand and only ever get linearly more powerful. Larger realm requiring more intensive management and there being a breaking point is something only very few games represent. Another game like this was Attila. In every other Total War (And Paradox) game expansion is always just snowballing with no negatives whatsoever.