I'm working on a balancing mod and I address some of the points you raise. Lets see:
herostratus said:
* Mages are ridiculously OP
Agree. One of the things I noticed is that fights drag for too long and defense is too low (that's more hit points less defense). But they also drag for a long time due to the damage output of an average character, given that critical hits are nerfed to some extent a fighter or thief will have a hard time when compared to a mage who can deal lots of damage sometimes in a short period, not to mention that the damage is inflicted on a group of enemies. To remedy that I've already modified the abilities script's (that includes all talents and spells) so that they can be easily modified by INI entries (this will facilitate my tuning afterward). I've also increase the critical hit modifier for all weapons. This should give me the necessary tools to make mages less overpowered at least. I've also given all creatures less starting hit points (about %60 less of what they received), the same goes for stamina/mana and I've increased their starting defense rating. Hit points do not go up per level and neither does mana, so if you reach a mage or able to hit him with an arrow then you could probably end the fight in one or two hits. Of course this will force me to add spells that can be used strategically to avoid such problems (similar to Protection against missiles of the Dungeons & Dragons setting).
* Archery is UP for a player character, and OP for enemies once they get scattershot (although less so than mages)
This is true. I think I'll be able to modify this in the same way as noted, if not I'll have to touch it specifically.
* Warriors run out of interesting choices once all talents on the combat tree are taken.
Well I can see how that's possible for a level 25 character, but I've already capped the experience points so you can only reach level 20 at most.
* The stats aren't balanced at all, there is typically one that is the clearly superior choice for your character and which you will end up putting all your points in
I do not agree with this. They're not perfectly balanced, but any character needs good defense, stamina and health (at least) so there you have three distinct corresponding stats which govern them (Dexterity, Willpower and Constitution). Then strength, dexterity and cunning are vital for rogues specifically, for example. On the other hand we've Magic, which of course you'll say that only serves mages, but that's not correct, the more Magic a character has the more beneficial magic affects him and the more his spell resistance increases (though I'm not so sure about the last). Of course, given that you need to focus on some attributes to get access to the best equipment, spending points on other attributes might be counter-productive.
* Also, all enemies (with the exception of the ogre?) are variations of the same three types: Rogue, tank or mage. Thus, all encounters can be fought with the same tactic.
I think that the main reason that all enemies can be fought using the same tactics (if you want to call them that) is because there's not a single resistance that they possess that you cannot break with enough persistance, immunities to some things for some enemies could remedy this (but I'm not about to touch it any time soon). I also suspect that making the battles shorter (because they're lethal) will prevent you from being so persistent. There's also another factor which I can say for sure that influences the sameness on battle approach across the game: you're almost always fighting against greater numbers, so the first thing you're going to do is either to lure them into an ambush so you can pick them one by one (of course this is not always possible since the encounters are sometimes designed to keep you in one place) or using your mages to manage the groups with area of effect spells.
* Level scaling made the bandit mobs of Denerim a fiercer opponent than a dragon.
Again this is due to number of opponents, not so much quality. All the high dragons you fight are at least level 20 and those bandits might be level 20 if you enter their area for the first time being level 23-24. I've also made the level scaling a little less rampant, enemies inside an area wont scale further or lower than a two fold range (10-12, or 5-7, and so on).
* With the exception of coercion, combat tactics and traps/poison/herbalism for one of your party members, skills were useless.
Yes. I was thinking of using survival ala Fallout, giving you an opportunity to avoid a random encounter if you've enough ranks, however the way random encounters work is a little obtuse (from the scripting perspective at least), so I left it for later. Beyond that I think that coercion could also have its uses inside combat now that you mention it, such as being able to intimidate enemies and inflict a fear effect on them, thanks for the tip. As for tactics I really thought that the system was well crafted, but the options are still too few and the fact that you've to buy ranks on it to do something basic as giving directives is not kosher.
Of course it will take a lot more of what I'm doing to "fix it" if such a thing can be said. I'll really like it to be turn based, alas, that's not possible without messing with some engine nuances.
EDIT: Magic, not Spellpower.