Can't really recall any random spawns beside ones in Oathbinder's Sanctum, where else have you encountered them?
Anyway, am currently playing with earlier mentioned XP nerf mod and economy overhaul - lvl 14 and difficulty is already falling apart. Even without metagaming cheese like rushing for Grave Calling etc, it's really hard not to completely break the game around lvl 10-12, which is a bummer with all those potions/explosives/drugs/traps/etc. Guess the only way to keep any challenge is fully cranked up DD or rolling with some wacky party build.
I'm level 9 on PoTD and I'm using the same mods (XP and economy overhaul). The only difficulti thing, as for the moment, are some ship battles (because I suck at strategy) and the big worm in the Temple in the Old City (but I tried it the first (and only) time two levels ago.
Generally speaking, yes, if it wasn't for me being utterly unable in strategizing or using potions/bombs/traps/drugs, the game is quite easy