Grunker , I think we're coming from different places when we talk about difficulty. Can't comment on the megabosses, because I haven't fought any one of them yet.
You said PotD is too easy for most of the game. I am again repeating my point - it may have been too easy for you, but I had set up my play in such a way that it was relatively hard most of the game. Maybe I was going to areas too early, maybe my party composition or tactics were worse than yours, never mind the reason, but my perception is that PotD was difficult.
Now we come to another measurement - was it the "fun kind of difficult" or the tedious grindy kind. I felt it was more the latter than the former. And I believe it felt this way for me because of the too many enemies which don't require new tactical approaches from me in order to defeat them.