While I agree that early game is the the best part, I don't understand why lots of people are saying that Act 2 and reaching Defiance Bay is beginning of the downfall.
The city itself, along with the quests there, is designed pretty well, plus the newly opened wilderness areas are also ok.
The downfall starts in Act 3, where it's obvious and in your face that devs were rushing to finish the game - best example are the quests in Twin Elms, where in one you kill dwarves in red dungeon, and in other you kill elves in blue dungeon - epitome of laziness.
Act IV is basically nonexistent and devs clearly didn't give a shit about anything at this stage, which lead to anti-climatic ending.
WM1&2 are top notch, but everyone knows this already.
Also, I don't get the complaints about extensive lore-dumps. Everyone that's shitting on PoE is acting like in each conversation you are bombarded with blocks of text about NPCs family history and the state of local agriculture. I don't get this feeling at all - dialogues are rather short, and the only lore dumps are simple and crucial ones - world's geography, some metahpysics (the Wheel, souls etc), major events (i.e Saint's War), gods, factions, etc. No one is forcing you to interact with golden plate NPCs, which seem to trigger people the most. On my first run, after checking few of them I figured out that it's meaningless and ignored them completely for the rest of the game.