Itself. Rogue in 3.X has a few terrible problems.
1: Sneak Attack stops working as soon as you start fighting undead, constructs, ooze and plants (plus incorporeal creatures, but most of them are undead anyways) without some major book diving, and the most common way only fixes one of the types and don't come cheap. The first 2, especially undead, are really common and losing almost all of your offense as soon as you hit one, with nothing to fall back on, is a very terrible experience.
2: "Lockpicker" or "Trapfinder", even with the full depth of 3.5, has very few alternate options. While there are countless "melee" "divine caster" and "arcane caster" alternates, most "skilled" base classes don't get open lock, disable device AND trapfinding (See: Bard), limiting you to, off the top of my head, Beguiler, Factotum, Scout and Ninja, the last two are effectively variant rogues, though the first two actually are pretty good. Unlike the other 3 roles, a Rogue isn't just expected, but actually required in most prewritten adventures (or even just a "classic" dungeon with traps and locked stuff), and when he isn't, he's pretty much dead weight.
(A wizard can fake it with Knock and summoning canon fodder/doing the opening with a cantrip to "blow" the traps, but what can't the wizard do?)
3: Like most non-casters, his power gain is linear, and unlike the fighter, he can't pull off things like trip that make being a beat stick worthwhile.
4: No capstone in core (a WotC web article gives them one, but it's only semi official): Nearly ANYTHING after 19 levels of rogue is better than rogue 20. Minor, but still bad design.