Some of what people think is great AI in games of the past is actually not AI at all. The marines in Half-Life were scripted if I recall correctly, and so were NPCs in Gothic games.
One big problem with real AI is that it doesn't really fit that well with static games which dominate the industry now. Do you really want your NPCs to think independently in the middle of a fixed storyline? The game might be about saving a princess, but the princess AI might decide to get a sex change, unite the peasantry WITH the ancient evil and march them all to attack your avatar because of some sexually insensitive jokes you made in a tavern earlier on. So real AI would belong better in emergent, procedural games, of which there aren't that many to begin with. With non-emergent games, you kind of want the AI to think a bit, but not too much, and only in certain directions, which I am sure would kill any enthusiasm for it.
And then of course, as already mentioned, AI is a very complex thing, and much like any other technology (like graphics, physics), it needs significat investment. The only companies that have the funds (big mainstream ones) would need to see a real incentive to do so, and right now, it's just not there. If one of them managed to do it first, they could make a killing, but these guys aren't the innovative types clearly, as seen from the derivative shit they've been making for over a decade now.