They did it for the simplest of reasons: To fit the third dimension into games.
The problem with that, is that they had to regress games considerably to pull that off, and then retread their steps to try to get back where they were. Unfortunately a certain depth of gameplay almost got permanently lost in the process, and only with the rise of the indie game scene around 2009 was it spared utter destruction.
What? Early 3D games had plenty of depth, its just that you needed a super computer to run it or it'd be absolute hell to play. Strike Commander is a notable example, but you can find examples all the way back to 1981 if you look hard enough. Part of the reason for Doom and Wolfenstein's success is that they actually ran on most machines without getting a frame rate in seconds per frame.
The game in the OP is in 3D...
Ah, I see that my post is confusing, since I failed to give it the proper attention (was juggling my attention across multiple places). I apologize for that. Allow me to elaborate.
The first issue is that I missed out on adding a second parameter when I said "fit the third dimension into games" to make things clearer, and that parameter is "that acted in real-time".
The second issue is that some posters seem to be confusing 'games with three dimensions' with 'games with first-person perspective'.
A 3D 'perspective' has been around in games since the early 1980s, but it wasn't until 1987 that games came out that had 'true' 3D environments, and even then they utilized basic geometric shapes, and they still couldn't nail down the 'real-time' part. It was more common to see games who convincingly faked it, than ones that actually tried. And yes, there were some very complex games released that dared to use these 'first-gen' 3D environments. The Freescape-games, the Mercenary-games, the Midwinter-games, the Elite-games, and many more. But all of them were faking it in part, even though some of them contributed technological progression towards the goal.
When Wolfenstein 3D came along it gained notice because it offered a fast-running first-person perspective game, with modern-day graphics. But it was a 2D-game, the z-plane was a single, constant value. World interaction was also minimal, though it did break free of being grid-based in movement, even though the gameworld itself is built on a grid. Ultima Underworld made a closer step, but both it (and the Doom-games released shortly after) were 2.5D-games as the z-plane was no longer a constant value, but it was still a single value, meaning you couldn't have one area on top of another, giving the games a 'flat' feeling.
It wasn't until games like Descent and Quake came about that the third dimension was truly and finally crammed in, and that in real time as well. (Compare that to the contemporary Build-engine games, who did a very convincing fakeout of full 3D.) But consider the 'cost', to pardon the term; those two games (and many of the ones that followed them) were simple games with simple gameplay, move about and shoot things. Neither game sports a 'Use'-function for example, instead you either push against or shoot anything that required "using". (Thief: the Dark Project deserves praise because it contains surprising depth for a full 3D game in these early days.)
But then we get to the third and final issue of my post, and that is me failing to define who "They" are, the ones who made the push to add in the third dimension proper. Are "They" the developers? Only in part. They were certainly pushing for it, but they were going about it at their own pace. But once Wing Commander 3 and Doom were released in 1993, the Big Money-people, the corporate suits, began to take notice; because those two games were absolute proof to them that you could put (big) money into video games, and they would make bigger money back. And it was all thanks to this tired cliché known as 'immersion'. Because with a first-person perspective the player
feels and
thinks like he's actually in the game, a notion that had been proven consistently since the mid-1980s, but only became 'financially viable' in the early 90s. And immersion
sells.
So the corporate suits moved into gaming, and they began to bark orders. Old licenses were exhumed and made into '3D' titles, deadlines became extra-strict, so games were cut down in depth and gameplay, rather than in looks and immersion. The technology advanced, but the games stagnated. And one way the suits tried to cover up this simplification process was to make the whole thing feel edgy. One moment we were (happily) mowing down Nazis in a simple first-person shooter game, the next moment we have this game where a guy is looking to make us their bitch. And despite massive leaps and bounds in the technology between those two games, they play eeriely similar... and it's obvious which one of them is more fun to play.
This overemphasis on pushing the "3D" part forward, at the cost of all else, just so that the suits could make more money, slowed down any real innovation in gaming to a crawl. One could argue that the late 90s were the last gasps of an era where innovative gaming was the leading light of a game's development, followed by a decade where even that light seemed to have been snuffed out. Only around 2009, where small studios releasing small games proved to be commercially viable (again), did this dark era come to an end... and even then most of the people daring to low-ball their way through a game's development were just trying to recreate a past game held dear to them. But a few devs were thinking bigger. Ironically, one of the key steps in the rise of the indie scene was a developer realizing, that in order for his vision of a game to come true, he'd have to scale back on the 3D engine he was using. But once Markus Persson did, he could make Minecraft a reality. Terraria pulled a similar thing by scaling itself back until it looks like a side-scrolling title on the Super Nintendo, but once you realize it's simultaneously micro-managing a few million blocks of terrain, while being a full-fledged platformer that doesn't break a sweat, then you realize that actual depth is starting to come back to gaming. And Terraria is ten years old now.
But yeah, gaming took a hit because of the third dimension.