Many reasons, most of them mundane and in the immediate sense financially motivated, but overall I think it is an expected and natural consequence in the industry reflecting the evolution of human consciousness in microcosm; I'm referring to the qualitative mutations undergone in perspective and of our increasing awareness of space and time. To explain this well would be a difficult task on a forum post but I'd recommend reading Jean Gebser's The Ever-Present Origin, wherein he covers this phenomenon from both art history and metaphysical positions. A very basic outline of this work could be described as such: The evolution of human consciousness has developed so far across five psychosocial structures; approximately they are the archaic, magic, mythic, mental and (not finally, but ultimately, as far as we're presently concerned) the integral, and these structures arose and transformed concurrent to our developing conscious acuity in relation to time and space which occurred in three phases he called the unperspectival, the perspectival and on to the aperspectival.
The growing realization of spatial dimensions and of time can be traced across art history, observable in the two dimensional and disproportionate profile perspective of Egyptian, or Mayan, or Persian wall paintings and reliefs and for instance ancient Grecian pottery, to the development of portraiture and the Renaissance era shift to realized landscape foregrounds and backgrounds with intricate details depicted to scale, to the abstract art of the early modern period where unto visible and invisible space is added the presentation of time (most markedly as Gebser points out in a work of Picasso's which somehow portrays a woman from all three dimensions simultaneously). Each conscious mutation, he contends is not superior nor inferior to its former structures and includes all those preceding it while adapting. To my thinking it appears that the development of video games embody these same successive developmental structures, from the early instances of flat, 2D perspectives where time only moves vertically or horizontally forward or the pseudo first-person 3D wireframe games of the mid-seventies, onto extensions out into space by first and third person perspectives, the advent of real-time multiplayer interactivity, onto enhanced "3D" and now virtual reality. The greater impulse moving through the art world is towards the next structure, inclusive of all that came before but primarily oriented towards realizing the mutation. Virtual reality can even be thought of as undergoing its own very similar but more advanced developmental process, where in its early phases no other perspective can generally be conceived of than the first person or over-the-shoulder third person, unless a Picasso of the medium appears and completely alters our visionary possibilities. Like the several millennia of observable art history, we've had breakthroughs of perspective in video games only a handful of times, such as with Ultima Underworld. The few genuinely innovative artists alive and working in the industry today have been trying to achieve the next breakthrough, for instance Kojima and his "strand" games concept (one of the worst instances of self-celebrating I've ever seen, and I genuinely like some of his games), but as far as I'm aware nothing of the sort has happened yet. As we move closer to it, we will have more and more iterations of the same game (narrative-focused open-world amusement park styled sandboxes) as the majority become less inspired, less capable of innovation, less creative etc. The so-called CRPG "renaissance" is among the more grievous atrocities perpetuated under these abhorrent circumstances, thinking a simple reverting in perspective is a sufficient alternative and completely overlooking what made the classics what they are. First-person CRPGs are among the peak of the entirety of video games for me, but it was so much more than the perspective that made them so remarkable, such as the unique aesthetics where today everything has the same generic hyper-realistic style, where even the hand painted inventory icons and other features of the user interfaces look like parts of colored photographs.
