My two cents:
1. It took less time to make experiments, which means they were cheaper. Cheaper experiments are easier to pitch.
2. The market had a shitload of small and mid-sized publishers. They couldn't compete by simply doing what everyone else was doing, they had to stand out somehow.
3. Game development was not mainstream and required knowledge that wasn't easy to access, which is why I think we got the eccentric geniuses you mention.
4. Cutting edge games were relatively cheaper and developed faster than they are today. This allowed devs to make "AAA" games that still managed to experiment and break new ground. The other advantage is that other studios can pick up on trends much faster.
The post-Wolfenstein/Doom period really caught my eye when I read about it. The rate at which devs made progress in the genre was astounding when compared to modern games.
Doom came out on December 10, 1993. By December 1994 we had Descent, which added six degrees of freedom, (almost) full 3D graphics and advanced lighting. Marathon came out in the same month, with a bigger focus on storytelling and adding freelook, dual-wielding and alt-fire modes. In February 1995 there was Dark Forces, which featured cutscenes, crouching, platforming, etc.
June 1996 Quake comes out and kills the sprite-based FPS. Less than 4 years after Wolfenstein and Ultima Underworld and the genre was completely different already.
I think WoW is a good example on the opposite end. Huge hit in 2004, many major publishers decide they want a piece of that pie. The most ambitious WoW clone was TOR, which took 5 years to make and was released 7 years after WoW. Not to mention it was the most expensive game ever at that point. TESO, the other major competitor, was started in 2007 and released in 2014, a full decade late. By the time it came out the subscription-based business model was already dying and WoW was in decline.
The 90s had the best of both worlds, because the more creative and experimental projects were still on par with the big hits, so they were more accessible. It had some things that won't happen again anytime soon.
To use MCA as an example: which modern publisher would give 2 years, free rein and 50 people to a young developer so he could lead an ambitious narrative-driven PC-exclusive RPG based on a cult D&D setting? That's without considering the right proportions. A game that could be to 2015 what PS:T was to 1999 would need at least twice the money and the team size. If it sold the same ~400k, the young director would probably get fired or demoted.
A game doesn't have to be pretty and have AAA production values to be just as good today, I know. I'm also aware that it's easier than ever to make or modify a game nowadays. However, games back then had both things going for them, a luxury 2016 indie games don't have.
On one hand, the small and mid-sized publisher graveyard doesn't help. On the other hand, the industry has changed in positive ways, too. We have crowdfunding and digital distribution, so who knows...
