However, i's still just "click on this/execute command".
And what exactly would be gained by having you move your fingers accurately in a well-simulated animation-simulation through oculus rift rather than looking at a big screen and clicking the left mouse button?
As Lyric Suite said, simple motions like picking things up are done with little effort in real life. As little effort as clicking a button in a game takes. So having to do a complex movement for it is jarring, and it becomes a tedious gimmick after a while where you wish that the game had simple button-clicking rather than taking so much effort to just pick up a coin.
Okay, let's assume there is a good and immersive first person game, using a normal LCD screen and mouse+kb controls. It is immersive because of its atmosphere, its story, its interactivity. You can interact with almost anything (by clicking mouse buttons and pressing keyboard keys) and every interaction has a visible impact on the game world that is even recognized by NPCs. This is why the game is so fucking immersive. It would likely be your dream game, too. Oh, and it took many years and a huge budget to develop, which brings us to:
Adding those SUPER REAL animations and actions and simulated movement, replacing every simple button click with intricate virtual finger movements, would raise the budget even higher and some of the features that make the game immersive in the first place have to be scrapped. If you need an animation for every single interaction, money has to be spent on creating these animations. Think Thief 1 and 2 with their simple "right click thingie to interact with it, interaction happens" compared to Thi4f's "every single thing is animated, from taking loot to opening a safe". As a consequence, every interaction the game has is animated and, by your definition, "more immersive", but the game actually ends up having less possible interactions because they ran out of animation budget, consequently
making the game less immersive due to less interaction being possible.
Actual
content is what makes a game immersive, not the presentation as you falsely believe.