I can agree with NPCs giving the player some agency and personality. But I do think it's a trade-off, and in the context of an immersive sim not one I personally would be willing to make. That said, Prey probably handles it as well as it could imho; NPCs will talk at you and ask you things but not require any player input to respond other than doing or not doing the task itself. Having text trees where you ask questions about what they think of the current situation and about their backstory and they answer you like you're some kind of newspaper interviewer just feels a little grating.
I can be immersed in games with abstract UIs too, and in fact I've been trained to do so over the past 25 or so years I've been playing video games (it helps if the game is good and/or interesting), but that doesn't mean games can't do better. If Thief had no health shields and the light gem was an actual in-game gem attached to your wrist you'd have to look down at, wouldn't that be good? That's what
Gloomwood is doing, and while paradoxically it ended up taking me out of the experience at first due to the realization that the UI is diegetic and I'm playing a video game, in the long term it was beneficial.
As for Half-Life, maybe I'm being too harsh on it, but I honestly do think a lot of what you attribute to storytelling is just thanks to the design of the game being incredibly linear. You might not directly be told what to do, but if you're presented with a door leading to a corridor leading to another door and so forth and there's no other way to progress, you're going to be walking that path. I will say that I prefer Half-Life's storytelling to Half-Life 2's though, as I don't like having information dumped at me (which is why I like old school dungeon crawlers), and I feel that Prey handles it pretty well because what the characters tell you serves to flesh out their personality (which is very important for the plot's main conceit) while giving you a straightforward goal and isn't overlong, irrelevant exposition.
Yeah, health is an abstraction in the first place. It doesn't differentiate between a broken arm or a broken leg and is just a measure of "average damage person can take before dying". You can do away with precise health values/bars ala Resident Evil (which does technically have numbers, but hidden from the player) and combine it with Prey's trauma system and that'd be relatively abstract and realistic. Although probably quite frustrating for players too.
Inventory screens can be done with a container of some sort (like the suitcase in Gloomwood). I liked the way the Ultima games did it. Backpacks were actual bags that you could freely move stuff around in and would cover up your screen/vision when you opened them, it was a good abstraction. ZombiU did a neat thing where your character would crouch and look down at their backpack while the secondary WiiU screen turned into an inventory screen and then you would have to look down and interact with it, but that's a very specific example. Prey I didn't like because it moved it to a clearly separate menu and paused the game logic, where it could easily have kept the game going ala Dark Souls and interacted via the Transtar device, the thing serving as the menu-opening animation in the first place.
Also, generally speaking, sci-fi games have a big advantage when it comes to making diegetic UIs in that technology can be advanced enough where information is shown in a HUD via the character's helmet or implants or whatnot. In a fantasy game you could do it with a magic spell that tells you your life force or something, but good luck explaining why it'd constantly be up in your line of sight draining your mana. Alternatively, you could do a life gem akin to Thief's light gem. In an otherwise realistic first-person game? You'll have to rely on a Fitbit unless you want red screen tinting and blood jam.
But yeah, this kind of stuff might become more important as VR and virtual worlds take off (or whatever is after that, holograms, full dive?), until then there's still the separation between player and computer screen.