I think the problem is not many writers do it well. Basically none. That is why people think light and darkness is tired; poor execution rather than inherent merit. Only Tolkien did it right, that's about it. Babylon 5 in terms of TV sci-fi series too. Star Wars to some degree. All spiritual. For most RPGs, their attempts at "epic", boil down to some stupid Manichean war between heaven and hell. Literal angels fighting literal demons, with horns and halos, the most prosaic idea. You could blame D&D and Diablo. Note how far this is from Tolkien. Sauron is an infernal spirit, who's twisted intellect can therefore be taken to angelic extremes of beyond human capacities.
Damn, this is the good stuff. The Tolkenian perspective here is interesting. So Sauron is an infernal spirit playing by the rules of the material universe, using arcane power alongside its knowledge of sciences beyond us? Is that why both Sauron and Saruman have industrial themes?
I've been struggling with this issue in my thoughts lately.
You see, I was raised and am Adventist. So my take on those things is a bit... different.
I mean, the idea of fighting demons and such is awesome. But also... banal? Feels to me that it reduces such incredible entities to mundane levels. I don't even like the idea of straight out demons and angels appearing in my stuff, not clearly, I mean. I prefer that such elements are implied/discrete, and rare and sublime. I'm really not sure how to show these things when it's time for ultraviolence.
For example, I'm biblically read enough to know that the "horns and hooves" image of demons is just something the medievals lifted from Pan and Satyrs. Actual biblical angels are borderline cosmic horror beings in appearance, so demons are likely similar. I prefer demons as being tempters, possessors, otherworldly beings who have "height, width, depth... and a couple of other things", and if you ever fight one, you're fighting it with a lot of limitations.
(personally I have an idea that demons
can't kill, because it is God who determines if someone lives or dies, but they can totally hurt people non-fatally and tempt them just fine. The people they tempt, through, have no such limitations)
So in my ideas, my "metaverse", my take on the "horns and hooves" demons of flesh, is that they are just humans who were genetically altered by the devil using super science (but I'm not going to say its straight out the Devil, only imply), and the traditional "fire and brimstone dimension" hell is actually just an alternate earth/dimension they were placed in.
They're not congenitally evil either (not for lack of trying), because anything non-supernatural at their level of intelligence can't be "made evil" from birth. Their society sucks, but then again, they literally worship The Devil as god.
(I just realized that this is pretty much what Sauron did to the Elves to create Orcs)
I have similar ideas with magic. I don't like the idea of "good magic", because I have been taught that "Magic" is just supernatural power from the Devil. So to me, Magic is Evil. Good people use technology and the odd miracle, but you can't "cast up" a miracle.
Been trying to square up pagan-style gods with a Christian/Adventist cosmology too. Not sure if I got the Jack Kirby way (super-advanced living ideas and concepts which are "above" humans but below sinless beings like angels and can be good or evil) or the Gandalf way (actually divine beings but using a guise to fool people, like how Gandalf is a "Wizard").
Oh man, it sounds like we have had very similar thoughts; I am going to respond to you in depth, hopefully some of this will be interesting.
I grew up with a world-view that was simultaneously quite spiritual, while also being metaphysically atheist, and felt pulled toward both science and spirituality. I wanted to understand why both aspects of human experience inspired me; scientific humanism and spiritual discipline. I wanted to reconcile the two. Why did The Lord of the Rings, or Star Wars, really touch something in my soul? I felt the incredible profundity just pouring from Tolkien's works. What, from a purely Darwinian or materialistic conception of the human race, makes us weep when we see faultless martyrs die for humanity and great saintly kings sweep away evil? For example, a lot of people wept when Luke Skywalker turned up in the Mandalorian to 'take away' a certain character (assume bodily into heaven); looking at what they did, it was obvious he was a messianic metaphor, the episode even appearing mere days before Christmas. As well as bridging atheism and spirituality, I also wanted to find a way of reconciling East and West; seemingly relativistic religions like Taoism teaching that both darkness and light are necessary with Christian view that we are in a war against entropic forces of darkness beyond human ken. Both traditions I could feel bore 'truth' and 'beauty' inside them.
One of the first perspectives I encountered on how they could be reconciled was Carl Jung's ideas. The man opined that humans contained within them a series of entrenched instincts that allowed them to not merely exist, but flourish, and be animated with new life. He framed it in terms of psychology. That a prophet or rishi was accessing the vastly more powerful subconscious that took in huge amounts more data and wisdom than the conscious portion of the mind; that these surreal images bore more truth than limited rational dictum. For materialists these were evolutionary in origin, but his work could also be read purely religiously just as easily. Why are there higher archetypes, patterns and orders of morality entrenched in the universe? To paraphrase Jordan Peterson, who has also tried to reconcile the scientific spirit with religious, when we feel momentary enlightenment, several orders of priorities from the personal to the species level are coming into alignment. To Jung, most myths are true, because they all reflect this surreal wisdom that is expressed in images. King Arthur is thus so inspiring because it reflects some deeper truth; exactly what a Christian knows.
A scientist believes there is one reality, which can be independently tested by completely discreet individuals in different parts of the material cosmos. A person performing a scientific experiment in Iowa, or on Mars, will get the same outcome. So clearly there is such a thing as objective truth; you arrive at the same conclusion independent of communication. The idea that the world is subjective is just as anathema to a scientist as to a Christian. This is where I finally began to see how the two can be completely reconciled. To me, religion begins with the idea that there is only one objective reality, the logos, which is a concept present in every successful culture; you can't perform science without it, and you can't agree on anything as a culture without it; law, government, society all require objectivity. Solipsism is the death of civilization. This concept is also imbued in every culture of note. The Greeks obviously termed it logos. The Egyptians termed it ma'at. The Persians asha. The Hindus rta. "In the beginning was the logos, and the logos was with God, and the logos was God." Next comes the natural laws that invariably must spring from a base reality, as consequences of limitation imposed by existence. 'Natural Law' in Christianity. 'Dharma' in Indian languages. 'Tao' in Chinese languages. When the recognition of these eternal sacred laws fall into degeneracy, an age of catastrophe results, which is the decline into Ragnarok or the Kali Yuga.
The best part is, you can test the natural law. Eastern mendicants sometimes tell their adherents to contemplate what they really 'feel' is right, after removing fear, pride, hubris, lust, envy, greed; they are left with a self-knowledge of what is 'actually' important to them. What is actually compassionate, when you are no longer afraid to offend? Family, nation, honor, beauty, truth. Take a dogmatic social activist who believes in something completely contrary to nature. Well, they can test their choice; if they have the self-awareness to see what flourishes the sum of life, and what brings civilization closer to heat death. Lies, calumny, corruption and fear, drive us toward entropy. Truth fortifies civilization. If they believe that a house fire can't hurt a human being; they can stand in a house fire, and reality will quickly burn them. Reality enforces the natural law. Testing various spiritual disciplines for myself, even though Buddhism can seem more rational, or Hinduism more universal, I find that Christianity is for some reason, the most emotionally fulfilling tradition. It somehow gets more right about the human race psychologically than the others. However they are themselves not without shining moments of superiority, sometimes more in tune with rta, and I have great respect for Indian and Chinese traditions. So there must be a way to reconcile pagan religions and high ones like Christianity. Some higher reality must unite them if there is one objective reality and they bear fragments of the truth.
I can't really give you a definitive idea of how pagan myths and Christianity align, since I am also working out these things still. I can only present how I think about some of these myths. For me, unironically, I use aspects of science fiction or fantasy in my spirituality; when they are masterly, speak of darkness and light, they inspire me. That is because, first of all, no matter what anyone else thinks, I 'know', that some aspect of them speak to something deeper in us; Tolkien uncovers something long buried. I have felt it. It may be that mythic adventure, as a genre, speaks to more of the total of human nature than any other more mundane genre. It might be, as I think Rene Guenon might have said, that the Western people of the world have truly antediluvian traditions ingrained within our cultures, that we barely understand, and that modern people have lost the meanings of them, where the medieval understood. Few have the skill to uncover it, and Tolkien was one. Plato believed, or perhaps just provoked, that there was a world of 'forms', in which pure concepts like 'the perfect table', 'the perfect chair' exist in a theoretical state, and we merely work toward these perfect ideals. Did he mean it literally, or figuratively? Tolkien's Elves, with their beauty that suggests an alignment with the eternal beyond their mere physical existence, are meant to be like expressions of a higher world of sorts. For a Buddhist they would be close to Buddhahood, and for a Christian close to sainthood. In Christianity, there are spirits. Angels which you rightly say are not the classical anthropomorphic paintings of St Michael we see in galleries, but disembodied cosmic entities. In a sense, to me a spirit exists in a metaphysical world of 'similitude', which is not unlike Plato's world of forms. If we are luminous beings, then the greater part of us also exists here. Is this atheism, or religion? For me, it is no longer easy to say, and I can't draw such easy distinctions anymore; they ultimately meet.
If you read Norse or Greek myth, the gods were not pleasant. Clearly they can't be reconciled with Christianity if they don't act in a way that would be congruent with what we know about the Lord. So I choose to think that the illiterate pagan cultures of Scandanavia or the Aegean Sea were remembered poorly, and indeed we know that oral folk traditions changed and evolved radically over time. It's akin to how in modern Hinduism, you have extremely high-minded philosophers, but also completely mis-remembered folk traditions that degenerated into mere celebrations of temporal things over time. You could as a Christian see pagan entities as some of the spirits that came to Earth after creation (Tolkien seems to have taken this approach in his world building with the Ainur), as nephilim or remembrance of long dead mortal heroes (as the Irish, English and Scandinavians later saw their mythology after Christianisation). What I think is important however is that we Westerners don't abandon our traditions; they have a place in Christian culture, and the desire to puritanically purge can sever truly ancient lines of initiation into wisdom. My favorite part of Christianity is the medieval spirit of warfare against darkness; it speaks to the Aryan man from Iceland to India.
I absolutely hear you about how banal most 'spiritual warfare' is in D&D inspired games. Hooves and horns. Tolkien was of a different league, and out-classes everyone. He understood how banal/childish that would be, before they had even conceived it. All of us feel how banal that depiction of a demon is, yet nobody in the industry moves beyond it; just more pitiful simulacra. It's the product of very limited metaphysical learning, though fantasy RPG cosmology is not entirely without some minor insights. Sauron was different. He was exactly what you are talking about; something otherworldly; merely a shadow of an inferno. There was also a really interesting discussion on Tolkien's concept of magic, by a Catholic on YouTube, who pointed out that unlike the bland Harry Potter elemental magic in most fantasy, the rare times magic is used in Tolkien are subdued, and Gandalf constantly warns people not to use powers they cannot explain or account for, like the Palantir.