Gandalf isn't a mage, as evidenced by his failure to ever cast fireballs and cloudkill and murder everything! He's almost certainly some kind of cleric/thief with a couple magic wands larping as a wizard.
Cept he did. Magic, at least wizard magic, is reactive and environmental in LOTR and not specifically requiring reagents. It's a point he makes when the Fellowship tries to cross the mountain pass before Moria. Gimli bitches asking why Gandalf isn't melting all the snow blocking their path and he replies that he needs something to do that with, and apparently something proportional to the act, as the firewood they brought wasn't enough for that but he was able to start a fire with it when no one else could get one going by just touching his staff to one of the logs.
Another from the Hobbit is when they're hiding from the huge army of wolves in trees he starts picking pine cones and firing them off aflame like fireworks to land on wolves and starts setting a bunch on fire, which then backfires when Orcs arrive and use the patches of fire he's started to build pyres around the trees everyone's hiding in to try to smoke them out.
There's also the physical exhaustion aspect he runs into trying to magically block a door in Moria and the check that the Balrog attempts to defeat his spell with almost killed him if it had succeeded and how he can't do many things when called upon because he says the spells require preparation in a Vancian manner.
Regardless of how it works, magic ala typical wizard use seems to be done in a way to prevent them simply standing around waving their hands and murdering all in their path and for a sword and physical action to be required, if only to fill in the times in between when magic is unavailable.
