Perkel , how did you do it?
If there is no clear direction open map and look for place. Once you get there you usually don't need to use map at all.
Secondly exploration itself helps a lot.
Like in quest of Keira Metz where she mentioned tower that she wanted to check and i already knew what tower she was talking about due to me being there.
Orchard garnison was the same thing. Herbalist was easy to find as she had shit ton of crops near her house. Problematic was hunter but garnison dude said he lived near the woods. So open map and look for secluded house near the woods.
Character often mention where you need to go and only occasionally they don't say anything about place.
Most problematic is/was novigrad as exploration of every little house there is kind of time consuming proccess. But as i started doing quests in Novigrad i didn't need to use later map as i know perfectly well where i can find characters.
Like for example Dendelion girls quests where i had to visit them. 2 of them lived in mansions which means high class district to the north. 2 of them were rather poor so either slums disctrict or village outside of novigrade near walls.
Biggest problem for me was to find first time and second time triss house as Yennefer literally says she live in novigrad and that is it. At least in second case Triss says where she lives but slums area isn't exactly small but it was hard to find right door to her house.
I know from my memory where bathhouse lies, markets, whole docks area, temple disctrict etc. Where differnt inns are, Vivaldi, hatori etc.
As for quests involving forests etc this is where exploration comes in. Most of quests given in villages are self contained to village area. So if you start a quest you know it is something near it. So intead of fallowing some quest marker i simply explore near vicinity of village.
Though i didn't do many sidequest (something probably like 10%) i fallowed mainly mc with occasional sidequests where it fit narrative.
Oxenfurt is the city which gives me problems but mostly probably because i didn't do much questing there and mc doesn't actually have much to do with Oxenfurt. Still finding Witch Hunters HQ and butchering motherfukers was amazing.