Working on a design on this type of game myself. Probably never make it but the design process itself is its own reward.
1) How will you make the quest line interesting?
IMO, stories need at least one of two things to be interesting (and it should preferably have a bit of both):
a) Intrigue. Political plots, manipulation, cat and mouse games, etc.
b) Mystery. Reveal the bare bones so any idiot can access them. Add some meat for the player to discover on his own for those so inclined. This could be anything from hidden motivations of antagonists or seemingly friendly players to discovering things that may or may not be related to the plot that will make things easier for you in the long run. Something like New Vegas's main plotline, only without being spoonfed details about helios, the boomers or hidden robot armies. Leaving those things for the player to discover and use to his advantage in the final battle would have added a great deal of extra satisfaction to the plot.
2) How will the compass work? How will the map work?
I actually intend to make maps and stuff like quest compasses a part of the skill system. IIRC I have this stuff under the Exploration skill. A character with little to no skill in it would have a (very) basic map with just his location and major settlements he discovered marked on it. As he gains skill points and related abilities he'll be able to map out more and more details. Eventually, once he has the equivalent tracking and mapping skills of a grandmaster ranger he'll be able to predict the location of dungeons and ruins from huge distances, as well as figure out the type and strength of creatures within the dungeon (abstracted tracking, basically). Detailed text description of each location, including likely patrol patterns and types of traps will be included. Wisdom and Intelligence will also be able to pinpoint some locations, based on assumed prior reading in the case of the former and deductions based on obscure hints in the latter. The possibility of buying maps from scouts and rangers or finding/looting them will also be available, with more complex maps being considerably more expensive.
3) How will you have large number of NPCs that actually look/say different things.
I probably won't as the scope of my game is huge and it just isn't realistic to voice what could amount to npcs in the millions. I'll probably just divide them into (a large number) of archetypes and give them generic lines for that. If I ever make it I would really like to use an advanced text to speech system so that I'm not dependent on voice actors and can throw in voiced stuff on the fly, but that's well into the future.
4) How will you populate the world?
Through the level editor.
5) Will you simulate an economy?
I'd very much like to. In fact one of the five archetypes the classes are built around (the agent) has the skills necessary to become a powerful lord/merchant with his money grubbing tendrils all over the world. The opportunity for the player to not just influence but be heavily invested in the economy would be there. This would also mix well with some of the other skills (like using engineering funded by your enterprises to build a massive steamwork creature army, or alchemy + merchant acquired labs to equip your mercenary army with grenades and potions).
6) How will you design the RPG elements so that they don't get repetitive or Boring?
Have lots of them. There's 10 primary attributes, about 16 races, some of which have subraces (15 for human is about as high as they get IIRC), all apart from commoner humans having specific (and more important, useful and highly distinct) special abilities. The skill system was running at near 100 skills, though I've toned it down a little as I would like each skill to have a lot of things you can do with it. In other words, variety and lots of options. The spell system is also immense, with about 15 basic domains, each of which has 8 circles you can learn. For example, fire magic starts off with the Heat circle, which allows you to singe enemies, melt ice, keep yourself warm in deadly cold environments or piss off enemies. A mid-tier aspect is the Blaze circle, which allows you to create actual fire. Getting set on fire is a big deal because it won't go away on its own. For weaker spells/flames you can just stop, drop and roll if you've got the skill and mental fortitude not to panic. For stronger fires you'll have to jump into water and for the most powerful fires even jumping into water won't help, you'll have to use powerful extinguish spells/potions or die. The 7th circle, lava will require you to have some knowledge of earth magic as well before you can even learn it, but lava is one of the most powerful attacks in the game, taking off a % of the enemy's health in addition to the normal damage, blinding enemies permanently if it hits the face, and latching on to them (and slowing them down in the process). It can also be used more subtly (like melting a lock so the door is permanently jammed). Each spell domain also unlocks a certain type of spell that increases tactical options considerably. At higher air magic levels you can get the ability to create cloud spells, which linger in an area affecting everything in them. Earth spells can create walls, ranging from earth's own metal walls which are very hard to destroy (and possibly spined), fire walls that burn and set on fire anything that walks through them, Order magic walls that create impenetrable force fields and space/time magic walls that teleport enemies to a pre-determined anchor (if you don't bother with that it flings them into outer space
).
7) What about the enemies? Remember that soon you will become too powerful for the enemies with lots of combat available early. Should there be respawning? Should there be level scaling?
Enemies in the wild will mostly spawn randomly. I might have a rarity or extinction meter that affects spawn rates, but it wouldn't be a priority and I won't cry myself to sleep if I don't have it.
Dungeons would be slightly different in that enemies will eventually respawn, but the player will be able to find and destroy a certain object (like an altar) which will permanently end respawning in that dungeon and maybe drop a useful item. I don't like the idea of finite enemies in an open world game, but I also want to allow the player to set up bases in obscure dungeons if they choose to, so this is a nice compromise.
8) Will you like a Enemy vs. Enemy system?
There are a lot of factions from rival countries to smaller cults. Some of them hate each other and will start hacking if they meet. There will be no weird stuff like bandits and wild animals only attacking the player. Anyway, sorry, I got a little carried away there.