It is moot to talk about drawing a sword since, as a rule, a sword would always be unsheathed when entering battle (this also applies to Japanese samurai, as opposed to the spergy movie depictions on the matter). Furthermore, the swords were typically carried in hand (sheathed in their scabbard), and very rarely worn.
Not sure where you are getting this info. I've seen a ton of depictions of europeans and japanese samurai with swords (in movies, paintings, statuettes, etc), and in pretty much all of them, the swords were "worn", not carried in hand. In fact, the point you make later on in one of your posts , that swords were primarily a secondary weapon in battle (which i agree with) supports them being "worn", since obviously a side-arm would be kinda useless if it had to be carried in your hands.
As far as weapons being unsheathed when entering battle, yeah that's probably true in most cases, but we are playing RPG/adventure games, not battle simulators. Just moving around the world as an adventurer/mercenary, you wouldn't have advance warning the same way a soldier entering battle would, so it would be very important for you to be able to draw out your weapon at a moment's notice.
In general, I am not obsessive about realism or anything, there are times when it can be compromised, but at the same time, I don't like the whole argument that since games have some unrealistic elements (fantasy, etc), we should just throw all realism out and let developers come up with whatever crap feels the coolest to them at the time. My take on it is that if realism clearly leads to bad gameplay, can it, but otherwise, try to stick with it. The reason being, based on my experiences with games, those that take a more realistic approach usually tend to have more fun, deeper gameplay (as long as they dont take it overboard as mentioned earlier). Think of it like this, realistic combat systems are already deep inherently, because of how they came about and for what aims, so if developers model them somewhat (removing the parts harming gameplay), good, challenging and deep gameplay is likely to result. On the other hand, if developers just pull some unrealistic combat system out of their ass, because they believe it's fun, sure, sometimes that might work, but generally speaking, that's how we end up with so many games with crappy combat.