Do not create drawbacks that are "opt-out" for the player if it still gives some benefit to the player. I.e. do not allow the player to take what is ostensibly a "drawback" that gives them a bonus to a skill pool, or some other sort of gameplay bonus, unless that drawback is very difficult/impossible to avoid. When people want to specialize a character in something, they already know what they want to do. What they don't want to do is pretty much everything but that activity. "You gain +4 to damage with broadswords but -20 to damage with wooden dowels and light maces," contains an effectively worthless drawback. The only way the drawback would ever arise in gameplay would be through some asinine heavy-handedness on the part of the game designer -- for which the player will almost assuredly resent you. A more even-handed drawback would be, "You gain +4 to damage with broadswords but attack 20% more slowly when using them." The benefit and the drawback are both realized within the same activity. The player cannot reap the benefit without suffering the penalty.
* When making trade-offs between items/skills/abilities, those trade offs must actually feel different in application or the player's choice isn't very important. For example, in the above case of +4 to damage with a 20% lower attack rate, there should be situations in which more damage per hit = better and situations in which faster attack rate = better. For example, if an armor system is threshold based (subtracts a flat damage value), doing more damage per shot always means that damage has a greater chance of getting through armor. In this case, the +4 bonus is better when used against opponents with armor. Against opponents with high health and no armor, raw DPS matters more than damage per shot. In such cases, having a 20% faster attack rate may be better if it outweighs the DPS value of the +4 in the overall equation.