Clock Tower, Strange Journey, Silent Hill 2, Devil May Cry 3, the Deception games, Devil Survivor, and The World Ends With You are the ones almost everyone likes. Nippon Ichi's games are also good, and they are pretty wacky. I liked Shadow Of Memories, but that's just me.
If you are into japanese myth or occult beliefs check Kuon, but otherwise leave it. I personally love Theresia, though that's mostly a game about wandering a huge creepy complex, solving puzzles, and getting killed by traps that spring from everything you touch. Checked the wrong book? Trap! Moved the chair in the wrong direction? Trap! Opened the wrong door? Trap! Steped on the wrong place? Trap! Etc, etc.
If you are into artsy stuff, Yumme Nikki. If you are into weird shit and really creepy stories, and don't mind lots of misplaced porn to get at it, check some Nitro+ games like Saya No Uta and Chaos;Head. If you are into horror games add The Calling to this list. Hellnight is fun if you can get it, a game where you mostly run around labyrinths solving puzzles and trying not to get slaughtered by an immortal monster that's very obsessed about doing funny things to you.
Is the Legend of Cao Cao japanese? If so, play that one too. At least, try to. That game's hardcore.