There are two haunted houses in the Temple Quarter of Vizima in The Witcher: the haunted house associated with the quest for some very old wine for Conrad, which is also identified as haunted by Coleman; and the haunted house that a thug from the slums entices Geralt to investigate.
The first house is on Harbor Street, the street which leads to the Gate to the Dike. It is west of detective Raymond's house, not far from the entrance to the sewers. On Geralt's first visit, he is met by two non-too-friendly wraiths. Later, whether he ventures downstairs immediately or comes back for a look at Conrad's request, he is set upon by graveirs.
The second house is in the slums area, west of Ramsmeat's place, just north of the Eager Thighs brothel. There, at least, Geralt is only met with "natural" forces, lowlife. He also gets the opportunity to rescue a half-elf who is being held captive.