Troi utilizes directed dreaming to communicate with the alien.
Directed dreaming was a therapeutic technique wherein a subject could learn to control their own dreams and retain a conscious memory, even while in REM sleep.
Deanna Troi had used directed dreaming when dealing with patients who were suffering from debilitating nightmares. In 2367, Troi called upon her experience with this technique in an attempt to communicate with a telepathic alien whose mental transmissions were interfering with the USS Enterprise-D crew's ability to dream. The attempt was successful, and both the Enterprise and the alien were able to pool their efforts in order to escape from a Tyken's Rift in which they had both become trapped. (TNG: "Night Terrors")
In 2370, Data described the linking of his dream program to the holodeck in order to study the images generated by it as similar to the concept of directed dreaming. (TNG: "Phantasms")