A compliment was an expression of praise or admiration.
In 2266, the cryptography section of the USS Enterprise was decoding a Romulan transmission from a Bird-of-Prey when Lieutenant Stiles "suggested" giving it to the Vulcan Spock to possibly translate, due to the resemblance between Vulcans and Romulans. Captain James T. Kirk assumed that Stiles complimented Spock on his ability to decode, but Stiles wasn't sure. Kirk then stated to leave his bigotry in his quarters. (TOS: "Balance of Terror")
In 2374, after Kyrians had taken Seven of Nine hostage and entered the USS Voyager's mess hall, The Doctor suggested to Captain Kathryn Janeway that he should go in first to draw any phaser fire if need be. As a result, Ambassador Daleth stated Janeway's crew was heroic. The Doctor clarified that he was invulnerable to phaser fire but he appreciated the compliment. (VOY: "Living Witness")