I'm trying to remember which missions/events gave you Wisdom Gems (including mandatory story ones)

You get one for learning all ultimate moves
You get one after beating Zhang Liao in the Hot-Blooded Route
You get another for completing the Meat Bun Man Sidequest
You get one for doing Mi Zhu's sidequest of giving her a pear in Chapter 3

I think (?) you got one in Chapter 3 to use Water Tactics but I'm not sure