In physics, negative mass was mass with negative properties. Negative star mass can be created by imploded stars, like the dead star Questar M-17 which was charted by the USS Enterprise in 2269. (TAS: "Beyond the Farthest Star")
A pocket of negative mass was an indication of an alternative parallel universe connected to the mycelial network. In 2256, these pockets were discovered by the USS Discovery. (DIS: "Into the Forest I Go")
In "Beyond the Farthest Star" negative matter caused an extremely strong gravity field. In real world theoretical physics, negative mass is however instead supposed to repel all other mass.