If you have a server attempting to process a steady stream of incoming transactions at the edge of its capacity and it begins turning memory IO operations into disk IO (which is 1000 times slower), that is harmful to response times.

There probably isn't a lot of sense in having swap for suspend to disk on a server anyway of course.

I've only heard the heuristic in the context of workstations/laptops.