I would argue 99% of people care a great deal, whether they like to admit it or not. We are social creatures. Caring about these things (within some range) also gives a higher chance for reproductive success, so as you said, we are hard-wired to care.