Caltech does have a kind of affirmative action (or at least it used to). For instance, when I was there they were trying hard to get more women. However, they did not do it by lowering standards for women compared to men.

Rather, they sought out qualified women and recruited them, much the way other schools recruit athletes. For instance, they might send someone to personally go visit the woman to try to sell her on the benefits of Caltech, and pay for the woman and her parents to visit and tour the campus.