Not everyone has a phone, or a phone supporting NFC. (Microsoft employs >100,000 people) Badges are a low cost solution which don't require charging and can be easily issued or replaced. Additionally, badges display pictures, so people wandering around the office can be visually checked by security. (No need to scan their badge)
Traders are not necessarily smarter than the average software engineer. (There are many very smart people in both fields) However, they are fairly irreplaceable. This is because to be good at finance you need to be both smart and experienced. The only way to get experience is to work in finance for several years, and as a result, the supply of traders is fairly limited. (compared to the demand) Generally, someone who starts finance will not make their firm any money for the first year, but after five years they will be earning their firm a lot of money. The supply of good traders is a lot smaller than the supply of smart people.
A second reason why they are compensated so highly is that they are working in a highly leveraged job where the labor cannot be easily be divided between multiple people. This is similar to how CEO's make a lot of money, because being a good CEO produces a lot of value for the company and the position cannot be split between multiple people. (I'm not trying to imply that traders produce as much value to society as CEO's do) Being a marginally better trader will make your firm millions more per year. Good hedge funds have 100-1000 million of assets under management per front-office employee. (For example, D.E. Shaw or Bridgewater Associates)
But why does transferring large amounts of dollars take days? There is nothing physical preventing banks from supporting fast transfers. They can do fast transfers with BTC or USD.
Transfers are slow because there is little need to transfer large amounts of money quickly, and slower transactions help address issues with fraud. If you are buying a car, or paying for a house there is little need for a fast transactions. So with BTC or USD, transferring large amounts of money will take days.
It's got nothing to do with fraud checks. Transfers take days because banks largely run on mainframes programmed in the 1970s. Because of the (by todays standards) peculiar designs those machines had, the bulk of the work is done at night in batch jobs. The vast size and complexity of the software, combined with archaic platforms, their business-critical nature, and almost total absence of competition means that banks don't upgrade to more modern systems unless forced (see the UK same day payments initiative). Then the banking network is not a perfectly connected system so sometimes payments must be routed via intermediate banks. That's why it can end up taking days.
Why are the other people creating value, while HFT traders do not? Both types of players extract value from the market, while improving market efficiency.
HFT definitely has high entry costs (10's of millions of dollars), and firms that aren't serious about HFT can't participate. Note that HFT only makes 2 billion a year, so they aren't actually missing out on that much. HFT just isn't a very big space, compared to long term investing, and doesn't represent very much movement of capital. (The wealthy don't really invest in HFT, since HFT firms generally don't accept outside capital, or need any substantial amount of capital to operate)
Generally, HFT firms end up decreasing volatility. They price stock more accurately, which makes them money and decreases volatility. If they were increasing volatility, they would make infinite money, since the source of their profit is the volatility.
The main scenario where HFT firms increase volatility is when their algorithms break (either due to a programming error, or because they cease to accurately model the market due to a change in market conditions). In that case, the misfiring algorithm will increase the market volatility, and lose the HFT firm money.
You absolutely right that firms without co-located data centers are at a disadvantage. Thankfully, this only really matters if you are trying to compete with HFT firms. So it's not a big issue for people who trade on a longer time scales. The HFT firms do make some money off of these long term investors, and in exchange provide increased liquidity, decreased spreads, and decreased volatility. (if the HFT firms weren't offering the best prices for the stocks, then the long term investors wouldn't be buying from them)
Great, could you explain to me as a retail investor without direct access to the exchanges how to place an ALO order and to buy or sell without enriching HFT shops? (Hint: you can't.)
With a retail broker it's moot - they charge you a flat $8/trade regardless of whether you add or remove liquidity. You never collect rebates or pay to take liquidity.
You can still choose not to cross the spread with a retail broker. Just place limit orders on the right side of the spread.
Most retail traders actually get slightly (insignificantly) better prices for liquidity taking orders through price improvement at the wholesaler. Retail traders can also send post-only orders and participate in exchange the rebate/fee program on Interactive Brokers.
That isn't really an example of market manipulation. Market manipulation is an intentional manipulation of the market. (http://en.wikipedia.org/wiki/Market_manipulation) The Knight Capital incident was an example of a firm performing bad trades due to an error in their systems. Their behavior certainly wasn't intentional or profitable.
HFT isn't based on market manipulation. Some HFT firms perform market manipulation, as do some non HFT firms, but this practice is illegal. (For example, newedge was recently fined for market manipulation)
