Not being subject to vendor lock-in is huge in itself.
You can save plenty of money if you have the scale to move out of S3. That’s important because you can usually trade CPU for storage by storing data in multiple formats, optimized for different access patterns.
But mostly, the Hadoop ecosystem is very open. The tools are still maturing and it’s easier to debug open source tools than dealing with the generally poor support in most managed solutions.
S3 is flexible and scalable, but it is not cheap. I'd be hard pressed to run the numbers now, but at some point it's cheaper to just do storage yourself.
But to be fair, you'll go on-premise due to the computing or bandwidth costs first. And you'll likely move data to the same datacenter to avoid expensive transfer costs.
I've also had to work in places where you simply could not put your data in the cloud due to regulatory reasons.
Amazon has to make a profit on selling services. You don’t have to make a profit providing services internally. There are certain inelasticities that both of you have to pay for: power, real estate, internet, etc. If you’re big enough, you can do it cheaper than Amazon.
S3 isn't cheap if used with other services. Either you use AWS for everything or you pay with bandwidth. It's cheap to get your data in, using it or getting it out isn't cheap at all.
You can save plenty of money if you have the scale to move out of S3. That’s important because you can usually trade CPU for storage by storing data in multiple formats, optimized for different access patterns.
But mostly, the Hadoop ecosystem is very open. The tools are still maturing and it’s easier to debug open source tools than dealing with the generally poor support in most managed solutions.