But my point is that you're still trusting your email provider with the password, and now if that get's leaked an attacker has access to (arguably/potentially) more sites than they would have before (via password resets).

How so? If an attacker gets your mail password, they can pretty much already password-reset every single site you use.

That is exactly why sites shouldn't provide password reset by email. Email shouldn't be used for authentication in any case. It's really insecure solution.

Unfortunately security questions aren't much better. The best solution is to expect the user to safely and securely store a reset-key (kind of like Mozilla's Sync).

However, to the average, non-techie user this is

* Bad UX * They won't store it securely * They'll lose it

Another option is using public keys with some form of transition mechanism.