Not if you also ask for the user email though the Facebook API. In this case people can get access by getting a password reset sent to their email. This is what Grooveshark is doing right now and what most people using Facebook for signing in do too.
Adding Facebook as your sign-on adds another significant point of failure.