Not if you also ask for the user email though the Facebook API. In this case people can get access by getting a password reset sent to their email. This is what Grooveshark is doing right now and what most people using Facebook for signing in do too.