I forget where I read it, many years ago, and I will probably butcher it but the gist of the idea was:

Freedom of Speech includes the Freedom of those whom the speech would be directed at to HEAR the speech. The abridgment of freedom of speech denies not only the speaker the right to speak but others the* right to hear what they may or may not disagree with.