John Stuart Mill and Freedom of Speech

From John Stuart Mill's On Liberty:

Society can and does execute its own mandates, and if it issues wrong mandates instead of right, or any mandates at all in things with which it ought not to meddle, it practices a social tyranny more formidable than many kinds of political oppression, since, though not usually upheld by such extreme penalties, it leaves fewer means of escape, penetrating more deeply into the details of life, and enslaving the soul itself. Protection, therefore, against the tyranny of the magistrate is not enough. We need protection also against the tyranny of the prevailing opinion and feeling, against the tendency of society to impose, by other means than civil penalties, its own ideas and practices as rules of conduct on those who dissent from them.

Mill is promoting absolute freedom of speech. How can we actually enforce such unfettered freedom without being tyrannical? If you have an idea, please post a comment.