A quote from Paul Crider, the author of a paper on this subject, right at the beginning of the paper:
Consider a world where enforcement of property rights is perfect and government violence is minimal, but where all the property is owned by a rich, cohesive majority group, large enough to function economically on its own. Further, members of the majority loathe the minority group, and uniformly refuse to sell or lease them property; neither will they employ them except perhaps for dangerous or degrading jobs for exploitative wages; neither will they educate them in their first class schools.Although this hypothetical supports his conclusion that the government needs to step in and protect the rights of the hypothetical minority, the problem is that he starts the argument with an (extremely) unlikely hypothetical in order to justify massive government intervention into what must ultimately be all aspects of people's lives.
Oh, and hey, before you get all "but America and slavery" on me, it was government that maintained the status quo when it came to slavery. It was the government who paid to track down your fugitive slaves, it was government who prevented you from freeing your slaves if you wanted to, it was government who prohibited freed blacks from living freely, in short, it was government that made slavery possible for so many years. Had it not been for the government slavery would have collapsed under its own weight long before it did.
Jim Crow? Oh wait, government. It was illegal for you to serve blacks in the front of your restaurant, even if you wanted to.
The reality is that absent government these atrocities would not have lasted nearly as long as they did.
I have issues with an argument that starts with such an extreme posit in order to justify an argument. If the argument is valid, it shouldn't need a starting point so far outside what actually exists in order to justify it. In a world with less government interference in our personal lives (the protection from force and from fraud is a good starting point) some individuals may find that they have more difficulty accessing some services, but I happen to believe that the majority of people would not allow that to happen merely by their individual actions, not at the point of a gun and by the power of the police state. Crider's argument rationally leads to our current situation whereby people who attempt to exercise their powers of conscience are deprived of their means to make a living and of their own rights to free action and association, if that exercise does not fit within the current paradigm (I think here of the current move to prevent Christians from refusing to serve at gay weddings, even when there are plenty of alternative service providers, and even in hypothetical situations that are also unlikely to arise).
When you allow a select group of people a monopoly on violence, regardless of how that group is chose, you give them the power to impose the views of whoever manages to grasp controls of the reins of that power. This is a dangerous road to go down and one in which we see all too well the end result since we are there. Justifying the current regime on libertarian principles (yet claiming that we just need less of it) is not a libertarian idea.