Libertarians and Immigration

A question that's been eating at me for a while:

Why do libertarians object to illegal immigrants?

Perhaps there are libertarians who do not, who extend their principles to encompass newcomers and their liberty to live and work where they please, without government interference. But my experience of libertarian pundits and of my own libertarian friends is that generally, they do not. The most anti-big-government libertarian of my friends also takes it as a given that illegal immigrants are a social ill.

Now, maybe some of these folks (my personal friend excepted, naturally) are merely using "libertarianism" as cover for another set of policy objectives. In that case, the explanation is that they're not sincerely libertarians. But what am I not grasping about the sincere ones?

If governments have no right to interfere in private economic activity and the pursuit of happiness, why can a government restrict the flow of labor from one place to another by erecting a border or, more intrusively still, by regulating how many immigrants are allowed to find jobs here. Immigration quotas don't seem to make any libertarian sense at all. And if not for the quotas, no one would be illegal. The "law" being broken is the law that government bureaucrats get to decided who can come in and who can't, while the government gets to set arbitrary numbers of immigrants from each group. ("Sorry, we've had all the Norwegians we can take for the year. Try us again in January.") The immigrants are only "illegal" because the very government authority that libertarians purport to despise labels those people as illegal.

Why shouldn't someone be able to get a job where there are jobs to be had? Why should someone be prevented from taking a job because too many other people from country X or Y have entered the country? (Talk about your identity politics....) Why shouldn't farmers be allowed to hire the people who want the harvest jobs? And why shouldn't an internet startup be free to hire a bunch of hotshots from IIT?

Seriously, I'd love any thoughts on this.