Immigrants are just a scapegoat. Modernity is the "culture fabric" culprit.
This anti-immigrant feeling is a conceptual mistake. We're in a big argument about which rain god caused last week's flooding, but it was the clouds.
As a thought experiment, suppose all the world's cultures peacefully sorted themselves geographically. Suppose we fed the world's land area into a computer and had the computer "fairly" divide the area into zones, one zone per categorizable culture. All the people of each culture could then move to their designated zone. Each zone would have a homogenous "cultural fabric."
Let everything else about modern life stay the same though. Everyone gets to keep their mobile phones, their Internet access, their Amazon Prime. There is still tourism and free trade among "culture zones." People still go about their jobs, their education, etc., exactly as they do now. The only new rule in the newly organized world would be "no immigration of any kind." If you are in a given culture zone, then you stay there.
Who in their right mind thinks that the "culture fabrics" in the world would then stabilize?
They simply wouldn't. A thousand homogenous culture fabrics with mobile phones, the Internet, free trade, tourism, etc., would almost immediately turn into a thousand heterogeneous culture fabrics. The big changes we see in culture, some very good, some inarguably horrible, don't have much to do with immigration. They have everything to do with modernity.
On edit: And I'm not arguing against modernity by the way.