So it is the level of "privilege" that does or does not allow the commission of -isms then. The better off the target is, the more acceptable discrimination is? That is also a very Western perspective. It would be ok to tell Muslims in the Middle East that terrorism is their responsibility because their country's power structure does put Islam firmly above others?
This "some animals are more equal than others" stuff is moral equivocating. If something is wrong if done to a group that isn't "in power", then it is also wrong to do it to the group "in power". This isn't a zero sum game. We don't have to weight the guilt by association for a black man when compared with a white man because systemic racism competes with systemic patriarchy. If you do think that the immutable characteristics a person is born with are the most important things about them, I would encourage you to self interrogate how messed up that is.
The invisible hand of the market is not all powerful, which is why regulation and safeguards are needed for a "free" market to function. Anti-monopoly laws, labor laws, etc. I lean libertarian, but do not embrace 100% laissez-faire economics. Immigration falls under this same framework.
The West has eliminated their manufacturing and blue collar base by outsourcing it overseas, which hurt large swaths of the working class. Outsourcing labor by importing labor from overseas to do the job cheaper here has similar results. See the agricultural sector in the US for this example. Everyone always says that the reason immigrants are needed is because Americans don't want to do those jobs, but leave out "for the wages paid".
Some regulation is needed, and we have had wholesale failure of meaningful regulation and complete regulatory capture by the oligarchy which started under Reagan and snowballed out of control since. Proper support networks and social safety nets have also failed, for the same reasons. Unrestricted immigration does not solve these issues, and with these holes in place does indeed hurt.
Things that aren't a problem when everything is healthy and working as intended can definitely hurt when things aren't healthy. Obviously the "health issues" need to be addressed to actually fix the problem, but ignoring symptoms while doing so doesn't help.