I can't speak to why the Irish instead of some other group. But generally, they're an example of a historically subjugated people who experienced hardship and discrimination when arriving in the US, but were eventually (more or less) accepted and brought into the fold as "white". Meanwhile, black people, on whose backs the US was literally built, many of whose lineage in the US stretches back hundreds of years further than many white people, still face some of the worst treatment and outcomes of any minority group in the US.
So my read is that while they had a common plight, the relative leg up the Irish had would render attempts to form community between the two groups fraught. And Remmick is clearly trying to exploit that common cause for his own purposes, and isn't too concerned with actual liberation, as demonstrated by his first recruits being Klan members.
I really enjoyed the interview he gave on The Big Picture about a month ago: https://youtu.be/Nh9H3EXM3jA?t=6641
(also available in their podcast feed if you prefer audio-only)