Just to be upfront, I'm a huge Tolkienite and have loved Tolkien since I was a little kid before the movies even came out, so I'm biased. And I'm having a rough night so I may not express this as eloquently and I'm not going to find sources at the moment.

But, I don't think Tolkien's work is racist the way people say and I do believe his, let's say, stances are years ahead of Rowling despite the fact that he was born the better part of a century earlier and was raised a White man in apartheid South Africa.

That being said, he isn't perfect or a Communist. And if you take Lord of the Rings as book by itself, then I can see how you would mostly draw that conclusion you mention. That it's racist and treats the White, Western men as the inherently "Good" men who fight the "evil" hordes of Black and Brown men. But, honestly, you get a much fuller picture if you also read other supporting literature, The Silmarillion as well as The History of Middle-earth, and so on, as well as his personal letters. There is a lot of humanizing and sympathy that happens towards the Haradrim and Men of Rhûn that is pretty tight. And even with Orcs, but I won't get into that.

There is a whole history of how the Men of the West, Aragorn's ancestors, not only were made so evil and corrupt by following Sauron themselves that they dared go to war with the fucking gods, which almost brought about their own utter annihilation, but also that they literally colonized, enslaved, pillaged, and murdered the indigenous population. This history of conflict between Men ("humans") caused so much hatred and resentment towards the West that when Sauron eventually came along as their enemy who promised to bring victory over them that the Southrons and Easterlings naturally allied with Sauron. Not because they were inherently evil but because of the material history. This only later developed into cults of worship, but the same Men of the West made Sauron-influenced cults of worship too so that is not unique either. So, Men of the West aren't in any way the inherently good men of the entire story—nor are Elves for that matter, they have a severely fucked history that is not discussed in Lord of the Rings. It just doesn't get brought up as deeply so people just get this Good v. Bad surface level view in that one book, similar to how taken by itself The Hobbit is just a fairytale but is very different when contextualized in its history. And I just see it as Men of the West already having had their own fall to Sauron's evil and now, because of their own actions, this same evil has passed to the Southrons and the Easterlings. The Men of the West are fighting Sauron, but they are not necessarily fighting the other Men—it's not like there is an attempt at genocide after Sauron is defeated. There is an understanding that Sauron is the true evil and manipulating the other Men—again, Sauron also corrupted the Men of the West and other races and they are constantly fighting the temptation to go back to him so it is not like they are totally above these 'evil' men just for not being under Sauron's sway this time. And even in Lord of the Rings itself you already at least see one humanizing moment with Sam when he and Frodo are in Ithilien and one of the Haradrim falls dead off the Mûmakil (Oliphaunt) and Sam wonders

what the man's name was and where he came from; and if he was really evil of heart, or what lies or threats had led him on the long march from his home; and if he would rather have stayed there in peace.

Oh yeah its pretty racist but much less than, say, Orwell and in some ways less than TH Lawrence.