I would actually prefer to be exposed to multiple POVs wherever possible, if it were delivered in a respectful manner. Likewise I enjoy learning more about various religions, and cultures, and all manner of things. However, I must draw the line against trolling behaviors. So for me, it is not just that they contain "content that I may not agree with", but that they outright encourage behaviors that have driven me and so many others away.
I got my start on Kbin.social, before it repeatedly kept going down for a week at a stretch, multiple times, and they had already defederated from those places, so fortunately I got to be exposed to the friendlier side of the Fediverse first, so I knew what was possible. But then after switching to try a couple of Lemmy instances, I gave serious thought to leaving the Fediverse entirely. There is so much else that I could do with my time, you know!? Read books, watch videos (like Hank & John Greene's Crash Course series on YouTube), go outside and touch grass - I don't need to be arguing with the emotional equivalent of toddlers online, acting as the recipient for their emo-venting aka vomiting all over me rather than having true conversations aka "communication".
That said, I might understand what you mean about the memes - if they violate the community rules then they are being disrespectful to the recipients who would have to spend time reading them, rather than enjoyable content. And if there is enough spamming of such, it inches closer to "trolling" behaviors, as in the same kind even if not quite degree. Though oftentimes people from that server also engage in actual trolling as well, in the form of responses that do not care about how the recipient would like to receive.
Anyway, I left Reddit over that shit, and I would leave the Fediverse too if it came down to it, though fortunately it works to just block those 2-3 places and the rest becomes a MUCH better place to play around in!:-)
(Translation: Klingon/pIqaD: Qapla'!)